1. Adding a Redis Shard Took Down Production Writes for Over an Hour

   Redis memory was full. After one round of clearing cold data, usage only dropped by 5 GB, so I decided to just add a shard and scale out directly. As a result, only 12 seconds after slot migration started, `ZADD` in the recommendation service began throwing errors like crazy. I thought the service just hadn’t picked up the new node and a restart would fix it, but it was still failing even after the restart. Following that `"reachable node:None"` message all the way down, I eventually traced it to a line in the `redis-py` client that basically says, “if a node times out, remove it from the list.”

   2026/09/03

2. A Synchronous Blocking Call Froze the Entire Event Loop

   The service was mysteriously getting drained and rebuilt in batches. The database looked pretty stable too. After digging into it for quite a while, I finally found that a third-party SDK was quietly making a synchronous blocking call inside an `async` function, freezing the event loop for several seconds and triggering a whole cascading failure chain.

   2026/08/22

3. A Database Avalanche Incident Investigation

   The first thing I saw after being pulled into the incident chat was: "Every API is timing out." We stopped the bleeding first and gave the database a chance to recover, then circled back to figure out why a hidden risk planted two and a half days earlier would suddenly blow up all at once at that exact moment. Following the trail, it turned out to be a complete chain of events: an analytical query that had been left running, a high-traffic endpoint with a missing index, and a distributed lock that got released at exactly the wrong time. All three lined up and detonated together.

   2026/08/22

4. How to Elegantly Connect a Local GUI Client to a Private Redis on AWS / GCP

   Say Goodbye to Tedious and Fragile SSM Port Forwarding: Use SSH Tunnels to Give Desktop Clients Smooth Direct Access to Redis Across Multiple Environments

   2026/08/21

5. An Emergency Production Incident: Redis Memory Exhaustion, Precise Cleanup, and an Architectural Postmortem

   Redis Memory Hit 99.98% in Production and Triggered Eviction Alerts: How We Precisely Cleaned Up 50% of Cold Data Using LRU Idle Time to Free 4GB of Memory, and a Postmortem on the Architectural Risks Behind It

   2026/08/21

6. Investigating a Slow Memory Leak: Creating a New OpenAI Client Every Time, with the Real Issue in the Event Loop

   While investigating a slow memory leak in a service, I came across a piece of background enrichment logic a teammate had written: on every call, it instantiated a brand-new `AsyncOpenAI` client, and the connection pool quietly piled up into a leak. I wanted to simply cache and reuse the client, but it turned out that didn’t work at all—the root cause was the lifecycle of the event loop.

   2026/08/19

Selected articles

Open source software