When a Production Incident Hit at Dawn: A Record of My Fumbling
An incident response story from last year, when I got called in at 3 a.m. and went out to handle it. I never want to go through it again, but I’m leaving a record of it.
1. First, the alert came in. A warning that the DB connection pool was full. At first, I started by restarting the server. Thirty minutes later, the same alert came back.
2. I looked at the slow queries. One particular API was taking 40 seconds each time. One index was missing. At 4 a.m., I added the index. Responses returned to normal.
3. But why all of a sudden? After checking, it turned out that a feature deployed that evening had made that table do a full scan for the first time.
4. I thought about rolling back, but decided that adding the index was safer, so I kept it.
I learned two things. First, when an alert comes in, the very first thing you need to check is “when it started.” Second, immediately after a deployment, you must check the slow query logs.
Now I’ve added a query plan check to the deployment pipeline. Once is enough for the same mistake.
10 answers
Agreed—one index can really wreck you in an instant.
That feeling when you get paged at 3 a.m.... just hearing the alarm makes my heart drop lol
But it's a bit unfortunate that in #1 you started with a restart. When an alert comes in, the order should always be to check 'since when' and 'what changed' first—though when you're in a hurry, your hands do move first. I've been there too. So these days, I've gotten into the habit of capturing the dashboard timeline first whenever an alert comes in.
Well, I don't know about that. Isn't it a bit risky to add an index to a live table at dawn? It could take a table lock and escalate into a bigger outage. We were just lucky traffic was low then; I wouldn't say that kind of judgment is always safe.
I ran into something similar. In our case, what we thought was a connection pool issue turned out to be a connection leak. We suspected slow queries first, but after checking the logs, a specific batch job wasn't returning connections. Some things aren't visible just by looking at the queries, so keep that in mind.
Got a source? How did you add the query plan check?
I think you made a good call on rollback vs. index, but I’m curious about the basis for judging it “safer” in #4. Were you confident that the feature deployed that day was only problematic for that query? If other queries were also doing full scans, rollback would have been the right move.
There are honestly plenty of cases where rollback is the right call. Adding an index just treats the symptom, and the deployment that caused the issue is still live. It could blow up again on the next deploy, right?
I’d also like to try adding query plan checks to CI—could you break down a bit how you implemented it? Is it by parsing the EXPLAIN output and failing the pipeline when a full scan shows up? Or is it threshold-based? I need evidence to convince my team ㅠㅠ I’m also curious whether it actually had any effect after adoption.
That saying, 'Once is enough for the same mistake,' is a good one. But in reality, people make the same mistake up to three times lol.