Things I Realized After Responding to a 3 A.M. Incident
The alarm went off early Friday morning. The payment service's error rate had surged to 40%, and I was the on-call engineer.
The first 30 minutes were complete panic. I just kept staring at the dashboard, unsure what to do. After agonizing over whether to roll back or scale out, I ended up calling a senior engineer. Calling for help right away was the right move.
The cause was nothing major. A single configuration value deployed that morning was causing infinite retries under certain conditions. We rolled back and were back up in 40 minutes.
Things I wrote down during the retrospective afterward:
- If you can't decide within 5 minutes of getting the alert, call someone, no matter what.
- Look at recent deployment history before the dashboard; that's usually where it is.
- Don't drink during the week you're on call.
And the biggest realization was that process, not skill, is what saves you during an incident. It's really important to have runbooks prepared in advance.
10 answers
True, procedure is a skill.
Where do you keep your runbooks? Wiki? Notion? Our team makes them, but nobody ever looks at them lol
A 40-minute recovery during a 3 a.m. on-call is a win. That's a seriously mentally draining time of day—nice work.
Well, isn't 'if you can't decide within 5 minutes, always call a human' a bit of a risky generalization? It depends on the service scale and team situation. If someone who can decide on their own unnecessarily escalates, it can create a structure where that person keeps getting called on. I'm also curious whether there's any basis for the number '5 minutes'.