The Night I Got Paged at 4 AM and Recovered from an Outage

Last Tuesday in the early hours, my phone went absolutely nuts. It was an alert about DB connections in production, and the dashboard was completely red.

For the first 30 minutes, I just sat there dazed. The connection pool had blown up, and the application was getting slower and slower. I turned on the circuit breaker and cut off the path where traffic was leaking.

The root cause turned out to be a missing connection return. There was one piece of code that didn't hit close on an exception path, and because it was a path that rarely got triggered normally, that made it even worse.

Here's what I wrote up after recovery:

  • We hadn't written any tests for exception paths at all.
  • Alerts fired, but there was no dashboard, so I wasted 5 minutes.
  • The rollback procedure existed only in docs, and nobody had ever tried it.

I fixed these three the following week. In the end, outages are all documentation problems. What do you all do first when you get an alert in the early morning?

by 호기심천국137

5 answers

For middle-of-the-night alerts, I first meditate for 5 seconds and drink a glass of water lol. I've seen a few cases where someone rolled back at dawn in that state and caused an even bigger incident.

by 데이터덕후848 · ▲0

I'm curious what your stack is. What connection pool are you using, and what did you use to enable the circuit breaker? Also, it's really dangerous that the rollback procedure exists only in the docs and nobody has actually tried it—but good job fixing that right away next week.

by 월급루팡409 · ▲0

I totally relate to wasting 5 minutes because there was no dashboard... that feeling of getting an alert and not knowing where to look first. We went through that too, so we pinned the dashboard link at the top of the alert channel.

by AI덕후528 · ▲0

If connections aren’t being returned, the fastest fix if you’re using HikariCP is to set leakDetectionThreshold. Set it to around 30 seconds and it’ll even print a stack trace for the connection that wasn’t returned, so you won’t lose 30 minutes next time. And adding runbook links to alerts is really effective. We put three lines of 'just turn this on first' and the dashboard URL in the alert body, and that alone significantly cut our middle-of-the-night response time. Good call turning on the circuit breaker first.

by 주말개발자616 · ▲0

"In the end, incidents always come down to documentation problems." I'm not so sure about that. This case wasn't a documentation problem—it was a failure to write tests for exception paths, and more precisely, a design problem that left resource cleanup to manual close. If it had been structurally prevented with try-with-resources or finally, wouldn't this be an incident that wouldn't have happened even without documentation? It's also a bit disappointing that all three root-cause items were lumped together as "we just didn't try it."

by 데이터덕후964 · ▲0