customer emailed everything is down while i was in a day job standup - what does solo on-call actually look like
Tuesday morning, in a meeting, phone face down. Customer emailed at 09:40 saying the app was returning errors. I saw it at 11:15. It was a database connection limit thing, fixed in twelve minutes once I looked.
So the outage was ninety minutes of which eighty-five were me not knowing. Nobody cancelled and one person was quite nice about it, but I do not want to run this on luck.
What does an honest on-call setup look like when you have a full time job and cannot be reachable during the day?
@costbasis_carl · 3mo ago · 3 replies
The connection limit detail matters more than the on-call process. That failure class is entirely preventable and it is the most common way small apps fall over.
Put a connection pooler in front of the database, cap the pool below the server limit, set a sane statement timeout so one bad query cannot hold connections open indefinitely, and add a health endpoint that acquires and releases a connection so your monitor is actually testing the thing that broke.
Spending an evening on that buys you more sleep than any alerting configuration, because the incident does not happen.
Reply
Report
@arraysformula · 3mo ago
Statement timeout is the one people skip and it is a one line configuration change. A single runaway query is what turns a slow morning into a full outage.
Reply
Report
@wick_and_wax · 3mo ago
Setting the pool maximum to a number you actually calculated rather than the default is most of the trick. Every default assumes it is the only client talking to that database.
Reply
Report