my side project paged me at 3am on a work night — what is the minimum alerting that still lets me sleep
Uptime checks every minute, push straight to my phone. Last month it woke me three times: once real, the connection pool was exhausted and the app was properly down, and twice for blips that healed inside forty seconds while I was still finding my glasses.
I have a standup at 9 and a manager who does not know about any of this. Honestly I would rather be down for six hours overnight than be useless at my day job twice a week.
So: when there is exactly one person and that person is employed elsewhere, what do you actually let wake you up?
@quiet_promotion · 5d ago · 3 replies
Two changes and my nights came back.
First, require consecutive failures. A single 60 second check firing on the first bad response is a blip detector, not an outage detector. Three consecutive failures means you find out about three minutes late and you stop hearing about every deploy, every certificate renewal, every provider hiccup. Nearly all of my noise was in that window.
Second, split the alert channels by what you would actually do. Down and staying down goes to push. Everything else — error rate up, queue backing up, disk at 80% — goes to an email that you read with coffee. If you would not get out of bed and open a laptop for it, it is not a page, it is a note.
Reply
Report
@scope_of_work_sky · 3d ago
Consecutive failures is right but do it from more than one region or you will trade false pages for false calm. I had a check region that got itself blocked and reported my app healthy for two days while it was returning 500s to actual humans in Europe. Two probe locations and requiring both to fail is the version of this that does not lie to you.
Reply
Report
@parquet_pile · 5d ago
Adding one: alert on the thing the customer feels, not the thing your infrastructure feels. A check that logs in and loads one real page catches ten times more real breakage than a check that pings /health and gets a hardcoded 200 back from a process that cannot reach its own database.
Reply
Report