Ask
312
@hotswap_hana ·

my side project paged me at 3am on a work night — what is the minimum alerting that still lets me sleep

Uptime checks every minute, push straight to my phone. Last month it woke me three times: once real, the connection pool was exhausted and the app was properly down, and twice for blips that healed inside forty seconds while I was still finding my glasses.

I have a standup at 9 and a manager who does not know about any of this. Honestly I would rather be down for six hours overnight than be useless at my day job twice a week.

So: when there is exactly one person and that person is employed elsewhere, what do you actually let wake you up?

9 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @quiet_promotion · 5d ago · 3 replies

    Two changes and my nights came back.

    First, require consecutive failures. A single 60 second check firing on the first bad response is a blip detector, not an outage detector. Three consecutive failures means you find out about three minutes late and you stop hearing about every deploy, every certificate renewal, every provider hiccup. Nearly all of my noise was in that window.

    Second, split the alert channels by what you would actually do. Down and staying down goes to push. Everything else — error rate up, queue backing up, disk at 80% — goes to an email that you read with coffee. If you would not get out of bed and open a laptop for it, it is not a page, it is a note.

    226
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @scope_of_work_sky · 3d ago

      Consecutive failures is right but do it from more than one region or you will trade false pages for false calm. I had a check region that got itself blocked and reported my app healthy for two days while it was returning 500s to actual humans in Europe. Two probe locations and requiring both to fail is the version of this that does not lie to you.

      87
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @parquet_pile · 5d ago

      Adding one: alert on the thing the customer feels, not the thing your infrastructure feels. A check that logs in and loads one real page catches ten times more real breakage than a check that pings /health and gets a hardcoded 200 back from a process that cannot reach its own database.

      64
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @backfill_bram · 4d ago

    Your real bug is the connection pool, and it will come back. Exhausted pools on a small box are almost always one of three things: a route that opens a connection per request and never returns it, a long-running job holding a transaction open, or your pool size multiplied by your process count being higher than the database will accept.

    Put a connection pooler in front of the database, cap the pool per process, and set a statement timeout so a single stuck query cannot hold a slot forever. Then add a restart policy so the process dies and comes back instead of sitting there wedged. That one turns a page into a log line.

    141
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @bedroomsynth · 4d ago · 2 replies

    Nothing wakes me. Nothing. The app restarts itself on failure, there is a status page that updates automatically, and there is a digest at 08:00 with everything that happened. Two years of that and I have lost exactly one customer to overnight downtime, which is fewer than I lost to being visibly exhausted in customer calls.

    158
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @seedstart_sim · 4d ago

      Works until you sell to a business in another timezone, and then "overnight" is their Tuesday afternoon. I would keep the philosophy but scope it: page for accounts above some dollar value or on a plan that promises anything, email for everyone else. My rule is that if a single account can leave and take more than 10% of revenue with it, that account gets to wake me.

      74
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @tripodwobble · 3d ago

    Cheapest setup that works, in order of what I would add: an external uptime check with a multi-failure threshold, an auto-restart on the process, a static status page on a different host than the app, and error tracking with a daily digest rather than instant alerts. All of that lands in the low tens of dollars a month, most of it on free tiers at your size.

    What I would not pay for yet is an incident management tool with schedules and escalation policies. You are the schedule. There is nobody to escalate to.

    62
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @solo_supper · 5d ago

    Separate thought: write the downtime expectation on your own site before someone asks. One line saying you aim for business-hours response and that overnight incidents are picked up in the morning. It costs you almost no customers at a small price point and it converts a 3am panic into a policy you already published.

    45
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report