Ask
267
@rollback_rae ·

customer emailed everything is down while i was in a day job standup - what does solo on-call actually look like

Tuesday morning, in a meeting, phone face down. Customer emailed at 09:40 saying the app was returning errors. I saw it at 11:15. It was a database connection limit thing, fixed in twelve minutes once I looked.

So the outage was ninety minutes of which eighty-five were me not knowing. Nobody cancelled and one person was quite nice about it, but I do not want to run this on luck.

What does an honest on-call setup look like when you have a full time job and cannot be reachable during the day?

10 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @costbasis_carl · 3mo ago · 3 replies

    The connection limit detail matters more than the on-call process. That failure class is entirely preventable and it is the most common way small apps fall over.

    Put a connection pooler in front of the database, cap the pool below the server limit, set a sane statement timeout so one bad query cannot hold connections open indefinitely, and add a health endpoint that acquires and releases a connection so your monitor is actually testing the thing that broke.

    Spending an evening on that buys you more sleep than any alerting configuration, because the incident does not happen.

    217
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @arraysformula · 3mo ago

      Statement timeout is the one people skip and it is a one line configuration change. A single runaway query is what turns a slow morning into a full outage.

      78
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @wick_and_wax · 3mo ago

      Setting the pool maximum to a number you actually calculated rather than the default is most of the trick. Every default assumes it is the only client talking to that database.

      52
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @rollback_rae · 3mo ago · 3 replies

    Separate the three problems, because you are trying to solve them with one mechanism and they have different answers.

    Detection: an external uptime check hitting a real endpoint that touches the database, not a static page, every minute, alerting to your phone with a distinct sound that gets through your focus mode. This is cheap or free and it turns eighty-five minutes of ignorance into two.

    Communication: a status page and an autoresponder. If the check fires it should post automatically. A customer who sees an acknowledged incident behaves completely differently from a customer shouting into a void.

    Response: this is the one you cannot fully solve, and that is fine. Being unable to respond for two hours is acceptable if detection and communication are instant. What is not acceptable is finding out from a customer.

    246
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @hot_heap_hana · 3mo ago

      The distinct alert sound is the underrated part. If your outage alert makes the same noise as a marketing email, you will have trained yourself to ignore it within a fortnight.

      71
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @probe_to_ground · 3mo ago

      Checking a real endpoint rather than the homepage. My monitor was hitting a static page the whole time, so it was green throughout.

      82
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @hemline_hana · 3mo ago

    Say it out loud on the pricing page. Mine reads: support is answered within one working day, and there is no guaranteed response time outside UK business hours. It has cost me exactly one deal, from a company that would have been miserable to have as a customer.

    Everything gets easier once the expectation is written down. You stop feeling guilty for being in a meeting, and customers stop guessing.

    174
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @vac_seal_vera · 3mo ago

    Also worth being realistic about what customers actually expect from a tool at this price. Ninety minutes on a Tuesday with an honest email afterwards is survivable for almost every product that is not payments or auth. People forgive downtime. They do not forgive silence, and they really do not forgive discovering it themselves twice.

    129
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @gainstage_gil · 3mo ago

    Degrade rather than die. If the database is unreachable, serve a cached read-only view with a banner instead of a 500. Most of my users on any given hour are looking at things, not changing them, and a read-only app looks like a minor blip while an error page looks like a company that has gone out of business.

    158
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @carryon_cass · 3mo ago

    One thing that cost me nothing: a five line runbook in a note on my phone. What to check first, the command to restart the worker, the URL of the hosting dashboard, and the login. Twelve minutes of fixing while standing in a corridor is achievable. Twelve minutes of remembering how the system works while standing in a corridor is not.

    141
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report