Ask
87
@pgpolicy_nadia ·

neon compute never scaled to zero because a 30s health check kept the branch awake Database Costs

Serverless Postgres, picked specifically because my traffic is bursty and I liked the idea of paying nothing overnight. Autosuspend is on with a 5 minute timeout.

Bill went from about $19 to $61. Compute hours for the month: 730. That is every hour in the month, which is a very suspicious number.

Eventually found it. My uptime monitor hits /health every 30 seconds, and /health runs SELECT 1 against the database to "prove" the app is healthy. So the compute never gets five idle minutes, ever, in any month, and autosuspend has literally never fired since I set it up.

Writing it down makes it obvious. Posting because I suspect I am not the only one, and because I would like opinions on what a health check should actually do - I do want to know if the database is unreachable.

7 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @egress_egon · 2mo ago · 2 replies

    Add to the list of things that quietly keep serverless databases awake:

    • migration tooling that opens an advisory-lock connection at boot and holds it
    • a background job runner polling a jobs table every few seconds
    • an admin dashboard tab someone left open with a 10 second refresh
    • log shippers or metrics exporters querying pg_stat_*
    • your own local development environment connected to production because of a stale .env

    The last one has happened to two people I know and the tell is compute activity in a timezone where you have no users.

    48
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @quietmargin · 2mo ago

      The open admin tab is so real. We had a dashboard on a wall-mounted TV in the office polling every 5 seconds for eight months. It cost more than the tool it was displaying.

      20
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @kraut_corner · 2mo ago · 2 replies

    730 compute hours in a 730 hour month is the cleanest diagnostic in this whole thread. Any time you see that number, something is polling.

    On what a health check should do: split it into two endpoints.

    • /health - liveness. Is this process running and able to respond. No dependencies, returns a constant. This is what your uptime monitor hits every 30 seconds and it should touch nothing.
    • /ready - readiness. Can it actually serve, including dependencies. Hits the database. Called by your orchestrator on deploy, or by you at a much lower frequency.

    A liveness check that fails when a dependency is down is actively harmful anyway - it tells your platform to restart a perfectly healthy process because the database is having a moment, and restarting will not help.

    If you want database monitoring, monitor the database. Do not infer it by poking your app, and definitely do not do it at a frequency that defeats the billing model you chose.

    One extra thing to check: any connection pooler, ORM keepalive, or long-lived connection also counts as activity on most of these platforms. Suspend usually needs no connections, not just no queries. A pooler holding one idle connection open forever will keep the compute up just as effectively as your health check.

    72
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @pgpolicy_nadia · 2mo ago

      Split the endpoints and compute hours for the last week are 41 instead of 168. The pooler warning is a good one, I checked and my connection string does go through the pooled endpoint but it drops idle connections, so I got lucky there.

      24
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @durable_ines · 2mo ago

    Worth asking whether you want scale-to-zero at all, rather than just fixing this instance of it.

    The economics only work if you have long genuine idle stretches. If you get a request every few minutes during waking hours, you are paying cold start latency all day - hundreds of milliseconds on the first query after a suspend, sometimes more - in exchange for saving money during hours nobody is awake anyway.

    Run the numbers on your actual traffic shape. For a lot of small products the honest answer is a small always-on instance with a hard ceiling on autoscaling, because the bill is predictable and every request is fast. Scale-to-zero is genuinely great for preview branches and dev environments, where the idle really is 95%.

    31
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @quietmargin · 2mo ago

    Whatever you do, set a budget alert at 2x your expected spend on day one. $61 is a cheap lesson. The same bug on a bigger instance class is a very expensive one and the failure mode is identical - you find out at the end of the month.

    16
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @probe_to_ground · 2mo ago

    Health checks are the only code we write specifically to run forever and then never think about again.

    7
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report