Ask
289
@adjunct_ora ·

still getting too many clients already on the pooled 6543 string at 120 concurrent Connection Pooling

Serverless functions against a managed Postgres. Under a burst of about 120 concurrent invocations I get FATAL: sorry, too many clients already on maybe 15% of them.

I already moved the connection string to the pooled endpoint on port 6543. Same error. The diagram is how I understood the two paths - I thought moving to the bottom lane was the whole fix.

What I have checked: only one connection string in the environment, no long transactions, functions finish in about 90ms. What am I still doing wrong?

9 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @two_vcpu_club · 2mo ago · 3 replies

    The pooler is not the only pool in the picture. Your ORM opens its own client-side pool inside every function instance, and the default size is usually derived from CPU count - commonly something like 5 to 9 connections per instance. 120 instances times 5 is 600 clients arriving at a pooler that will happily accept a bounded number and then refuse the rest.

    Append the limit to the pooled URL:

    postgres://...:6543/db?pgbouncer=true&connection_limit=1
    

    One connection per instance is correct for serverless. The instance handles one request at a time, so a pool inside it buys you nothing and costs you 4 to 8 sockets.

    While you are there, confirm the host really is the pooled one. On most providers the pooled endpoint is a different hostname, not just a different port, and it is very easy to change 5432 to 6543 on the direct host and get a connection refused or a silent fallback depending on the provider.

    246
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @adjunct_ora · 2mo ago

      connection_limit=1 took the error rate from 15% to zero in the same load test. I had read that parameter three times and assumed it was a maximum I would never reach.

      74
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @two_vcpu_club · 2mo ago

      It reads like a safety limit and it is actually a per-instance allocation. The mental model that fixes it: in serverless you are not sizing a pool for your app, you are sizing it for one request.

      81
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @crimp_not_solder · 2mo ago · 2 replies

    Also check whether something else in the same deployment is quietly using the direct connection. Migrations, a seed script, a health check, or the ORM itself for schema introspection - several of them insist on a direct URL and will use DIRECT_URL if it exists in the environment. If a migration runs on deploy while the burst is happening, that is 20 direct connections arriving at the worst moment.

    Easiest way to prove it: select count(*), application_name from pg_stat_activity group by 2 during the burst. If you see anything other than what you expect, there is your second client.

    132
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @adjunct_ora · 2mo ago

      pg_stat_activity showed 22 rows with an application_name I did not recognise, which turned out to be the ORM's introspection running on every deploy. Not the main bug but definitely a contributor during deploy windows.

      49
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @schema_drift_lu · 2mo ago

    Once you are on transaction mode, do also check for prepared statement errors under load. Different symptom, prepared statement "s0" already exists, and it shows up intermittently under exactly this kind of burst because the pooler hands you a different server connection than the one that prepared it. Most drivers have a flag to disable prepared statements for this reason. Not your current bug but you will meet it next week.

    63
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @cast_iron_carl · 2mo ago

    One more ceiling people forget: the pooler has its own limit, and on smaller plans it is lower than you would guess. 120 concurrent invocations each holding a client connection can exceed the pooler's client limit even at one connection each - at which point the fix is not on your side, it is a plan size or a queue in front of the burst. Worth reading which of the two numbers you are actually hitting before buying anything.

    78
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @flashcard_fen · 2mo ago

    For what it is worth, 90ms functions and a 120 burst is exactly the profile where an HTTP-based database driver stops being a novelty. No sockets to run out of, no pool to size, and the connection ceiling stops being something you think about. Bigger change than a query string parameter, but it deletes the entire category.

    44
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @pivot_pilot · 2mo ago

    Every serverless database story is the same story: something you did not know was a pool was a pool.

    19
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report