Ask
187
@edge_runtime_bo ·

node container exits 137 under load on a 2gb vps with mem_limit 512m Dockerfile

Small API, Node 22, running under compose on a 2 GB Hetzner box next to postgres and caddy. Under any real traffic the api container dies:

api    Exited (137) 8 seconds ago

docker inspect --format '{{.State.OOMKilled}}' api says true. I set mem_limit: 512m in compose specifically to protect postgres.

RSS sits around 300 MB idle and climbs to about 540 under load, then it's killed. What I don't understand is why node doesn't garbage collect harder as it approaches the limit instead of walking straight into it. Is 512m just too small for Node, or am I holding this wrong?

13 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @coworking_cass · 6h ago · 3 replies

    Node doesn't know about your cgroup limit. V8 sizes its old-space heap from what it believes total system memory to be — on a 2 GB box that's roughly 1 GB — so it happily grows toward 1 GB while the cgroup kills you at 512 MB. It isn't failing to GC, it thinks it has plenty of room left.

    Tell it the truth:

    environment:
      NODE_OPTIONS: --max-old-space-size=384
    mem_limit: 512m
    

    Leave headroom, because old space isn't the whole RSS — buffers, the code cache, native modules and stacks all live outside it. 384 under a 512 limit is about right; if you do a lot of Buffer work, leave more.

    Then watch it. If it now GCs constantly and CPU spikes instead of dying, the limit really is too small and you raise it. Nine times out of ten this is the entire bug.

    176
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @edge_runtime_bo · 13h ago

      Two days. That's how long I spent reading my own code looking for a leak. Set it to 384, ran the same load test, peak RSS 470, no kills.

      52
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @wonder_why_wren · 3h ago

      Same class of bug hits the JVM, Go (GOMEMLIMIT) and Python with big allocators. Any runtime that autotunes off host memory needs telling when it's inside a cgroup.

      31
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @flux_and_solder · 3h ago · 2 replies

    To be precise about the diagnostic, because it's worth having for next time: 137 is 128 + 9, so SIGKILL. That on its own doesn't mean OOM — you also get it when compose gives up waiting during docker stop and kills the container. The thing that told you it was OOM is the OOMKilled: true you already checked, so good instinct.

    If it had been the stop-timeout case, the fix would be handling SIGTERM properly rather than touching memory at all.

    74
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @edge_runtime_bo · 3h ago

      Noted, and mildly relevant — I currently don't handle SIGTERM at all, which explains why every deploy takes exactly ten seconds.

      18
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @negativesplitz · 9h ago

    Since postgres is on the same 2 GB box: give it an explicit limit too, and check shared_buffers. The default in the official image is sane, but people bump it to 25% of host RAM without registering that three services share that host. Two containers both sizing themselves from 2 GB is how you get a machine that OOMs at the kernel level and takes down something unrelated.

    Also check whether you have any swap. Most VPS images ship with none, and with no swap there's no soft failure mode — you go from fine to killed with nothing in between.

    41
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @mulch_mule · 3h ago

    Measure before you tune. docker stats for the crude version, or run cAdvisor and get real numbers. I've watched people set max-old-space-size by feel and then spend a week wondering why p99 got worse — a heap that's slightly too small gives you constant major GCs, which is death by a thousand 40 ms pauses instead of one clean crash.

    26
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @muslin_mira · 3h ago · 2 replies

    Also worth confirming there isn't a real leak underneath the misconfiguration, because 300 MB idle is high for a small API. Take two heap snapshots ten minutes apart under steady load and diff retained size. If it's flat, you're fine and this was purely heap sizing.

    19
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @edge_runtime_bo · 24h ago

      Idle is mostly a big in-memory geo lookup table, about 180 MB. Legitimate, but I should probably move it out of the API process entirely.

      11
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @oncall_omar · yesterday · 3 replies

    just drop mem_limit, the box has 2gb and the kernel will sort it out

    18
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @oncall_omar · 13h ago

      That's a fair point I hadn't thought about — who gets picked, not whether something gets picked.

      12
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @coworking_cass · 3h ago

      Then the kernel OOM killer picks a victim by badness score, and it will frequently pick postgres, because postgres is the fattest process on the box. You'd be trading a clean four-second API restart for database downtime. Limits are how you choose who dies.

      15
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @rubberduck_rae · 3h ago

    Set restart: unless-stopped at minimum while you sort this out, so a kill at 3am is a four-second blip rather than your morning.

    8
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report