Ask
203

Node API holds 400 rps fine but memory climbs all day, is Go the fix or am I hiding a leak?

Express in front of Postgres, one 2 vCPU box, restarts itself nightly which is the only reason we do not fall over. Resident memory starts around 200MB after a deploy and is comfortably over a gigabyte by the evening, every day, whether traffic is heavy or not. The team is JavaScript native and rewriting in Go is being floated as the solution. I would like to know whether anyone has actually made this exact move and whether the memory problem followed them.

10 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @two_vcpu_club · 2w ago

    Also worth separating leak from heap growth, because they look identical from the outside. A garbage collected runtime will happily let the heap grow toward whatever ceiling it thinks it has before it works hard at collecting, and on a small box the defaults are not always what you want. Try pinning the old space limit to something well under your container limit and watch whether the curve flattens and stabilises rather than climbing. If it flattens, you never had a leak, you had a runtime doing exactly what it was told. If it still climbs and then the process dies, it is a real leak.

    152
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @parquet_pile · 2w ago

    I did make this exact move, Express to Go, for a service doing similar numbers. The honest report: memory went from over a gigabyte to a couple of hundred megabytes and stayed flat, p99 latency improved but not dramatically because we were database bound, and the whole thing took a team of two about seven weeks including the tail of bugs.

    The part I did not expect is that we shipped the same leak on day one. We ported a cache with no eviction straight across, and it grew slower because Go's memory representation is more compact, so it took three days to fall over instead of one. That was a fairly humbling deploy. Go changed the constant, it did not change the bug.

    141
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @flakey_test · 2w ago · 4 replies

    The nightly restart is a bandage over a leak, and a leak follows you into any language, because the usual cause is a data structure that only ever grows. Take two heap snapshots four hours apart under load, diff them, and look at what grew. Nine times out of ten it is an in-process cache with no eviction, a Map keyed by request or user id, or event listeners being attached per request and never removed. Fix that in Node and you can decide about Go on the merits rather than out of desperation.

    188
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @edge_runtime_bo · 2w ago

      We do have a cache keyed by user id with no eviction. I feel slightly stupid now.

      47
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @two_vcpu_club · 2w ago

      Do not feel stupid, feel relieved. That is a one afternoon fix instead of a one quarter rewrite.

      96
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @flakey_test · 2w ago

      And write it down in the postmortem, because someone will add another unbounded map in six months unless there is a rule about it.

      41
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @plainly_pat · 2w ago · 2 replies

    Going to disagree with the pure leak-hunting advice, or at least add to it. At 400 rps on 2 vCPU with Postgres behind it, you are almost certainly not CPU bound, so the rewrite will not buy you throughput. But the memory profile of a Go service genuinely is different in kind, not just degree, and if your constraint is a small box then that difference has real operational value. I would still find the leak first, because you cannot make an honest comparison against a broken baseline, and I would then evaluate Go on operational grounds like a single static binary and a much smaller image rather than on rps.

    118
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @flakey_test · 2w ago

      No disagreement from me. Find the leak first, then have the language argument on facts. My objection is only to rewriting because a graph goes up and to the right.

      57
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @chmod_confused · 2w ago

    Practical middle path we took: kept Node for the API surface, moved the one endpoint doing heavy work into a small Go service behind the same load balancer. Two weeks of work, no rewrite, and it let the team learn Go on something with a small blast radius. If the memory turns out to be concentrated in one code path, this is a much cheaper experiment than committing the whole service.

    96
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @quietpackets · 2w ago

    A team that is JavaScript native writes better JavaScript than Go for at least a quarter. Fix the leak. Revisit the language when the reason is not a graph you could have fixed on a Tuesday.

    84
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report