Rae

Nine people and no platform person: pay for hosted. The real cost of self-hosted isn't hardware, it's the Tuesday where a runner has a full disk and nobody knows why the queue is stuck.

Two things to try before spending anything. Run the integration Postgres on tmpfs, which routinely cuts DB-heavy suites by a third because fsync is what's eating you. Then shard the suite across four parallel jobs. If 22 minutes becomes 7 on standard runners you've solved it for free and the hosting question goes away.

Probably true long term, but that's migrating a whole monorepo to fix a two minute install. Worth doing on purpose, not as a debugging step.

setup-node's cache only stores the npm download cache (~/.npm), not node_modules. So on a hit you skip the network, but npm still unpacks and links every package, which is the slow part in a monorepo. Two fixes, pick one: cache node_modules yourself keyed on the lockfile hash and skip npm ci entirely when the key hits, or change the install to npm ci --prefer-offline --no-audit --fund=false. The second one usually buys 30-40% and takes one line.

Endpoint propagation. When a pod goes Terminating, kube-proxy and your ingress controller find out asynchronously, and there is a window where the Service still lists it. Your app exiting in 2 seconds is actually the bug: it dies while load balancers are still handing it work.

Standard fix is a preStop hook that sleeps 5-15 seconds before the app starts shutting down, so the pod stops being advertised, in-flight requests finish, then you exit. Also make readiness start failing on SIGTERM, not just liveness.

The mental model that fixed this for me: readiness controls whether you get new work, SIGTERM handling controls whether you finish the work you already have. Two separate switches, and they have to flip in that order.

Rotate the token first, today, before you touch anything else. Once it's rotated the leaked value is worthless and the rest is hygiene rather than an incident.

Then yes, the value lives in the layer metadata forever for those images. You can't edit history in place, only delete tags and garbage collect. Order I'd work in: rotate, delete affected tags, run registry GC, then check whether the registry replicates anywhere or gets backed up, because that copy needs purging too. Don't spend a day scrubbing images before the rotation is done.

Very normal, and the cause is almost always data rather than code. Staging has 400 clean rows, production has four million with a decade of edge cases in them: the null in a column that's supposedly not null, the account created in 2014 before a field was required.

Two things move the needle far more than making staging more prod-shaped. Feature flags with a real progressive rollout, so production is your last test environment but only for 1% of traffic. And making rollback boring and fast, because if you can go back in 90 seconds you can afford to learn in production.

Keep staging for smoke tests and integration wiring. Stop expecting it to find behavioural bugs.