We run a full staging stack that mirrors prod's services and I can't remember the last time it caught anything real. Everything that breaks breaks in production, usually on data shapes we don't have in staging. Six months in and I'm wondering if we're maintaining an expensive copy of nothing. Is this just how it goes or are we using it wrong?
One thing that does help: periodically restore an anonymised copy of the production database into staging. Not live, not continuous, a monthly snapshot with PII scrubbed is enough to surface most shape problems. The migration that took 40 minutes and locked the table in prod would have taken 40 minutes in staging too, and we'd have known.
Seconded, but flag the legal side. Depending on where you operate, anonymised has a specific meaning and a quick scrub of names isn't it. Worth a conversation with whoever owns that before you copy anything.
Check whether staging carries any load at all. Ours got no traffic except deploys, so nothing warmed up, no cache eviction, no connection pool pressure, no interesting concurrency. We started replaying a sample of production traffic at it and it immediately started finding things.
Blunt version: staging catches wiring bugs, production catches reality bugs. If you built it hoping for the second thing you built the wrong tool, but the first thing is still worth having.
@rollback_rae · 10mo ago · 3 replies
Very normal, and the cause is almost always data rather than code. Staging has 400 clean rows, production has four million with a decade of edge cases in them: the null in a column that's supposedly not null, the account created in 2014 before a field was required.
Two things move the needle far more than making staging more prod-shaped. Feature flags with a real progressive rollout, so production is your last test environment but only for 1% of traffic. And making rollback boring and fast, because if you can go back in 90 seconds you can afford to learn in production.
Keep staging for smoke tests and integration wiring. Stop expecting it to find behavioural bugs.
Reply
Report
@shortcut_sam · 10mo ago
This reframing helped more than anything else here. Staging keeps the job of proving the services talk to each other, everything else moves behind flags.
Reply
Report
@two_calendars · 10mo ago
Same conclusion at my last place. The tell was that every incident review had a line saying staging didn't have this data. After the fourth one we stopped pretending.
Reply
Report