Pattern is always the same. Sunday evening is my longest block, I finish something around 22:30, ship it because it feels wasteful not to, go to bed, wake to error emails, fix it at 6am, then do a full day of work on five hours of sleep. Four times in eight weeks. The product is fine, I am the failing component.
The other half is being able to undo it in sixty seconds. One command that puts the previous release back, and a health check the deploy itself watches — if the check fails, it rolls back without you. Once rollback is genuinely trivial, a Sunday night deploy stops being a risk, because the worst case is a two minute recovery rather than a 6am debugging session.
Plus a smoke test that curls three real URLs after the deploy and shouts if any of them is not 200. Twenty lines. Catches the "forgot to set the env var in prod" deploy, which in my experience is over half of them.
Also get the product into a state where being down for eight hours is survivable. Status page, an autoresponder on support, an honest changelog. Solo products get a surprising amount of grace when the person behind them is visible and straightforward about it — silence is what makes people leave.
Four out of eight is not bad luck, it is an untested area. What broke each time? If it is the same subsystem twice, that is one focused afternoon of tests, not a character flaw. People reach for discipline when the answer is a fixture.
The 6am fix is the thing that will eventually end this project. The outages will not.
@lena_arbogast · 2mo ago · 2 replies
Stop deploying at the end of a session. Deploy at the start of the next one.
Right now you ship at the exact moment you are most tired and have the least remaining time to fix anything. Write it Sunday, ship Monday evening with two hours in front of you and a clear head. It costs you a day of latency on a product nobody is waiting on, and it removes this entire category of problem.
I did this after a similar run and have not had a night-time incident since. It is not discipline, it is not deploying while impaired.
Reply
Report
@airflow_amir · 2mo ago
Embarrassingly obvious once written down. Added a rule that nothing ships that I have not run locally twice, and shifted deploys to Monday. Three weeks clean.
Reply
Report