231
back after seven months: build fails, ci is red, and i don't remember how any of it works
Left this alone in December. Opened it last night, ran the install, and got a wall of errors before anything even started. CI has been red since some point in the spring. There are eleven paying customers still on it, which is the only reason I came back.
I genuinely do not remember how the deploy works. There is a script called deploy.sh and I am afraid of it.
Where do you start with something like this? My instinct is to update everything and start clean, but I have a feeling that is the wrong instinct.
@scopecreep_sam · 3mo ago · 3 replies
Your instinct is wrong and you already suspect it. Updating is how a broken build becomes an unrecognisable broken build.
Order that works: get it building exactly as it was. Lockfile untouched, same runtime version. If there is no file pinning the runtime, infer it from the lockfile format and whatever your CI config says it used, then install that specific version locally. Only once it builds green do you touch a single dependency, and then one at a time.
Second thing, before any code: write a page of notes about what is actually true. Where it deploys, which service holds the database, what the deploy script does line by line, where the secrets come from, what happens on a new signup. You are the only person who can write that document and last night proved you are also losing it.
Third: do not deploy anything until you have reproduced a deploy of the unchanged code. Finding out the pipeline is broken while you also have a change in flight is how a bad evening becomes a bad week.
Reply
Report
@errwrap_elle · 3mo ago
Came back to something after five months and did it in the wrong order, so this is what not to do. I ran the update command first, on everything, because it felt like tidying. That turned about four real errors into thirty, and I could no longer tell which were from the gap and which were from me. I ended up resetting to the last commit and starting again with pinned versions, exactly as described here, and it built on the second attempt. The whole detour cost me a weekend and taught me nothing.
Reply
Report
@darning_dot · 3mo ago
The dependency you did not change is rarely what broke. Your CI image moved underneath you, or your local runtime got upgraded by an unrelated update. Check what version is actually running before blaming the code.
Reply
Report