Ask
29

Everything has run fine for six months and I no longer remember how any of it was set up

About a dozen services on one box. It all works. I have not touched it since spring.

Last week something needed changing and I spent an evening working out where the configuration lived, which of two reverse proxy config files was actually in use, and why a container had an odd environment variable that I must have had a reason for.

Nothing was broken. I simply could not remember any of it, and I built it.

The uncomfortable version of this is that if the box died I am not certain I could rebuild it. What do people actually do here, is it documentation, is it configuration in git, or is there a way to set things up so this does not happen?

8 answers Share
Report

Answering anonymously, a moderator will review it first.

  • @readme_in_the_repo · 3w ago

    The answer that fixes it structurally rather than by trying to remember harder: put the configuration in git and treat the box as disposable.

    Concretely, one repository containing:

    • Every compose file
    • Every reverse proxy config
    • Any service configuration that is a file
    • A short readme per service
    • A single script that brings the whole thing up on a fresh machine

    The box then holds only data, and everything else is reproducible. That is the same split you already have for your code, applied to the infrastructure.

    What it solves beyond memory:

    Your two reverse proxy config files stop being ambiguous, because only one is in the repository and the box is built from it.

    The odd environment variable gets a comment next to it at the moment you add it, which is the only time you know why.

    Rebuilding becomes clone, restore data, run script: which turns your uncomfortable version into an afternoon.

    Changes get a history. Six months from now, why is this here has an answer with a date on it.

    Start by copying what is currently on the box into a repository as it is, warts included. Tidying can come later; capturing it is the urgent part.

    30
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
  • @why_not_what · 3w ago · 3 replies

    The documentation half, and the distinction that makes it worth writing: what is recoverable from the machine. Why is not.

    You can always read a config file and see what it says. What you cannot recover is that you set that odd variable because the default broke something else at three in the morning in March.

    So write down the whys, and skip most of the whats:

    Decisions with reasons. Why this service rather than the obvious alternative. Why this port. Why this filesystem.

    Things that bit you. The workaround, and what it works around. These are the highest-value lines in any homelab document.

    Dependencies between services. What must start before what, and what breaks if you stop this one.

    External things. Which domain, which registrar, where DNS is hosted, which email address the certificates are registered to. This category is entirely absent from the machine and it is what stops a rebuild dead.

    Where the data is and how it is backed up.

    One file per service, a few lines each, alongside the config in the repository. And write it at the moment you do the thing, because a documentation session scheduled for later never happens and would be reconstructing from memory anyway: which is exactly the problem you have now.

    26
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @wrote_the_readme · 2w ago · 2 replies

      The what-versus-why split is the part that makes this sustainable. I stopped documenting what is installed, because the machine knows that better than I do and my notes go stale within a month.

      What I write down is why: why this port, why this service instead of the obvious one, what I tried that did not work. None of that is recoverable from the box and all of it is what I need six months later.

      22
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
      • @why_not_what · 2w ago

        Recoverable from the machine versus only in your head. That is the whole filing rule.

        13
        Share
        Reply

        Answering anonymously, a moderator will review it first.

        Report
  • @readme_in_the_repo · 2w ago

    Configuration in git means the box is disposable, which is the actual goal behind all of this.

    10
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
  • @wrote_the_readme · 3w ago

    Six months of everything working is precisely how you end up unable to change anything.

    6
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
  • @rebuild_drill_ruth · 3w ago · 2 replies

    The thing that tells you whether any of the above is actually working: rebuild something on purpose.

    Pick one service, ideally a middling one rather than the most important, and rebuild it from your repository onto a fresh container or a spare machine. Not a thought experiment, do it.

    What it finds, every time:

    • A file that was never in the repository
    • A manual step you did once and never recorded
    • A secret that lives only on the box
    • A dependency on something you set up years ago and forgot existed
    • A restore procedure that does not work

    Each of those is a landmine you have now defused on a calm evening rather than during an outage.

    Do it once or twice a year, on a different service each time. It takes an hour and it is the only honest test of whether you could recover.

    A smaller version, if a full rebuild is too much: read your own documentation as though you were somebody else and try to follow it. You will find the steps that only make sense to you.

    And on your immediate situation, the useful order: capture what exists into git first, write the whys as you go, then do one rebuild drill. The first step alone removes most of the risk you are worried about.

    1
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @wrote_the_readme · 2w ago

      The rebuild drill is the test and it is uncomfortable in a useful way. Pick the least important service, destroy it, bring it back from what you have written. Whatever you had to remember instead of read is the gap.

      16
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report