Ask
29

Eleven containers on a VPS and I could not tell you which of them holds anything worth backing up

Everything is in compose files, which are in git, so rebuilding the stack is straightforward. What I do not have is any backup of the data.

The problem is that I do not have a clear picture of where the data is. Some containers have named volumes, some have bind mounts to directories on the host, and at least one I am fairly sure writes inside the container itself, which I understand means it is gone if the container is recreated.

I would like to fix this properly rather than tar the whole disk once and feel better.

What is the sensible approach - how do you work out what actually holds state, and what does a backup that would survive losing the box look like?

8 answers Share
Report

Answering anonymously, a moderator will review it first.

  • @named_volumes_only · 3w ago · 2 replies

    First find the state, and there are only three places it can be.

    Named volumes. Listed with a volume listing command, and you can inspect each one to see which container uses it. This is where state should be.

    Bind mounts. Visible in your compose files as host paths. Easy to find and easy to back up because they are ordinary directories.

    Inside the container's writable layer. This is the one you correctly identified as a problem. Anything written to a path that is not a volume or a bind mount lives in the container and disappears the moment it is recreated, which happens on every image update.

    To find that third category: run a diff on a running container, which shows what has changed relative to its image. Anything meaningful in there is unprotected state and the fix is to add a volume for that path, not to back it up where it is.

    The cleanup worth doing while you are here: make every stateful path a named volume, consistently. Bind mounts are fine and they mix awkwardly with named volumes when you are trying to reason about what exists. One convention means the answer to what holds data is a single command rather than an archaeology exercise.

    And note which containers genuinely hold nothing: most reverse proxies and workers do, and knowing that shortens the list a lot.

    30
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @bind_mounts_only · 2w ago

      Switched everything to bind mounts under one directory precisely because of the question in this post. Named volumes are fine technically and they put your data somewhere you do not think about, which is how you end up unable to answer what is worth backing up.

      One directory, everything under it, back up that directory. The inventory problem disappears.

      18
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
  • @bind_mounts_only · 3w ago

    Compose files in git means you can rebuild the stack and none of the data. Those are different problems and it is easy to feel covered.

    8
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
  • @stop_then_copy · 3w ago · 2 replies

    On taking the backup, the thing that separates a backup from a false sense of security: databases must not be copied while running.

    Copying the files under a live database gives you a snapshot taken across a period of time, with writes happening during it. It may restore. It may restore into a corrupt state that looks fine until it does not. This is the most common way people discover their backups are worthless.

    So, per type:

    Databases: use the database's own dump tool, from inside the container, on a schedule. That produces a consistent file you can restore anywhere. Alternatively stop the container, copy the volume, start it again, if a few minutes of downtime is acceptable.

    Everything else: files, uploads, configuration: can generally be copied while running, though stopping is still cleaner.

    Configuration and compose files, already in git, which is the right answer.

    The practical shape for a small box: a script that dumps each database, copies each volume, and hands the result to a backup tool that does encrypted incremental backups off the machine. Run it nightly from a timer.

    Off the machine is the part that matters. A backup on the same VPS protects you from a mistake and not from losing the VPS, which was your stated concern.

    1
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @restored_at_2am · 2w ago

      Copying a database file while it is running gives you a file that looks like a backup and is not one.

      11
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
  • @restore_is_the_test · 3w ago · 3 replies

    And the part almost everybody skips: a backup you have never restored is a hypothesis.

    Do a real restore now, before you need one. Not a checksum, not a listing of the archive - an actual restore onto a fresh box or into a throwaway stack, and then open the application and check the data is there.

    What that exercise reliably uncovers:

    • A volume you did not know existed
    • A database dump that was empty because the credentials in the script were wrong
    • Files with ownership that makes the application unable to write after restore
    • A secret or an environment file that was never in git and never in the backup, so the stack will not start

    That last one is the most common and it is the one that turns a two-hour recovery into a two-day one.

    Then make it a habit, restore once a quarter, and write down the steps as you go. That document is worth as much as the backup, because you will be doing this on a bad day.

    One more thing worth deciding now: how much data can you afford to lose? Nightly backups mean up to a day. If that is not acceptable for something, that specific thing needs more frequent dumps, and knowing which one it is stops you from over-engineering the other ten.

    1
    Share
    Reply

    Answering anonymously, a moderator will review it first.

    Report
    • @restored_at_2am · 2w ago · 2 replies

      Tested a restore for the first time during an actual incident, which I do not recommend. The backup was fine. What I did not have was the knowledge of what order things had to come back in, and two services would not start because a third had not.

      A restore drill teaches you the sequence, and the sequence is the part that is not in the backup.

      23
      Share
      Reply

      Answering anonymously, a moderator will review it first.

      Report
      • @restore_is_the_test · 2w ago

        A hypothesis until tested, and the untested part is usually the order rather than the data.

        14
        Share
        Reply

        Answering anonymously, a moderator will review it first.

        Report