265

Nightly restic backup takes nine hours and the repo grew 400GB in a month Backups

Backing up about 3TB from the server to a remote object store with restic, nightly, single job. It started at around 90 minutes and it now takes nine hours, and the repository has grown by roughly 400GB in the last month even though I have not added anywhere near that much new data. Retention is set to keep daily for 30 days. There is a photo library, a media folder, and a couple of database dumps in there. Where is that growth coming from?

8 answers Share
Report

Answering anonymously — a moderator will review it first.

  • @cloze_kai · 7mo ago · 2 replies

    For the nine hours, check whether you are running forget with prune in the same window. Prune on a large remote repository is enormously slower than the backup itself and it is easy to have scheduled them together. Splitting prune to weekly took my nightly job from six hours to 40 minutes.

    198
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @quietpackets · 6mo ago

      Also worth separating the media folder into its own repository with a different schedule. Backing up rarely-changing bulk data nightly is mostly a waste of scanning time.

      82
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @chmod_confused · 7mo ago

    Time the phases so you know what is slow: scanning for changes, uploading, or the index work at the end. If scanning dominates, the metadata cache on the client is probably being lost between runs, which happens a lot inside containers with no persistent volume for the cache directory.

    164
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @stdlib_stef · 7mo ago · 3 replies

    The database dumps, near certainly. A full dump written as a fresh file every night deduplicates badly because compression changes the byte layout of the whole file even when 99 percent of the data is identical. If they are compressed dumps, that is your 400GB. Two fixes: dump uncompressed and let restic handle the compression and dedup, or use a format that produces stable output. People are always surprised how much a 10GB nightly dump costs over a month.

    246
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
    • @back_of_envelope · 7mo ago

      They are gzipped, about 12GB a night. So roughly 360GB a month, which matches the growth almost exactly.

      108
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
    • @stdlib_stef · 7mo ago

      There it is. Switch to plain dumps and the same month will cost you a few gigabytes. Also check whether gzip is set to include a timestamp in the header, because that alone can defeat dedup on otherwise identical files.

      124
      Share
      Reply

      Answering anonymously — a moderator will review it first.

      Report
  • @flakey_test · 7mo ago

    Whatever you change, do a restore test afterwards. A backup you have never restored is a hypothesis.

    152
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report
  • @expense_ratio_ed · 7mo ago

    One financial note since you are on object storage: check whether your provider charges per API request. A nightly job on millions of small files can cost more in requests than in storage, and it does not show up until the bill.

    138
    Share
    Reply

    Answering anonymously — a moderator will review it first.

    Report