Dilek

Find it before deleting anything. Two commands answer almost every case.

df -h                    # which filesystem is full
sudo du -xh / | sort -h | tail -40

The -x matters — it stops du wandering into other mounted filesystems and giving you a total that means nothing. If you have an interactive tool available, ncdu -x / is the same information you can walk around in, and it is worth installing while you still have the space to install it.

The usual culprits, in the order I find them:

• Logs. Systemd's journal grows until told not to. journalctl --disk-usage reports it and journalctl --vacuum-size=200M trims it, permanently once you set the limit in the configuration.
• The package cache. Every distribution keeps downloaded packages after installing them. Clearing it is entirely safe.
• Old kernels, which is the same problem as a full boot partition.
• Container images and volumes, which is usually the answer on a machine running anything containerised.

The three that catch people, because the space does not show up where they look:

A deleted file still held open. A process holding a deleted log file keeps the space allocated until it closes it, so du says the space is free and df says it is not. lsof +L1 lists them, and restarting the offending service releases the space. This is the classic "I deleted it and nothing changed".

Files hidden under a mount point. If something was written to a directory before a disk was mounted over it, the data is still there and invisible. Mounting the root filesystem elsewhere and looking is the only way to see it.

Snapshots. On filesystems with snapshots, deleting a file frees nothing while a snapshot references it. df says full, everything you can see is small, and the answer is to expire snapshots.

Any of those three explains the case where you delete a lot and the free space does not move.

The rule that has saved me data: before running any repair, mount read-only and copy off what matters.

sudo mount -o ro,noload /dev/sdX1 /mnt

Many filesystems will mount read-only when they refuse to mount writable, precisely so the data can be rescued. Copy first, repair second.

And if the drive is making noises, showing read errors in the kernel log, or the checker is reporting large numbers of problems, stop and take an image of the whole device with a tool that tolerates bad sectors before doing anything else. Repairs write to the disk, and a failing disk is the worst place to be writing.

For the machine you already have, the way back is an inventory rather than a rebuild.

List what the package manager knows about, and then treat everything else as a manual item. On the systems I run I keep a plain text file recording anything installed outside the package manager, with where it came from and how to update it. It takes a minute per item and it is the difference between a machine you can hand over and one only you can maintain.

The test for whether you have a problem: could somebody else patch this machine after a vulnerability announcement, without asking you? If the answer is no, the inventory is what fixes that, not reinstalling.

One portability note, since it connects to the storage question: check how the application writes links.

If they are ordinary relative links in plain files, they survive moving to another application. If they are identifiers resolved through a database, they do not, and a migration turns your linked collection into disconnected text.

That is worth checking before investing years of writing in one, and it is not something the marketing pages mention.

Worth knowing about compressed swap in RAM, since it changes the calculation on small machines and is now a default on several distributions.

Instead of a partition on disk, a block of memory is used as compressed swap. Pages are compressed rather than written out, which is far faster than a disk and gives you an effective memory increase of roughly two or three times for compressible pages.

On a small single-board machine or a laptop with limited memory, this is the single most effective change available, and it is often just installing a package. It combines fine with a small disk swap as a second tier for genuinely cold pages.

It is also much kinder to a solid-state disk than heavy conventional swapping, which is the other reason it has become the default.