Saving a Server with a Full Disk

My self-hosted server with some Docker Compose services kept humming along nicely until today. Upon logging in to the server, it turned out that the disk usage was at 99%! Of course, I should have set up notifications for the disk usage.

Like a true full disk, most programs didn't start anymore including docker and apt. To fix the issue, a first thing is to regain some space so that apt works again:

$ rm -rf /var/lib/apt/lists/*

$ rm -rf /var/cache/apt

Next, the Docker logs are likely to be the culprit, so clean them too (thanks to https://github.com/moby/moby/issues/33775):

$ find /var/lib/docker/containers -type f -name "*.log" -print0 | du -shc --files0-from - | tail -n1
5.1G    total

$ truncate -s 0 /var/lib/docker/containers/*/*-json.log

$ find /var/lib/docker/containers -type f -name "*.log" -print0 | du -shc --files0-from - | tail -n1
4.0K    total

This should make the server functional again. It's very likely that some services like the nameserver are down To get all services running again, reboot the server.

$ reboot now

and then prune Docker:

$ docker system prune -af

Now, everything should work again. To clean up further, install ncdu and run:

$ ncdu /

This will order all folders by size. You can navigate the list with the arrow keys and Enter.