Automating Server Maintenance

As a self hoster, server maintenance is important. Running backups is important in case something goes wrong and updating software is important so that security fixes are used. However, this takes time and we would preferably spend time on life instead of server maintenance. So, the question is what to do and how much to automate?

My answer is to accept some manual work. Even the best automation cannot capture all things that can go wrong. For example, if you make a robot which checks your car for rust on the doors, then this robot may not notice the hole in the roof. If you would personally walk past the car and check it, then you would certainly notice a hole in the roof. It's the same with servers, there is no point in trying to automate everything. Nothing beats an occasional look; you might spot something that no automation would have ever spotted.

There is also a benefit to doing manual software updates and a moment which is right for you. If you have to update, say, software A, B, and C, then take the following strategy for each update:

1. Update the software.

2. Test whether the relevant parts of the server still work.

If the second step fails, you immediately know what caused the server to break and you are ready to fix it straight away. Alternatively, an automated update could have occurred without you knowing it and then you have to first figure out what caused the breakage.

Another benefit is that you'll get a feeling for the server. You'll know how to log in to the server, what range the memory usage should be, and what range the disk usage should be.

That's why is advice to schedule a regular maintenance round, which should take only about 15 minutes or so. To keep track of it, most calendars allow you to create a recurring event and receive an email on the event date. With that, you'll get an email reminding you to conduct a maintenance round.

During the round:

1. Pull a manual backup to your system as described in the backup post and verify the backup.

2. Check memory and disk usage.

3. Update system wide packages and reboot if necessary (and test as described above).

4. Check for newer versions of the specified Docker images in the docker-compose.yml files. See the Docker Compose Cheatsheet for more information about restarting services.

That's it. All should be safe now 🔐.