This blog and just about all of my systems are powered by OpenBSD, the most secure operating system on the planet. While OpenBSD might not be the snappiest (in terms of speed), you can stand up a secure server and really don’t have to worry about it as chances are good the hardware will fail before the operating system does. As luck would have it, the hardware did fail on the server where my virtual machine was located. I did have backups of the most critical stuff but I went about doing it in a very inefficient way. In the post mortem, I learned a few things that I thought I would share here. There are tools like tar(1) and dump(1) and restore(1). There are merits to both. Dump and restore are nice tools for backing up and restoring a filesystem and doing incremental backups. Tar is nice for creating archives of files.
My backups were separate tarballs for /etc, /var/www, /home, and a mysql database dump. Needless to say this created a lot of extra work for me as inevitably I missed things. When I created the tarballs, I created them on the server and then copied them via scp(1) down to my laptop. Of course when I went to rebuild my server, after reinstalling OpenBSD, I copied the tarballs back to the server and then extracted them in my home directory and manually copied everything over. It was a lot of needless extra work. So here is what I ended up doing and testing once I rebuilt my server and everything was functioning. Time to create and test a disaster recovery plan.
One of the very powerful aspects of SSH is the ability to run commands remotely and pipe input and output to it. I read about people doing this but never really tried it myself until today. In the end I chose to use tarballs because I don’t really need to rebuild a filesystem. SSH is just one of those real swiss army tools. So the first step is to backup the current server. The real challenge for me was figuring out how to take the output coming from the SSH session and dump that to file. There are several ways to do it but I chose to use dd(1). The dd command seems purpose built to do this kind of thing.
ssh user@server "cd / && doas tar -cpvz -f - /" | dd of=backup.tgz
The command above executes tar on the server. The portion with the ‘-f -‘ tells tar to send the output to the standard output instead of a file. In this case the standard output is being sent through the SSH session and then piped to dd which writes the data to my local disk. As a nice extra bonus, I discovered that tar did not archive the socket files in /var which really helped. After the backup completed, I stood up a server just for a test restore. This time I effectively did the reverse.
dd if=backup.tgz | ssh user@testserver "cd / && doas tar -xpvz -f - "
The command above sends the backup archive to the standard output and pipes it to the SSH session to the server. From there, the tar ‘-f -‘ command tells tar to extract the files from the archive being read by the standard input. In the end this worked perfectly. After the restore, I rebooted the server and everything came up without error. In about 25 minutes, I had a perfectly restored and ready server. Thank you to SSH and the swiss army of tools available!