Notes from my server upgrades to Ubuntu 20.04 LTS

This past weekend I learned that Ubuntu 20.04.1 LTS was out. This meant that I could start upgrading my servers from Ubuntu 18.04 LTS to Ubuntu 20.04 LTS as that is when do-release-upgrade makes the next LTS release available for upgrade.

I started with my web server i.e the server this blog is hosted on. Mostly because I’ve been wanting PHP 7.4 for my WordPress sites and Ubuntu 20.04 would give me that. The upgrade went without issue and after a small configuration change to nginx to make sure it used the new Unix socket for PHP 7.4 FPM, we were all good to go.

From then on, I went ahead and started doing more of the servers, in the order of the servers that were least likely to break with the upgrade.

Breakages and Manual Interventions

Some of my servers failed to upgrade the grub-pc package during the dist-upgrade process. dpkg simply returned a error code. This was fixed by purging grub-pc and installing the grub2 package and then configuring it according to Linode’s documentation on GRUB2. I’m not entirely sure why the grub-pc package failed to upgrade.

All my servers also needed the apt sources list entry for the Icinga repository manually updated to use focal. My Mastodon servers needed repository entries for nodesource manually updated as well.

Ubuntu 20.04 brought with it PostgreSQL 12 and that means manually upgrading the installs and databases on my PostgreSQL 10 servers. This involves dropping the newly created empty 12 ‘main’ cluster and then running pg_upgradecluster on the 10 ‘main’ cluster. After that completed, the 10 ‘main’ cluster can be dropped. I didn’t run into any issues here. I did take a manual backup of database(s) before I started the process and I would recommend everyone do that if possible.

I will however note here that since the process creates a copy of the database into the 12 cluster, you will nearly double your disk usage. This can be a problem if you have large pgsql databases and do not have enough spare disk space on your servers to complete the process. Something to keep in mind if you are planning on doing one of these.

Since PostgreSQL was upgraded to 12, the pgbackrest configuration and stanza needed to be updated. Their documentation goes through the process but I didn’t need to use it as it is fairly straight forward.

Bits, Bobs and Final Thoughts

Overall the process of upgrading my servers to Ubuntu 20.04 was fairly smooth. I didn’t encounter any catastrophic failures or data loss. If I had I could have reverted to the manual snapshots I took of the servers before I started.

I’m glad to finally have PHP 7.4 on my web server and WordPress no longer complaining about having to use PHP 7.2.

Another neat thing to note is that one of my servers was initially provisioned on 14.04 and has over the years been upgraded through LTS Ubuntu releases. So 14.04 -> 16.04 -> 18.04 -> 20.04.

That’s all from me!

TestDisk and your lost partition table

Have you ever tried to resize a ext4 partition using fdisk but end up accidentally wiping your partition table instead?

It can be a panic inducing moment but all is not lost (including your data) .

Turns out that there is a FOSS tool called TestDisk that is designed to be used to recover from such human errors.

From their web page:

TestDisk is powerful free data recovery software! It was primarily designed to help recover lost partitions and/or make non-booting disks bootable again when these symptoms are caused by faulty software: certain types of viruses or human error (such as accidentally deleting a Partition Table). Partition table recovery using TestDisk is really easy.

I can attest to the easy part, I have in the past quite easily recovered a ext4 partition that I accidentally deleted using fdisk.

TestDisk is distributed as static binaries for various operating systems so it is handy to keep a copy of it on any sort of recovery/rescue media that you use.

A protip about fdisk

Do not use fdisk unless you are absolutely certain you know what you are doing with it. While TestDisk can save you if you fuck up, it is better to not take that risk.

Use something safer like parted or my favourite its GUI front-end gparted. In fact, these days I prefer to boot into the gparted LiveCD and perform my partition creations/resizes there.

Booting into a live CD is usually a slower process but it is better to be safe rather than sorry when it comes to things like partitions.

 

Adiós self-hosted mail server

As the title suggests, I will be retiring my self-hosted mail server (mx1.staticsafe.ca) soon, most likely by next month.

It has been a lot of fun and an excellent learning experience but I simply cannot devote the time and/or effort into active maintenance of the server.

I am moving all of the mailing lists that I used this server for to the extremely competent folks over at Fastmail.

The retirement will be done in a few stages as follows:

  1. Stage 1 – Personal mailing list subscription move (completed)
  2. Stage 2 – Change of MX, SPF records for staticsafe.ca to Fastmail’s servers and add a wildcard alias there.
  3. Stage 3 – Discontinuation of relay/non-personal services on mx1.staticsafe.ca. One person is using this server as a submission relay for their server. I will be asking them to move to my new MSA only server. Services for caffeine-linux.org will also have to be discontinued.
  4. Stage 4 – Stop all relevant daemons. Postfix, Dovecot, amavisd-new.

My Thanks

A lot of people and/or companies have provided direct and/or indirect support for this learning experience.

  • Linode Library for providing initial inspiration for the current configuration of the server
  • The folks in the Postfix and Dovecot communities. (#postfix, postfix-users, #dovecot, Wietse for the wonderful software)
  • purrdeta for telling me mail servers are a lot of fun

Addendum

I have put my server configuration files on Github if anyone is interested, you can see them here.