Optimising the Linux upgrade process

There are a number of ways to make Linux maintenance easier. The approach described here is designed to minimise the effort involved in upgrading to the next release of your distro. Since this will be done once or twice a year its worthwhile organising things to make the process as fast and simple as possible and also to keep track of the extra software you'll want to continue using after the upgrade. Few things are more annoying than finishing a clean install only to find that some package you use a lot is missing and then you don't remember where you got it from when you installed it last year.

This guide is written round RedHat's Fedora distro but should be readily adaptable to most other distros.

Why only doing clean installs is a good idea

If you've added much stuff to your setup you may well find that upgrading to the next release of your distro is (a)slow and (b)doesn't work, so personally I only do clean installs, but there are ways of organising yourself and the installation so that's easier.

Preparing to use the clean install approach

You have some preparatory work to do on the existing installation before you can use this approach for the first time. The essentials are:

1. Move the /usr/local directory to /home/local and replace it with a symlink to /home/local.
2. Do the same for anything else under /usr that isn't part of the standard distro, e.g. Oracle Java. See below for details.
3. Do the same for anything under /var that you'd like to back up alongside the contents of /home. Suitable candidates are the Apache root directory, Postgres or MySQL database storage, cvs repository, etc. You'll also need to change their configurations, check that they still work and make copies of the modified configurations somewhere that will be safe. Putting the copies in your development user solves that problem. See below for details.
4. Back up the contents of the, now extended, /home directory tree.
5. Run 'df -h' and print or otherwise preserve its output. You'll need it when you're deciding how big to make the various partitions in your customised disk layout. You'll probably also want to refer to it as you install each new release of your distro as a reminder of what partition goes on each mount point and which of them MUST NOT be reformatted. You don't have to repartition the disk next time but you do have to tell the installer how to map file systems to partitions.

How to do a clean install without losing any data

The key idea is to make /home into a separate partition. Doing this means that you can format every other Linux partition on the disk and do a clean install without losing any personal data. The steps are:

1. Start the installer and test the installation media as usual.
2. Tell the installer you want to do a custom install. This lets you describe your chosen partitioning scheme to it rather than blindly using whatever the installer's designer thought you'd want. When you're happy it will format the partitions and do the rest of the installation as usual except that it configures the system to auto-mount all the partitions. The default for most distros is to use as few partitions as possible. Often that means three:
   • /boot
   • and a giant partition occupying the rest of the disk that contains the rest of the Linux filing system.
   • a swap partition
   I think a more sensible scheme involves at least four partitions:
   • /boot
   • /
   • /home
   • swap
   In addition I separate out /tmp so a runaway process is less likely to crash the system by filling it. I add a few extra touches that make post installation configuration much faster and avoids the need to restore any data if the install is replacing the version of Fedora on the same disk:
   • move the contents of /usr/local to /home/local and replace 'local with a symlink, so after a clean install I can do:

     cd /usr; rm -rf local; ln -s /home/local local

     and immediately have all locally developed scripts and programs available. This doesn't lose anything because a clean Fedora install creates the /usr/local directory structure but leaves it empty.
   • I do the same with /usr/java
   • I keep copies of everything I've edited in /etc in my main development user (backed up with cvs) so they can also be dropped back in place after a clean install after checking that the program(s) in the new version haven't totally changed their config file formats.
   • I run PostgreSQL and Apache By default these put their data in /var. I've moved their directory structures into directories in /home and changed their configurations to match because both deal with large amounts of data.
   • I also run a private DNS service which is definitive for my LAN. The named zone files are small, so I simply keep copies of them which are dropped back into /var/named after the install.
3. After a clean install and an immediate update to pick up the latest kernel and updated packages:
   • manually add your /home logins back with the users & groups tool, taking care that the logins point to the original home directories and that you manually set uid and gid to their previous values. Running "ls -l /home" shows you the old UID and GID values alongside the home directory names.
   • I delete /usr/local/*, which is always empty after a clean Fedora install, and replace it with the symlink to /home/local, which immediately makes all my locally developed programs and scripts available.
   • I use Oracle Java, not OpenJava. After the fresh install I simply recreate the Java symlink and Java is back.
   • I keep copies of every config file I change in /etc in a mirror structure under a user in /home. After a clean install these are copied back to /etc. Where there's a clash I make sure there are no new parameters for the new program versions and fix and/or merge the config file versions as necessary.
   • Make good use of separate scripts in /etc/profile.d to set environment variables, search paths etc for each utility or application that needs this treatment. Using them really saves a lot of hassle. Save these scripts along with everything else you've manually edited in /etc so you can put them back after a clean install. This will also save your bacon after some package upgrade wipes out your customised less,cvs or vim configuration.
   • Use the Services GUI and/or chkconfig and service to mark your usual daemons active and start them. By now you're pretty much back where you were before the reinstall.
   • Keep a list of the packages you add to the system and where they came from. Sounds obvious, but not doing it is a real pain.

I've now scripted everything described above that can be automated apart from checking config files, putting them back and starting the services, so all the 'just do it' steps take about 5-10 minutes to run. Its that long because the script also installs the 30-40 'extra' Fedora packages I use plus a few from rpmfusion and atrpms. Without this yum activity the scripts would run in about 30 seconds.

The process of restoring configuration files for the major services and then starting them remains a manual procedure because distro upgrades typically install a later version of these services and this often means that the configuration parameters have changed as well: I use diff to see what has changed and then decide whether to continue using my original configuration files or to apply my changes to the newer files and, after verifying that everything is working correctly, replace my backup configuration file set with the new one.

As I said above, another reason for working this way is that an upgrade seems to take around twice as long as a clean install. This is not surprising when you remember that it has to compare all the files before it can delete and reinstall any file that has changed. I could probably live with that if it worked, but almost every time I've tried an upgrade the system has been trashed and I've had to do a clean install after all. In fact RedHat has now realised that upgrading across a version boundary usually fails and has removed that choice. However, a cross-version upgrade is still the preferred option for Debian stable installations and, presumably, for Debian-derived distros too.