Updating Ubuntu over ssh
Installing over SSH
It is not recommended to install the OS over ssh. I fully support that thought, however, the reality is that there is offsite hardware and no one is scheduled to visit it for a long time and no one lives close enough to use it. Ideally, it would be a system on top of bare metal and virtual machines would be provisioned on top of that, but in this client's case it was all bare metal with the OS installed, then a Kubernetes Cluster built on top of that.
The oldest OS was Ubuntu 12.04, the newest was 16.04. They wanted everything updated to 20.04. It can be problematic to update the operating system from old versions that are currently unsupported, but the system is basically the same for each one:
Update the packages
First I updated the packages to get the latest and see if apt was even working
$ sudo apt-get update
$ sudo apt-get upgrade -y
There were some errors with old apt repositories not working with jenkins and postgres, so I removed those until everything was updated. The client wasn't conscerend with keeping anything running except a file server which required nginx and the server folder to be maintained. Normally, if I was at the server, I would just plug in a USB drive, grab the files and nginx config files off of it, and wipe the server and redo it from scratch as that is fairly straight forward, but remotely that's not an option.
Run do-release-upgrade
Now that the packages were updated, it's time to run ubuntu's do-release-upgrade to see if it will upgrade the OS.
$ sudo do-release-upgrade
...
Calculating the changes
Calculating the changes
Could not calculate the upgrade
An unresolvable problem occurred while calculating the upgrade.
If none of this applies, then please report this bug using the
command 'ubuntu-bug ubuntu-release-upgrader-core' in a terminal. If
you want to investigate this yourself the log files in
'/var/log/dist-upgrade' will contain details about the upgrade.
Specifically, look at 'main.log' and 'apt.log'.
Restoring original system state
Aborting
Reading package lists... Done
Building dependency tree
Reading state information... Done
=== Command detached from window (Wed Dec 28 16:46:39 2022) ===
=== Command terminated with exit status 1 (Wed Dec 28 16:46:49 2022) ==
So it doesn't work, time to check what's wrong:
Finding errors when upgrading
As the log said, look into the main log and see what you find.
$ grep ERROR /var/log/dist-upgrade/main.log
2022-XX-XX XX:XX:XX,795 ERROR Dist-upgrade failed: 'The package 'postgresql-10-postgis-2.4' is marked for removal but it is in the removal blacklist.'
So there is a problem with the PostGIS package. It is simple enough to remove it and continue. If this was a needed package, I would reinstall after running the Operating System upgrade. Sometimes this means installing an old package from source or something. In this case it wasn't important to keep the PostgreSQL database around so I can just uninstall.
$ sudo apt-get remove -y postgresql-10-postgis-2.4
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following packages will be REMOVED:
postgresql-10-postgis-2.4
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 1,693 kB disk space will be freed.
Do you want to continue? [Y/n] Y
(Reading database ... 201150 files and directories currently installed.)
Removing postgresql-10-postgis-2.4 (2.4.3+dfsg-4) ...
Eventually, three versions of the PostGIS package were uninstalled before the operating install was able to continue.
Reconnecting after ssh termination
One of the problems with installing over ssh is what happens when you lose connection in the middle of the install?
$ Timeout, server server.to.upgrade.com not responding.
To reconnect just ssh into the server and run the screen reattach command
$ ssh server.to.upgrade.com
$ sudo screen -r
Note that you must include sudo as the do-release-upgrade is running under sudo, not the logged in user. When do-release-upgrade starts, it starts a screen session to run the upgrade.
Often times, the install is waiting on a question to be answered.
Checking the version installed
After installing the version can be checked with lsb_release and uname
$ lsb_release -a
No LSB modules are available.
Distributor ID: Ubuntu
Description: Ubuntu 20.04.5 LTS
Release: 20.04
Codename: focal
$ uname -a
Linux little 5.4.0-135-generic #152-Ubuntu SMP Wed Nov 23 20:19:22 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux