Support non-reinstall migration to Ubuntu 24.04 LTS (noble)

[legoktm]

Description

Once we support noble (#7210), we should provide a non-reinstall upgrade path for admins currently on focal. The process may not be 100% automated, but should not require a full reinstall as we have in the past.

[legoktm]

I did a fresh installation of 2.10.1 on a focal system, and then started upgrading the mon server:

• edit sources.list to point to noble
• apt upgrade --without-new-pkgs
• before starting the full-upgrade, I noticed it was trying to remove iptables-persistent. a bit of debugging showed that ufw now conflicts with that. So let's remove ufw. apt remove ufw.
• Now apt full-upgrade Only packages being removed are fuse, a bunch of old library versions, lz4, mime-support and python3.8
• After the full upgrade, cat /etc/os-release shows noble! The only systemd unit I see failing is systemd-networkd, because it doesn't support the ProtectProc setting? Possibly because this is the old version of systemd running, or we're still on an old kernel? Networking still works though, because I'm clearly still connected :p
• After rebooting, I could no longer ssh in. I got a console on it directly, edited the sshd_config to handle the group rename of ssh to _ssh, restarted sshd and then I was in.
• systemctl status ossec looks good. I'm not getting any emails but I forgot to test ossec config before upgrading so I might've set it up wrong.
• iptables -L was empty, also had to change the name of the ssh group there and then restart netfilter-persistent and then they were back.

Now to upgrade app.

• removed ufw first off.
• then tried apt upgrade --without-new-pkgs and first couldn't delete /opt/venvs/securedrop-app-code/lib/python3.8/ because of all the __pycache__ folders (just a warning though) but then it failed when enabling the tor apparmor profile:

ERROR: Syntax Error: Unknown line found in file tunables/home line 24:
    include if exists <tunables/home.d>

• This is because apparmor was held back since it had a library upgrade and so we're using the new profile syntax with old apparmor. Ideally the tor package should have a Pre-Depends on a specific version of apparmor to force it to be later, but we don't control the tor package so we can stick the pre-depends in securedrop-app-code. Let's just go ahead with a full apt full-upgrade now.
• I forgot to rebuild the packages with new apparmor files, but after I fixed that manually, apache2 started up and everything was installed.
• Then I tweaked the sshd_config for the new group name, rebooted, and it worked! The service was up! I did forget to update the iptables config so it booted with nothing, but fixed that and then it's back.

Overall I think this is a viable path for an upgrade. Immediate next steps: patch to remove ufw package from focal and rename the ssh group, plus the apparmor dependency.

Then we can try the upgrade part as a headless script, take a snapshot and see how it differs from a fresh install or something. I also want to grab a snapshot of a long-running (dev) instance and see how that differs from a focal fresh install.

[zenmonkeykstop]

It will be interesting to see that diff - previously this approach ended up with a system that was pretty divergent from a fresh install, which would have made support more complicated. But it sounds like it's in a lot better state this time round.

[legoktm]

We should make sure to temporarily disable OSSEC when upgrading, otherwise you'll get like 500 notices about files changing and new packages being installed.

[legoktm]

I think we can temporarily raise the email_alert_level to like 15, and then lower it once we're done. Our upgrade script should log somewhere at a higher level so it can still send info to administrators if something goes wrong.