Create focal -> noble upgrade script

[legoktm]

Description

Part of #7211.

The workflow of the script will be:

• apply any pending updates and reboot
• run migration check to make sure we're all set to go
• take a backup
• check disk space, again??
• raise OSSEC email_alert_level config to 15
• disable unattended upgrades
• edit sources.list to point to noble
• apt-get update
• apt-get upgrade --without-new-pkgs
• apt-get full-upgrade
• re-enable unattended-upgrades
• re-enable OSSEC(?)
• reboot
• switch sources.list -> ubuntu.sources
• some basic verification
• delete backup

Everything the script does should be aggressively logged and each major step should trigger an OSSEC notification.

I think we can trigger the script from a systemd timer/service. Each step should be recorded in some kind of state file so it can recover and resume if interrupted.

We should also leave behind a marker file like /etc/securedrop-upgraded-from-focal so if we detect some bug in the future, we can conditionally apply logic based on whether it's a fresh noble install or an upgrade.

[legoktm]

One question I've been debating in my head is whether to write the script in Rust or Python. The pros for Rust are pretty clear, this is a place we want to have solid error handling and recovery. The cons are that I think it'll be a little harder to get review for and be a bit more friction for others to contribute.

My plan is to start sketching out the script in Python and then see if it can do all the error handling stuff in a not-crazy way. I expect any Python<-->Rust porting to be trivial.

[legoktm]

raise OSSEC email_alert_level config to 15

This is doable on the mon server but going to be an issue on the app server.

[zenmonkeykstop]

Doesn't OSSEC have a message threshold anyway? Given this is a one-time event I think you could probably do without the alert level change.

[legoktm]

Yes, after it hits some limit it stops sending messages and just queues them up...which means there's a backlog of like 1000 emails and any new alerts (e.g. using the send OSSEC test button) don't get sent. When I upgraded the mon server I got ~27 emails before it stopped sending new ones. The queue seems to be stored in memory though, because I restarted the ossec-server service and then it dropped everything and started sending my newly triggered OSSEC test alerts.

I'll put it into the nice to have bucket for now and when we do test upgrades we can see how bad the impact is and decide if we want something else.

[legoktm]

One question I've been debating in my head is whether to write the script in Rust or Python. The pros for Rust are pretty clear, this is a place we want to have solid error handling and recovery. The cons are that I think it'll be a little harder to get review for and be a bit more friction for others to contribute.

Now that I've started writing the code, there's actually a bigger issue: we are uninstalling Python 3.8 and installing Python 3.12 during the migration. If something fails midway through, we could easily have a broken Python installation. A statically compiled Rust binary will avoid all of that.

[legoktm]

For reference: https://github.com/freedomofpress/securedrop/blob/6f5ef9e69fb1ac87ce4414e92bd1481347f11795/securedrop/debian/config/usr/bin/securedrop-noble-migration.py is what I had sketched out in Python. I'm going to redo that all in Rust now.

[legoktm]

And here's the Rust port: https://github.com/freedomofpress/securedrop/blob/94b84b7894d1bb6f21e93cd61fda3f793363dba8/noble-migration/src/bin/upgrade.rs

It's still very basic with lots of FIXMEs inline, but you can see the rough state machine and how it'll handle reboots, etc. Next is:

• finishing said FIXMEs
• implementing logging in an OSSEC-friendly way (i.e. error!("foo") should trigger an OSSEC notification)
• error handling and identifying most likely failure points
• trying it! and integrating it into the staging CI job