Support Ubuntu 24.04 LTS (noble)

[legoktm]

Description

As our current Ubuntu version, 20.04 LTS aka focal, approaches end of life, SecureDrop should support Ubuntu 24.04 LTS, aka noble numbat.

A separate issue will track work for the upgrade path.

[zenmonkeykstop]

Probably should consider:

• impacts of Python bump
  • (yaay dependency updates for all)
  • picking a minimal Debian/Ubuntu version required for development
  • resyncing version of Ansible used in dev and Tails envs
• moving to nftables over iptables for server fw rules
• bumping OSSEC (or switching to Wazuh or similar)
• setting up sftp on the servers and updating Ansible to use it instead of scp
• TBD (probably a lot of TBDs)

[legoktm]

Note that there are no Tor packages for noble yet: https://gitlab.torproject.org/tpo/tpa/team/-/issues/41605

[legoktm]

Overall things seem smoother than expected on the Python application side (haven't tested ansible, etc. yet). Everything basically works, with small updates needed for alembic and six and some other non-prod dependencies. With the noble-dev branch, basic SI and JI functionality work and all tests pass in CI.

There are a lot of warnings being emittted from flask, etc., so we may want to upgrade them regardless (#6963). But I think we should be able to do everything in a dual-compat way, keeping both focal and noble support without much branching.

[legoktm]

Two updates for today:

• Support noble dev environment and pass tests #7249 is now ready for review-ish, it still needs a diff review and is waiting on the lint change first.
• I have package building "working" in the noble-build branch. I haven't tested them yet, but they build! I need to do further investigation on the OSSEC packages since we need a build hack for it to compile (Compiling on Debian Bullseye (11) fails with cryptic error message -- collect2: error: ld returned 1 exit status ossec/ossec-hids#2022 (comment)).
  • We also need to mirror paxctld into our apt projects now. Need to figure out which version we want to use, SDW is on 1.2.5, but I noticed Upgrade to paxctld 1.2.6 securedrop-workstation#1193 today

[legoktm]

The noble installer is basically the same as focal, except it'll now prompt you whether you want to install the normal Ubuntu server or if you want to install a "minimized" version of it. From [https://askubuntu.com/questions/1511204/what-is-the-difference-between-ubuntu-24-04-default-minimal-installation-and-f](looking online), it seems the difference is https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.noble/server vs https://ubuntu-archive-team.ubuntu.com/seeds/ubuntu.noble/server-minimal. For now in my testing I'm going with the full version, I don't think we need to bother with the headache minimal versions bring.

[legoktm]

Also the installer no longer runs a system update, so you have to do it yourself. Our playbooks do that pretty early on so I don't think it's an issue, but just a heads up.

[legoktm]

I naively tried setting up a noble CI staging job, it didn't work and really didn't even get very far. The bento/ubuntu-24.04 box appears to be a "version 2" box as described at https://vagrant-libvirt.github.io/vagrant-libvirt/boxes.html#box-formats. Running it with our old-ish vagrant version turns into "No image virtual size specified for box" (which is technically correct, it's not in the metadata.json file).

I tried a few workarounds, but nothing easy worked. Tomorrow I'm planning to just try a real prod install instead of fiddling with staging.

[legoktm]

I've been slowly working through running an install on real hardware and it hasn't been super bad (but also I'm not done yet), all of my fixes are being iteratively pushed to the noble-install branch.

Main gotchas so far:

• in linux 6.6 or noble, the behavior of sysctl has changed in that once the grsec_lock is set, running sysctl -p to reload settings that are locked will error even if the change is a no-op. I've worked around this for now, but I think we should be moving most of our settings to the kernel package itself so we don't need to manually set them via ansible.
• iptables is now nftables. This is what I'm working through right now and where I'm taking a break, because the 6.6 kernel I built didn't have all the necessary modules, so rebuilding it now.

[legoktm]

Now with a fixed kernel package, I was able to get through the initial iptables rules and after fixing some apparmor issues, successfully installed SecureDrop on noble \o/

Definitely a lot of things under the surface that need fixing but it's functional.

[legoktm]

Definitely a lot of things under the surface that need fixing but it's functional.

I was iteratively working through these in a noble-install branch, I've now started splitting that up into bite-sized PRs that are ready for review. The ones tagged noble are actual unblocking of things, the rest are cleanup that I spotted.

[legoktm]

As an update, I think the last thing we need for fresh install to work on noble is:

• Fix sysctl handling for noble/Linux 6.6 #7323

To get testinfra checks to pass:

• Expand some testinfra checks to expect noble #7306
• Remove tests checking that no apparmor profiles are complaining #7308

Nice to have but not blockers:

• Figure out sshd algorithms for noble  #7281
• Tests under noble / Python 3.12 are far slower #7311