Add support for RockyLinux9

[sjpb]

Make the appliance compatible with RockyLinux 9-based images.

Note that the CI and CaaS environments will continue to use RL8 at present. CI is only carried out using RL9 if a PR branch name starts with rl9 or RL9 is selected when running CI workflows manually.

Additional notes:

• This also removes the Packer template for "environment-specific" compute image builds. No deployments are using this functionality and it has limitations which would be better addressed by making the fatimage usable on boot via cloud-init.
• Podman commands in systemd units are changed to use --cgroup-manager=cgroupfs. This was demonstrated to be the default in RL8; for RL9 the default is systemd which leads to log warnings like:

  The cgroupv2 manager is set to systemd but there is no systemd user session available"
  For using systemd, you may need to login using an user session"
  Alternatively, you can enable lingering with: `loginctl enable-linger 1001` (possibly as root)"
  Falling back to --cgroup-manager=cgroupfs"
  "unlinkat /run/podman/libpod/tmp: permission denied"
  

  However enabling user-lingering leads to mysql failing to start at all, with the relevant error probably being:

  [email protected]: Got notification message from PID 5388, but reception only permitted for main PID 4270
  

Replaces #323

[sjpb]

FAILED Fat image build: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/7643335283

Edit: had wrong image type in glance

[sjpb]

Tests at 0982f41 on a "local" cluster:

• deploy OK
• reboot control: services OK
• hpctests OK
• OOD shell: OK
• slurm jobs: OK
• hpl timehistory: OK
• slurm exporter: OK
• ondemand exporter: some missing data, see below
• OOD jupyter OK
• OOD desktop: failed, no dbus-launch command

ondemand exporter:

[rocky@rl9-login-0 ~]$ systemctl status ondemand_exporter.service
Jan 24 16:26:24 rl9-login-0.rl9.invalid ondemand_exporter[36589]: ts=2024-01-24T16:26:24.868Z caller=collector.go:171 level=error msg="Error collecting apache information" err="Get \"http://localhost:81/server-status\":>

[rocky@rl9-login-0 ~]$ cat /usr/lib/systemd/system/ondemand_exporter.service
Environment="APACHE_STATUS_URL=http://localhost:81/server-status"

[rocky@rl9-login-0 ~]$ curl localhost:9301/metrics
# shows this is working at least

[sjpb]

Fat image build: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/7653662893

Edit: currently failing due to CVMFS repo 503-ing
Edit: repo appears up, retrying

[sjpb]

Checked locally that e5608d9 works on both a) a cluster with existing non-system users b) fresh image

[sjpb]

NB: Currently CI doens't get past the os-manila-mount install task b/c the rpm-reef URL at https://download.ceph.com/ has been broken/renamed.

Edit: see https://tracker.ceph.com/issues/64718

[sjpb]

Repos fixed, lets try again

[sjpb]

Rebuilding fat image: https://github.com/stackhpc/ansible-slurm-appliance/actions/runs/8263123087

Built

• openhpc-RL8-240313-1028-15f9ab38
• openhpc-RL9-240313-1057-15f9ab38

[sjpb]

Tests at 43d43f2 on "local" cluster:

• deploy OK
• hpctests OK
• OOD shell: OK
• slurm jobs: OK
• hpl timehistory: OK although appear to be hitting ~100% mem usage
• slurm exporter: OK
• ondemand exporter: OK
• OOD jupyter: OK
• OOD desktop: failed, partition name was wrong, fixed, OK

[m-bull]

Minor comment but otherwise LGTM