Fix for /var content (ostreedev PR #569) doesn't seem to be in the latest centos-bootc-dev base image

[nzwulfin]

Intalling Postgres in the latest centos-bootc-dev base container fails on the ostree commit due to content being placed in /var/lib by the rpm install. This container should include the code from ostreedev PR #569

Error from podman build:

STEP 3/3: RUn ostree container commit
Found file: var/lib/pgsql/.bash_profile
error: Found content in var
Error: building at STEP "RUN ostree container commit": while running runtime: exit status 1

Simple containerfile to reproduce

FROM quay.io/centos-bootc/centos-bootc-dev:stream9

RUN dnf install -y postgresql-server && \
    rm /var/log/*.log /var/lib/dnf -rf

RUN ostree container commit

Output of podman images
quay.io/centos-bootc/centos-bootc-dev stream9 96188766724f About an hour ago 1.45 GB

ostree info from inside container:
bash-5.1# ostree --version

libostree:
 Version: '2023.9'
 Git: f9d013632d88db3a2a89a8c4924417b3cfbdabec
 DevelBuild: yes
 Features:
  - inode64
  - libcurl
  - gpgme
  - composefs
  - ex-fsverity
  - libarchive
  - selinux
  - openssl
  - sign-ed25519
  - libmount
  - systemd
  - devel
  - p2p

bash-5.1# rpm -qa | grep ostree
ostree-libs-2023.8.54.g19cd8cf1-%autorelease.x86_64
ostree-2023.8.54.g19cd8cf1-%autorelease.x86_64
rpm-ostree-libs-2024.1.23.gefe57a65-1.el9.x86_64
rpm-ostree-2024.1.23.gefe57a65-1.el9.x86_64

[cgwalters]

The short version is you can just drop RUN ostree container commit. We aren't trying to do anything really important there anymore because basically no one ever used it correctly, it has hard to explain semantics, and we are just going to try harder to Just Work with whatever's in the container image.

Just did ostreedev/ostree-rs-ext#584 to fully codify this.

[nzwulfin]

Removed the ostree commit, everything built fine, can start / stop / modify the running container, including more changes to /var once running. Thanks!

[cgwalters]

We need to try to re-land #186 again, once osbuild/bootc-image-builder#18 is fixed

[cgwalters]

That's a related problem, but the real issue here is that the change in ostreedev/ostree#3103 doesn't work on c9s because systemd there doesn't understand C+, so it degrades to C which is just a no-op because /var already exists.

Probably the simplest thing is to change ostree-ext to synthesize tmpfiles.d snippets instead, i.e. instead of a global /var we introduce /usr/share/container-image-var or so, and our tmpfiles.d snippets target subdirectories of it. (And ideally, we avoid generating duplicate tmpfiles.d entries, but that's its own complex mess)

This would also fix a cosmetic issue of a conflict between our /usr/share/factory/var and /usr/lib/tmpfiles.d/var.conf from systemd, but that's currently just a warning because the ostree one wins.

[cgwalters]

OK, I've verified that with the latest changes from ostreedev/ostree#3166 and ostreedev/ostree-rs-ext#602 that have now landed in the latest dev image a layered httpd example works when run through bib.

[nzwulfin]

Local testing on the reproducer works great, thanks!

[cgwalters]

Thanks so much for testing this!

[nzwulfin]

I'm running into an issue that I can't tell if it's a local problem with what I'm doing or something else. I didn't notice until I booted from the derived container and nmcli was complaining about dbus files.

In the derived container with the simple postgres install, i wind up with a /var/run that only includes the postrgesql directory and the sshd.pid file. Starting the base container with systemd looks "normal" and dbus related commands like nmcli conn show function as expected.

Output of podman images
quay.io/centos-bootc/centos-bootc-dev stream9 0208a9f98bc5 9 hours ago 1.53 GB

/var/run of the base container

podman run -d --name base centos-bootc-dev:stream9 /sbin/init

podman exec -it base nmcli conn show
NAME  UUID                                  TYPE      DEVICE 
tap0  a1bb241a-78e4-4482-92ae-f90088be4033  tun       tap0   
lo    22dcf453-2cac-4100-a488-eded3dde5e6b  loopback  lo  

podman exec -it base ls /var/run/
NetworkManager	criu		 faillock  mdadm   podman	 secrets   systemd
bootupd.sock	cryptsetup	 initctl   media   rhsm		 sepermit  tpm2-tss
chrony-dhcp	dbus		 lock	   motd    rpcbind	 setrans   udisks2
console		dmeventd-client  log	   motd.d  rpcbind.sock  sshd.pid  user
credentials	dmeventd-server  lvm	   ostree  samba	 sudo	   utmp

/var/run of the derived container immediately after build

podman run -d --name repro repro /sbin/init

podman exec -it repro nmcli conn show
Error: Could not create NMClient object: Could not connect: No such file or directory.

podman exec -it repro ls /var/run
postgresql  sshd.pid

Containerfile

FROM quay.io/centos-bootc/centos-bootc-dev:stream9

RUN dnf install -y postgresql-server && \
    dnf clean all

[nzwulfin]

From reading the linked PRs I think i see what's happening but not why. In the base image, /var/run is properly symlinked to /run based on /usr/lib/tmpfiles.d/var.conf, but in the derived image it's a regular directory.

[core@bootc-builder repro]$ podman exec -it base ls -ald /var/run
lrwxrwxrwx. 1 root root 6 Feb 15 18:30 /var/run -> ../run
[core@bootc-builder repro]$ podman exec -it repro ls -ald /var/run
drwxr-xr-x. 1 root root 22 Feb 15 18:26 /var/run

There's a tmpfile.d config that creates the directory for postgresql:
d /run/postgresql 0755 postgres postgres -

Looks like the tmpfile definition for postgres is interfering with the /var/run definition in var.conf?

I tried with httpd like you did and it works fine with a similar tmpfile.d config for httpd

[core@bootc-builder repro]$ podman exec -it repro ls /var/run
NetworkManager	criu		 faillock  lvm	   ostree	 samba	   sudo      utmp
bootupd.sock	cryptsetup	 httpd	   mdadm   podman	 secrets   systemd
chrony-dhcp	dbus		 initctl   media   rhsm		 sepermit  tpm2-tss
console		dmeventd-client  lock	   motd    rpcbind	 setrans   udisks2
credentials	dmeventd-server  log	   motd.d  rpcbind.sock  sshd.pid  user

[core@bootc-builder repro]$ podman exec -it repro nmcli conn show
NAME  UUID                                  TYPE      DEVICE 
tap0  8735de76-ed36-4f09-8322-0da20c68ec07  tun       tap0   
lo    71d32f61-f5b3-4000-8da3-1b0e8754943b  loopback  lo     

[core@bootc-builder repro]$ podman exec -it repro cat /usr/lib/tmpfiles.d/httpd.conf
d /run/httpd   710 root apache
d /run/httpd/htcacheclean   700 apache apache

[core@bootc-builder repro]$ podman exec -it repro ls -al /var/run
lrwxrwxrwx. 1 root root 6 Feb 15 18:55 /var/run -> ../run

[cgwalters]

Hmm not having /var/run be a symlink is indeed going to cause a lot of problems.

And yep, I see dnf -y install postgresql-server causes this. It should be easy to work around this with a RUN rm /var/run -rf.

And yep I see, the problem is that the RPM includes /var/run/postgresql as physical content. This is a serious bug in the package.

[cgwalters]

https://issues.redhat.com/browse/RHEL-25756

[cgwalters]

https://gitlab.com/bootc-org/tests/container-fixtures/-/merge_requests/3

[nzwulfin]

I looked for it in the rpm file list first and missed it somehow.. oh well :)

Tested the postgresql reproducer with the suggested workaround and it works as it should. The tmpfile.d config shipped in the package works, and /var/run is properly linked.

[cgwalters]

I believe we've done everything we can do in the base image for this issue now.

[cgwalters]

➡️ #354 (comment)