Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Wrong directory when using a tenant container and w/ a mount namespace #2751

Open
jeromegn opened this issue Apr 9, 2024 · 2 comments
Open

Comments

@jeromegn
Copy link

jeromegn commented Apr 9, 2024

I'm unable to use a tenant container for executing commands in a container because, as soon as setns(mnt_ns) is called, the tenant's container_init_process ends up in / of my host.

The situation is different from most usage of libcontainer:

  • I'm running it as part of an VM's init program
  • That program runs from a initramfs and therefore I have to create a dummy bind mount (/newroot) and chroot there so that pivot_root will work in the future (else it returns EINVAL because called from an initramfs, consistent with the docs)
  • I'm creating and starting containers from the init process directly, not by invoking the youki CLI

Much debug logging later: I found that after calling apply_rest_namespaces, my current directory shifts to the root / outside of my dummy bind mount. To be clear, the directory listing after setns returns ["/", "/newroot"]. I'm expecting the process to "be" in /containers/my-container/rootfs.

Logs I added

Main container:

DEBUG unsharing for Mount
DEBUG after ns Mount entry: Ok(DirEntry("./upper"))
DEBUG after ns Mount entry: Ok(DirEntry("./lower"))
DEBUG after ns Mount entry: Ok(DirEntry("./bundles"))
DEBUG after ns Mount entry: Ok(DirEntry("./containers"))
DEBUG after ns Mount entry: Ok(DirEntry("./etc"))
DEBUG after ns Mount entry: Ok(DirEntry("./root"))
DEBUG after ns Mount entry: Ok(DirEntry("./run"))
DEBUG after ns Mount entry: Ok(DirEntry("./sys"))
DEBUG after ns Mount entry: Ok(DirEntry("./proc"))
DEBUG after ns Mount entry: Ok(DirEntry("./dev"))
...
DEBUG prepare rootfs rootfs="/bundles/sleep/rootfs"
DEBUG mount root fs "/bundles/sleep/rootfs"
...
DEBUG pivoting root to /bundles/sleep/rootfs

Tenant:

DEBUG unshare or setns: LinuxNamespace { typ: Cgroup, path: Some("/proc/242/ns/cgroup") }
DEBUG setns for Cgroup at /proc/242/ns/cgroup
DEBUG after ns Cgroup entry: Ok(DirEntry("./upper"))
DEBUG after ns Cgroup entry: Ok(DirEntry("./lower"))
DEBUG after ns Cgroup entry: Ok(DirEntry("./bundles"))
DEBUG after ns Cgroup entry: Ok(DirEntry("./containers"))
DEBUG after ns Cgroup entry: Ok(DirEntry("./etc"))
DEBUG after ns Cgroup entry: Ok(DirEntry("./root"))
DEBUG after ns Cgroup entry: Ok(DirEntry("./run"))
DEBUG after ns Cgroup entry: Ok(DirEntry("./sys"))
DEBUG after ns Cgroup entry: Ok(DirEntry("./proc"))
DEBUG after ns Cgroup entry: Ok(DirEntry("./dev"))
DEBUG unshare or setns: LinuxNamespace { typ: Mount, path: Some("/proc/242/ns/mnt") }
DEBUG setns for Mount at /proc/242/ns/mnt
DEBUG after ns Mount entry: Ok(DirEntry("./newroot"))
DEBUG after ns Mount entry: Ok(DirEntry("./fly"))
DEBUG after ns Mount entry: Ok(DirEntry("./root"))
DEBUG after ns Mount entry: Ok(DirEntry("./dev"))

I don't understand how that's happening. I tried changing various settings and code in my youki fork, but I can't get it to do anything else. I've confirmed that the process is getting a different mount namespace based on its inode. I've tried spawning threads to better isolate libcontainer operations (and all the syscalls it is making), but that didn't change anything.

At this point I'm wondering if this is happening because of the weird bind mount I'm making from my initrd program. It could also be related to the fact that I'm creating containers all from the same process.

I'm using the default oci_spec::Spec with minimal changes to support host networking (#2745).

This is all leading to this exec error because it can't find the uname program I'm trying to run (rightly slow, it's not in the right mount namespace!):

ERROR executable for container process not found in PATH executable="uname"
ERROR failed to initialize container process: executable 'uname' not found in $PATH
ERROR failed to wait for init ready: exec process failed with error error in executing process : executable 'uname' not found in $PATH
ERROR failed to run container process err=Channel(ExecError("error in executing process : executable 'uname' not found in $PATH"))
@jeromegn
Copy link
Author

I figured it out. If you're running from an initramfs (e.g. as part of an initrd program) with no "real" filesystem, you need to use mount --move and chroot instead of pivot_root: https://github.com/opencontainers/runc/blob/5e0ec3fbbff07ef4016c04ae584f18b27bbcfdc1/libcontainer/SPEC.md?plain=1#L118-L125

I might do a PR that detects this niche case if it's of any interest.

@utam0k
Copy link
Member

utam0k commented Jun 16, 2024

Hi, @jeromegn. This issue may have been fixed by #2780. May I ask you to confirm it by the latest libcontainer?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants