Privileged run does not allow mounting shm tmpfs

[saschagrunert]

/kind bug

Description/Steps to reproduce the issue

If I run a container in privileged mode, then I am not able to do something like this:

> sudo ./bin/podman run --privileged -it saschagrunert/crio-playground
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
5456fb2c96ff:~ # crictl runp sandbox.yml
FATA[0002] run pod sandbox failed: rpc error: code = Unknown desc = error creating pod sandbox with name "k8s_sandbox_default__0": Error committing the finished image: error adding layer with blob "sha256:67ddbfb20a22d7c0ea0df568069e7ffc42378467402d04f28ecfa244e78c5eb8": ApplyLayer exit status 1 stdout:  stderr: permission denied

Describe the results you received:

The image saschagrunert/crio-playground contains a running crio instance in a background tmux session, whereas the session can be attached via tmux at. There we see now:

ERRO[2019-05-24 09:41:07.408490952Z] Error while applying layer: ApplyLayer exit status 1 stdout:  stderr: permission denied
DEBU[2019-05-24 09:41:07.408486735Z] received signal                               signal=broken pipe

Describe the results you expected:

It should work in privileged mode, like with docker:

> docker run --privileged -it saschagrunert/crio-playground
1185702fbcca:~ # crictl runp sandbox.yml
35dca868509a81793feeda06cf668953f93063cd583fdda392facd65e596915a

Output of podman version:

> sudo ./bin/podman version
Version:            1.3.1
RemoteAPI Version:  1
Go Version:         go1.12.5
Git Commit:         7210727e205c333af9a2d0ed0bb66adcf92a6369
Built:              Fri May 24 11:30:31 2019
OS/Arch:            linux/amd64

Output of podman info --debug:

debug:
  compiler: gc
  git commit: 7210727e205c333af9a2d0ed0bb66adcf92a6369
  go version: go1.12.5
  podman version: 1.3.1
host:
  BuildahVersion: 1.8.2
  Conmon:
    package: podman-1.2.0-1.1.x86_64
    path: /usr/lib/podman/bin/conmon
    version: 'conmon version 1.14.0, commit: '
  Distribution:
    distribution: '"opensuse-tumbleweed"'
    version: "20190521"
  MemFree: 6324269056
  MemTotal: 16691032064
  OCIRuntime:
    package: runc-1.0.0~rc8-1.1.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc8
      spec: 1.0.1-dev
  SwapFree: 15950409728
  SwapTotal: 16693325824
  arch: amd64
  cpus: 8
  hostname: nb
  kernel: 5.1.3-1-default
  os: linux
  rootless: false
  uptime: 3h 10m 34.77s (Approximately 0.12 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - quay.io
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 2
  GraphDriverName: btrfs
  GraphOptions: null
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Build Version: 'Btrfs v4.20.1 '
    Library Version: "102"
  ImageStore:
    number: 2
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

[rhatdan]

Must be something different in the way we are setting up /dev versus Docker?
If you volume mount in /dev does it work?
-v /dev:/dev

[saschagrunert]

Must be something different in the way we are setting up /dev versus Docker?
If you volume mount in /dev does it work?
-v /dev:/dev

Hm, no this does not work either:

> sudo podman run --privileged -v /dev:/dev -it saschagrunert/crio-playground
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
WARNING: The same type, major and minor should not be used for multiple devices.
119d46a94f7a:~ # crictl runp sandbox.yml
FATA[0002] run pod sandbox failed: rpc error: code = Unknown desc = error creating pod sandbox with name "k8s_sandbox_default__0": Error committing the finished image: error adding layer with blob "sha256:67ddbfb20a22d7c0ea0df568069e7ffc42378467402d04f28ecfa244e78c5eb8": ApplyLayer exit status 1 stdout:  stderr: permission denied

Although, the warnings should not happen at all, right? Can this be the problem here?

They seem to originate from:
https://github.com/opencontainers/runtime-tools/blob/095789df6c2bc53a9dd4464cda8c22616d66e0d6/generate/generate.go#L1490-L1504

[rhatdan]

Worked for me on Fedora 30

# podman run --privileged -it saschagrunert/crio-playground
Trying to pull docker.io/saschagrunert/crio-playground...Getting image source signatures
Copying blob 4a7b8e0a5b23 done
Copying blob 70823703d6b7 done
Copying blob 3ce75c405591 done
Copying blob 241c1ade42a5 done
Copying blob 4b3fbce106c6 done
Copying blob d74d3269eed7 done
Copying config ae25b00e57 done
Writing manifest to image destination
Storing signatures
3e410cab4ec6:~ # crictl runp sandbox.yml
266e1208e3fb34369b67a293f2d0afbafd92b34c8b050350b5e7f68c54eba357
3e410cab4ec6:~ # 

# podman version
Version:            1.3.1
RemoteAPI Version:  1
Go Version:         go1.12.2
OS/Arch:            linux/amd64

# podman info
host:
  BuildahVersion: 1.8.2
  Conmon:
    package: podman-1.3.1-1.git7210727.fc30.x86_64
    path: /usr/libexec/podman/conmon
    version: 'conmon version 1.12.0-dev, commit: c9a4c48d1bff85033b7fc9b62d25961dd5048689'
  Distribution:
    distribution: fedora
    version: "30"
  MemFree: 460554240
  MemTotal: 16450428928
  OCIRuntime:
    package: runc-1.0.0-92.dev.gitc1b8c57.fc30.x86_64
    path: /usr/bin/runc
    version: |-
      runc version 1.0.0-rc7+dev
      commit: d38f8a2d00ff444e52d16ab1ea5bbe5c1cb471e8
      spec: 1.0.1-dev
  SwapFree: 8291348480
  SwapTotal: 8296329216
  arch: amd64
  cpus: 8
  hostname: localhost.localdomain
  kernel: 5.0.16-300.fc30.x86_64
  os: linux
  rootless: false
  uptime: 144h 19m 56.8s (Approximately 6.00 days)
registries:
  blocked: null
  insecure: null
  search:
  - docker.io
  - registry.fedoraproject.org
  - quay.io
  - registry.access.redhat.com
  - registry.centos.org
store:
  ConfigFile: /etc/containers/storage.conf
  ContainerStore:
    number: 67
  GraphDriverName: overlay
  GraphOptions:
  - overlay.mountopt=nodev,metacopy=on
  GraphRoot: /var/lib/containers/storage
  GraphStatus:
    Backing Filesystem: xfs
    Native Overlay Diff: "false"
    Supports d_type: "true"
    Using metacopy: "true"
  ImageStore:
    number: 22
  RunRoot: /var/run/containers/storage
  VolumePath: /var/lib/containers/storage/volumes

[saschagrunert]

Thanks for trying it out on fedora. I think it could be related to btrfs, but I have to dig deeper into the code now.

[vrothberg]

@saschagrunert, can you try it out with the latest podman? I assume your're running on openSUSE and we just recently fixed a bug where podman applied the apparmor profile despite the --privileged flag.

[saschagrunert]

@saschagrunert, can you try it out with the latest podman? I assume your're running on openSUSE and we just recently fixed a bug where podman applied the apparmor profile despite the --privileged flag.

Ah nice thanks for the hint, the issue seems fixed on the master branch.

I still get the warnings (only on btrfs driver usage):

WARNING: The same type, major and minor should not be used for multiple devices.

Should I look deeper into that or is it already known?

[saschagrunert]

Closing for now since I assume the fix is included in one of the following versions.

[vrothberg]

Should I look deeper into that or is it already known?

It would be great to check what's behind this warning. Thanks 🙏

[rhatdan]

So this is Overlayfs running on a BTRFS disk?

[saschagrunert]

I opened up a new issue about the warning: opencontainers/runtime-tools#695