Sunshine KMS screen capture not working due to cap_sys_admin+p not working from Hyprland in Linux container

[falk4243]

Basic Setup (more details, see below): Arch Linux with Linux containers (incus - unprivileged with device passthrough) and latest Sunshine version (native) on a headless host (iGPU with forced EDID monitor).

Problem: From what I understand Sunshine should work with Hyprland when using a real monitor and KMS capture, but I haven't been able to get it to work due to Sunshine always reporting that "cap_sys_admin+p" has not been set on the sunshine binary when in fact it has. The app works if I start it before starting Hyprland, coming up properly and finding the h.264 and h.264 va-api encoders, though obviously has no Hyprland screen to attach to. Let me know if you have any idea how to solve this, if it can be solved at all. I'll gladly provide more info if you need it.

This is what it looks like when I start Sunshine from within the Hyprland session -

[capture] -- [kms]
[2024-10-30 18:04:40.210]: Info: Sunshine version: v2024.911.215654
[2024-10-30 18:04:40.210]: Info: Package Publisher: LizardByte
[2024-10-30 18:04:40.210]: Info: Publisher Website: https://app.lizardbyte.dev
[2024-10-30 18:04:40.210]: Info: Get support: https://app.lizardbyte.dev/support
[2024-10-30 18:04:40.235]: Info: /dev/dri/card0 -> i915
[2024-10-30 18:04:40.236]: Error: Couldn't get handle for DRM Framebuffer [260]: Probably not permitted
[2024-10-30 18:04:40.236]: Fatal: You must run [sudo setcap cap_sys_admin+p $(readlink -f $(which sunshine))] for KMS display capture to work!
If you installed from AppImage or Flatpak, please refer to the official documentation:
https://docs.lizardbyte.dev/projects/sunshine/en/latest/about/setup.html#install
[2024-10-30 18:04:40.236]: Info: Found display [wayland-1]
[2024-10-30 18:04:40.236]: Info: Found display [wayland-1]
[2024-10-30 18:04:40.236]: Info: Found interface: zxdg_output_manager_v1(9) version 3
[2024-10-30 18:04:40.236]: Info: Found interface: wl_output(49) version 4
[2024-10-30 18:04:40.236]: Info: Resolution: 3840x2160
[2024-10-30 18:04:40.236]: Info: Name: HDMI-A-2
[2024-10-30 18:04:40.236]: Info: Found monitor: Iiyama North America PL3288UH 1169603601071 (HDMI-A-2)
[2024-10-30 18:04:40.236]: Info: Offset: 0x0
[2024-10-30 18:04:40.236]: Info: Logical size: 3840x2160
[2024-10-30 18:04:40.236]: Info: -------- Start of KMS monitor list --------
[2024-10-30 18:04:40.236]: Warning: Mismatch on expected Resolution compared to actual resolution: 0x0 vs 3840x2160
[2024-10-30 18:04:40.236]: Info: Monitor 0 is HDMI-A-2: Iiyama North America PL3288UH 1169603601071 (HDMI-A-2)
[2024-10-30 18:04:40.236]: Info: --------- End of KMS monitor list ---------
[2024-10-30 18:04:40.236]: Error: Unable to initialize capture method
[2024-10-30 18:04:40.236]: Error: Platform failed to initialize
[2024-10-30 18:04:40.237]: Warning: Gamepad ds5 is disabled due to No such file or directory
[2024-10-30 18:04:40.237]: Info: // Testing for available encoders, this may generate errors. You can safely ignore those errors. //
[2024-10-30 18:04:40.237]: Info: Trying encoder [nvenc]
[2024-10-30 18:04:40.262]: Info: System tray created
(sunshine:1222): Gtk-CRITICAL **: 18:04:40.366: gtk_widget_get_scale_factor: assertion 'GTK_IS_WIDGET (widget)' failed
[2024-10-30 18:04:40.637]: Info: Encoder [nvenc] failed
[2024-10-30 18:04:40.637]: Info: Trying encoder [vaapi]
[2024-10-30 18:04:41.037]: Info: Encoder [vaapi] failed
[2024-10-30 18:04:41.038]: Info: Trying encoder [software]
[2024-10-30 18:04:41.438]: Info: Encoder [software] failed
[2024-10-30 18:04:41.438]: Fatal: Unable to find display or encoder during startup.
[2024-10-30 18:04:41.438]: Fatal: Please check that a display is connected and powered on.
[2024-10-30 18:04:41.438]: Error: Video failed to find working encoder
[2024-10-30 18:04:41.439]: Error: Failed to create client: Daemon not running
[2024-10-30 18:04:41.440]: Info: Configuration UI available at [https://localhost:47990]

Setup: I've been using a headless sway setup for a while now on a Hetzner server with an Intel GPU. sway worked well enough with a headless display by setting "WLR_BACKENDS,headless", but this doesn't work anymore for Hyprland from version 42.0 onward and setting up a virtual display is cumbersome and doesn't work without a real display attached afaik (?). I can't easily attach a dummy plug, so have added a suitable EDID file via the kernel command line according to this Reddit post, which is recognized as a 4K monitor.

The iGPU is passed through to an Incus (formerly LXD) container along with a few other devices to make using the GPU / console / screen from the container possible. Within the container, a regular user has been added to the relevant groups (input, video, seat, tty). Hyprland happily starts with this and I can use wayvnc with H.264 hardware encoding to attach to the display. The session is started via SSH login into the container with its "desktop" user and then simply bringing up Hyprland. I have also added a systemd script for the container to automatically start the SSH connection + Hyprland on boot.

Here's the container config:

config:
  boot.autostart: "true"
  boot.autostart.priority: "1"
  boot.stop.priority: "1"
  limits.cpu.priority: "2"
  linux.kernel_modules: wireguard
description: desk profile
devices:
  desk-keyboard:
    mode: "666"
    productid: dead
    type: unix-hotplug
    vendorid: beef
  desk-mouse:
    mode: "666"
    productid: "4038"
    type: unix-hotplug
    vendorid: 046d
  desk-tty:
    mode: "666"
    path: /dev/tty
    type: unix-char
  desk-tty0:
    mode: "666"
    path: /dev/tty0
    type: unix-char
  desk-tty1:
    mode: "666"
    path: /dev/tty1
    type: unix-char
  desk-tty2:
    mode: "666"
    path: /dev/tty2
    type: unix-char
  desk-uinput:
    mode: "666"
    path: /dev/uinput
    type: unix-char
  desk.igpu:
    gid: "985"
    pci: "0000:00:02.0"
    type: gpu
  eth0:
    ipv4.address: yyy
    ipv6.address: zzz
    limits.priority: "2"
    nictype: bridged
    parent: desk
    type: nic
name: desk
used_by:
- /1.0/instances/desk
project: default

And here's my (mostly default) sway config (only relevant parts):

################                                                                                                                                                                                                   
### MONITORS ###                                    
################                                                                                                                                                                                                   
                                                                                                         
# See https://wiki.hyprland.org/Configuring/Monitors/                                                    
monitor = HDMI-A-2, 3840x2160@60, 0x0, 1 

#################                                                                                        
### AUTOSTART ###                                   
#################                                                                                        
                                                                                                         
# Autostart necessary processes (like notifications daemons, status bars, etc.)
# Or execute your favorite apps at launch like this: 
                                                                                                         
# exec-once = $terminal                                                                                  
# exec-once = nm-applet &                           
# exec-once = waybar & hyprpaper & firefox                                                               
                                                                                                         
exec-once = /usr/lib/polkit-kde-authentication-agent-1                                                   
exec-once = /usr/bin/wayvnc --output=HDMI-A-2 --max-fps=60 --gpu --render-cursor --log-level=info --config /home/desk/.config/wayvnc/config
exec-once = /bin/sleep 5 && /usr/bin/sunshine /home/desk/.config/sunshine/sunshine.conf

#############################                                                                                                                                                                                      
### ENVIRONMENT VARIABLES ###
#############################

# See https://wiki.hyprland.org/Configuring/Environment-variables/

env = XCURSOR_SIZE,24
env = HYPRCURSOR_SIZE,24

env= XKB_DEFAULT_LAYOUT,de

env = XDG_SESSION_TYPE,wayland
env = XDG_CURRENT_DESKTOP,Hyprland
env = XDG_SESSION_DESKTOP,Hyprland
env = XDG_DATA_DIRS,/usr/local/share:/usr/share:/var/lib/flatpak/exports/share:/home/desk/.local/share/flatpak/exports/share

env = GDK_BACKEND,wayland,x11,*
env = QT_QPA_PLATFORM,wayland;xcb
env = SDL_VIDEODRIVER,wayland
env = CLUTTER_BACKEND,wayland

env = QT_AUTO_SCREEN_SCALE_FACTOR,1
env = QT_QPA_PLATFORM,wayland;xcb
env = QT_WAYLAND_DISABLE_WINDOWDECORATION,1
env = QT_QPA_PLATFORMTHEME,qt5ct