undefined symbol: cuDeviceGetCount - inside of a charliecloud container

[VladimirFokow]

Hello,

I get undefined symbol: cuDeviceGetCount inside of a charliecloud container:
isaac-sim/OmniIsaacGymEnvs#145

I'm not sure if this is an issue with charliecloud or not.
I'm writing it here in case it may be relevant to charliecloud, or just if you have some thoughts about it.

Thanks!

(more details if needed)

(charliecloud version: 0.32)

Dockerfile:
(requires the nvidia's api key to pull)

FROM nvcr.io/nvidia/isaac-sim:2023.1.1
COPY ./OmniIsaacGymEnvs/ /files/omniisaacgymenvs/
RUN /isaac-sim/python.sh -m pip install -e /files/omniisaacgymenvs

Building the container:

module load charliecloud

converted_images=path/to/converted_images
ch-image build -t sim-image -f Dockerfile ./
ch-convert sim-image $converted_images/sim-image
ch-fromhost --nvidia $converted_images/sim-image

entryscript.sh:

#!/bin/bash

alias PYTHON_PATH='/isaac-sim/python.sh'
alias python=PYTHON_PATH
alias pip='python -m pip'

Running the container:

module load charliecloud

# Directories expected by this specific image to mount:
mounted_directories=path/to/mounted_directories
mkdir -p $mounted_directories
cd $mounted_directories
mkdir -p ./cache/kit
mkdir -p ./cache/ov
mkdir -p ./cache/pip
mkdir -p ./cache/glcache
mkdir -p ./cache/computecache
mkdir -p ./logs
mkdir -p ./data
mkdir -p ./documents

# Folder where sim-image is:
converted_images=path/to/converted_images
# Our personal folder to mount:
my_dir=path/to/my_dir
# A starting directory in the container:
start_dir=/files/omniisaacgymenvs/omniisaacgymenvs
# entryscript.sh will be sourced by bash when entering the container:
cp path/to/entryscript.sh $my_dir

ch-run \
    -w \
    -u0 -g0 \
    --set-env \
    --set-env=ACCEPT_EULA=Y \
    --set-env=PRIVACY_CONSENT=Y \
    -b ./cache/kit:/isaac-sim/kit/cache \
    -b ./cache/ov:/root/.cache/ov \
    -b ./cache/pip:/root/.cache/pip \
    -b ./cache/glcache:/root/.cache/nvidia/GLCache \
    -b ./cache/computecache:/root/.nv/ComputeCache \
    -b ./logs:/root/.nvidia-omniverse/logs \
    -b ./data:/root/.local/share/ov/data \
    -b ./documents:/root/Documents \
    -b $my_dir:/my_dir \
    --cd=$start_dir \
    $converted_images/sim-image \
    -- bash --rcfile /my_dir/entryscript.sh

Inside, run:

PYTHON_PATH scripts/rlgames_train.py task=Cartpole headless=True

get the error:

/isaac-sim/kit/python/bin/python3: symbol lookup error: /isaac-sim/extsPhysics/omni.physx-105.1.12-5.1/bin/libomni.physx.plugin.so: undefined symbol: cuDeviceGetCount

[reidpr]

Smells like some kind of library mis-match. Thanks for the nice reproduction recipe. I'll try to find the right person internally to think about it.

[j-ogas]

@VladimirFokow which version of nvidia-container-cli is being used by ch-fromhost?

[VladimirFokow]

~$ nvidia-container-cli --version

cli-version: 1.13.1
lib-version: 1.13.1
build date: 2023-04-24T12:23+00:00
build revision: 6f4aea0fca16aaff01bab2567adb34ec30847a0e
build compiler: x86_64-linux-gnu-gcc-7 7.5.0
build platform: x86_64
build flags: -D_GNU_SOURCE -D_FORTIFY_SOURCE=2 -DNDEBUG -std=gnu11 -O2 -g -fdata-sections -ffunction-sections -fplan9-extensions -fstack-protector -fno-strict-aliasing -fvisibility=hidden -Wall -Wextra -Wcast-align -Wpointer-arith -Wmissing-prototypes -Wnonnull -Wwrite-strings -Wlogical-op -Wformat=2 -Wmissing-format-attribute -Winit-self -Wshadow -Wstrict-prototypes -Wunreachable-code -Wconversion -Wsign-conversion -Wno-unknown-warning-option -Wno-format-extra-args -Wno-gnu-alignof-expression -Wl,-zrelro -Wl,-znow -Wl,-zdefs -Wl,--gc-sections

[VladimirFokow]

if needed:

nvidia-container-cli list

/dev/nvidiactl
/dev/nvidia-uvm
/dev/nvidia-uvm-tools
/dev/nvidia-modeset
/dev/nvidia1
/dev/nvidia0
/dev/nvidia3
/dev/nvidia2
/usr/bin/nvidia-smi
/usr/bin/nvidia-debugdump
/usr/bin/nvidia-persistenced
/usr/bin/nvidia-cuda-mps-control
/usr/bin/nvidia-cuda-mps-server
/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.525.105.17
/usr/lib/x86_64-linux-gnu/libnvidia-cfg.so.525.105.17
/usr/lib/x86_64-linux-gnu/libcuda.so.525.105.17
/usr/lib/x86_64-linux-gnu/libcudadebugger.so.525.105.17
/usr/lib/x86_64-linux-gnu/libnvidia-opencl.so.525.105.17
/usr/lib/x86_64-linux-gnu/libnvidia-ptxjitcompiler.so.525.105.17
/usr/lib/x86_64-linux-gnu/libnvidia-allocator.so.525.105.17
/usr/lib/x86_64-linux-gnu/libnvidia-compiler.so.525.105.17
/usr/lib/x86_64-linux-gnu/libnvidia-nvvm.so.525.105.17
/usr/lib/x86_64-linux-gnu/libnvidia-ngx.so.525.105.17
/usr/lib/x86_64-linux-gnu/libnvidia-encode.so.525.105.17
/usr/lib/x86_64-linux-gnu/libnvidia-opticalflow.so.525.105.17
/usr/lib/x86_64-linux-gnu/libnvcuvid.so.525.105.17
/usr/lib/x86_64-linux-gnu/libnvidia-eglcore.so.525.105.17
/usr/lib/x86_64-linux-gnu/libnvidia-glcore.so.525.105.17
/usr/lib/x86_64-linux-gnu/libnvidia-tls.so.525.105.17
/usr/lib/x86_64-linux-gnu/libnvidia-glsi.so.525.105.17
/usr/lib/x86_64-linux-gnu/libnvidia-fbc.so.525.105.17
/usr/lib/x86_64-linux-gnu/libnvidia-rtcore.so.525.105.17
/usr/lib/x86_64-linux-gnu/libnvoptix.so.525.105.17
/usr/lib/x86_64-linux-gnu/libGLX_nvidia.so.525.105.17
/usr/lib/x86_64-linux-gnu/libEGL_nvidia.so.525.105.17
/usr/lib/x86_64-linux-gnu/libGLESv2_nvidia.so.525.105.17
/usr/lib/x86_64-linux-gnu/libGLESv1_CM_nvidia.so.525.105.17
/usr/lib/x86_64-linux-gnu/libnvidia-glvkspirv.so.525.105.17
/usr/lib/i386-linux-gnu/libnvidia-ml.so.525.105.17
/usr/lib/i386-linux-gnu/libcuda.so.525.105.17
/usr/lib/i386-linux-gnu/libnvidia-opencl.so.525.105.17
/usr/lib/i386-linux-gnu/libnvidia-ptxjitcompiler.so.525.105.17
/usr/lib/i386-linux-gnu/libnvidia-compiler.so.525.105.17
/usr/lib/i386-linux-gnu/libnvidia-nvvm.so.525.105.17
/usr/lib/i386-linux-gnu/libnvidia-encode.so.525.105.17
/usr/lib/i386-linux-gnu/libnvidia-opticalflow.so.525.105.17
/usr/lib/i386-linux-gnu/libnvcuvid.so.525.105.17
/usr/lib/i386-linux-gnu/libnvidia-eglcore.so.525.105.17
/usr/lib/i386-linux-gnu/libnvidia-glcore.so.525.105.17
/usr/lib/i386-linux-gnu/libnvidia-tls.so.525.105.17
/usr/lib/i386-linux-gnu/libnvidia-glsi.so.525.105.17
/usr/lib/i386-linux-gnu/libnvidia-fbc.so.525.105.17
/usr/lib/i386-linux-gnu/libGLX_nvidia.so.525.105.17
/usr/lib/i386-linux-gnu/libEGL_nvidia.so.525.105.17
/usr/lib/i386-linux-gnu/libGLESv2_nvidia.so.525.105.17
/usr/lib/i386-linux-gnu/libGLESv1_CM_nvidia.so.525.105.17
/usr/lib/i386-linux-gnu/libnvidia-glvkspirv.so.525.105.17
/run/nvidia-persistenced/socket
/lib/firmware/nvidia/525.105.17/gsp_ad10x.bin
/lib/firmware/nvidia/525.105.17/gsp_tu10x.bin

[j-ogas]

Hmm, unfortunately the GPU systems I have access to only have nvidia-container-cli 1.0.0. I am unable to reproduce your symbol error with what I have; about 350s in I get:

2024-03-18 17:04:03 [347,474ms] [Error] [omniisaacgymenvs.robots.articulations.cartpole] Could not find Isaac Sim assets folder
2024-03-18 17:04:03 [347,474ms] [Error] [omni.kit.app._impl] [py stderr]: Error executing job with overrides: ['task=Cartpole', 'headless=True']

However, I think the error I encountered is probably fixable, I just don't know what I am doing.

So I wonder if the nvidia-container-cli version has something to do with your error.

[VladimirFokow]

Thank you very much for the attempt!
In the future I can try investigating on other systems, and with other versions,
this looks like the good direction to go.

[VladimirFokow]

Possibly, Isaac Sim isn't supported on A100 at all (A100 have no RT cores):
https://forums.developer.nvidia.com/t/isaac-sim-orbit-docker-on-hpc/269377
so possibly that's the cause of error..