There was a need to run GPU-accelerated containers under a dedicated rootless Podman user (limu), with --userns isolation and an unprivileged --user inside the container, like that:
podman -c limu run --rm \
--device=nvidia.com/gpu=all \
--userns auto:size=65536,gidmapping=0:@9931:1 \
--user 1729 \
nvidia/cuda nvidia-smi # ollama/ollamaLegend:
limu(9929) – host user, member ofshgshg(9931) – host supplementary group for shared resources- 1729 – in-container UID to run as
Running GPU workloads in Docker/Podman is straightforward until you start tightening isolation. In our case the default NVIDIA Container Toolkit CDI setup fell apart.
Right out of the box with a generated /var/run/cdi/nvidia.yaml, container startup failed immediately on createContainer hooks:
Error: preparing container … for attach: crun: {“msg”:“error executing hook
/usr/bin/nvidia-cdi-hook(exit code: 1)”,…}: OCI runtime error
A quick-and-dirty wrapper around nvidia-cdi-hook showed everything was fine up to pivot_root, and failed somewhere after, in the unprivileged namespace. Whether the root cause was the rootless + fuse-overlayfs on ZFS combination, or --userns auto, or something else in hook execution – it was tempting to wrap things further and reach for strace, but digging deeper into crun/CDI internals wasn’t the goal. Thankfully, disabling the hooks via NVIDIA_CTK_CDI_GENERATE_DISABLED_HOOKS=all let the container start:
root@227100f97548:/# nvidia-smi
NVIDIA-SMI couldn't find libnvidia-ml.so library in your system.Good news, in fact. Even with hooks disabled, nvidia-ctk still provides all the required libs (libnvidia-ml.so.580..., libcuda.so.580..., …), only the *.so.1 soname symlinks are missing. The rest of the problem is reduced to a simple ldconfig.
--read-only
Running /sbin/ldconfig as a hook or entrypoint works only when the container starts as root (UID 0), and fails once --user 1729 or --read-only is used.
Instead of creating symlinks inside the container at runtime, we’ll duplicate each lib*.so.<driver_version> mount entry in /var/run/cdi/nvidia.yaml directly at its lib*.so.1 target path. A small post-processing script does the trick, we just ask the CDI config refresh systemd-service to post-exec it.
The recap
Assuming you already have nvidia-container-toolkit installed
Disable default hooks:
/etc/nvidia-container-toolkit/nvidia-cdi-refresh.env
NVIDIA_CTK_CDI_GENERATE_DISABLED_HOOKS=allAdd a CDI config post-processing script
e.g.
/usr/local/sbin/patch-nvidia-cdi, see belowHook it into nvidia-cdi-refresh.service, so the solution survives reboots and driver updates:
sudo systemctl edit nvidia-cdi-refresh.service/etc/systemd/system/nvidia-cdi-refresh.service.d/override.conf
[Service] ExecStartPost=/usr/local/sbin/patch-nvidia-cdiDon’t forget to daemon-reload and restart the nvidia-cdi-refresh.service to apply the changes and regenerate the CDI spec.
Least Privilege after-party
By default, the NVIDIA kernel driver creates /dev/nvidia* with mode 0666 (crw-rw-rw-), exposing the GPU to any local user or unrelated rootless container. Limit access to the render group (say, GID 992):
/etc/modprobe.d/nvidia-permissions.conf
options nvidia NVreg_DeviceFileUID=0 NVreg_DeviceFileGID=992 NVreg_DeviceFileMode=0660 NVreg_ModifyDeviceFiles=1Check your local render GID with getent group render
Followed by update-initramfs -u, usermod -aG render ..., and a reboot.
P.S. Once /dev/nvidia* is locked down to root:render, 0660, --userns=auto will immediately remind you that host GID doesn’t exist inside the container’s user namespace. RTFM on gidmapping in podman-run(1) ;) And don’t fall into another compromise with keep-groups, which forwards all supplementary groups of the calling user rather than just render.