GPU passthrough to a rootless sandbox

Taming Nvidia CDI in rootless Podman with UserNS isolation – without compromises

There was a need to run GPU-accelerated containers under a dedicated rootless Podman user (limu), with --userns isolation and an unprivileged --user inside the container, like that:

podman -c limu run --rm \
  --device=nvidia.com/gpu=all \
  --userns auto:size=65536,gidmapping=0:@9931:1 \
  --user 1729 \
  nvidia/cuda nvidia-smi  # ollama/ollama

Legend:

  • limu (9929) – host user, member of shg
  • shg (9931) – host supplementary group for shared resources
  • 1729 – in-container UID to run as

The numbers are carefully selected purely for the docs/examples. The 9929, 9931 pair is as nice as twin primes, 1729 is a pseudoprime for a pseudouser.

  • Ubuntu-24.04 (aka Noble Numbat, LTS)
  • Podman-5.4.1 (yeah, on Noble)
  • Nvidia Driver 580 series, CUDA 13.0

Running GPU workloads in Docker/Podman is straightforward until you start tightening isolation. In our case the default NVIDIA Container Toolkit CDI setup fell apart.

Right out of the box with a generated /var/run/cdi/nvidia.yaml, container startup failed immediately on createContainer hooks:

Error: preparing container … for attach: crun: {“msg”:“error executing hook /usr/bin/nvidia-cdi-hook (exit code: 1)”,…}: OCI runtime error

A quick-and-dirty wrapper around nvidia-cdi-hook showed everything was fine up to pivot_root, and failed somewhere after, in the unprivileged namespace. Whether the root cause was the rootless + fuse-overlayfs on ZFS combination, or --userns auto, or something else in hook execution – it was tempting to wrap things further and reach for strace, but digging deeper into crun/CDI internals wasn’t the goal. Thankfully, disabling the hooks via NVIDIA_CTK_CDI_GENERATE_DISABLED_HOOKS=all let the container start:

root@227100f97548:/# nvidia-smi
NVIDIA-SMI couldn't find libnvidia-ml.so library in your system.

Good news, in fact. Even with hooks disabled, nvidia-ctk still provides all the required libs (libnvidia-ml.so.580..., libcuda.so.580..., …), only the *.so.1 soname symlinks are missing. The rest of the problem is reduced to a simple ldconfig.

CautionMind best practices: non-root users and --read-only

Running /sbin/ldconfig as a hook or entrypoint works only when the container starts as root (UID 0), and fails once --user 1729 or --read-only is used.

Instead of creating symlinks inside the container at runtime, we’ll duplicate each lib*.so.<driver_version> mount entry in /var/run/cdi/nvidia.yaml directly at its lib*.so.1 target path. A small post-processing script does the trick, we just ask the CDI config refresh systemd-service to post-exec it.

The recap

  1. Assuming you already have nvidia-container-toolkit installed

  2. Disable default hooks:

    /etc/nvidia-container-toolkit/nvidia-cdi-refresh.env
    NVIDIA_CTK_CDI_GENERATE_DISABLED_HOOKS=all
  3. Add a CDI config post-processing script

    e.g. /usr/local/sbin/patch-nvidia-cdi, see below

  4. Hook it into nvidia-cdi-refresh.service, so the solution survives reboots and driver updates:

    sudo systemctl edit nvidia-cdi-refresh.service
    /etc/systemd/system/nvidia-cdi-refresh.service.d/override.conf
    [Service]
    ExecStartPost=/usr/local/sbin/patch-nvidia-cdi

    Don’t forget to daemon-reload and restart the nvidia-cdi-refresh.service to apply the changes and regenerate the CDI spec.

WARN: python3-yaml (apt) or pyyaml (pypi) required

/usr/local/sbin/patch-nvidia-cdi
#!/usr/bin/env python3

import os
import re
import yaml


def add_soname_mounts(ce):
    mounts = ce.get('mounts', [])
    existing = {m['containerPath'] for m in mounts}
    extra = []
    for m in mounts:
        cp = m.get('containerPath', '')
        # /usr/lib/.../libfoo.so.580.178.04
        match = re.match(r'^(.*/lib[^/]+\.so)\.\d+\.\d+.*$', cp)
        if match:
            soname1 = match.group(1) + '.1'
            if soname1 not in existing:
                new_m = dict(m)
                new_m['containerPath'] = soname1
                extra.append(new_m)
                existing.add(soname1)
    if extra:
        ce['mounts'] = mounts + extra


CDI_FILE = '/var/run/cdi/nvidia.yaml'
cdi_file_patched = f'{CDI_FILE}.patched'
cdi_file_backup = f'{CDI_FILE}.bak'

with open(CDI_FILE) as f:
    d = yaml.safe_load(f)

if 'containerEdits' in d:
    add_soname_mounts(d['containerEdits'])

for dev in d.get('devices', []):
    if 'containerEdits' in dev:
        add_soname_mounts(dev['containerEdits'])

with open(cdi_file_patched, 'w') as f:
    yaml.safe_dump(d, f, sort_keys=False)

os.rename(CDI_FILE, cdi_file_backup)
os.rename(cdi_file_patched, CDI_FILE)

Least Privilege after-party

By default, the NVIDIA kernel driver creates /dev/nvidia* with mode 0666 (crw-rw-rw-), exposing the GPU to any local user or unrelated rootless container. Limit access to the render group (say, GID 992):

/etc/modprobe.d/nvidia-permissions.conf
options nvidia NVreg_DeviceFileUID=0 NVreg_DeviceFileGID=992 NVreg_DeviceFileMode=0660 NVreg_ModifyDeviceFiles=1

Check your local render GID with getent group render

Followed by update-initramfs -u, usermod -aG render ..., and a reboot.

P.S. Once /dev/nvidia* is locked down to root:render, 0660, --userns=auto will immediately remind you that host GID doesn’t exist inside the container’s user namespace. RTFM on gidmapping in podman-run(1) ;) And don’t fall into another compromise with keep-groups, which forwards all supplementary groups of the calling user rather than just render.