Installing the NVIDIA Container Toolkit: Docker, containerd, CRI-O, Podman and Kubernetes

This technical note is for engineers who need GPUs inside containers: platform and MLOps teams, Kubernetes administrators, and anyone handed a new GPU server and asked to get Docker working on it. It covers what the NVIDIA Container Toolkit does, how to install it on Debian, Ubuntu, RHEL-family and SUSE systems, how to configure Docker, rootless Docker, containerd, nerdctl, CRI-O and Podman, how to test it, and what goes wrong. Commands follow NVIDIA's install guide for toolkit version 1.20.1, the current release when we wrote this. The explanations, checks, troubleshooting and fleet advice are our own, from building and supporting GPU servers.

Contents

1. What the toolkit does
2. Before you start
3. Install on Ubuntu and Debian (apt)
4. Install on RHEL, Rocky, Fedora and Amazon Linux (dnf)
5. Install on openSUSE and SLES (zypper)
6. Pin versions and plan upgrades
7. Configure Docker
8. Configure rootless Docker
9. Configure containerd for Kubernetes
10. nerdctl
11. Configure CRI-O
12. Podman and the Container Device Interface (CDI)
13. Change the toolkit's own settings safely
14. Test GPU access in a container
15. Troubleshooting
16. Running it across a fleet
17. How Workstation can help

1. What the toolkit does

A container image should not carry a GPU driver. The driver is tied to the kernel module on the host, and an image built for one driver version would break on the next. The NVIDIA Container Toolkit solves this: when a container asks for a GPU, the toolkit finds the host's driver libraries, device nodes and utilities, such as nvidia-smi, and mounts them into the container just before it starts. The image only needs the CUDA user-space libraries the application was built with.

The toolkit is a small set of packages that work together:

Package or tool	What it does
libnvidia-container1 and libnvidia-container-tools	The library and CLI that find the driver components on the host and set them up inside the container.
nvidia-container-toolkit-base	The nvidia-container-runtime, a thin wrapper around runc (or crun) that adds the GPU set-up to the container spec, and the nvidia-ctk command.
nvidia-container-toolkit	The hook that container engines call when a container starts, plus everything above as dependencies.
nvidia-ctk	The command you use to configure container engines, edit the toolkit's settings and generate CDI specifications.

You do not need the CUDA Toolkit on the host. The host needs the NVIDIA driver; the CUDA version lives in the container image. The one rule is that the host driver must be new enough for the CUDA version in the image. A newer driver runs images built for older CUDA versions.

2. Before you start

• A supported distribution. NVIDIA publishes the list of supported platforms. In practice: current Ubuntu LTS and Debian releases, RHEL and its rebuilds (Rocky, AlmaLinux), Fedora, Amazon Linux, openSUSE and SLES, on x86_64 or arm64.
• A working NVIDIA driver. Install it from your distribution's package manager, as NVIDIA recommends, or from NVIDIA's .run installer. Check it before going further:

  $ nvidia-smi

  If this doesn't list your GPUs, stop and fix the driver first. The toolkit can't work without it. On servers with Secure Boot, the kernel module must be signed, or the driver will install but not load.
• A container engine for the configuration steps: Docker, containerd, CRI-O or Podman. You can install the toolkit first and the engine afterwards.
• Root or sudo access, and outbound HTTPS to nvidia.github.io, or a local mirror of NVIDIA's repository on air-gapped sites.

A known issue to plan for. On hosts where the container engine uses the systemd cgroup driver, running systemctl daemon-reload can make running containers lose access to their GPUs. They then report Failed to initialize NVML: Unknown Error. Package upgrades often trigger a daemon reload, so this tends to appear days after installation. NVIDIA's troubleshooting guide covers it; the usual fixes are to use CDI (section 12) or to create the /dev/char symlinks the driver devices need:

$ sudo nvidia-ctk system create-dev-char-symlinks --create-all

3. Install on Ubuntu and Debian (apt)

These steps work on most Debian-derived distributions.

Step 1: install the tools the repository set-up needs.

$ sudo apt-get update && sudo apt-get install -y --no-install-recommends \
      ca-certificates curl gnupg2

Step 2: add NVIDIA's signing key and the production repository. The key goes in its own keyring and the repository is restricted to it with signed-by, so this key can't sign packages from any other repository.

$ curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
    | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
$ curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
    | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
    | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

NVIDIA also publishes an experimental channel. Leave it off on production machines; to turn it on for testing, uncomment it:

$ sudo sed -i -e '/experimental/ s/^#//g' /etc/apt/sources.list.d/nvidia-container-toolkit.list

Step 3: refresh the package lists.

$ sudo apt-get update

Step 4: install the four packages at one version. Naming the version keeps every host in a fleet identical, and keeps the four packages in step with each other.

$ export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.20.1-1
$ sudo apt-get install -y \
    nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}

Check what you installed:

$ nvidia-ctk --version

4. Install on RHEL, Rocky, Fedora and Amazon Linux (dnf)

These steps work on most RPM-based distributions that use dnf.

Step 1: make sure curl is present.

$ sudo dnf install -y curl

Step 2: add the production repository.

$ curl -s -L https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo \
    | sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo

To try experimental packages on a test machine:

$ sudo dnf config-manager --enable nvidia-container-toolkit-experimental

Step 3: install the packages at one version. Note that dnf joins the name and version with a hyphen, not an equals sign.

$ export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.20.1-1
$ sudo dnf install -y \
    nvidia-container-toolkit-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    nvidia-container-toolkit-base-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    libnvidia-container-tools-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    libnvidia-container1-${NVIDIA_CONTAINER_TOOLKIT_VERSION}

dnf asks you to accept NVIDIA's signing key the first time. Check the fingerprint against NVIDIA's published key before you accept it.

5. Install on openSUSE and SLES (zypper)

Step 1: add the production repository.

$ sudo zypper ar https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo

To try experimental packages:

$ sudo zypper modifyrepo --enable nvidia-container-toolkit-experimental

Step 2: install the packages at one version.

$ export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.20.1-1
$ sudo zypper --gpg-auto-import-keys install -y \
    nvidia-container-toolkit-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    nvidia-container-toolkit-base-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    libnvidia-container-tools-${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    libnvidia-container1-${NVIDIA_CONTAINER_TOOLKIT_VERSION}

--gpg-auto-import-keys trusts the repository's key without asking. That suits automation; on a single server you may prefer to import and check the key by hand first.

6. Pin versions and plan upgrades

An unattended upgrade that moves the toolkit, or moves the driver without the toolkit, is a common cause of GPU containers failing overnight. Hold the packages and upgrade them on purpose:

Distribution	Hold the toolkit packages
Ubuntu, Debian	sudo apt-mark hold nvidia-container-toolkit nvidia-container-toolkit-base libnvidia-container-tools libnvidia-container1
RHEL family, Fedora	sudo dnf versionlock add nvidia-container-toolkit nvidia-container-toolkit-base libnvidia-container-tools libnvidia-container1 (needs the versionlock plugin)
openSUSE, SLES	sudo zypper addlock nvidia-container-toolkit nvidia-container-toolkit-base libnvidia-container-tools libnvidia-container1

When you upgrade, upgrade all four packages together, on one test host first, then follow the order in section 16.

7. Configure Docker

Step 1: register the NVIDIA runtime with Docker.

$ sudo nvidia-ctk runtime configure --runtime=docker

This edits /etc/docker/daemon.json and adds an nvidia runtime. Existing settings in the file are kept. The result looks like this:

{
  "runtimes": {
    "nvidia": {
      "args": [],
      "path": "nvidia-container-runtime"
    }
  }
}

Step 2: restart Docker.

$ sudo systemctl restart docker

Restarting Docker stops running containers unless live restore is enabled, so do this in a maintenance window on a busy host. If you manage daemon.json with configuration management, add the runtime there instead, or your tool will remove it on its next run.

Docker's --gpus flag works once the runtime is registered. If you want every container to use the NVIDIA runtime by default, for example on a dedicated GPU host, add --set-as-default to the configure command.

8. Configure rootless Docker

Rootless Docker runs the daemon as an ordinary user, with its own configuration file. Three steps:

Step 1: register the runtime in the user's Docker configuration (no sudo):

$ nvidia-ctk runtime configure --runtime=docker --config=$HOME/.config/docker/daemon.json

Step 2: restart the user's Docker daemon.

$ systemctl --user restart docker

Step 3: tell the toolkit not to manage cgroups, which a rootless daemon isn't allowed to do. This is a host-wide setting, so it needs sudo:

$ sudo nvidia-ctk config --set nvidia-container-cli.no-cgroups --in-place

That setting affects every engine on the host. Don't apply it on a host that also runs rootful containers that rely on the toolkit for cgroup set-up.

9. Configure containerd for Kubernetes

Step 1: register the runtime with containerd.

$ sudo nvidia-ctk runtime configure --runtime=containerd

By default this writes a drop-in file, /etc/containerd/conf.d/99-nvidia.toml, and adds it to the imports list in /etc/containerd/config.toml, creating that file if it doesn't exist. Keeping the NVIDIA settings in a drop-in means your own config.toml stays readable and the change is easy to remove.

Step 2: restart containerd.

$ sudo systemctl restart containerd

Kubernetes then needs to know about the GPUs. Most clusters use the NVIDIA GPU Operator, which can install the toolkit and device plugin for you, or the NVIDIA device plugin on its own. Pods select the runtime through a RuntimeClass named nvidia, or use it by default if you set it as containerd's default runtime. If you use the GPU Operator with its built-in toolkit, don't also install the toolkit by hand on the nodes, or the two will fight over the same configuration.

Distributions that ship their own containerd, such as k3s and RKE2, keep its configuration in a different place and often detect the NVIDIA runtime automatically. Follow their GPU documentation rather than editing /etc/containerd.

10. nerdctl

nerdctl, the Docker-style command line for containerd, needs no configuration. Once the toolkit is installed, nerdctl run --gpus=all works with or without root. You don't need the containerd configuration from section 9 for nerdctl on its own.

11. Configure CRI-O

Step 1: register the runtime with CRI-O.

$ sudo nvidia-ctk runtime configure --runtime=crio

This writes a drop-in file, /etc/crio/conf.d/99-nvidia.toml, so CRI-O can use the NVIDIA runtime.

Step 2: restart CRI-O.

$ sudo systemctl restart crio

12. Podman and the Container Device Interface (CDI)

For Podman, NVIDIA recommends the Container Device Interface. CDI is an open specification: a YAML file describes each device and everything a container needs to use it, and any engine that supports CDI reads that file. Nothing has to hook into the engine at start-up.

Step 1: generate the CDI specification for the GPUs in this host:

$ sudo nvidia-ctk cdi generate --output=/etc/cdi/nvidia.yaml

Step 2: check the device names it created:

$ nvidia-ctk cdi list
nvidia.com/gpu=0
nvidia.com/gpu=1
nvidia.com/gpu=all
...

Step 3: run a container with a GPU.

$ podman run --rm --device nvidia.com/gpu=all --security-opt=label=disable \
    ubuntu nvidia-smi -L

--security-opt=label=disable is needed on SELinux systems such as RHEL, where the default label stops the container reading the device files.

The specification records the driver version and file paths at the time it was generated. Regenerate it after every driver upgrade, and after adding or removing GPUs or changing MIG partitions. Recent toolkit releases can do this automatically through a systemd unit; check whether your version includes it before relying on it. Docker and containerd can use CDI too, and it avoids the cgroup issue described in section 2.

13. Change the toolkit's own settings safely

The toolkit keeps its settings in /etc/nvidia-container-runtime/config.toml. Edit it with nvidia-ctk config rather than by hand. Without --in-place or --output, the command only prints the result, which gives you a safe preview.

Preview a change:

$ sudo nvidia-ctk config --set nvidia-container-runtime.log-level=debug

Apply it once you're happy with the output:

$ sudo nvidia-ctk config --in-place --set nvidia-container-runtime.log-level=debug

Set several values in one go by repeating --set. This turns on debug logging and sends it to a file:

$ sudo nvidia-ctk config --in-place \
    --set nvidia-container-runtime.log-level=debug \
    --set nvidia-container-runtime.debug=/var/log/nvidia-container-runtime.log

For a list setting, separate the items with colons. This makes crun the first low-level runtime to try, with runc as the fallback:

$ sudo nvidia-ctk config --in-place --set nvidia-container-runtime.runtimes=crun:runc

Settings you don't name are kept. If the same key is given twice, the last value wins. Turn debug logging off again when you've finished: the log grows quickly on a busy host.

14. Test GPU access in a container

Run nvidia-smi in a plain Ubuntu container. The image has no NVIDIA software in it; the toolkit mounts the utility in.

$ docker run --rm --gpus all ubuntu nvidia-smi

You should see the same GPU table as on the host. Then check the cases you'll use in production:

What to check	Command
All GPUs	docker run --rm --gpus all ubuntu nvidia-smi -L
Specific GPUs by index	docker run --rm --gpus '"device=0,1"' ubuntu nvidia-smi -L
A number of GPUs	docker run --rm --gpus 2 ubuntu nvidia-smi -L
Through the runtime directly	docker run --rm --runtime=nvidia -e NVIDIA_VISIBLE_DEVICES=all ubuntu nvidia-smi
Through CDI	docker run --rm --device nvidia.com/gpu=all ubuntu nvidia-smi
A real CUDA workload	Run a CUDA sample or your framework's GPU check (for example torch.cuda.is_available()) in the image you'll actually deploy.

Note the quoting in the second example: the device list needs both sets of quotes, or Docker splits it at the comma.

Containers get a limited set of driver features by default: compute and utility. Graphics, video encode and decode, and display need more. Set them with the NVIDIA_DRIVER_CAPABILITIES environment variable, for example -e NVIDIA_DRIVER_CAPABILITIES=compute,utility,video for a transcoding container.

15. Troubleshooting

Symptom	Likely cause	Fix
could not select device driver "" with capabilities: [[gpu]]	Docker doesn't know about the NVIDIA runtime.	Run nvidia-ctk runtime configure --runtime=docker and restart Docker. Check that configuration management hasn't overwritten daemon.json.
Failed to initialize NVML: Unknown Error in a container that used to work	The systemd cgroup issue after systemctl daemon-reload.	Restart the container. To stop it recurring, use CDI or create the /dev/char symlinks (section 2).
Failed to initialize NVML: Driver/library version mismatch on the host	The driver package was upgraded but the old kernel module is still loaded.	Reboot, or unload and reload the module in a maintenance window.
nvidia-smi works on the host but not in the container	The container started without a GPU request, or the capabilities exclude utility.	Add --gpus or a CDI device, and check NVIDIA_DRIVER_CAPABILITIES.
CUDA error saying the driver is too old	The image's CUDA version needs a newer driver than the host has.	Upgrade the host driver, or use an image built for an older CUDA version.
CDI device not found after a driver upgrade	The CDI specification describes the old driver.	Regenerate it with nvidia-ctk cdi generate.
Permission denied on /dev/nvidia* with Podman	SELinux labelling.	Add --security-opt=label=disable, or use a policy that allows the devices.
Errors from the hook with rootless Docker	The toolkit is trying to manage cgroups.	Set nvidia-container-cli.no-cgroups (section 8).

For anything else, turn on debug logging (section 13), reproduce the failure, and read the log. It records every file and device the toolkit tried to mount.

16. Running it across a fleet

• Automate it. Put the repository, the pinned packages, the engine configuration and the CDI generation in Ansible, Salt or your image build, not in a runbook someone types by hand.
• Upgrade in rings. Upgrade the driver and toolkit on one test host, run your real workloads and the checks in section 14, then move to a small group of production hosts, then the rest. Our own Ring Promoter works this way for software releases.
• Keep driver and toolkit changes separate. Change one at a time, so you know which change caused a problem.
• Regenerate CDI specifications as part of every driver upgrade.
• Mirror the repository for air-gapped or regulated sites, and sign or checksum what you mirror.
• Monitor the GPUs, not just the containers. NVIDIA DCGM and its Prometheus exporter show GPU health, errors and use, so you can tell a failing GPU from a broken container.
• Record versions. Keep the driver, toolkit, CUDA image and engine versions for each host in your inventory. It's the first thing anyone asks when something breaks.

17. How Workstation can help

Every AI server and workstation we build ships with the NVIDIA driver, the Container Toolkit and the container engine installed, configured and tested, with versions pinned and recorded. We can also:

• set up Kubernetes with the GPU Operator on your servers, with upgrades that roll out in rings;
• build the Ansible or image pipeline that keeps a GPU fleet consistent;
• find leased GPU capacity if you'd rather not own the hardware (see GPU Leasing);
• support the platform afterwards, including driver and toolkit upgrades.

See our AI workstations and GPU clusters, or contact us to talk to an engineer.

Sources

• NVIDIA, Installing the NVIDIA Container Toolkit, toolkit version 1.20.1. Install and configuration commands follow this guide.
• NVIDIA, Container Toolkit documentation on supported platforms, CDI support and troubleshooting.

NVIDIA, CUDA and DGX are trademarks and/or registered trademarks of NVIDIA Corporation in the U.S. and other countries. Docker, Kubernetes, Podman and other names are trademarks of their respective owners.