AIForum.

How to Run Ollama in Docker with GPU Support

On this page

THE SHORT ANSWER

Ollama runs in Docker with a single command, and GPU acceleration is a host-configuration problem rather than a different image. On NVIDIA hardware you install the NVIDIA Container Toolkit and add --gpus=all; on AMD hardware you use the rocm tag with /dev/kfd and /dev/dri. This guide covers the documented commands, verification, persistence, and the failures that most often send a container back to CPU.
Quick facts

Ollama runs comfortably inside Docker, and adding GPU acceleration is a host-configuration problem rather than a different image. On NVIDIA hardware you install the NVIDIA Container Toolkit, configure the Docker runtime, and start the container with --gpus=all. On AMD hardware you switch to the rocm image tag and pass the /dev/kfd and /dev/dri devices. The model store, the API port, and the CLI stay identical to a CPU-only deployment.

This guide is written for a technically curious beginner or intermediate practitioner who already has Docker installed and wants a reproducible GPU setup instead of a one-off experiment. The commands below come from the official Ollama Docker documentation; they are documented procedures, not the results of an editor-run benchmark on specific hardware.

What Docker adds to a local Ollama install

Running Ollama in a container buys three things that matter for local AI work. First, version pinning: the image tag defines the runtime, so an upgrade is a deliberate docker pull rather than an accidental side effect. Second, isolation: the container has its own filesystem and does not touch a host-level Ollama installation. Third, persistence through a named volume — the official image stores models under /root/.ollama, so mounting -v ollama:/root/.ollama keeps multi-gigabyte weights across container restarts and image upgrades.

The tradeoff is that the container does not include GPU drivers. It depends on the host driver and, for NVIDIA, on a runtime hook installed by the NVIDIA Container Toolkit. That dependency is the most common reason a container appears to work but quietly falls back to CPU.

Prerequisites

  • Docker Engine installed and functional on the host.
  • NVIDIA path: a working NVIDIA driver plus the NVIDIA Container Toolkit, installed from NVIDIA’s package repository.
  • AMD path: a host configuration that exposes /dev/kfd and /dev/dri to containers.
  • Enough disk space for model weights; they live in the Docker volume, not the image.
  • Port 11434 free on the host, or a different published port.

The CPU-only baseline

Start with the CPU-only command, because it separates Docker problems from GPU problems. If this works, the Docker plumbing is fine and any later failure is GPU-specific.

docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

The flags do the obvious work: -d detaches, -v ollama:/root/.ollama persists the model store in a named volume, -p 11434:11434 publishes the API port, and --name ollama gives the container a stable handle. Once the container is running, you can execute the CLI inside it:

docker exec -it ollama ollama run <model>

Replace the placeholder with whichever model you intend to use. The first run downloads the weights into the volume; subsequent runs reuse them.

Running Ollama with an NVIDIA GPU

NVIDIA acceleration requires three host-side steps before the container command changes at all: install the toolkit, install its packages, and tell Docker to use the NVIDIA runtime.

1. Configure the NVIDIA Container Toolkit repository (Apt)

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -fsSL https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update

2. Install the toolkit packages

sudo apt-get install -y nvidia-container-toolkit

3. Configure Docker to use the NVIDIA driver

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

The restart is not optional. Docker reads its runtime configuration at startup, so a container launched before the restart will not see the NVIDIA runtime.

4. Start the container with GPU access

docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

The only difference from the CPU command is --gpus=all. Everything else — volume, port, name — is unchanged.

Yum and Dnf hosts

On RPM-based distributions the repository setup differs, but the install and configuration steps follow the same shape:

curl -fsSL https://nvidia.github.io/libnvidia-container/stable/rpm/nvidia-container-toolkit.repo | sudo tee /etc/yum.repos.d/nvidia-container-toolkit.repo
sudo yum install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

NVIDIA Jetson boards

On JetPack systems, Ollama cannot detect the correct JetPack version automatically. The documentation instructs you to pass JETSON_JETPACK=5 or JETSON_JETPACK=6 to the container to select version 5 or 6 respectively. Add it as an -e environment variable in the run command.

Running Ollama with an AMD GPU

AMD support uses a separate image tag and device passthrough instead of a runtime hook:

docker run -d --device /dev/kfd --device /dev/dri -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama:rocm

Two details are easy to miss. The tag is ollama/ollama:rocm, not the default ollama/ollama, and both device flags are required — /dev/kfd for the compute interface and /dev/dri for the render nodes.

Vulkan

The Docker documentation also notes that Vulkan support is bundled into the ollama/ollama image. Because the available documentation excerpt stops at that point, treat Vulkan as a documented capability and check the current page for the exact host requirements before relying on it in production.

Verifying that the GPU is actually in use

A container that starts successfully is not proof of GPU acceleration. Work through these checks in order:

  1. Run a model inside the container with docker exec -it ollama ollama run <model> and confirm generation completes.
  2. Inspect container logs with docker logs ollama. Ollama reports the compute device it selected during model load; a CPU-only fallback is usually visible there.
  3. Check the running model list inside the container to see which model is loaded and which processor is handling it.
  4. Compare generation speed against the CPU-only baseline. A large, immediate difference is the practical confirmation.

Persistence, updates, and daily operation

The named volume is what makes the container replaceable. To update, pull the newer image and recreate the container against the same volume:

docker pull ollama/ollama
docker rm -f ollama
docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama

Because models live in the volume, the new container starts with the same model library. If you skip the volume mount, every recreation re-downloads everything.

Troubleshooting: the container runs but the GPU does not

Treat each of these as a hypothesis to confirm, not a diagnosis to assume.

The NVIDIA runtime was configured but Docker was not restarted

This is a frequent cause. Re-run sudo nvidia-ctk runtime configure --runtime=docker followed by sudo systemctl restart docker, then recreate the container.

The container was started without --gpus=all

Check the original run command or docker inspect output. Without the flag, the container has no GPU devices regardless of host configuration.

AMD: wrong tag or missing device flags

Confirm the image is ollama/ollama:rocm and that both --device /dev/kfd and --device /dev/dri are present. A container started from the default tag will run, but on CPU.

Jetson: version not declared

If the board is a JetPack system, add -e JETSON_JETPACK=5 or -e JETSON_JETPACK=6 matching the installed JetPack major version.

Port 11434 is already in use

A native Ollama installation or another container may already own the port. Either stop the other service or publish a different host port, for example -p 11435:11434, and point clients at the new port.

Device permissions on AMD hosts

If the container starts but cannot open the render devices, the container user may lack access to /dev/dri. This depends on host group configuration and is worth checking against your distribution’s documentation.

Memory sizing: weights are not the whole story

When deciding whether a model fits, do not treat the download size as the memory requirement. Runtime memory includes the quantised weights, the key/value cache that grows with context length, and the runtime’s own overhead. A model that loads comfortably at a short context can run out of memory at a long one. If a model fails to load or the system starts swapping, reduce the context length or move to a smaller quantisation before assuming the GPU is at fault.

Exposing the API safely

Ollama’s HTTP API does not include built-in authentication. Publishing -p 11434:11434 binds the port on all host interfaces by default, which is convenient on a trusted workstation and risky on a shared or internet-facing machine. If the container only needs to serve local tools, bind it to the loopback interface with -p 127.0.0.1:11434:11434. For remote access, place it behind a reverse proxy that handles authentication and TLS rather than exposing the port directly.

Where to go next

Pick the path that matches your hardware, run the CPU baseline first, then add GPU support and verify it with a log check and a speed comparison. If something does not behave as documented, share your host OS, GPU model, Docker version, the exact docker run command, and the relevant container log lines in the Ai-Forum community — that combination is usually enough for another member to identify whether the problem is the toolkit, the runtime configuration, or the image tag.

FAQ

Can I run Ollama in Docker without a GPU?

Yes. The CPU-only command is docker run -d -v ollama:/root/.ollama -p 11434:11434 --name ollama ollama/ollama. It is a useful baseline because it isolates Docker problems from GPU configuration problems.

How do I keep downloaded models between container restarts?

Mount a named volume at /root/.ollama, as in -v ollama:/root/.ollama. Models are stored there rather than in the container filesystem, so they survive docker rm and image upgrades.

Why is my NVIDIA GPU not being used by the container?

Check three things in order: the NVIDIA Container Toolkit is installed, nvidia-ctk runtime configure --runtime=docker has been run and Docker restarted, and the container was started with --gpus=all. Missing any one of these leaves the container running on CPU.

Is there a separate Ollama image for AMD GPUs?

Yes. Use the rocm tag, ollama/ollama:rocm, and pass both --device /dev/kfd and --device /dev/dri when starting the container.

How do I update Ollama running in Docker?

Run docker pull ollama/ollama, remove the existing container, and start a new one from the same image with the same volume mount and GPU flags. The model library in the volume is preserved.

References

  1. https://docs.ollama.com/docker
  2. https://hub.docker.com/r/ollama/ollama

Related terms

Ollama
A tool for downloading and running large language models locally, distributed as a CLI, a local HTTP API, and an official Docker image.
NVIDIA Container Toolkit
The package set that lets Docker containers access NVIDIA GPUs, configured through nvidia-ctk runtime configure and a Docker restart.
ROCm
AMD's GPU compute stack; the Ollama Docker image ships a rocm tag for AMD acceleration.
Docker named volume
Persistent storage managed by Docker and mounted into a container; used here to keep model weights at /root/.ollama across container recreation.
GPU passthrough
Exposing host GPU devices to a container, done with --gpus=all on NVIDIA or explicit --device flags on AMD.
JetPack
NVIDIA's software stack for Jetson boards; Ollama cannot detect its version automatically, so JETSON_JETPACK must be set explicitly.

Leave a Reply

Your email address will not be published. Required fields are marked *