What you need first
Docker Engine on Linux, plus one thing that depends on your card. On NVIDIA you need the NVIDIA Container Toolkit, and installing the package is only half of it. `nvidia-ctk runtime configure` registers the runtime with Docker, and Docker has to restart before it notices. Skip either step and containers start fine but cannot see the GPU. On AMD there is no toolkit: the container is handed the `/dev/kfd` and `/dev/dri` devices directly. Prove the GPU is visible before you touch Ollama, because "does my container see the GPU" is far easier to debug than "Ollama is slow in Docker".
# NVIDIA: after installing nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker
# NVIDIA's own check: the toolkit injects nvidia-smi into a plain image
sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi
# AMD: these must exist on the host
ls -l /dev/kfd /dev/driThe compose file
Most copy-pasted examples get the GPU part wrong or leave it out. On NVIDIA it is the `deploy.resources.reservations.devices` block. On AMD it is the `devices` list and the `:rocm` image tag. Ollama’s default image also includes Vulkan, which it uses when it can reach the GPU devices, so a Radeon that ROCm does not support can try the plain image with the same two devices. Bind the port to 127.0.0.1 unless you mean to expose the API. Ollama has no authentication, and Docker’s published ports bypass ufw.
# NVIDIA
services:
ollama:
image: ollama/ollama
container_name: ollama
ports:
- "127.0.0.1:11434:11434"
volumes:
- ollama_data:/root/.ollama
restart: unless-stopped
deploy:
resources:
reservations:
devices:
- driver: nvidia
count: all
capabilities: [gpu]
volumes:
ollama_data:
# AMD: same file, but replace image + deploy with
# image: ollama/ollama:rocm
# devices:
# - /dev/kfd
# - /dev/driStart it and pull a model
Bring the container up, then pull a model into it. The model store lives in the named volume, so it survives `down` and `up` without a re-download. Llama 3.1 8B at Q4_K_M needs about 5.6 GB at 4K context, which is Ollama’s default window, so it is a reasonable first pull on any card with 8GB or more.
docker compose up -d
docker exec ollama ollama pull llama3.1:8bCheck it actually ran on the GPU
This is the step containers make easy to skip. The container starts successfully either way, on the CPU or the GPU, and only speed tells you which. Ask the container, not the host: `ollama ps` inside it reports where the loaded model actually sits. The container log also names the GPU Ollama found when it started, which is the quickest check on AMD.
docker exec ollama ollama run llama3.1:8b "hi" >/dev/null
docker exec ollama ollama ps
# NAME ID SIZE PROCESSOR UNTIL
# llama3.1:8b … 6.1 GB 100% GPU 4 minutes from now
docker logs ollama 2>&1 | grep -i -E "gpu|rocm|cuda|vulkan" | head
# NVIDIA, on the host:
nvidia-smi --query-gpu=memory.used,utilization.gpu --format=csv -l 1What the numbers should look like
A container passes the GPU through rather than emulating it, so once passthrough works there is no reason to expect a different speed. This site has not measured a containerised run against a native one, so take that as an expectation, not a measurement. What it does have is a ceiling: Llama 3.1 8B at Q4_K_M is 4.6 GB of weights, and token generation reads all of them for every token. On an RTX 4090 at 1,008 GB/s that caps generation near 218 tok/s. The index’s own llama.cpp run of that model and quant on that card measured 148, and Ollama uses llama.cpp underneath. A result an order of magnitude lower means the container is on the CPU.
When it does not work
`ollama ps` in the container shows a CPU share while `nvidia-smi` on the host works. The toolkit check from the first step tells you which part is missing. If it fails, the toolkit is not installed or `nvidia-ctk runtime configure` was never run, and nothing in Ollama’s configuration works around that. If it passes, the compose file is missing its `deploy` block. On AMD the container starts but finds no GPU: check that `/dev/kfd` and `/dev/dri` exist on the host and are listed under `devices`. On SELinux distributions, Ollama’s docs give `sudo setsebool container_use_devices=1` to let containers use them. Models disappear after `docker compose down`: no named volume was used, or it was removed with `-v`. On Windows this needs Docker Desktop’s WSL2 backend with GPU support for NVIDIA. On a Mac, run Ollama natively, because Docker containers on macOS cannot reach the Apple GPU.
Common questions
Why does Ollama in Docker use my CPU instead of my GPU?
On NVIDIA it is almost always the container toolkit or the compose file. Either the NVIDIA Container Toolkit is missing, it was installed but `sudo nvidia-ctk runtime configure --runtime=docker` and a Docker restart never followed, or the compose file has no `deploy.resources.reservations.devices` block. Test the toolkit on its own with `sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi`. If that fails, the problem is not Ollama. On AMD, the container needs `/dev/kfd` and `/dev/dri` passed in and the `ollama/ollama:rocm` image.
Can I run Ollama in Docker on an AMD GPU?
Yes, on Linux. Ollama’s docs use the `ollama/ollama:rocm` image with `--device /dev/kfd --device /dev/dri`, and no container toolkit is involved. The default `ollama/ollama` image also includes Vulkan and uses it when it can reach the same devices. That is the option to try for a Radeon that ROCm does not support. Check `docker logs ollama` to see which GPU it found.
Does running Ollama in Docker slow it down?
It should not once the GPU is passed through, because the container hands the real device to Ollama rather than emulating it. This site has not measured a containerised run against a native one, though. The large slowdowns people report are almost always a container that has fallen back to the CPU, and `ollama ps` inside the container shows that immediately.
How do I keep my downloaded models after recreating the container?
Use a named volume mapped to `/root/.ollama`, as in the compose file above. The model files then live outside the container’s lifecycle, so `docker compose down` followed by `up` does not trigger a re-download. Removing the volume with `down -v` does delete them, so that is the one command to avoid.
What this guide uses
- Models
- Llama 3.1 8B Instruct8B
- Format
- GGUF
Next steps
Models covered in this guide
Did this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
Docker Compose LLM Stack: Ollama + Open WebUI
A production-ready Docker Compose stack that gives you a local ChatGPT experience with one command.
Ollama on Windows (Native, No WSL)
Install the Windows Ollama app for the simplest path — NVIDIA and AMD Radeon GPUs both work, with no WSL and no toolkit to install.
Multi-Model API Server on RTX 4090 with vLLM
Serve multiple AWQ-quantized models with vLLM's continuous batching for production-grade throughput.