What you need first
Windows 10 22H2 or newer and a current GPU driver. Ollama’s documented minimum for NVIDIA is driver 551.61. AMD Radeon cards work too, not just NVIDIA. The RX 7600, 7600 XT, 7700 XT, 7800 XT and 7900 series run on ROCm, which needs AMD’s ROCm 7-capable driver. Other Radeons, the RX 6000 series included, run through Vulkan, which Ollama enables by default and which ships inside AMD’s normal driver. The installer brings its own GPU runtime, so there is no CUDA Toolkit or ROCm SDK to install and nothing to build, and it does not need administrator rights.
# Download and run OllamaSetup.exe from ollama.com, then:
ollama --version
# NVIDIA: driver version and VRAM
nvidia-smi
# AMD: Task Manager → Performance → GPU shows "Dedicated GPU memory"Pull a model and run it
Start with an 8B at Q4_K_M. Llama 3.1 8B at Q4_K_M needs about 5.6 GB at 4K context on this index’s numbers, and 4K is Ollama’s default context window, so it fits an 8GB card with room to spare. Ollama picks the GPU itself; there is no flag to turn it on. A longer window costs memory: set OLLAMA_CONTEXT_LENGTH for every model, or num_ctx for one session, and check the size in the calculator first.
ollama pull llama3.1:8b
ollama run llama3.1:8b
# Inside the chat, for this session only:
/set parameter num_ctx 8192Where the models actually go
Models land in .ollama\models under your user profile, which is usually on the C: drive, and a few models will fill a small SSD without saying why. Move the store before you download, not after. OLLAMA_MODELS is a user environment variable that Ollama reads when it starts. Set it (Settings → search "environment variables" → Edit environment variables for your account, or setx below), then quit Ollama from the tray and relaunch it from the Start menu. A terminal that was already open does not see the new value either. Logs live elsewhere, in %LOCALAPPDATA%\Ollama, and server.log there is the first place to look when something goes wrong.
# Default store: %HOMEPATH%\.ollama\models
setx OLLAMA_MODELS "D:\ollama\models"
# Quit Ollama from the tray, then start it again from the Start menu.
# Logs (server.log names the GPU it found):
explorer $env:LOCALAPPDATA\OllamaCheck it actually ran on the GPU
When a model does not fit, Ollama quietly puts part of it on the CPU, and the only symptom is slowness. ollama ps shows what it decided for each loaded model. The PROCESSOR column is the answer: anything other than 100% GPU on a model that should fit is worth chasing. On NVIDIA, nvidia-smi shows the memory in use. On AMD, Task Manager’s GPU page shows dedicated GPU memory climbing as the model loads, and server.log names the device and the backend (ROCm or Vulkan) Ollama chose.
ollama ps
# NAME ID SIZE PROCESSOR UNTIL
# llama3.1:8b <model id> 6.1 GB 100% GPU 4 minutes from now
# NVIDIA only:
nvidia-smi --query-gpu=memory.used,utilization.gpu --format=csv -l 1Using it from other programs
Editor plugins and desktop clients talk to the HTTP API Ollama serves on port 11434, which runs whether or not you ever open a terminal. In Windows PowerShell 5.1, curl is an alias for Invoke-WebRequest, so a curl -d example copied from a Linux guide fails with a parameter error. Use Invoke-RestMethod, or type curl.exe to get the real curl. Ollama listens on 127.0.0.1 only by default. OLLAMA_HOST changes that, and exposing it to your network lets anyone on that network use your GPU, so make that choice deliberately rather than by copying a command.
Invoke-RestMethod -Method Post -Uri http://localhost:11434/api/generate `
-Body '{"model":"llama3.1:8b","prompt":"hello","stream":false}'
# Keep a model loaded for an hour instead of the 5-minute default:
Invoke-RestMethod -Method Post -Uri http://localhost:11434/api/generate `
-Body '{"model":"llama3.1:8b","keep_alive":"1h"}'When it does not work
ollama ps shows a CPU share on a model that should fit. Either something else is holding VRAM (a browser with hardware acceleration is the usual culprit), or on NVIDIA, System Memory Fallback in the control panel is spilling VRAM into system RAM, which turns "does not fit" into "inexplicably slow". An AMD card runs on the CPU: on RX 6000-class cards, current Windows drivers may not expose ROCm 7, and Ollama’s documented fallback is Vulkan, which is on by default. If a laptop or desktop with integrated graphics picks the iGPU, set GGML_VK_VISIBLE_DEVICES to the discrete card’s index. The C: drive fills up: that is the model store, and OLLAMA_MODELS has to be set before the download. A plugin cannot reach the API: the server binds to localhost by default, which is correct. Point the plugin at 127.0.0.1:11434, or change the binding deliberately. A model that ran yesterday will not load today: another model is usually still resident. Models unload after five idle minutes, or immediately with ollama stop.
Common questions
Do I need WSL to run Ollama on Windows?
No. The native Windows app uses NVIDIA and AMD Radeon cards directly. It ships its own GPU runtime, so there is no CUDA Toolkit, no ROCm SDK and no build step. WSL is still a reasonable choice if you want a Linux environment for other reasons, but it adds a layer rather than removing one.
Does Ollama on Windows use an AMD Radeon GPU?
Yes. Ollama’s docs list the RX 7600, 7600 XT, 7700 XT, 7800 XT and 7900 series (XTX, XT, GRE) for ROCm on Windows, which needs a ROCm 7-capable AMD driver. Other Radeons, the RX 6000 series included, run through Vulkan, which is enabled by default and ships inside AMD’s regular driver. Run ollama ps to confirm: the PROCESSOR column reads 100% GPU when the model is on the card.
How do I move Ollama models off the C: drive?
Set OLLAMA_MODELS as a user environment variable pointing at the folder you want. Then quit Ollama from the system tray and start it again from the Start menu, because it reads the variable only at start. Do this before downloading, not after. The default store is under your user profile, and a few models will fill a small system drive without saying why.
How do I know whether Ollama is using my GPU?
Run ollama ps while a model is loaded. The PROCESSOR column reads 100% GPU when the whole model is on the card, and shows a CPU share otherwise. If it shows a CPU share on a model that should fit, check what else is holding VRAM. On NVIDIA, also check whether System Memory Fallback is enabled in the control panel. On AMD, check that server.log in %LOCALAPPDATA%\Ollama names your discrete card rather than the integrated GPU.
What this guide uses
- Format
- GGUF
Next steps
Models covered in this guide
Did this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
WSL2 + Ollama GPU Passthrough on Windows
Run Ollama with NVIDIA GPU acceleration inside WSL2 — the most reliable Windows path for local LLMs.
llama.cpp on Windows with CUDA
Build llama.cpp with NVIDIA GPU support on Windows 11 — the path of least resistance for PC gamers.
8GB GPU Starter Guide: 3060 / 4060 / 3070
The most common local LLM hardware tier — which models, quants, and context lengths actually fit in 8GB VRAM.