Who this is for
A Windows machine with an NVIDIA GPU, where you want the Linux tooling (Ollama, llama.cpp, Python) without dual-booting. If you only want to chat with a model and never touch a terminal, the native Windows Ollama app is simpler — this guide is for the case where you also want the Linux side.
Prerequisites
Windows 11 (or Windows 10 21H2+), an NVIDIA GPU, and a current NVIDIA driver installed on Windows. The one rule that matters: install no NVIDIA driver inside WSL. NVIDIA's CUDA on WSL guide is explicit — the Windows driver is stubbed into WSL as libcuda.so under /usr/lib/wsl/lib, and a Linux driver installed on top overwrites it. The easy way to do that by accident is apt install cuda (or cuda-drivers): those meta-packages pull in the Linux driver. If you need the CUDA toolkit inside WSL for building things, NVIDIA says to install the cuda-toolkit-12-x package only.
# PowerShell (Administrator)
wsl --install
wsl --update
wsl --status # want: default version 2
# The check that matters — GPU visible from inside the WSL VM
wsl nvidia-smiGive the WSL VM enough RAM
WSL2 runs in a lightweight VM whose memory limit defaults to 50% of the host's RAM (Microsoft's documented default for the memory setting). That ceiling is invisible until a model needs CPU offload and the VM runs out well before Windows does. Set it explicitly in %UserProfile%\.wslconfig, then wsl --shutdown to apply. Leave several GB for Windows itself.
# %UserProfile%\.wslconfig (example for a 32GB machine)
[wsl2]
memory=20GB
swap=8GB
# then, in PowerShell:
# wsl --shutdownInstall Ollama inside WSL
Install the Linux build inside Ubuntu, not the Windows app — running both leaves two servers competing for port 11434 and for the GPU. On WSL2 the install script installs no GPU driver at all; it only checks for the passthrough and prints "Nvidia GPU detected." when nvidia-smi works, so if that line is missing, fix the passthrough before going further. The script also starts Ollama as a systemd service. WSL's documented default is systemd off, and without it the script warns "systemd is not running" and nothing is serving — enable it in /etc/wsl.conf, or run ollama serve in a second terminal. Keep models on the WSL filesystem (~/.ollama); under /mnt/c every read crosses into the Windows filesystem and loads are dramatically slower.
curl -fsSL https://ollama.com/install.sh | sh
# look for: "Nvidia GPU detected."
# If it warned "systemd is not running":
# printf '[boot]\nsystemd=true\n' | sudo tee -a /etc/wsl.conf
# then in PowerShell: wsl --shutdown
# Most library tags default to Q4_K_M
ollama pull qwen3:8b
ollama run qwen3:8bVerify it is actually on the GPU
Ollama falls back to CPU rather than failing, so a model that answers slowly is the symptom of a broken passthrough, not of a slow card. Two checks: ollama ps shows a PROCESSOR column that reads 100% GPU when the whole model is on the card, and nvidia-smi inside WSL shows the ollama process holding roughly the model's size. For Qwen3 8B at Q4_K_M and Ollama's default 4,096-token context, this site's calculator puts that at about 5.8 GB. If PROCESSOR shows a CPU share on a model that should fit, the passthrough is the problem, not the model.
ollama ps
# NAME ID SIZE PROCESSOR UNTIL
# qwen3:8b <id> ... 100% GPU 4 minutes from now
nvidia-smi --query-compute-apps=pid,process_name,used_memory --format=csvReaching the API from Windows
WSL2 forwards localhost by default, so http://localhost:11434 from a Windows browser or PowerShell reaches the server inside WSL with no extra configuration. In WSL's default NAT networking, nothing on your LAN can reach that server — Microsoft's docs say LAN access needs a netsh portproxy rule. That changes in mirrored networking mode (networkingMode=mirrored, Windows 11 22H2+), which connects WSL directly to the LAN: there, an Ollama bound to 0.0.0.0 is reachable by every machine on the network, with no authentication in front of it. Leave OLLAMA_HOST at its default unless you mean that.
# From Windows PowerShell (curl.exe — plain curl is an alias there)
curl.exe http://localhost:11434/api/tagsCommon problems
wsl nvidia-smi fails: update the Windows driver, then wsl --update and wsl --shutdown; do not install a driver inside WSL. It worked and then stopped after an apt upgrade: check whether a cuda or cuda-drivers package got installed — that is the Linux driver overwriting the stub. "could not connect to ollama app": the server is not running, almost always because systemd is off (see the install step). nvidia-smi works but Ollama uses CPU: usually a second Ollama (the Windows app) already holding the port, or a model too large for the card — check ollama ps. Disk fills up: the WSL virtual disk grows to hold pulled models and does not shrink by itself; ollama rm removes a model, and getting the space back needs the vhdx compacted (sparseVhd=true in .wslconfig only applies to newly created disks). First load painfully slow: the model is probably under /mnt/c.
Next steps
Put your card into the VRAM calculator to see what else fits at the context you actually use before pulling a larger model — on Windows, subtract the 0.5–1.5GB the desktop already holds.
Common questions
Do I need to install the CUDA toolkit inside WSL to run Ollama?
No. Ollama ships its own CUDA libraries and uses the driver that Windows projects into WSL. The toolkit is only for compiling CUDA code yourself — and if you do install it, install cuda-toolkit-12-x alone, never the cuda or cuda-drivers meta-packages, which bring a Linux driver that breaks passthrough.
Is WSL2 slower than running Ollama natively on Windows?
This site has not measured the difference, so it will not quote a number. What it can say: once the model is on the GPU, generation runs on the same card through the same driver either way. The real WSL2 costs come from the edges — a 50% RAM ceiling that bites when a model spills to the CPU, and slow loads when models live under /mnt/c — both covered above.
Can other machines on my network use this Ollama server?
Not in WSL's default NAT mode — that needs a netsh portproxy rule on Windows. In mirrored networking mode they can, as soon as Ollama listens on 0.0.0.0, and the API has no authentication. If you want LAN access, put an authenticating proxy in front of it rather than exposing port 11434 directly.
What this guide uses
- Format
- GGUF
- Picks for it
- Best local LLM for 16GB
Next steps
See everything that fits 🟢 RTX 4060 Ti 16GReverse lookup — 16GB at 4096 context, ranked by qualityModels covered in this guide
Did this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
Ollama on Windows (Native, No WSL)
Install the Windows Ollama app for the simplest path — NVIDIA and AMD Radeon GPUs both work, with no WSL and no toolkit to install.
llama.cpp on Windows with CUDA
Build llama.cpp with NVIDIA GPU support on Windows 11 — the path of least resistance for PC gamers.
Run GPT-OSS 20B (and 120B) locally without re-quantizing
GPT-OSS ships natively in MXFP4, so the usual "download the Q4_K_M" habit makes it bigger and worse. Sizing, the right flags, and how MoE expert-offload puts the 120B on a 24GB card.