Who this is for
An 8GB NVIDIA card (RTX 3060 8G, 3070, 4060, 4060 Ti 8G) on Windows or Linux, with a working driver and roughly 16GB of system RAM. Every figure below is for a single stream — batch 1 — at the context length stated. The commands use Ollama; llama.cpp works the same way and is noted where it differs.
What fits, and what the numbers mean
These are estimates from this site’s VRAM calculator — model weights at the level’s measured bits-per-weight, plus the KV cache for the stated context, plus a 10% activation buffer. They are not measured file sizes and they exclude whatever your desktop is already using, which on Windows is commonly 0.5–1.5GB. That is why the green band stops at 88% of the card rather than 100%. At 4K context an 8GB card runs 7–8B at Q4_K_M with real room to spare; 14B does not fit at Q4_K_M at any context, because the weights alone are over the card.
Model Level Context Estimate On 8GB
----------------------------------------------------------------------
Phi-4 Mini 3.8B Q4_K_M 4K 3.0 GB fits, large margin
Qwen3 4B Q6_K 4K 4.0 GB fits
Gemma 4 E4B (image, audio) Q4_K_M 4K 4.9 GB fits
Qwen2.5 7B Q4_K_M 4K 5.1 GB fits
Llama 3.1 8B Q4_K_M 4K 5.6 GB fits
Qwen3 8B Q4_K_M 4K 5.8 GB fits
Qwen3-VL 8B (image) Q4_K_M 4K 6.2 GB fits
Llama 3.1 8B Q4_K_M 16K 7.3 GB tight — 91% of the card
Qwen3 14B Q4_K_M 2K 9.7 GB over the card — see below
Qwen2.5 32B any 4K 21.7 GB not on 8GBWhat "over the card" actually means
A 14B at Q4_K_M needs about 9.7GB against 8GB of VRAM. That is not "tight" — it will not load fully on the GPU. Two real options: use the AWQ/GPTQ INT4 build of the same model, which the index puts at about 8.1GB and which is still above the comfortable band, — and note vLLM, which serves those, runs on Linux only (its docs say it does not support Windows natively), so on Windows that route goes through WSL2 — or keep GGUF and offload part of the model to system RAM. Partial offload works and is what Ollama does by default when a model does not fit, but the layers left on the CPU are read over PCIe every token, so throughput falls sharply — how far depends on your PCIe link and RAM speed, and this guide has not measured it on your hardware. Budget for the offloaded weights in system RAM on top of what the OS needs: roughly 2GB for the 14B case above, and keep 16GB total system RAM as the floor.
# llama.cpp now picks the split itself by default: with -ngl left unset it
# fits as many layers as it can while keeping 1 GiB of the card free
llama-server -m qwen3-14b-Q4_K_M.gguf -c 2048 --host 127.0.0.1
# Either way, the log line says what actually landed on the card:
# load_tensors: offloaded N/M layers to GPU
# To choose yourself, pass -ngl (e.g. -ngl 28): fewer layers = less VRAM,
# slower generation.Quick start with Ollama
Ollama does not pick a quantization to match your VRAM. Pulling a bare tag gives you that tag's default build — Q4_K_M for most models — on a 24GB card and an 8GB one alike; ask for a different level by tagging it explicitly. What Ollama does decide automatically is how many layers to put on the GPU, and it will silently fall back to partial CPU offload rather than refuse to load.
# Default tag — Q4_K_M, regardless of your card
ollama pull qwen2.5:7b
# Ask for a level explicitly
ollama pull qwen2.5:7b-instruct-q5_K_M
ollama run qwen2.5:7bCheck it actually ran on the GPU
A model that quietly fell back to CPU offload still answers — it is just slow, and that is the single most common "why is local inference so bad" report. Two checks: `ollama ps` prints a PROCESSOR column that reads 100% GPU when the whole model is on the card, or splits (for example 70%/30% CPU/GPU) when it is not; and `nvidia-smi` should show a process holding roughly the estimate above. If PROCESSOR shows any CPU share, drop to a smaller model or a lower level rather than living with it.
ollama ps
# NAME ID SIZE PROCESSOR UNTIL
# qwen2.5:7b <id> ... 100% GPU 4 minutes from now
nvidia-smi --query-compute-apps=pid,used_memory --format=csvCommon problems
Out of memory on load: the context is usually the cause, not the weights — the KV cache grows linearly with it, so 16K costs an 8B model about 1.7GB over its 4K figure, and puts it at 91% of an 8GB card. Lower the context first. Slow generation with a model that should fit: check `ollama ps` as above; something else on the card (a browser, a game, a second model still resident) is the usual culprit, and `ollama stop <model>` frees the previous one. Windows specifically: the desktop compositor holds VRAM that never appears in your model's own accounting, so treat the 88% band as the real ceiling.
Next steps
Put your own card into the VRAM calculator to see the full list of what fits at the context you actually use — the numbers above are one row of that table. If you are on Windows, the WSL2 + Ollama GPU guide covers driver passthrough, which is where most 8GB setups actually get stuck.
Common questions
Can an 8GB GPU run a 14B model?
Not entirely on the card. Qwen3 14B at Q4_K_M is about 9.7GB even at 2K context, so the weights alone are over 8GB. It still runs with part of the model offloaded to system RAM — Ollama and llama.cpp both do this by default — but the layers on the CPU slow every token down. The AWQ INT4 build is about 8.1GB, which still leaves no room for the desktop. For a model that sits fully on the card, stay at 7–8B.
Which models run on an 8GB card with image input?
Two in this index fit comfortably at 4K context: Gemma 4 E4B at about 4.9GB, which also takes audio, and Qwen3-VL 8B at about 6.2GB. Both are Q4_K_M estimates from this site's calculator. Image input adds a vision encoder, which llama.cpp loads from a separate file and which these figures do not include, so leave some margin.
How much context can an 8GB card handle?
For an 8B model at Q4_K_M, about 16K before it leaves the comfortable band: Llama 3.1 8B is 5.6GB at 4K and 7.3GB at 16K, which is 91% of the card — tight, with nothing left for the desktop. The KV cache is the only part that grows with context. A lower quant frees room for it by shrinking the weights; a smaller model does that and also has a smaller cache per token, so it buys more window for the same memory.
What this guide uses
Next steps
See everything that fits 🟢 RTX 4060Reverse lookup — 8GB at 4096 context, ranked by qualityModels covered in this guide
- Llama 3.1 8B Instruct8B · per-quant VRAM & speed
- Qwen3 8B Instruct8B · per-quant VRAM & speed
- Qwen3 4B Instruct4B · per-quant VRAM & speed
- Phi-4 Mini Instruct3.8B · per-quant VRAM & speed
- Qwen3-VL 8B Instruct8B · per-quant VRAM & speed
- Gemma 4 E4B ITE4B · per-quant VRAM & speed
- Qwen3.5 9B9B · per-quant VRAM & speed
- Granite 4.2 8B8B · per-quant VRAM & speed
Did this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
What to Run on a 12GB GPU (RTX 3060 12G, 4070, 5070)
Which models fit a 12GB NVIDIA card at 8K–32K context, where 14B stops fitting, how to run them with Ollama or llama.cpp, and how to prove they are on the GPU.
Run GPT-OSS 20B (and 120B) locally without re-quantizing
GPT-OSS ships natively in MXFP4, so the usual "download the Q4_K_M" habit makes it bigger and worse. Sizing, the right flags, and how MoE expert-offload puts the 120B on a 24GB card.
Intel Arc: llama.cpp (SYCL) or Ollama (Vulkan)
Run GGUF models on an Arc B580, B570, A770 or A750 — the SYCL build, the Ollama route, how to prove the GPU is doing the work, and what fits in 8–16 GB.