The one thing to get right: MXFP4 is the original
Almost every other model on this site is published in BF16 and quantized by the community afterwards, so "find the Q4_K_M" is the right reflex. GPT-OSS breaks that reflex. OpenAI post-trained it with the MoE weights already in MXFP4 (~4.25 bits), and those MoE weights are over 90% of the parameters. The MXFP4 checkpoint is not a lossy copy of something better — it is the model. Converting it up to Q8_0, or sideways to Q4_K_M, gives you a file that is larger and no more accurate, because the precision it is padding back was never there.
gpt-oss-20b MXFP4 (native) 11.3 GiB file ← use this
gpt-oss-120b MXFP4 (native) 59.0 GiB file ← use this
(file sizes from llama.cpp's own gpt-oss guide)
Rule: for GPT-OSS, "bigger quant" buys you nothing.
Spend the VRAM on context length instead.Sizing it for your card
The 20B fits a 16GB card with room for a useful context window. Note that GPT-OSS uses a head dimension of 64 rather than the usual 128, which halves its KV cache compared to a same-layer-count model — and half its layers use a 128-token sliding window, so only the other half grow a cache with context. Long context is unusually cheap here. Use the VRAM calculator with the MXFP4 level selected. MXFP4 is about 4.25 bits on the expert weights only — attention and embeddings are stored wider — so the whole 20B file works out to about 4.6 bits per weight, which is the rate the calculator uses for it. llama.cpp's own gpt-oss guide lists a higher total than this calculator does (14.9 GB at 8K context for the 20B, against about 12.9 here), because its figure includes the runtime's compute buffers in full; treat the calculator as the lower bound and leave room.
gpt-oss-20b @ MXFP4, batch=1 (this site's calculator)
weights ~11.5 GB
KV cache @ 8K ctx ~0.2 GB
KV cache @ 32K ctx ~0.8 GB
KV cache @ 131K ctx ~3.0 GB
total @ 8K / 32K / 131K 12.9 / 13.5 / 16.0 GB
llama.cpp's own guide, same model: 14.9 / 15.5 / 17.9 GB
16GB card → comfortable to ~32K ctx; the full 131K window does not fit
24GB card → full 131K ctx with headroomFastest path: Ollama
Ollama pulls the MXFP4 build by default, so there is no quant tag to choose and no way to accidentally get a re-quantized one. This is the right starting point unless you need custom flags.
ollama pull gpt-oss:20b
ollama run gpt-oss:20b
# 120B — needs ~61GB of combined VRAM+RAM
ollama pull gpt-oss:120b
# OpenAI-compatible endpoint stays on :11434
curl http://localhost:11434/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"model":"gpt-oss:20b","messages":[{"role":"user","content":"hi"}]}'llama.cpp: pass --jinja or the output gets strange
GPT-OSS was trained on OpenAI's "harmony" response format, which separates the reasoning channel from the final answer. That structure lives in the model's chat template, so llama.cpp needs --jinja to apply it. Skip the flag and you get raw channel markers bleeding into replies, or a model that never stops talking — a failure that reads like a broken quant but is purely a template problem.
# 20B, all layers on a 16GB+ GPU
llama-server \
-hf ggml-org/gpt-oss-20b-GGUF \
--jinja \
-ngl 99 \
--ctx-size 32768 \
--host 127.0.0.1 --port 8080
# --jinja applies the harmony chat template (do not omit)
# -ngl 99 offload every layer to the GPU
# --ctx-size 32K is ~13.5 GB by this site's estimate, 15.5 GB by
# llama.cpp's own table — fine on 16GB with nothing else
# on the card. llama.cpp's guide uses --ctx-size 0 (the
# full 131K) and -ub 2048 -b 2048, which needs ~18 GB.Running the 120B on a 24GB consumer card
This is where the MoE architecture pays off. Only 5.1B parameters are active per token, so the expert weights are read sparsely — which makes them the ideal thing to leave in system RAM. llama.cpp's own guidance is to offload the whole model and use --n-cpu-moe to keep as many layers' experts on the CPU as necessary; whatever stays on the CPU lives in system RAM, so budget most of the 59 GiB file there. The same flag brings the 20B to smaller cards: llama.cpp's guide runs it on an 8GB RTX 2060 with --n-cpu-moe 16. This site has not measured either setup, so it gives no speed for them.
# 120B on 24GB: keep the experts of N layers on the CPU
llama-server \
-hf ggml-org/gpt-oss-120b-GGUF \
--jinja \
-ngl 99 \
--n-cpu-moe 28 \
--ctx-size 16384 \
--host 127.0.0.1 --port 8080
# 20B on an 8GB card (llama.cpp's own RTX 2060 example)
llama-server -hf ggml-org/gpt-oss-20b-GGUF --jinja \
--ctx-size 32768 --n-cpu-moe 16 --host 127.0.0.1
# Lower N = more experts on the GPU = faster. Lower it until
# loading fails for lack of VRAM, then step back up.Reasoning effort is a dial, not a fixed cost
GPT-OSS exposes low / medium / high reasoning effort. High spends far more tokens thinking before answering, which on local hardware is the difference between a snappy assistant and one that pauses for a minute. Set it low for chat and autocomplete, high only for problems that actually need the chain of thought.
llama.cpp (from its gpt-oss guide):
llama-server ... --chat-template-kwargs '{"reasoning_effort": "low"}'
Any client that can only set the system message:
System: Reasoning: low
Rough local cost on a 16GB card (20B):
low fast, chat-grade latency
medium noticeably more thinking tokens
high can multiply time-to-first-answer several times over
Start at low. Raise it per-task, not globally.Common failure modes
Most GPT-OSS problems reported locally are one of five things, and none of them are the quantization. Check these before hunting for a different build.
Channel markers in the output, or it never stops
→ missing --jinja (harmony template not applied)
"unknown model architecture" on load
→ llama.cpp / Ollama predates gpt-oss support; update
Slower than expected on the 120B
→ --n-cpu-moe too high; lower it until GPU VRAM is nearly full
Repetitive or oddly degraded answers
→ check sampling: OpenAI recommends temperature 1.0 and
top_p 1.0, and llama.cpp's guide says not to use a
repetition penalty
File much bigger than ~11.3 GiB (20B)
→ you downloaded an upcast build; get the MXFP4 oneCommon questions
Can GPT-OSS 20B run on an 8GB or 12GB GPU?
Not entirely on the card — this site sizes it at about 12.8GB at 4K context, and llama.cpp's own table is higher. It runs with some of its experts kept in system RAM: llama.cpp's gpt-oss guide shows it on an 8GB RTX 2060 with --n-cpu-moe 16, and reports about 67 tokens per second at the start of generation on a 12GB RTX 3060 with offloading. Because only about 3.6B parameters are active per token, it slows down far less from offloading than a dense 20B would.
How fast is GPT-OSS 20B on an RTX 4090?
llama.cpp's own benchmark in its gpt-oss guide reports about 222 tokens per second of generation and about 8,000 tokens per second of prompt processing on an RTX 4090. Those are the llama.cpp developers' measurements, not this site's; this site's own figure for the same card is listed on the model page with its source marked.
What sampling settings should I use for GPT-OSS?
OpenAI recommends temperature 1.0 and top_p 1.0, and llama.cpp's gpt-oss guide adds: do not use a repetition penalty. Many front-ends apply their own defaults, often a lower temperature and a repetition penalty, so set these explicitly if answers come out repetitive or strangely clipped.
What this guide uses
- Hardware
- RTX 409024GBRTX 3090
- Format
- GGUF
- Picks for it
- Best local LLM for 24GB
Next steps
See everything that fits 🟢 RTX 4090Reverse lookup — 24GB at 32768 context, ranked by qualityModels covered in this guide
Did this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
Intel Arc: llama.cpp (SYCL) or Ollama (Vulkan)
Run GGUF models on an Arc B580, B570, A770 or A750 — the SYCL build, the Ollama route, how to prove the GPU is doing the work, and what fits in 8–16 GB.
RTX 5090: What 32GB Actually Buys for Local LLMs
What fits on an RTX 5090 that does not fit on a 24GB card: not many more models, but much longer context and higher speed ceilings. Sizes from the calculator, setup for Blackwell.
DGX Spark 64GB vs 128GB: What Each One Runs
Which models fit the 64GB and 128GB DGX Spark, from the calculator: GPT-OSS 120B needs the 128GB model, 70B dense fits both, and 273 GB/s sets the speed. Setup for Ollama and llama.cpp.