What you need first
An RTX 4090 or another 24GB card, and two decisions before you download anything: which format, and how much context you actually need. On this card the formats Qwen2.5-Coder 32B ships land in different verdict bands, and context length moves each of them across a line. If you are on Windows, the AWQ route means vLLM, which is Linux-only, so plan on WSL2.
nvidia-smi --query-gpu=name,memory.total,memory.used --format=csvThe format and the context window decide whether it fits
Qwen2.5-Coder 32B at GGUF Q4_K_M is about 21.7 GB at 4K context, 90% of a 24GB card, which this site calls tight. At 8K it is 95%, and at 16K it no longer fits. Qwen2.5-Coder 32B at AWQ INT4 is about 18.1 GB at 4K context, 75% and comfortable. It stays comfortable at 8K (80%), becomes tight at 16K (89%), and does not fit at 32K. The index also lists an EXL2 3.5bpw build at about 16.0 GB at 4K, the smallest of the three, but ExLlamaV2 is archived and offers no server, so it suits local chat only.
# GGUF via Ollama: simplest path; Ollama's default 4K window is the one that fits
ollama pull qwen2.5-coder:32b
# AWQ via vLLM (Linux / WSL2): more headroom, 8K stays comfortable
vllm serve Qwen/Qwen2.5-Coder-32B-Instruct-AWQ \
--max-model-len 8192 \
--host 127.0.0.1 --port 8000If you need long context: the MoE coder
For coding across whole files or repositories, the 32B dense model runs out of room on 24GB before the context gets useful. Qwen3-Coder 30B-A3B, also in this index, is the one that stretches. Qwen3-Coder 30B-A3B at GGUF Q4_K_M is about 19.7 GB at 4K context (82%, comfortable) and 22.6 GB at 32K (94%, tight). Qwen3-Coder 30B-A3B at AWQ INT4 is about 16.4 GB at 4K context and 19.2 GB at 32K, 80% of the card and still comfortable. It is a mixture-of-experts model that reads only about 3B active parameters per token, so its bandwidth ceiling is far higher than a dense 32B. This site has no measured run of it, and runs no coding benchmarks to say which model writes better code.
# Long-context coding on one 24GB card (Linux / WSL2).
# Substitute a real AWQ conversion — this site could not confirm
# which repo publishes one, so it does not guess a name.
vllm serve <an-AWQ-build-of-Qwen3-Coder-30B-A3B-Instruct> \
--max-model-len 32768 \
--host 127.0.0.1 --port 8000
# Or GGUF Q4_K_M at 32K (tight) through llama.cpp:
# llama-server -m <Qwen3-Coder-30B-A3B Q4_K_M .gguf> -ngl all -c 32768Check it actually ran on the GPU
At 90% of the card, GGUF Q4_K_M leaves little slack for a display or a second process. If anything else is using VRAM, this is the model that will not fit, and recent Ollama and llama.cpp then quietly move layers to the CPU instead of failing. Watch memory at load time, and check where the layers went.
nvidia-smi --query-gpu=memory.used,utilization.gpu --format=csv -l 1
# Ollama: PROCESSOR should read 100% GPU
ollama ps
# llama.cpp: confirm every layer offloaded
# load_tensors: offloaded 65/65 layers to GPUWhat the numbers should look like
This index has a measured run on the base Qwen2.5 32B, which has the same size and architecture: 44 tok/s at GGUF Q4_K_M on an RTX 4090 in llama.cpp. That is not a Coder-specific measurement. But generation is bound by memory bandwidth divided by weight size, and the two files are the same size, so the same ceiling applies: 1,008 GB/s over 18.7 GB of weights is about 54 tok/s. AWQ\u2019s smaller weights imply a higher ceiling, but this index has no measured AWQ run at this size.
When it does not work
GGUF Q4_K_M runs out of memory, or Ollama shows a CPU share, while anything else is running: expected at 90% of the card. Close other GPU consumers or switch to AWQ. It breaks only once you raise the context: GGUF has almost no room for KV cache growth (95% at 8K, out of room at 16K), and AWQ is tight by 16K and out of room at 32K. For long context, use the MoE coder above. vLLM fails at start-up on a desktop: `--gpu-memory-utilization` (default 0.9) is a share of the whole card, so a display or browser holding VRAM comes out of vLLM\u2019s budget. Close them or lower the value slightly. Prompt processing is slow on long files: expected on a 32B dense model in any format, because prefill is compute-bound.
Common questions
Does Qwen2.5-Coder 32B fit comfortably on an RTX 4090?
It depends on the format and the context. Qwen2.5-Coder 32B at GGUF Q4_K_M is about 21.7 GB at 4K context, 90% of the card, which this site calls tight, and it does not fit at 16K. Qwen2.5-Coder 32B at AWQ INT4 is about 18.1 GB at 4K context (75%). It stays comfortable to 8K, is tight at 16K, and does not fit at 32K.
Which coding model in this index handles 32K context on a 24GB card?
Qwen3-Coder 30B-A3B. At AWQ INT4 it is about 19.2 GB at 32K, 80% of the card and comfortable. At GGUF Q4_K_M it is 22.6 GB at 32K, which is tight but fits. Qwen2.5-Coder 32B does not fit 32K in AWQ or GGUF on this card. This site runs no coding benchmarks, so it does not rank the two models by quality, only by whether they fit.
How many tokens per second should I expect?
This index measured 44 tok/s for the same-size base Qwen2.5 32B at GGUF Q4_K_M on an RTX 4090 in llama.cpp. That is a reasonable proxy for the Coder build, since the file sizes and so the bandwidth ceiling (about 54 tok/s) are the same. There is no measured AWQ run at this size, and none for Qwen3-Coder 30B-A3B.
What this guide uses
- Hardware
- RTX 409024GBRTX 3090
- Picks for it
- Best local LLM for 24GB
Next steps
See everything that fits 🟢 RTX 4090Reverse lookup — 24GB at 4096 context, ranked by qualityModels covered in this guide
Did this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
DeepSeek-R1 Distill 14B: EXL2 vs GGUF
Head-to-head on RTX 4090 — when to pick turboderp EXL2 over bartowski GGUF.
ExLlamaV2 on RTX 4090: Full Setup Guide
Install ExLlamaV2, load an EXL2 quant, and serve an OpenAI-compatible API in under 10 minutes.
Run GPT-OSS 20B (and 120B) locally without re-quantizing
GPT-OSS ships natively in MXFP4, so the usual "download the Q4_K_M" habit makes it bigger and worse. Sizing, the right flags, and how MoE expert-offload puts the 120B on a 24GB card.