AWQ explained
What AWQ is, which runtimes read it, what it costs in VRAM, and all 53 of the 112 models in this index that ship in it.
What AWQ is
Activation-Aware Weight Quantization: 4-bit weights quantized with the activation statistics in hand, served mainly by vLLM on NVIDIA and on the AMD cards vLLM's ROCm build supports. On every model in this index that ships both, the AWQ INT4 file is smaller than GGUF Q4_K_M (median 18% smaller) and its published quality loss is higher (median 1.1 points more). AutoAWQ, the tool most AWQ builds were made with, is deprecated in favour of vLLM's llm-compressor; existing AWQ checkpoints still load. 53 of the 112 models in this index ship in it, across 1 quant level — INT4.
What reads it
vLLM · AutoAWQ · TGI. Hardware: NVIDIA GPU (CUDA 11.8+) or AMD via vLLM ROCm. Best suited to: High-throughput API server. Where it is strong: Smaller than GGUF Q4_K_M on every paired model here; Served by vLLM with continuous batching; Measured 218 tok/s (Llama 3.1 8B, RTX 4090, vLLM). Where it is not: Higher published loss than Q4_K_M on every paired model here; GPU only — no CPU or Apple path; AutoAWQ deprecated (llm-compressor replaces it).
What it costs
Llama 3.1 8B Instruct at AWQ INT4 is 4 bits per weight, which works out to 3.8 GB of weights plus 0.5 GB of KV cache at 4K context and a 0.4 GB activation buffer — 4.8 GB in total. The smallest card in this index that clears that comfortably is an RTX 5060 Ti 8G. Across the 53 models here with a published perplexity figure at this level, the median loss against FP16 is 4.0%.
AWQ levels in this index
| Level | bits/weight | Models | Median loss |
|---|---|---|---|
| INT4 | 4 | 53 | 4.0%median of 53 |
Bits per weight is the model row's own figure, so a range means different models ship the same level at slightly different sizes. The median loss is computed only over the models with a published perplexity figure at that level; the sample size is printed beside it.
Models in this index that ship AWQ
53 of 112, largest first, each with its smallest AWQ level and that level's published perplexity loss where one exists.
- Llama 3.1 405B Instruct405B · INT4 · 3.5%
- Qwen3 235B-A22B Instruct235B-A22B · INT4 · 3.2%
- Llama 4 Scout 17B (16E)109B MoE · INT4 · 3.2%
- Qwen2.5 72B Instruct72B · INT4 · 3.5%
- Llama 3.1 70B Instruct70B · INT4 · 3.9%
- Llama 3.3 70B Instruct70B · INT4 · 3.7%
- DeepSeek-R1-Distill-Llama-70B70B · INT4 · 3.5%
- Jamba 1.5 Mini52B-A12B · INT4 · 4.8%
- Mixtral 8x7B Instruct47B MoE · INT4 · 3.8%
- Seed-OSS 36B Instruct36B · INT4 · 3.8%
- Yi 1.5 34B Chat34B · INT4 · 4.0%
- Qwen3 32B Instruct32B · INT4 · 3.6%
- Qwen2.5-Coder 32B Instruct32B · INT4 · 3.5%
- Qwen3 30B-A3B Instruct30B-A3B · INT4 · 3.4%
- Qwen3-Coder 30B-A3B Instruct30B-A3B · INT4 · 3.2%
- Qwen3-VL 30B-A3B Instruct30B-A3B · INT4 · 3.7%
- Gemma 3 27B IT27B · INT4 · 3.8%
- Gemma 2 27B Instruct27B · INT4 · 4.0%
- Mistral Small 24B Instruct24B · INT4 · 3.8%
- Devstral Small 1.1 24B24B · INT4 · 4.0%
- Magistral Small 1.2 24B24B · INT4 · 4.0%
- Codestral 22B22B · INT4 · 4.2%
- DeepSeek-Coder-V2-Lite Instruct16B-A2.4B · INT4 · 4.1%
- DeepSeek-V2-Lite Chat16B-A2.4B · INT4 · 4.0%
- Qwen3 14B Instruct14B · INT4 · 3.5%
- Qwen2.5 14B Instruct14B · INT4 · 3.8%
- DeepSeek-R1-Distill-Qwen-14B14B · INT4 · 3.6%
- Phi-4 14B14B · INT4 · 3.6%
- Phi-3 Medium 14B Instruct14B · INT4 · 4.2%
- Mistral Nemo 12B Instruct12B · INT4 · 4.4%
- Gemma 3 12B IT12B · INT4 · 3.9%
- Stable LM 2 12B Chat12B · INT4 · 4.5%
- Solar 10.7B Instruct11B · INT4 · 4.2%
- Gemma 2 9B Instruct9B · INT4 · 4.6%
- GLM-4-9B-Chat9B · INT4 · 4.2%
- Qwen3-VL 8B Instruct8B · INT4 · 4.2%
- Qwen2-VL 7B Instruct7B · INT4 · 5.0%
- Qwen3 8B Instruct8B · INT4 · 3.8%
- Llama 3.1 8B Instruct8B · INT4 · 4.5%
- Aya 23 8B8B · INT4 · 4.6%
- DeepSeek-R1-Distill-Llama-8B8B · INT4 · 3.5%
- InternLM2 7B Chat7B · INT4 · 4.3%
- Qwen2.5 7B Instruct7B · INT4 · 4.2%
- Qwen2.5-Coder 7B Instruct7B · INT4 · 4.0%
- WizardLM-2 7B7B · INT4 · 4.3%
- Mistral 7B Instruct v0.37B · INT4 · 4.5%
- Gemma 3 4B IT4B · INT4 · 4.2%
- Qwen3 4B Instruct4B · INT4 · 3.8%
- Phi-4 Mini Instruct3.8B · INT4 · 4.8%
- Phi-3.5 Mini Instruct3.8B · INT4 · 5.1%
- Llama 3.2 3B Instruct3B · INT4 · 4.8%
- Gemma 2 2B Instruct2B · INT4 · 5.0%
- Qwen3 1.7B Instruct1.7B · INT4 · 4.0%
Common questions
What is AWQ?
Activation-Aware Weight Quantization: 4-bit weights quantized with the activation statistics in hand, served mainly by vLLM on NVIDIA and on the AMD cards vLLM's ROCm build supports. On every model in this index that ships both, the AWQ INT4 file is smaller than GGUF Q4_K_M (median 18% smaller) and its published quality loss is higher (median 1.1 points more). AutoAWQ, the tool most AWQ builds were made with, is deprecated in favour of vLLM's llm-compressor; existing AWQ checkpoints still load. It is read by vLLM · AutoAWQ · TGI, needs NVIDIA GPU (CUDA 11.8+) or AMD via vLLM ROCm, and 53 of the 112 models in this index ship in it.
How much VRAM does Llama 3.1 8B Instruct need in AWQ?
About 4.8 GB at AWQ INT4 with a 4K context window — 3.8 GB of that is the weights themselves, so an 8 GB card clears it comfortably. Longer context adds KV cache on top; the calculator will size any combination.
AWQ or something else?
Depends on what reads it — this one belongs where the job is high-throughput API server. This site compares it directly against GGUF, EXL2, using only the models that ship both formats — the only place the comparison is measurable rather than editorial.