GPTQ explained
What GPTQ is, which runtimes read it, what it costs in VRAM, and all 4 of the 112 models in this index that ship in it.
What GPTQ is
GPT Quantization, one of the first mainstream post-training methods, with wide framework compatibility. AutoGPTQ, its original tooling, is unmaintained, and its README points to GPTQModel. No model in this index ships both a GPTQ and an AWQ build, so this site has no paired figure to say which loses less quality. 4 of the 112 models in this index ship in it, across 1 quant level — INT4. At 4 of 112 it is the thinnest coverage of any format here; treat it as a format you may meet rather than one to target.
What reads it
GPTQModel · vLLM · TGI. Hardware: NVIDIA GPU (CUDA) or AMD via vLLM ROCm. Best suited to: Legacy server deployment. Where it is strong: Wide compatibility; Mature ecosystem; Works with HF transformers. Where it is not: Slow quantization process; AutoGPTQ unmaintained (GPTQModel replaces it).
What it costs
Granite 3.1 8B Instruct at GPTQ INT4 is 4 bits per weight, which works out to 3.9 GB of weights plus 0.6 GB of KV cache at 4K context and a 0.5 GB activation buffer — 5.0 GB in total. The smallest card in this index that clears that comfortably is an RTX 5060 Ti 8G. Across the 4 models here with a published perplexity figure at this level, the median loss against FP16 is 4.8%.
GPTQ levels in this index
| Level | bits/weight | Models | Median loss |
|---|---|---|---|
| INT4 | 4 | 4 | 4.8%median of 4 |
Bits per weight is the model row's own figure, so a range means different models ship the same level at slightly different sizes. The median loss is computed only over the models with a published perplexity figure at that level; the sample size is printed beside it.
Models in this index that ship GPTQ
4 of 112, largest first, each with its smallest GPTQ level and that level's published perplexity loss where one exists.
- Command R 35B35B · INT4 · 4.5%
- StarCoder2 15B15B · INT4 · 4.8%
- Falcon 3 10B Instruct10B · INT4 · 4.8%
- Granite 3.1 8B Instruct8B · INT4 · 4.5%
Common questions
What is GPTQ?
GPT Quantization, one of the first mainstream post-training methods, with wide framework compatibility. AutoGPTQ, its original tooling, is unmaintained, and its README points to GPTQModel. No model in this index ships both a GPTQ and an AWQ build, so this site has no paired figure to say which loses less quality. It is read by GPTQModel · vLLM · TGI, needs NVIDIA GPU (CUDA) or AMD via vLLM ROCm, and 4 of the 112 models in this index ship in it.
How much VRAM does Granite 3.1 8B Instruct need in GPTQ?
About 5.0 GB at GPTQ INT4 with a 4K context window — 3.9 GB of that is the weights themselves, so an 8 GB card clears it comfortably. Longer context adds KV cache on top; the calculator will size any combination.
Is GPTQ still worth using in 2026?
It still works, and GPTQModel still reads it. But 4 of 112 models here ship it against 112 for GGUF, and that ratio is the honest answer: it is maintained rather than developed. This site compares it directly against GGUF, using only the models that ship both formats — the only place the comparison is measurable rather than editorial.