What you need first
A fine-tuned or merged model in Hugging Face safetensors format, a built copy of llama.cpp, and enough free disk for three copies of the model at once: the original weights, an intermediate high-precision GGUF, and the final quantized file. For a 7B model in BF16 that is roughly 14 GB + 14 GB + 5 GB — budget 35–40 GB of scratch space, not just the size of the model you started with.
pip install -r requirements.txt # inside your llama.cpp checkout
df -h .Convert to GGUF
The conversion script is `convert_hf_to_gguf.py`. The older `convert.py` name that circulates in older tutorials no longer exists in the repository — if a guide you are reading elsewhere still uses it, that guide predates the rename. Convert to a high-precision intermediate first; converting straight to a low bit-depth from safetensors is not how the pipeline works. llama.cpp’s own example uses `--outtype bf16`, and `auto` (the default) picks the highest-fidelity 16-bit type for you. Forcing `f16` on a model released in BF16 can overflow values that BF16 holds fine. Newer architectures such as Gemma 4 need transformers 5, while the requirements file installs 4, so `pip install -U transformers` if the script does not recognise your model.
python convert_hf_to_gguf.py /path/to/my-model \
--outfile my-model-bf16.gguf \
--outtype bf16Quantize it
The quantization binary is `llama-quantize` — the bare `quantize` name from older tutorials was renamed along with the rest of the CLI tools. Q4_K_M is the level almost everyone should start with: this index’s own median published quality loss at that level is 2.9% against FP16 (across the 79 models that report one), and 106 of the 112 models here ship it — the exceptions are released only in other forms, such as a native MXFP4 or 2-bit build or ggml-org's Q4_0.
./build/bin/llama-quantize my-model-bf16.gguf my-model-Q4_K_M.gguf Q4_K_MDo it properly: an importance matrix
A plain quantization treats every weight the same. An importance matrix (imatrix), computed by running representative text through the F16 model first, tells the quantizer which weights matter more and protects them — the same idea behind why Q4_K_M already beats a naive 4-bit round-off, taken further. This step is what separates a quant you would publish from one that is merely usable.
# .gguf extension = current format; any other extension writes the legacy one
./build/bin/llama-imatrix \
-m my-model-bf16.gguf \
-f calibration-data.txt \
-o my-model.imatrix.gguf \
-ngl 99 # offload to the GPU if the model fits; much faster
./build/bin/llama-quantize \
--imatrix my-model.imatrix.gguf \
my-model-bf16.gguf my-model-Q4_K_M.gguf Q4_K_MCheck it actually worked
Load the quantized file and run a real prompt through `llama-cli`, which applies the model’s chat template, so it tests the template too. Current `llama-cli` is a chat program, so add `-st` to answer once and exit. A successful conversion that produces garbage output is a real failure mode, usually from a tokenizer or chat-template mismatch during conversion rather than from quantization itself. Compare the file size against expectation: a Q4_K_M file should land near params × 0.6 bytes per parameter, and a figure wildly off that suggests the conversion picked up the wrong precision.
# -st = single turn: answer once and exit. Without it llama-cli is a chat
# session and waits for your next message.
./build/bin/llama-cli -m my-model-Q4_K_M.gguf -p "Explain quantization in one sentence." -n 64 -st
ls -lh my-model-Q4_K_M.gguf # sanity-check the sizeWhen it does not work
The conversion script fails with an unknown architecture: your model uses a layer type `convert_hf_to_gguf.py` does not yet recognise — check the script’s supported-architecture list before assuming your model is fine and the tool is broken. Output is fluent but wrong (repeats itself, ignores the prompt): usually a chat-template mismatch carried over from the original model, not a quantization artifact — verify with the BF16 intermediate before blaming the quant step. Running out of disk mid-conversion: this is the three-copies problem from the first section: clean up the BF16 intermediate once the quantized file is verified, but not before. And if you would rather not run any of this yourself, the GGUF-my-repo Space on Hugging Face runs the same pipeline and syncs from llama.cpp’s main branch every six hours.
Common questions
What is convert_hf_to_gguf.py, and where did convert.py go?
`convert_hf_to_gguf.py` is the current script for turning Hugging Face safetensors weights into a GGUF file. `convert.py` was the older name and no longer exists in the repository — any tutorial still referencing it predates the rename.
What is llama-quantize, and is it the same as quantize?
It is the same tool under its current name. The binary was renamed from the bare `quantize` along with the rest of llama.cpp’s CLI tools (`llama-server`, `llama-cli`), so an older tutorial referencing `./quantize` is naming a file that no longer exists in a fresh build.
Do I need an importance matrix (imatrix) to quantize a model?
No — quantization works without one. An imatrix improves the result by telling the quantizer which weights matter most, computed by running representative text through the F16 model first. Skip it for a quick personal test; use it before publishing a quant anyone else will rely on.
What this guide uses
- Format
- GGUF
Did this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
llama.cpp on Windows with CUDA
Build llama.cpp with NVIDIA GPU support on Windows 11 — the path of least resistance for PC gamers.
Run GPT-OSS 20B (and 120B) locally without re-quantizing
GPT-OSS ships natively in MXFP4, so the usual "download the Q4_K_M" habit makes it bigger and worse. Sizing, the right flags, and how MoE expert-offload puts the 120B on a 24GB card.
Intel Arc: llama.cpp (SYCL) or Ollama (Vulkan)
Run GGUF models on an Arc B580, B570, A770 or A750 — the SYCL build, the Ollama route, how to prove the GPU is doing the work, and what fits in 8–16 GB.