What you need first
A Linux VPS with at least 16 GB RAM — 32 GB gives more comfortable headroom for the OS and a longer context window. CPU-only inference on a small model is genuinely usable for personal use, which is the honest case for this whole guide: it is not a production API, it is a private assistant that costs less than a coffee subscription.
# Ubuntu 22.04/24.04 LTS. Check what the plan actually gives you:
free -h
lscpu | grep -E "^CPU\(s\)|^Thread|^Core|^Socket"
# vCPUs are often hyperthreads: "Thread(s) per core: 2" means
# half of nproc is the number of real cores.Build llama.cpp
A plain CPU build works out of the box. OpenBLAS is worth adding only if you plan to process long documents in large batches — llama.cpp’s own documentation states BLAS acceleration helps prompt processing above batch size 32 and does not change generation speed, so skip it for a pure chat API.
sudo apt update && sudo apt install -y build-essential cmake git
git clone https://github.com/ggml-org/llama.cpp
cd llama.cpp
cmake -B build
cmake --build build --config Release -j$(nproc)Download the model and start the server
Q4_K_M is the right default — 4.6 GB of weights, comfortable in 16 GB of RAM with room for the OS and a real context window. Leave `-t` out: llama.cpp already uses the physical core count, while `$(nproc)` counts vCPUs, which on many plans are hyperthreads and add contention rather than speed. Bind to loopback, and put an API key in front of it if this box has any public exposure at all.
pip install -U huggingface_hub
hf download bartowski/Meta-Llama-3.1-8B-Instruct-GGUF \
--include "Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf" --local-dir ./models
./build/bin/llama-server \
-m ./models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf \
--host 127.0.0.1 --port 8080 \
-c 8192 \
--api-key "your-secret-key"Check it actually worked
A successful start is not the same as a usably fast one on a VPS — confirm both: one request proves the server answers, and `llama-bench` measures generation (`tg128`) on the hardware you are actually renting. The binary is `llama-server`; older tutorials referencing a bare `server` binary predate the CLI rename.
curl -H "Authorization: Bearer your-secret-key" \
http://127.0.0.1:8080/v1/chat/completions \
-H "Content-Type: application/json" \
-d '{"messages":[{"role":"user","content":"hi"}]}'
# Watch resident memory while it runs:
ps aux | grep llama-server
# Generation speed on this box (tg128 row = tokens/s):
./build/bin/llama-bench -m ./models/Meta-Llama-3.1-8B-Instruct-Q4_K_M.ggufWhat the numbers should look like
Llama 3.1 8B at Q4_K_M is 4.6 GB of weights. Generation speed is bandwidth divided by that figure, and system RAM bandwidth on a shared VPS varies far more than GPU VRAM bandwidth does — it depends on what the host actually allocates you, not just the advertised vCPU count — so this guide will not quote a specific tok/s figure nobody here measured on your specific provider. Memory is the number worth watching. On this site’s calculator, Llama 3.1 8B at Q4_K_M at 8K context comes to about 6.2 GB, and at 32K about 9.5 GB. Either fits a 16 GB box with the OS beside it. The model’s full 128K window needs about 22.7 GB and does not fit, so set `-c` deliberately rather than leaving it at the trained length.
When it does not work
Out of memory on a 16 GB box: check what else is running — a database or a web server sharing the same VPS competes for the same RAM the model needs. Generation is slower than you expected: system RAM bandwidth on budget VPS tiers is often the real bottleneck, and no amount of thread tuning fixes a shared, oversubscribed memory bus — a bigger instance tier is the actual fix, not a flag. The server binds but nothing outside the box can reach it: that is correct if you bound to `127.0.0.1`, which you should have — put a reverse proxy with TLS in front rather than binding to `0.0.0.0` directly. And if you skipped `--api-key` on a box with any public exposure, that is the first thing to fix, not the last.
Common questions
Is a budget 16 GB VPS actually usable for running an 8B model?
For personal use, yes. Llama 3.1 8B at Q4_K_M needs 4.6 GB of weights and about 6.2 GB at 8K context, comfortable on a 16 GB instance. Generation speed depends on the provider’s system RAM bandwidth, which this index has not measured across providers — treat it as usable for a private assistant, not as a production API with guaranteed throughput.
Should I build llama.cpp with OpenBLAS on a VPS?
Only if you process long documents in large batches. llama.cpp’s documentation states BLAS acceleration helps prompt processing above batch size 32 and has no effect on generation speed — for a chat API, a plain CPU build is enough.
How do I secure a llama.cpp server on a public VPS?
Bind the server to `127.0.0.1`, not `0.0.0.0`, always pass `--api-key`, and put a reverse proxy with TLS in front for anything reachable from the internet — llama.cpp’s built-in server has no rate limiting or user management of its own.
What this guide uses
- Hardware
- 16 GB RAM (CPU)16GB
- Format
- GGUF
Next steps
See everything that fits 💻 16 GB RAM (CPU)Reverse lookup — 16GB at 4096 context, ranked by qualityModels covered in this guide
Did this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
CPU Inference: OpenBLAS Tuning for llama.cpp
Maximize tokens/sec on a CPU-only VPS with thread count and BLAS backend tuning.
Nginx Reverse Proxy for Local LLM APIs
Put Ollama or llama.cpp behind Nginx with TLS, rate limiting, and a stable /v1 endpoint for your apps.
Multi-Model API Server on RTX 4090 with vLLM
Serve multiple AWQ-quantized models with vLLM's continuous batching for production-grade throughput.