Deployment Cookbook
Battle-tested guides for running LLMs on real hardware
Run Llama 3.1 8B on a €20/month VPS
A complete guide to running a private LLM API on a budget Linux VPS using llama.cpp server mode.
Mac M3 Max: The Ultimate Local LLM Setup
Maximise your Apple Silicon with Ollama. Run multiple models, set up an OpenAI-compatible API, and tune Metal GPU layers.
Multi-Model API Server on RTX 4090 with vLLM
Serve multiple AWQ-quantized models with vLLM's continuous batching for production-grade throughput.
Docker Compose LLM Stack: Ollama + Open WebUI
A production-ready Docker Compose stack that gives you a local ChatGPT experience with one command.
What Can You Run on RTX 4060 Ti 16G?
A practical guide to picking the right model and quant level for NVIDIA's best budget 16GB card.
DeepSeek-R1 Distill 14B: EXL2 vs GGUF
Head-to-head on RTX 4090 — when to pick turboderp EXL2 over bartowski GGUF.
ExLlamaV2 on RTX 4090: Full Setup Guide
Install ExLlamaV2, load an EXL2 quant, and serve an OpenAI-compatible API in under 10 minutes.
Running 70B on Dual RTX 3090 with llama.cpp
Split a 70B GGUF across two 24GB cards with llama.cpp — how much fits, which flags matter now that auto-fit is on, and what to change when it does not.
Qwen2.5-Coder 32B on a Single RTX 4090
Which Qwen2.5-Coder 32B build fits a 24GB card at which context length, and the MoE coder to use when you need 32K.
Mac M3 Pro: Realistic Model Limits
Unified memory is not all yours — what an 18GB and a 36GB M3 Pro actually hold, and the Metal limit that decides it.
llama.cpp on Windows with CUDA
Build llama.cpp with NVIDIA GPU support on Windows 11 — the path of least resistance for PC gamers.
TabbyAPI: OpenAI-Compatible Server for EXL3
TabbyAPI now serves ExLlamaV3 (EXL3 and FP16) rather than EXL2 — what changed, how to run it today, and what to do with an EXL2 model.
Quantize Your Own Model to GGUF
Use llama.cpp's quantize tool to convert any HF model to GGUF Q4_K_M for local inference.
vLLM + AWQ in Production: Tuning Guide
gpu-memory-utilization, max-model-len, and batching knobs for stable API serving.
CPU Inference: OpenBLAS Tuning for llama.cpp
Maximize tokens/sec on a CPU-only VPS with thread count and BLAS backend tuning.
8GB GPU Starter Guide: 3060 / 4060 / 3070
The most common local LLM hardware tier — which models, quants, and context lengths actually fit in 8GB VRAM.
M1 / M2 Mac 8GB: Realistic Ollama Limits
Unified memory is shared with macOS — here is what actually works on base MacBooks without swapping.
WSL2 + Ollama GPU Passthrough on Windows
Run Ollama with NVIDIA GPU acceleration inside WSL2 — the most reliable Windows path for local LLMs.
Docker: Ollama with NVIDIA or AMD GPU Passthrough
Containerised Ollama with GPU access — isolate models, pin versions, and run alongside other services.
Nginx Reverse Proxy for Local LLM APIs
Put Ollama or llama.cpp behind Nginx with TLS, rate limiting, and a stable /v1 endpoint for your apps.
AMD GPU + llama.cpp via ROCm (Quick Start)
Run GGUF models on Radeon RX 9070 / 7900 / 6800 series with llama.cpp HIP backend — what works and what does not.
Ollama on Windows (Native, No WSL)
Install the Windows Ollama app for the simplest path — NVIDIA and AMD Radeon GPUs both work, with no WSL and no toolkit to install.
Run GPT-OSS 20B (and 120B) locally without re-quantizing
GPT-OSS ships natively in MXFP4, so the usual "download the Q4_K_M" habit makes it bigger and worse. Sizing, the right flags, and how MoE expert-offload puts the 120B on a 24GB card.
Intel Arc: llama.cpp (SYCL) or Ollama (Vulkan)
Run GGUF models on an Arc B580, B570, A770 or A750 — the SYCL build, the Ollama route, how to prove the GPU is doing the work, and what fits in 8–16 GB.
What to Run on a 12GB GPU (RTX 3060 12G, 4070, 5070)
Which models fit a 12GB NVIDIA card at 8K–32K context, where 14B stops fitting, how to run them with Ollama or llama.cpp, and how to prove they are on the GPU.
RTX 5090: What 32GB Actually Buys for Local LLMs
What fits on an RTX 5090 that does not fit on a 24GB card: not many more models, but much longer context and higher speed ceilings. Sizes from the calculator, setup for Blackwell.
DGX Spark 64GB vs 128GB: What Each One Runs
Which models fit the 64GB and 128GB DGX Spark, from the calculator: GPT-OSS 120B needs the 128GB model, 70B dense fits both, and 273 GB/s sets the speed. Setup for Ollama and llama.cpp.