Local LLM FAQ

About 5.1 GB at Q4_K_M with a 4K context window — 4.2 GB of weights, 0.5 GB of KV cache and a 0.5 GB activation buffer, measured on WizardLM-2 7B (7B). Every 8 GB card in this index clears that with room left. The figure rises with context, not with how hard you use the model: the same file at 32K needs about 9.0 GB, because the KV cache grows to 4.0 GB.

At 4 bits and short context, yes — Mistral Small 24B Instruct at AWQ INT4 needs about 13.2 GB of 16 GB, leaving 2.8 GB. The level matters more than the parameter count: the same model at Q4_K_M is 15.9 GB, which is 99% of the card — it loads on a card with nothing else on it and has no margin for a longer window. A 16 GB card runs 64 of the 112 models here comfortably.

Yes. On a card that is also driving a desktop, subtract roughly 0.5–1.5 GB before comparing against any figure here — more with a browser open. That is why the calculator's "comfortable" verdict stops at 88% of the card rather than 100%: the last eighth is the margin that keeps a model from failing to allocate on a machine you are also using. The looser 105% rule is shown separately and always named.

Because a level like Q4_K_M is a recipe, not a fixed bit width — it quantizes different tensors to different precisions, and the mix depends on the model's shape. Across the 112 models here that ship Q4_K_M the effective rate runs from 4.1 to 5.7 bits per weight. This site stores each model's own measured rate rather than one table value, which is why two 8B models at the same level can differ by several hundred megabytes.

It depends on the model's attention shape, not on its parameter count. WizardLM-2 7B goes from 0.5 GB of KV cache at 4K to 4.0 GB at 32K — roughly linear, because every layer keeps a cache that grows with the window. 2026 hybrid-attention models break that: a model running only a fraction of its layers on full attention keeps a fixed-size recurrent state on the rest, so its cache barely moves. The calculator accounts for both.

At Q4_K_M the median published perplexity loss across the 79 models here that report one is 2.9% — 97.1% retained — with the range running 1.4% to 5.2%. Two caveats that matter more than the median: smaller models lose more than larger ones at the same level, and perplexity is a language-modelling measure, not a measure of whether the model still does your task. Models without a published figure are shown as a dash here rather than assumed to be lossless.

Q4 is the nominal bit width, K is llama.cpp's k-quant family, and M is the medium variant of it (there are S and L siblings). The K family does not quantize every tensor the same way — attention and feed-forward tensors get different treatment, and some stay at higher precision — which is why the effective rate lands above 4 bits. In this index Q4_K_M is the level every one of the 112 GGUF models ships, which is why it is the level the site quotes its headline quality figure against.

On the 16 models here that publish a figure at both levels, moving Q4_K_M → Q5_K_M recovers a median of 1.5 percentage points of perplexity loss for about 17% more file. Paired deliberately: the models reporting Q5 are not the same set as those reporting Q4, so two separate medians would compare the groups rather than the levels. In practice the decision is usually made for you by what fits — take the higher level when your card has the headroom, and do not drop below Q4 to squeeze in a bigger model without trying both.

This index cannot tell you. Every quality figure here is perplexity on WikiText-2, which measures next-token prediction on general prose — it is not a coding benchmark, and a site that cannot run task evaluations should not convert one into a claim about another. The widely reported pattern is that tasks with a single correct answer degrade more visibly than open-ended ones, but that is not something measured here, so it is not stated here as a number.

Usually yes at the same bit width, and the reason is mechanical: quantization-aware training runs the quantization in the forward pass during fine-tuning, so the weights adapt to it, where a community post-training quantization compresses a finished model and can only minimise the damage. What this site cannot yet tell you is which build you are looking at — the index records the format and level of each quant but not its publisher, so there is no "official" flag on any row here. That is a known gap, not a judgement that it does not matter.

GGUF unless you are running a server. GGUF runs on anything — CPU, NVIDIA, AMD, Apple — and 112 of the 112 models here ship it, against 53 for AWQ. AWQ's case is throughput under vLLM with batched requests, on an NVIDIA card or one of the Radeon cards vLLM's ROCm build supports; for one person talking to one model, it buys you nothing GGUF does not already do. 53 models here ship both, which is the only set where the two can be compared on the same weights.

Not usefully. Both are lossy compressions of the original FP16 weights, so converting one to the other stacks a second round of loss on top of the first — you would be quantizing an already-quantized model. The path is to go back to the original weights and quantize those, which is what the people publishing each format already did. If a model has no build in the format you need, that is a real gap, not something a conversion fixes.

Because it is already 4-bit — the released checkpoint is itself the quantized one. The 3 models here shipping MXFP4 were released at 4 bits rather than converted down, so there is no FP16 original sitting behind them to recover. Re-quantizing to Q4_K_M costs quality and saves nothing: both land near the same size, and you have added a lossy step for it.

GGUF, through llama.cpp or Ollama with the Metal backend — 112 of the 112 models here. AWQ, EXL2 and GPTQ all require CUDA and will not run on Apple silicon at all, so three of the four formats this index tracks are unavailable to you before you start. The practical constraint on a Mac is not the format but how much of the unified memory the GPU is allowed to wire down, which is below the number on the box.

GGUF via llama.cpp's ROCm or Vulkan backend is the reliable answer, on every Radeon. AWQ and GPTQ run through vLLM's official ROCm builds — but only on the cards vLLM supports: RX 7700 XT and up, RX 9000 and Instinct MI200 or newer. An RX 6000 card, the RX 7600 XT or an MI100 is not on that list, so this site does not recommend those formats there. EXL2 is CUDA-only. Beyond the format, what decides whether an AMD setup works at all is whether your kernel and card have a working ROCm build, which is a setup question rather than a hardware ceiling.

GGUF. llama.cpp runs it on Arc through its SYCL backend, whose documentation lists the A770, A750 and B580 as verified, and Ollama runs it through Vulkan with no build step. This site recommends nothing else on Arc: EXL2 is CUDA-only, and which quantized formats vLLM's Intel backend serves on consumer Arc cards has not been checked here. The index has 5 Arc cards; the largest consumer one, the Arc A770 16G, fits 60 of the 112 models comfortably at 4K context.

Whichever one has the most memory in your budget — capacity decides what you can run at all, and nothing else does. Of the 78 cards in this index the largest consumer option is the RTX 5090 at 32 GB and 1,792 GB/s, which runs 90 of the 112 models here comfortably against 44 for an 8 GB card. Bandwidth is the second question, not the first: it sets how fast a model that already fits will generate, and two cards with the same memory can differ by more than 2.5× on it.

Memory decides most of it, but not all: the formats a vendor's software can load trim the list too. At 16 GB, the Arc A770 16G fits 60 of the 112 models comfortably at 4K and the RTX 5080 fits 64, because some models only squeeze in at a 4-bit AWQ size that Arc cannot load. The biggest retail cards in this index are the RTX 5090 (32 GB, 90 of 112 models comfortably at 4K), the Radeon RX 7900 XTX (24 GB, 86) and the Arc A770 16G (16 GB, 60). NVIDIA runs every tracked format (AWQ, EXL2, GGUF, GPTQ); a Radeon runs AWQ, GGUF, GPTQ — AWQ and GPTQ only through vLLM, on the cards vLLM supports; Intel Arc runs GGUF. If GGUF through llama.cpp or Ollama is all you need, buy the most memory your budget allows from any of them.

For a large part of what people actually run, yes — 44 of the 112 models in this index fit an 8 GB card comfortably at 4K context, including 7B-class models at Q4_K_M (5.1 GB). What 8 GB does not give you is headroom: a long context window, a second process, or a model above about 13B. The step to 16 GB takes you from 44 models to 64.

Mostly, but not all of it. The GPU addresses the same pool as the CPU, so a 64 GB Mac can hold models a 24 GB discrete card cannot — that part is real. What is not real is treating the whole number as available: macOS reserves part of the pool for the system and caps how much a single process may wire down. Budget meaningfully below the figure on the box, and remember bandwidth differs by tier far more than capacity suggests.

One 32 GB card, for a single model. Splitting a model across two cards means every token crosses the bus between them, and the layers on the second card wait on the first — you get the capacity but not the throughput, and the setup is materially harder. Two cards earn their place when you want to run two things at once, or when a 32 GB card is not in the budget. A 32 GB card runs 90 of the 112 models here comfortably against 64 for a single 16 GB card.

For capacity, yes: 88 of the 112 models here fit comfortably in 32 GB the same way they would on a 32 GB card, since the sizing math only cares about the byte count. What it will not give you is a speed figure — token generation on CPU depends on your own DIMM count and channel configuration, not on "32 GB", so this index does not publish a system-RAM bandwidth the way it does for a GPU. Expect noticeably slower generation than the same model on a discrete card, and budget the difference before committing to a CPU-only setup.

Not on a consumer graphics card, no. Llama 3.1 70B Instruct at Q4_K_M needs about 46.1 GB, and no consumer GPU in this index has that much memory — the cards that do are 14 datacentre, Apple and CPU entries. Its smallest build here, EXL2 3.5bpw, still needs 33.6 GB. The realistic routes are a large-memory Mac, system RAM with CPU inference and the speed that implies, or two cards.

Three sources, kept apart. Model architecture (layer count, KV heads, head dimension) comes from each model's own published config and drives the VRAM arithmetic. Perplexity figures are the ones the quantizers published; models without one show a dash rather than a guess. Speed figures are runs recorded on Meta Llama 3.1 8B Instruct against WikiText-2 on a named stack — and only on hardware that stack actually ran on. Everything else on the site, including every VRAM number, is calculated rather than observed, and labelled as such.

Measured means someone ran it and recorded the number on named hardware with a named runtime version. Estimated means it was computed from the model's architecture and quant level using the same formula the calculator uses. The site labels which is which on every surface that shows a figure, and a hardware page with no runs on it says so rather than letting its estimates read as measurements. Removing a row is preferred to adjusting one — three benchmark rows were deleted after arithmetic showed the card they named could not physically reach the speed claimed.

The data carries its own date — it currently reads 2026-10-20 — and every change that alters what the site says is written into a changelog entry on the same day. The rhythm is a data refresh most weeks and a batch of new models every couple of weeks, weighted toward models a constrained setup can actually run rather than toward whatever launched loudest.

Check three things first: the context length (the default here is 4K and the KV cache grows with it), whether your card is also driving a display, and whether your build is the same quant level rather than the same nominal bit width. If it still disagrees, that is worth reporting — a figure that does not survive contact with real hardware is a figure this site would rather fix than defend, and three benchmark rows have already been removed for exactly that reason.