What actually fits on 48GB
A 70B at Q4_K_M is about 40.7GB of weights; with a 4K KV cache and the activation buffer the estimate lands near 46GB against 48GB of combined VRAM. That is 96% of the pair — it loads, but it is the amber band, not a comfortable fit, and it leaves nothing for a desktop session on either card. At 8K context the same model is around 47.5GB. Treat 70B Q4_K_M on 48GB as a headless configuration, and know that llama.cpp's own defaults will not even try to fit it entirely: its auto-fit keeps 1 GiB free on each card, which puts the pair's budget at about 46GB before the model is weighed.
Llama 3.3 70B Q4_K_M @4K ctx ≈ 46.1 GB of 48 GB amber — loads, no margin
Llama 3.3 70B Q4_K_M @8K ctx ≈ 47.5 GB of 48 GB amber — at the ceiling
Llama 3.3 70B Q3_K_M* @8K ctx ≈ 38.4 GB of 48 GB comfortable
Seed-OSS 36B Q4_K_M @8K ctx ≈ 25.0 GB of 48 GB comfortable
(estimates from this site’s calculator: weights + KV cache + 10% activations)
* generic Q3_K_M rate — this index lists no Q3_K_M build of Llama 3.3 70BPrerequisites
Two CUDA cards visible to the driver, a llama.cpp build with CUDA enabled, and enough system RAM to read the file while it is distributed. The flag is GGML_CUDA=ON. The old names are caught rather than ignored — LLAMA_CUDA still works with a deprecation warning and LLAMA_CUBLAS stops the configure step — but a misspelt flag only shows up under "Manually-specified variables were not used" at the end of configure, and you get a CPU-only build. The cards do not need NVLink: in the default layer split each card holds a run of whole layers, so only the activations at the boundary cross between them, and llama.cpp's multi-GPU documentation says this mode tolerates slow interconnects.
nvidia-smi --query-gpu=index,name,memory.total --format=csv
cmake -B build -DGGML_CUDA=ON -DCMAKE_BUILD_TYPE=Release
cmake --build build -j$(nproc)
./build/bin/llama-server --list-devices # both cards should be listedSplit across the cards
Two things changed in llama.cpp that this step depends on. --tensor-split is now optional: left out, the layer split is proportional to each card's memory, which on two identical 3090s is already an even split. When you do set it, it takes proportions, not gigabytes — 3,1 gives the first card 75%. And -ngl defaults to auto, with --fit on: llama.cpp adjusts any memory setting you did not set yourself so the model fits with a 1 GiB margin per card, which on a 46GB model means quietly running some layers on the CPU. Set -ngl all and -c explicitly, so a model that does not fit fails at load instead of loading slowly. Without -c, llama.cpp starts from the model's full trained context — 128K for Llama 3.3 — and lets auto-fit cut it down. Bind to loopback: llama-server has no authentication of its own.
./build/bin/llama-server \
-m ./models/Llama-3.3-70B-Instruct-Q4_K_M.gguf \
-ngl all -c 4096 \
--host 127.0.0.1 --port 8080
# Mismatched pair (e.g. 24GB + 16GB)? The default split already follows memory;
# override it with proportions only if one card also drives a display:
# --tensor-split 3,2Confirm both cards are carrying the model
The startup log states how many layers were offloaded; anything short of all of them means part of the model is on the CPU and generation will be slow regardless of what the cards are doing. nvidia-smi should show both GPUs holding roughly half the weights each. A single card at ~23GB while the other sits near zero means the second card was never used — check --list-devices and CUDA_VISIBLE_DEVICES.
# In the startup log:
# load_tensors: offloaded 81/81 layers to GPU
watch -n1 nvidia-smi --query-gpu=index,memory.used --format=csvThe other split mode
llama.cpp now also has --split-mode tensor, which splits every layer across both cards instead of giving each card whole layers. Its documentation describes it as experimental, aimed at faster token generation on NVIDIA cards, and more dependent on the link between the cards than the default. It needs flash attention, will not take a quantized KV cache, and turns auto-fit off, so you size the context yourself. The older row mode is deprecated. This site has not measured either against the default on a pair of 3090s, so it makes no speed claim; if you try tensor mode, compare tokens per second against the layer split on your own machine before keeping it.
When 46 of 48 GB is not enough
Three changes, in the order worth trying. Lower the context: the KV cache is the only part that scales with it, and going from 8K to 4K gives back over a gigabyte. Step down a quant level: at the calculator's generic Q3_K_M rate the same model is about 38.4GB at 8K, a comfortable fit on the pair — but this index lists no Q3_K_M build of Llama 3.3 70B, so that is an estimate from the generic rate, and no quality-loss figure for it has been published here. Or run a smaller model — Seed-OSS 36B at Q4_K_M is about 25GB at 8K on the same pair and leaves room to actually use the machine. What will not help is adding system RAM: once layers spill to the CPU, throughput falls off a cliff on a model this size.
Next steps
The calculator sizes one card at a time, so check a 70B against a single 24GB card to see the per-card half, then double it — and read the amber verdict as the warning it is.
Common questions
Do two RTX 3090s need NVLink to run a 70B in llama.cpp?
No. The default split mode gives each card a run of whole layers, so only the activations at the boundary cross between them, and llama.cpp's multi-GPU documentation says this mode tolerates slow interconnects. The link matters more for the experimental tensor split mode, which exchanges data inside every layer.
Can I pair two different cards, like an RTX 3090 and a 16GB card?
Yes — the default split is proportional to each card's memory, so you do not have to set --tensor-split. But 24 + 16 is 40GB, and a 70B at Q4_K_M needs about 46GB at 4K, so it does not fit. At the calculator's generic Q3_K_M rate it is about 37.1GB at 4K — 93% of the pair, which this site rates tight rather than comfortable.
Why does llama.cpp put some layers on the CPU when the model should fit?
Because auto-fit is on by default and keeps 1 GiB free on each card, and -ngl defaults to auto. Any setting you leave unset is adjusted to stay inside that margin, and on a model within a couple of gigabytes of the limit that means moving layers to the CPU without an error. Pass -ngl all and an explicit -c so it either loads entirely on the GPUs or fails with an out-of-memory error you can act on.
What this guide uses
- Hardware
- RTX 309024GBRTX 4090
- Format
- GGUF
- Picks for it
- Best local LLM for 24GB
Next steps
See everything that fits 🟢 RTX 3090Reverse lookup — 24GB at 4096 context, ranked by qualityModels covered in this guide
Did this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
Run Llama 3.1 8B on a €20/month VPS
A complete guide to running a private LLM API on a budget Linux VPS using llama.cpp server mode.
CPU Inference: OpenBLAS Tuning for llama.cpp
Maximize tokens/sec on a CPU-only VPS with thread count and BLAS backend tuning.
Nginx Reverse Proxy for Local LLM APIs
Put Ollama or llama.cpp behind Nginx with TLS, rate limiting, and a stable /v1 endpoint for your apps.