What changed
This guide used to describe TabbyAPI as the server for EXL2. It no longer is one. TabbyAPI now calls itself the official API server for ExLlamaV3, its dependencies install exllamav3 and nothing else, and its README lists two supported model types: EXL3 (recommended) and plain FP16/BF16 weights. ExLlamaV2 — the library behind EXL2 — is itself archived. TabbyAPI publishes no tagged releases, so there is no version this site can point you to that still loads EXL2. If you have EXL2 models, the options are in the last section.
What you need first, and what this is not
An NVIDIA card: the ExLlamaV3 wheels TabbyAPI installs are CUDA builds. Python 3.10 or newer. And the project’s own framing, which is worth reading before you build on it — its README says it is "a hobby project made for a small amount of users" and "not meant to run on production servers". It is a good way to put an EXL3 model behind an OpenAI-style API on your own machine; for a service other people depend on, use vLLM.
git clone https://github.com/theroyallab/tabbyAPI
cd tabbyAPI
# Linux (start.bat on Windows) — sets up the environment, uses uv if installed
./start.shGet a model and point the config at it
The start script has a downloader. EXL3 repositories keep each bitrate on its own branch, so pass it with --revision; the example is the one from TabbyAPI’s own documentation. Models are directories under models/. A config.yml is optional — copy config_sample.yml only if you want to change defaults. The ones that matter: model_name (the directory to load at startup) and max_seq_len, which decides how much KV cache is allocated and therefore whether the model loads at all. The network section defaults to 127.0.0.1, port 5000.
./start.sh download turboderp/Qwen2.5-VL-7B-Instruct-exl3 --revision 4.0bpw
cp config_sample.yml config.yml
# then in config.yml:
# model:
# model_dir: models
# model_name: Qwen2.5-VL-7B-Instruct-exl3
# max_seq_len: 8192Or run the container
TabbyAPI publishes a CUDA image. Keep the shared-memory flag from its README: ExLlamaV3 uses /dev/shm for tensor parallelism and CPU MoE offload, Docker’s default of 64 MiB is too small, and loading fails with an error naming the limit. The README’s own command publishes port 5000 on every interface; bind it to 127.0.0.1 unless you mean to expose it.
docker run --gpus all --shm-size=8g --name tabbyapi \
-p 127.0.0.1:5000:5000 \
-v /path/to/models:/app/models \
ghcr.io/theroyallab/tabbyapi:latestCheck it actually worked
Authentication is on by default: on first start TabbyAPI generates keys into api_tokens.yml, and requests need one in an x-api-key header or as Authorization: Bearer. The health endpoint answers without the model doing any work; the models endpoint tells you what is loaded. Watch nvidia-smi while it loads — a model that should fit but fails at load is almost always max_seq_len.
curl http://127.0.0.1:5000/health
KEY=$(grep api_key api_tokens.yml | cut -d" " -f2)
curl http://127.0.0.1:5000/v1/models -H "x-api-key: $KEY"
nvidia-smi --query-gpu=memory.used --format=csv -l 1What the numbers should look like
This site has not measured EXL3, and its model index carries no EXL3 builds, so it gives no speed or size figure here rather than borrowing EXL2’s. The structure of the cost is the same as any other server: the weights are fixed by the bitrate you downloaded, and the KV cache grows with max_seq_len. For a sense of scale, look the model up in the VRAM calculator at a GGUF or EXL2 level of similar bits per weight — and treat it as an approximation, not a measurement.
If what you have is an EXL2 model
Three honest options. Run it locally with ExLlamaV2’s own chat script — the ExLlamaV2 guide on this site covers that, and it still works; it just has no maintained API server. Serve the same model as GGUF with llama.cpp, which has an OpenAI-compatible server and is actively developed; every model in this index ships a GGUF build. Or move to the EXL3 build of the model, if its publisher has made one, and use TabbyAPI as above.
Common questions
Does TabbyAPI still load EXL2 models?
Not the current version. Its dependencies install exllamav3 only, and its README lists EXL3 and FP16/BF16 as the supported model types. TabbyAPI publishes no tagged releases, so there is no version to pin that is known to still load EXL2. Run EXL2 locally with ExLlamaV2’s own scripts, or serve the model’s GGUF build with llama.cpp.
Is TabbyAPI suitable for production?
Its own README says no: "a hobby project made for a small amount of users… not meant to run on production servers." That is the maintainers’ statement and it is worth respecting. Use it to make a model reachable from editor plugins and desktop clients on your own machine; use vLLM when the thing you are building is a service.
Why does the TabbyAPI container fail to load a model?
The usual cause is shared memory. ExLlamaV3 keeps tensor-parallel and CPU MoE offload buffers in /dev/shm, and Docker gives a container 64 MiB by default — TabbyAPI’s documentation says loading then fails with an error naming the limit. Start the container with --shm-size=8g, as its README does, or with --ipc=host. If that is not it, lower max_seq_len.
What this guide uses
- Hardware
- RTX 409024GBRTX 3090
- Picks for it
- Best local LLM for 24GB
Next steps
See everything that fits 🟢 RTX 4090Reverse lookup — 24GB at 8192 context, ranked by qualityDid this actually run?
Copying a command is not the same as it working, so this is the only place the site asks. Nothing is collected beyond the answer itself.
Related guides
Multi-Model API Server on RTX 4090 with vLLM
Serve multiple AWQ-quantized models with vLLM's continuous batching for production-grade throughput.
Nginx Reverse Proxy for Local LLM APIs
Put Ollama or llama.cpp behind Nginx with TLS, rate limiting, and a stable /v1 endpoint for your apps.
Run Llama 3.1 8B on a €20/month VPS
A complete guide to running a private LLM API on a budget Linux VPS using llama.cpp server mode.