Two numbers, not one. The weights are the floor — they sit in GPU memory whether you serve one user or a thousand, and they cost the parameter count times the bytes per weight. Whatever is left after weights and engine overhead becomes conversation cache, and that is what decides how many people you can serve at once. A model can fit on a card and still be useless at the context length you need.

Every token in a conversation leaves behind key and value tensors the model reuses for the next token, and they stay in GPU memory until the request ends. The cost is linear: a fixed number of bytes per token per GPU, multiplied by the tokens in the conversation — which is why context length drives the number more than anything else you can set. This calculator shows that per-token figure so you can check the arithmetic yourself. Sliding-window layers are the exception: past their window they stop charging for a longer conversation.

Close to the ratio of the bits per weight, but never exactly. Half precision costs 16 bits per parameter, FP8 costs 8, and INT4 at group size 128 costs about 4.375 once the scales and zero points are counted. The shortfall comes from tensors that stay at full precision — under FP8, embeddings, lm_head, norms and the router usually do. A real NVFP4 checkpoint measures 4.936 bits per weight rather than the 4.5 you would estimate, because every non-expert tensor is still bf16.

No. Weight precision and KV precision are separate settings, and running FP8 KV against bf16 weights is ordinary. FP8 KV halves the per-token cost against bf16 — one byte per element instead of two. AWQ, GPTQ and NVFP4 checkpoints carry no KV dtype of their own, so a KV precision has to be picked alongside them.

Every running request keeps its own cache. There is one pool, and a request takes bytes-per-token times its context length out of it, so concurrency is the pool divided by the per-request cost. Two other ceilings can bind before memory does — the engine's cap on running requests, and on hybrid models a separate per-conversation state pool. The number shown here is the lowest of the three.

Only when the model has at least as many KV heads as it has GPUs; then each rank holds ceil(heads / GPUs) of them. With fewer KV heads than GPUs the heads replicate, and per-GPU cache stops shrinking no matter how many cards you add. MLA replicates its compressed cache across the whole tensor-parallel group by design. Dividing the cache by GPU count is the most expensive mistake available in this arithmetic — 8-16× optimistic on an MLA model.

The engine sizes its cache inside a static budget, a fraction of the card rather than all of it. CUDA graph capture, peak prefill activations, the logits buffer and fragmentation headroom are allocated outside that fraction. Weights plus cache plus those can exceed the card while the budget arithmetic still balances — the server runs out of memory at startup on a configuration that added up. This calculator counts them against the card, not the budget, which is why it sometimes says no where a naive sum says yes.

From each model's own config.json and safetensors headers, read by a build script — never typed in by hand, never taken from a model card. The KV formulas are checked a second way, against real SGLang startup logs, and the engine's overhead constants are fitted to those logs rather than guessed. Every result carries a badge for which of the two it rests on: Measured means a log exists for that exact model and GPU; Calculated means the formula is verified but GPU capacity comes from the vendor spec sheet.

Enough that there is a page for it. It covers SGLang 0.5.x only — defaults and overhead constants move between releases. torch.compile piecewise CUDA graphs and the overlap scheduler's double buffering are not modeled at all. Checkpoint geometry has been checked on bf16 and FP8 layouts, not on AWQ or GPTQ. Every gap on that list runs in the direction that can burn you: real memory above what the tool shows, or real capacity below it.