Table of Contents
An on-premise LLM needs enough GPU memory for three things: the model weights, a key-value (KV) cache for every request in progress, and runtime overhead. A 72-billion-parameter model at 8-bit precision takes about 73 GB for its weights alone, so even a pilot needs one 96 GB card or two 48 GB cards, and more users need more cache.
Key takeaways
- Weights take roughly 2 GB per billion parameters at 16-bit precision, about half that at 8-bit and a quarter at 4-bit (Hugging Face).
- The KV cache grows with every token held in memory. For Qwen2.5-72B it is about 0.33 MB per token at 16-bit, so one 6,500-token document question needs about 2.1 GB, calculated from the model's configuration.
- Size for requests in flight, not for named users. A few hundred users may send only a handful of requests at the same moment.
- Generation speed is limited mainly by memory bandwidth, not compute (NVIDIA). Mixture-of-experts models use only part of their weights for each token and answer faster (Hugging Face).
- Measure with your own prompts and serving engine before you order hardware.
What takes up GPU memory when an LLM runs?
Three things share the GPU's memory.
Weights. The model's parameters. Loading a model with X billion parameters takes roughly 2 × X GB at bfloat16 precision (Hugging Face). Quantization to 8-bit or 4-bit reduces this; our guide to sovereign, on-premise AI gives a table for common model sizes.
KV cache. While generating, the model stores the keys and values computed for every previous token, so it does not recompute them at each step. This cache grows linearly with the number of tokens (Hugging Face). Every request in progress has its own.
Overhead. Activations, the CUDA runtime and the serving engine's buffers. In this article we reserve 10% of GPU memory for them. That is an assumption; check the real figure in your serving engine's logs.
Weight memory is fixed once you pick a model and precision. Cache memory depends on how your people use the system, which is why sizing from the model alone goes wrong.
How do you calculate the KV cache?
NVIDIA gives the formula: KV cache per token in bytes = 2 × number of layers × (number of heads × head dimension) × bytes per value (NVIDIA). The 2 counts keys and values. In NVIDIA's example, Llama 2 7B at 16-bit needs about 2 GB of cache for a single 4,096-token sequence.
Many recent models, including the one in the example below, use grouped-query attention, which stores a small number of key-value heads, such as 2, 4 or 8, instead of one per query head, and keeps almost all of the memory savings of the more aggressive multi-query attention (Hugging Face). For these models, use the number of key-value heads in the formula. You find it in the model's config.json.
A worked example with Qwen2.5-72B-Instruct, a 72.7-billion-parameter model that lists French and Arabic among its supported languages (model card). Its configuration gives 80 layers, 64 attention heads, 8 key-value heads and a hidden size of 8,192, so each head has 8,192 ÷ 64 = 128 dimensions.
- Per token, at 16-bit: 2 × 80 × 8 × 128 × 2 bytes = 327,680 bytes, about 0.33 MB.
- Per document question: a retrieval (RAG) prompt of 6,000 tokens plus a 500-token answer, the same workload we priced in our cost comparison, holds 6,500 tokens: about 2.1 GB.
- Per long request: at 32,768 tokens, the length set in the configuration file, one request would need about 10.7 GB.
- For 20 document questions in flight: about 42.6 GB, more than half the size of the 8-bit weights.
Two settings change these numbers. Storing the cache in FP8 instead of 16-bit halves the bytes per value; vLLM supports this and notes that it lets the GPU hold more tokens (vLLM). Limiting the maximum context length and the maximum number of simultaneous sequences also reduces memory use (vLLM).
Language matters too. The same text can produce very different numbers of tokens in different languages, with differences of up to 15 times in some cases (Petrov et al., 2023). Run a sample of your Arabic and French documents through the model's tokenizer before you fix the tokens per request.
How many requests will a plant send at the same time?
Requests in flight equal the arrival rate multiplied by the time each request takes. For 300 users asking 20 questions per working day, with 15% of the day's 6,000 questions in the busiest hour (assumption), the peak is 900 questions per hour, or 0.25 per second. If each answer takes 25 seconds, about 6 requests are in progress on average.
Arrivals are not smooth, so in the example below we size for 20 requests in flight, a little over three times the average (assumption). Replace these figures with your pilot's logs: users, questions per day, tokens per question and busiest hour.
How fast will the model answer?
Generating each token requires reading the model's weights from GPU memory, and NVIDIA describes inference as frequently "memory-bandwidth bound", in particular in the weights (NVIDIA). A rough upper limit for one request is memory bandwidth divided by the bytes of weights read per token.
For a dense 72.7-billion-parameter model at 8-bit, about 73 GB, on a card with 1,597 GB/s of bandwidth, that gives at most about 22 tokens per second for a single request, or roughly 23 seconds for a 500-token answer. Serving engines batch requests so that the cost of loading the weights is shared across many requests (NVIDIA). That raises total throughput; the speed of a single answer stays under this limit.
Mixture-of-experts (MoE) models change the trade-off. All their parameters must sit in memory, but only some are used for each token (Hugging Face). OpenAI's gpt-oss-120b has 117 billion parameters with 5.1 billion active and fits on a single 80 GB GPU (model card). On one RTX PRO 6000 Blackwell card, Hardware Corner measured 148 tokens per second of generation with a 12,000-token context and 62 GB of memory in use, using llama.cpp with a single user.
Which data-centre GPUs fit which models?
Specifications are from NVIDIA's product pages, checked in November 2026. The last column is our arithmetic for an 8-bit model of about 72 billion parameters, using 90% of memory.
| GPU | Memory | Bandwidth | Max power | GPU-to-GPU link | 72B model at 8-bit |
|---|---|---|---|---|---|
| L40S | 48 GB GDDR6 | 864 GB/s | 350 W | No NVLink; PCIe Gen4 | Needs 2 cards; about 14 GB for cache |
| H100 SXM | 80 GB | 3.35 TB/s | Up to 700 W | NVLink 900 GB/s | Does not fit on 1 card |
| H100 NVL | 94 GB | 3.9 TB/s | 350 to 400 W | NVLink 600 GB/s | Fits; about 12 GB for cache |
| RTX PRO 6000 Blackwell Server Edition | 96 GB GDDR7 | 1,597 GB/s | Up to 600 W | PCIe Gen5; no NVLink listed | Fits; about 14 GB for cache |
| H200 NVL | 141 GB | 4.8 TB/s | Up to 600 W | 2- or 4-way NVLink bridge, 900 GB/s | Fits; about 54 GB for cache |
NVIDIA lists the H200 figures as preliminary. Prices change often; our cost comparison gives sourced figures for one configuration.
When do you need more than one GPU?
vLLM's documentation sets out the order: if the model fits on a single GPU, distributed inference is probably unnecessary; if it fits on one server with several GPUs, use tensor parallelism; beyond one server, combine tensor and pipeline parallelism (vLLM). Tensor parallelism splits individual layers across GPUs (NVIDIA), so the link between the cards matters. For servers whose GPUs lack NVLink, such as the L40S, vLLM recommends pipeline parallelism instead, citing lower communication overhead.
A second GPU is also worth having for reasons other than size: it gives redundancy and room for the retrieval side of a RAG system, such as the embedding model. The RTX PRO 6000 Server Edition supports MIG, which divides one card into up to four isolated instances (NVIDIA).
Worked example: a document assistant for 300 users
Target: Qwen2.5-72B, 6,500-token requests, 20 requests in flight, 16-bit KV cache, 10% of memory reserved for overhead. GB here means 10⁹ bytes.
| Configuration | Total memory | Usable (90%) | Weights | Left for cache | Requests that fit |
|---|---|---|---|---|---|
| 1 × RTX PRO 6000, 8-bit weights | 96 GB | 86.4 GB | 72.7 GB | 13.7 GB | about 6 |
| 2 × RTX PRO 6000, 16-bit weights | 192 GB | 172.8 GB | 145.4 GB | 27.4 GB | about 12 |
| 2 × RTX PRO 6000, 8-bit weights | 192 GB | 172.8 GB | 72.7 GB | 100.1 GB | about 47 |
| 1 × H200 NVL, 8-bit weights | 141 GB | 126.9 GB | 72.7 GB | 54.2 GB | about 25 |
Only the last two meet the target of 20 with margin. The single card roughly covers the average load of 6 but not the bursts. The 8-bit weight figures are approximate, since quantized formats store some extra scaling data. Hugging Face found almost no degradation at 8-bit in its example (Hugging Face), but test quality on your own questions before you settle on it.
What else does the server need?
- Power and cooling. Four cards at up to 600 W each can draw 2.4 kW before processors, memory and fans. Check rack power, the UPS and cooling capacity in the room where the server will sit before you order.
- Storage. Model files are about the size of the weights; Hardware Corner describes the gpt-oss-120b file it tested as 65 GB. Keep the previous model version on disk so you can roll back.
- A test environment. A smaller GPU for trying new model versions and prompts before they reach users.
- Network design. If the site is isolated, plan how models and updates get in. See our guide to air-gapped AI deployment.
How should you size your own server?
- Pick two or three candidate models and check their licences and languages.
- Read each model's
config.jsonand calculate the KV cache per token. - Measure tokens per request on a sample of real Arabic, French and English questions with retrieved passages. Our RAG guide for maintenance manuals covers how prompts are built.
- Estimate requests in flight from expected users, questions per day and the busiest hour.
- Add weights, cache for the peak and overhead, then choose a GPU configuration with margin.
- Load-test the shortlisted configuration with your serving engine before signing the order.
If you are still choosing between retrieval and training on your documents, read fine-tuning vs RAG for an industrial assistant first, since the two lead to different hardware needs.
What we have seen at KABAS AI
DALILE runs on the client's own servers and answers questions from documents, spreadsheets and databases, so sizing the hardware is part of planning every deployment. The figures in this article are public ones, not measurements from client sites. The practical lesson from this method is to measure before buying: a sample of real questions, their token counts and the expected peak decide the server more than the model's name does. Our consulting team can run this calculation with your own documents and usage figures.
Sources
Specifications checked in November 2026. They change with new product versions; recheck before you order.
- NVIDIA: Mastering LLM Techniques, Inference Optimization
- Hugging Face: Optimizing LLMs for speed and memory
- Hugging Face: Mixture of Experts explained
- Qwen2.5-72B-Instruct model card
- Qwen2.5-72B-Instruct config.json
- openai/gpt-oss-120b model card
- vLLM: Quantized KV cache
- vLLM: Conserving memory
- vLLM: Parallelism and scaling
- Petrov et al., Language Model Tokenizers Introduce Unfairness Between Languages, 2023
- Hardware Corner: Testing GPT-OSS 120B on RTX Pro 6000 Blackwell
- NVIDIA L40S GPU
- NVIDIA H100 Tensor Core GPU
- NVIDIA RTX PRO 6000 Blackwell Server Edition
- NVIDIA H200 Tensor Core GPU
Frequently asked questions
How much GPU memory does a 70B model need?
For the weights alone, about 140 GB at 16-bit precision, about 70 GB at 8-bit and about 35 GB at 4-bit, following Hugging Face's rule of thumb of roughly 2 GB per billion parameters at bfloat16. You then need extra memory for the KV cache of every request in progress and for runtime overhead, so a 70B model at 8-bit does not fit usefully on a single 80 GB GPU.
Can I run an on-premise LLM on a gaming or workstation GPU?
Smaller models can run on cards with 16 to 32 GB of memory; OpenAI documents that gpt-oss-20b runs within 16 GB. For a shared assistant serving many users, look at server cards: NVIDIA's L40S and RTX PRO 6000 Server Edition, for example, have 48 GB and 96 GB of ECC memory and are built for rack servers. Whatever the card, test it under the load you expect.
Is it better to buy one large GPU or several smaller ones?
If the model and its cache fit on one GPU, one GPU is simpler: vLLM's documentation says distributed inference is probably unnecessary in that case. Several GPUs let you run larger models or more concurrent requests, but the cards must exchange data, and vLLM recommends a different way of splitting the model when they have no NVLink. Two cards also give some redundancy.
Does quantization reduce answer quality?
It can. Hugging Face reports almost no degradation at 8-bit in its example and notes that 4-bit results can differ more from full precision. Run your own test set of real questions on the quantized and unquantized versions before choosing, and check Arabic and French answers separately.
How many users can one GPU server support?
It depends on how many requests are in progress at the same time, how long each prompt is and how fast answers must arrive, not on the number of named users. Estimate requests in flight as requests per second multiplied by seconds per answer, size the KV cache for that number with margin, then confirm with a load test using your own prompts.



