Table of Contents
A 7-billion-parameter model is often enough when the task is narrow and the facts come from your own documents: classifying maintenance reports, extracting fields, or answering from retrieved passages. It falls short when the model must supply facts itself, read very long inputs or write well in languages it was not mainly trained on. Your own test set decides.
Key takeaways
- There is no official size threshold. NVIDIA researchers count most models below 10 billion parameters as small language models (SLMs) (Belcak et al., 2025).
- Small models have improved quickly. Mistral 7B outperformed Llama 2 13B on every benchmark its authors evaluated (Jiang et al., 2023), and the 3.8-billion-parameter Phi-3-mini scored 69% on MMLU (Abdin et al., 2024).
- Their main weakness is stored knowledge. Phi-3's authors say the model lacks the capacity to store much factual knowledge and point to search augmentation as the fix, which is what retrieval (RAG) provides.
- On your own servers, size drives cost: an 8-billion-parameter model needs about 16 GB for its weights at 16-bit, against about 145 GB for a 72-billion-parameter model, and it generates faster on the same GPU.
- Choose by testing: start with the smallest serious candidate on your own questions and move up only when it fails.
What counts as a small language model?
"Small" is relative, and it moves every year. A 2025 paper by researchers at NVIDIA Research and Georgia Tech gives a working definition: a small language model is one that "can fit onto a common consumer electronic device" and answers fast enough to serve one user. As of 2025, the authors treat most models below 10 billion parameters as small (Belcak et al., 2025).
For an industrial buyer, the useful line is practical: can the model run on one modest GPU, or even a workstation, with room left for users? Several open-weight families now offer such sizes. The table lists examples, not a ranking; details were checked on the model pages in November 2026.
| Model | Parameters | Licence | Context length | Arabic and French |
|---|---|---|---|---|
| Qwen3-4B | 4B | Apache 2.0 | 32K | Qwen3 lists 119 languages and dialects, including French and Standard Arabic |
| Qwen3-8B | 8.2B | Apache 2.0 | 32,768 native; 131,072 with YaRN | Same family list |
| Phi-4-mini-instruct | 3.8B | MIT | 128K | Both listed; the card warns that non-English languages perform worse |
| Gemma 3 4B | 4B | Gemma Terms of Use | 128K | "Over 140 languages"; not named on the card |
| ALLaM-7B-Instruct-preview | 7B | Check the repository's LICENSE file | 4,096 | Arabic and English; French not listed |
ALLaM, developed by Saudi Arabia's SDAIA, was trained on 4 trillion English tokens followed by 1.2 trillion mixed Arabic and English tokens, according to its model card. Our guide to choosing an open-weight LLM covers licences and larger models.
Why are small models better than they used to be?
Published results show small models closing much of the gap with larger ones on public benchmarks:
- Mistral 7B (2023) outperformed Llama 2 13B "across all evaluated benchmarks" and Llama 1 34B in reasoning, mathematics and code generation, according to its authors (Jiang et al., 2023).
- Phi-3-mini, with 3.8 billion parameters trained on 3.3 trillion tokens, scored 69% on MMLU, which Microsoft describes as rivalling Mixtral 8x7B and GPT-3.5. Quantized to 4-bit, it needs about 1.8 GB and generated over 12 tokens per second offline on an iPhone 14 (Abdin et al., 2024).
- Qwen3: the developers state that "even a tiny model like Qwen3-4B can rival the performance of Qwen2.5-72B-Instruct" (Qwen).
- Distillation: DeepSeek fine-tuned small open models on samples generated by its large DeepSeek-R1 model. The 7B version reports 55.5% on the AIME 2024 maths benchmark, against 9.3% for GPT-4o-0513 in the same table (DeepSeek on Hugging Face).
Read these with care. They are the developers' own results, mostly on English, mathematics and code benchmarks. None of them measures how a model answers questions about your compressor manuals or your HSE procedures.
What does a smaller model save on your own servers?
Two things: memory and speed.
Memory. Loading a model with X billion parameters needs roughly 2 × X GB at 16-bit precision (Hugging Face). Each request in progress also needs a key-value (KV) cache. From the Qwen3-8B configuration (36 layers, 8 key-value heads, 128 dimensions per head), the cache takes 2 × 36 × 8 × 128 × 2 bytes, about 0.15 MB per token at 16-bit. Our GPU sizing guide explains the method and gives 0.33 MB per token for Qwen2.5-72B.
Speed. Generating text is memory-bound: the time taken to move weights and cache from memory dominates latency (NVIDIA). A rough upper limit for one request is the GPU's memory bandwidth divided by the size of the weights. On a card with 1,597 GB/s, such as the RTX PRO 6000 Blackwell Server Edition, our arithmetic gives:
| Qwen3-8B | Qwen2.5-72B | |
|---|---|---|
| Weights at 16-bit | about 16.4 GB | about 145 GB |
| Weights at 8-bit | about 8.2 GB | about 73 GB |
| Weights at 4-bit | about 4.1 GB | about 36 GB |
| KV cache for one 6,500-token document question, 16-bit | about 0.96 GB | about 2.1 GB |
| Upper limit for one request at 8-bit, 1,597 GB/s | about 195 tokens per second | about 22 tokens per second |
These are calculated limits, not measurements; real throughput depends on the serving engine, batching and prompt length.
In practice, Qwen3-8B at full 16-bit precision fits on a single 48 GB NVIDIA L40S with about 27 GB left for cache once 10% is reserved for overhead, enough for roughly 28 document questions of that length in progress at once. A 72B model needs two such cards even at 8-bit.
The NVIDIA authors go further and estimate that serving a 7-billion-parameter SLM is 10 to 30 times cheaper in latency, energy and computation than a 70 to 175-billion-parameter model (Belcak et al., 2025). That is their estimate in a paper arguing for small models, not a measured result for your workload.
Where do small models fall short?
Stored knowledge. Phi-3's authors write that the model "simply does not have the capacity to store too much factual knowledge", which shows in low scores on a trivia benchmark, and suggest augmenting it with a search engine (Abdin et al., 2024). The Phi-4-mini card repeats the warning (Microsoft). For a plant this matters less than it seems: the facts that count, such as torque values, alarm codes and procedure revisions, should come from your documents at question time anyway. See fine-tuning vs RAG.
Languages other than English. The Phi-4-mini card states that languages other than English "will experience worse performance". On the Open Arabic LLM Leaderboard v2, larger models tend to achieve higher average scores. Its authors note that 7B models in two Arabic-focused families did not improve between versions, and suggest the cause is the lower performance of smaller models on ALRAGE, a generative Arabic RAG task. That is close to what a document assistant does.
Long inputs. In the RULER benchmark, larger model size correlated with better long-context performance, and Yi-34B did significantly better than Yi-6B with the same training context (Hsieh et al., 2024). Small models may also have shorter native windows: the three smallest Qwen3 models offer 32K tokens (Qwen).
Which industrial tasks suit a small model?
The table is a starting point for your own tests, based on the strengths and limits above, not on benchmark results for these tasks.
| Task | Small model as first candidate? | What to check |
|---|---|---|
| Classifying maintenance reports by failure mode or criticality | Yes | Narrow, repetitive output; a fine-tuned adapter can help if prompting is not enough |
| Extracting fields from work orders, certificates or datasheets | Yes | The facts are in the input; check output format and scanned-document quality |
| Answering from retrieved procedure passages (RAG) | Often | Retrieval quality; test Arabic and French answers separately |
| Summarising short shift logs and incident notes | Often | Real input length against the model's context; mixed-language text |
| Plain-language questions on databases (text-to-SQL) | Test it | Schema size and query complexity; see text-to-SQL on production databases |
| Long reports drawing on many documents, or long drafting in Arabic | Larger model usually safer | Long-context and Arabic generative results above |
| Planning multi-step agent actions | Start larger, then test smaller models step by step | Each step's accuracy, with human approval for actions |
Narrow tasks are where small models gain most from fine-tuning. In LoRA Land, 310 models were fine-tuned on 31 tasks; 4-bit LoRA fine-tuned models outperformed their base models by 34 points and GPT-4 by 10 points on average (Zhao et al., 2024).
Can you combine small and large models?
Yes, and that is often the most economical design.
Routing. A router sends each request to a stronger or a weaker model. The RouteLLM authors report cost reductions of over 2 times in certain cases without compromising response quality (Ong et al., 2024).
Agents. The NVIDIA paper observes that, in agentic systems, language models "perform a small number of specialized tasks repetitively", and advocates systems "where SLMs are used by default and LLMs are invoked selectively and sparingly" (Belcak et al., 2025).
The trade-off. Every extra model is one more thing to evaluate, update, secure and keep in your inventory. Add a second model when your tests show a gain worth that work.
How do you decide, step by step?
- List the tasks separately. "A plant assistant" is several tasks: search, extraction, classification, summaries, database questions.
- Build a test set per task, with real questions from engineers in Arabic, French and English, reference answers and source passages. Our guide to evaluating an LLM on your documents sets out the method.
- Shortlist one small and one larger model that pass your licence and language checks.
- Run both at the precision you will deploy, with the same retrieval and prompts, at real input lengths.
- Keep the small model where it passes with margin. Where it fails, check whether better retrieval or a few examples in the prompt fix it before moving up.
- Size the hardware from the result, including KV cache for the expected peak.
- Re-run the tests whenever a new model version appears.
What we have seen at KABAS AI
DALILE runs on the client's own servers, offline if needed, and answers questions from technical documents, spreadsheets and databases through a set of specialised agents. Model size is therefore a planning decision in every deployment, because it sets the hardware a site must buy, power and cool. The figures in this article come from the public sources cited, not from benchmarks we ran.
The practical guidance that follows from them: keep the application independent of any one model, so a smaller or newer one can replace it after testing; evaluate each task on its own rather than the system as a whole; and let the client's own questions in Arabic, French and English decide between model sizes. For the wider infrastructure and governance picture, see our guide to sovereign, on-premise AI. If you want help building the test set and comparing model sizes on your documents, our AI consulting team can run it with you.
Sources
Model details checked in November 2026. They change with new releases; recheck before you decide.
- Belcak et al., Small Language Models are the Future of Agentic AI, arXiv 2506.02153, 2025
- Jiang et al., Mistral 7B, arXiv 2310.06825, 2023
- Abdin et al., Phi-3 Technical Report, arXiv 2404.14219, 2024
- Qwen, Qwen3: Think Deeper, Act Faster
- Qwen3-8B model card, Hugging Face
- Qwen3-8B config.json, Hugging Face
- Phi-4-mini-instruct model card, Hugging Face
- Gemma 3 4B model card, Hugging Face
- ALLaM-7B-Instruct-preview model card, Hugging Face
- DeepSeek-R1-Distill-Qwen-7B model card, Hugging Face
- Hugging Face: Optimizing LLMs for speed and memory
- NVIDIA: Mastering LLM Techniques, Inference Optimization
- NVIDIA RTX PRO 6000 Blackwell Server Edition
- NVIDIA L40S GPU
- Hugging Face, The Open Arabic LLM Leaderboard 2
- Hsieh et al., RULER: What's the Real Context Size of Your Long-Context Language Models?, 2024
- Zhao et al., LoRA Land: 310 Fine-tuned LLMs that Rival GPT-4, arXiv 2405.00732, 2024
- Ong et al., RouteLLM: Learning to Route LLMs with Preference Data, arXiv 2406.18665, 2024
Frequently asked questions
What is a small language model?
There is no official threshold. NVIDIA researchers define a small language model as one that fits on a common consumer device and answers fast enough for a single user, and count most models below 10 billion parameters as small. The line moves as hardware and models improve, so treat it as a guide rather than a standard.
Can a 7B model run without a data-centre GPU?
Often, yes. Following Hugging Face's rule of thumb, a 7-billion-parameter model needs about 14 GB for its weights at 16-bit precision and roughly a quarter of that at 4-bit. Microsoft reports that its 3.8-billion-parameter Phi-3-mini runs on a phone at 4-bit. A shared assistant for many users still needs memory for each request in progress, so load-test before choosing the hardware.
Do small models handle Arabic well?
It varies by model. Phi-4-mini lists Arabic and French but warns that languages other than English perform worse, while ALLaM-7B was trained specifically on Arabic and English. On the Open Arabic LLM Leaderboard, larger models tend to score higher, and the authors link weaker 7B results to a generative Arabic RAG task. Test candidates on your own Arabic documents.
Do we need to fine-tune a small model?
Not always. Start with a good prompt and retrieval over your documents, then fine-tune only if a narrow task still fails. When fine-tuning is justified, small models respond well: in one study, 4-bit LoRA fine-tuned models beat their base models by 34 points and GPT-4 by 10 points on average across 31 tasks.
Should we run one large model or several small ones?
One model is simpler to test, update and secure. Several models make sense when the workload splits into clear tasks of different difficulty, with a router or an agent design sending each request to the smallest model that handles it. Every extra model adds evaluation and maintenance work, so add them only when your tests show a gain.



