Table of Contents
To evaluate an LLM on your own documents, write a test set of real questions from your users, each with a reference answer and the passage that supports it. Then run every candidate model through the same pipeline and grade the answers with code, engineers and a calibrated LLM judge. Public benchmark scores do not replace this step.
Key takeaways
- Public scores can be inflated by contamination: on a new set of grade-school maths problems built to mirror the public GSM8k benchmark, researchers observed accuracy drops of up to 8% (Zhang et al., 2024).
- Test the model and the retrieval separately: first with the correct passage supplied, then end to end.
- Small test sets have wide error margins. With 100 questions and a score of 80%, the 95% confidence interval is about ±8 points, using the formula recommended by Miller (2024).
- Strong LLM judges agree with human preferences over 80% of the time but have position, verbosity and self-enhancement biases (Zheng et al., 2023). Calibrate a judge against engineers' grades before trusting it.
- Evaluate the exact build you will deploy: same quantization, inference server, prompt and context length.
Why are public benchmark scores not enough?
Leaderboards rank models on public question sets. Three problems appear when you carry those scores over to a plant.
Contamination. Benchmark questions leak into training data. The authors of GSM1k wrote new grade-school maths problems designed to mirror GSM8k and found that several model families showed signs of systematic overfitting, with accuracy drops of up to 8%. They suggest some models may have partly memorised GSM8k (Zhang et al., 2024). Your maintenance manuals were never on a leaderboard, which is what makes them a fair test.
Wrong domain and language. ArabicMMLU contains 14,575 multiple-choice questions in Modern Standard Arabic, taken from school exams across North Africa, the Levant and the Gulf. In 2024, the best Arabic-centric model tested scored 62.3% (Koto et al., 2024). It is a serious benchmark, but school exams say little about whether a model can read a French pump datasheet and answer an Arabic question about it.
Claimed versus usable context. The RULER benchmark tested models that all claim a context of 32K tokens or more. Only half kept satisfactory performance at 32K, and almost all degraded as the context grew (Hsieh et al., 2024). If your procedures are long, test with their real length.
Use public scores to build a shortlist, and your own documents to decide. Our guide to choosing an open-weight LLM covers the shortlist.
What exactly are you evaluating?
A document assistant is a system: OCR, retrieval, model and prompt. To know which part failed, test at three levels.
| Level | What the model receives | What it tells you |
|---|---|---|
| Closed book | The question only | What the model knows on its own, and how readily it invents answers |
| Correct passage supplied | The question plus the passage that holds the answer | Whether the model can read, reason and answer when retrieval is perfect |
| End to end | The question, through your real retrieval pipeline | What users will actually get |
The gap between the last two levels is your retrieval loss. If a model fails with the right passage in front of it, better retrieval will not save it. If it succeeds with the passage supplied but fails end to end, fix OCR, chunking or search before changing models; our guide to RAG for maintenance manuals covers that pipeline. If the facts are right but the form of the answer is wrong, that is where fine-tuning may help.
How do you build a test set from your own documents?
- Collect real questions. Ask technicians, process engineers, planners and HSE staff for questions they actually asked recently, in the language they asked them. Generated questions can fill gaps later; check them by hand, because they tend to reuse the document's own wording.
- Record the reference answer and its source. For each question, an engineer writes the correct answer with the document number, revision and page. The source lets you score retrieval separately.
- Cover the question types that break systems (see the table below).
- Include unanswerable questions. The correct response is "not found in the documents", not a plausible guess.
- Mark critical questions. Torque values, setpoints, isolation steps, chemical compatibilities. One wrong answer there matters more than the average score.
- Freeze and version the set. Keep a held-out part that nobody tunes prompts against, and store the set with the list of document revisions it was written from.
- Keep it on your infrastructure. A good test set is a summary of your plant's know-how.
Anthropic's evaluation guidance makes the same points in general form: design evaluations that mirror the real task distribution, include edge cases, and structure questions so they can be graded automatically (Anthropic).
| Question type | Plant example | What it tests |
|---|---|---|
| Single fact | "What is the coupling bolt torque on pump P-101?" | Retrieval and exact reading of a value |
| Table lookup | "What is the gearbox oil change interval at high ambient temperature?" | Table parsing |
| Exact identifier | "What does alarm code TS-104 mean?" | Search on codes and part numbers |
| Several documents | "Does the site procedure use the same clearance as the OEM manual?" | Combining and comparing sources |
| Cross-language | An Arabic question whose answer is in a French procedure | Multilingual retrieval and answer language |
| Revision-sensitive | "What are the current isolation steps for the compressor?" when two revisions exist | Filtering of superseded documents |
| Unanswerable | A value that is in no document | Saying "not found" instead of inventing |
Mixed Arabic, French and English corpora deserve their own share of the set; see handling multilingual documents.
How many questions do you need?
OpenAI's accuracy guide treats 20 or more questions with ground-truth answers as the starting point for a baseline (OpenAI). That shows gross failures, but cannot rank two good models.
Every score has an error margin. Miller recommends reporting the standard error with every score, and a 95% confidence interval of the mean plus or minus 1.96 standard errors (Miller, 2024). For a pass or fail score p on n independent questions, the standard error is √(p(1−p)/n), which gives:
| Questions | Score | 95% confidence interval (approx.) |
|---|---|---|
| 50 | 80% | ±11 points |
| 100 | 80% | ±8 points |
| 300 | 80% | ±4.5 points |
| 1,000 | 80% | ±2.5 points |
Two refinements from the same paper. Questions written from the same passage are not independent, and clustered standard errors can be over three times larger than naive ones. When two models answer the same questions, compare them question by question (paired differences), which reduces noise. In Miller's worked example, detecting a 3-point difference with 80% power at 5% significance needs about 969 questions, and he concludes that new evaluations should contain at least 1,000.
In practice, a few hundred questions show large differences and failure patterns; small gaps between two strong models are noise. Break the results down by question type.
How do you grade the answers?
Anthropic ranks grading methods this way: code-based grading is the fastest and most reliable but lacks nuance; human grading is the most flexible but slow and expensive; LLM-based grading is fast and flexible but should be tested for reliability before it is scaled (Anthropic).
| Grading method | Use it for | Watch out for |
|---|---|---|
| Code (exact match, string match, numeric tolerance) | Values, units, codes, part numbers, "not found" replies | Formatting variants such as "25 N·m" and "25 Nm" |
| Engineer review | Critical questions, a sample of the rest, calibrating the judge | Time, and disagreement between reviewers |
| LLM judge | Free-text answers, faithfulness to the cited passage | Known biases; it must run on-premise for confidential documents |
LLM judges are useful but biased. In the MT-Bench study, GPT-4 as a judge reached over 80% agreement with human preferences, the same level humans reach with each other, while showing position, verbosity and self-enhancement biases (Zheng et al., 2023). In another study, simply changing the order in which two answers were shown let Vicuna-13B beat ChatGPT on 66 of 80 queries with ChatGPT as the judge (Wang et al., 2023).
Practical rules for a judge:
- Give it a detailed rubric and the reference answer, and ask for a clear verdict such as correct or incorrect, or a 1 to 5 score, as Anthropic recommends.
- Use a different model as the judge from the one under test; Anthropic's examples call this generally best practice.
- Have engineers grade a sample of answers, run the judge on the same sample, and check how often they agree before trusting the judge on the rest.
- Measure faithfulness separately. Ragas defines it as the share of an answer's claims supported by the retrieved context, from 0 to 1 (Ragas). For retrieval, Ragas context recall includes an ID-based variant that compares retrieved passage IDs with reference IDs without calling an LLM (Ragas).
The judge sees the same data as the model under test. A judge reached through a foreign cloud API sends confidential documents out with every grading call.
Why do results change between the test and production?
Non-determinism. Thinking Machines Lab sampled the same prompt 1,000 times at temperature 0 on Qwen3-235B and received 80 different completions. They trace this mainly to server load changing the batch size (He, 2025). Run each question more than once, or at least re-run the failures, before drawing conclusions.
Quantization. A study of over 500,000 evaluations found FP8 effectively lossless across model scales and well-tuned INT8 losing only 1 to 3% accuracy (Kurtic et al., 2024). Those are results on public benchmarks; test the quantized build you will actually serve. Hardware choices are covered in on-premise LLM hardware and GPU sizing.
Context and prompt. Test with the number of passages and document lengths you will use in production. A new system prompt, chat template or retrieval setting makes a new system: re-run the test set.
How do you compare two candidate models fairly?
- Use the same test set, retrieval index, prompt (adapted only to each model's chat format) and generation settings.
- Run all three levels: closed book, correct passage supplied, end to end.
- Grade with code first, then the judge, then engineer review of critical and disputed answers.
- Report scores by question type with confidence intervals, and the paired difference between models.
- Treat any wrong answer on a critical question as a blocking finding, whatever the average.
- Record model version, quantization, prompt version and index version with every run, so it can be repeated.
For standard benchmarks on local candidates, EleutherAI's MIT-licensed lm-evaluation-harness implements over 60 academic benchmarks and supports models served with vLLM or loaded with Hugging Face transformers (EleutherAI). It complements your test set; it does not replace it.
What we have seen at KABAS AI
DALILE runs fully on-premise and answers with exact sources. That makes the method above practical: each answer can be checked against its cited page, and the test set, the judge model and the logs can stay on the same servers as the documents.
When we scope a document assistant with a client, our advice is:
- Write the test set with the engineers who will use the assistant, before choosing a model.
- Agree in advance which questions are critical, and which failures block go-live.
- Keep the test set, the results and any judge model on the client's infrastructure and under the client's ownership.
- Re-run the full set after every change of model, quantization, prompt, documents or index, and keep the history.
If your assistant will also act on plant systems, our agentic AI services cover design and deployment, and human-in-the-loop approval covers which actions still need a person.
Sources
- Zhang et al., A Careful Examination of Large Language Model Performance on Grade School Arithmetic (arXiv, 2024)
- Koto et al., ArabicMMLU: Assessing Massive Multitask Language Understanding in Arabic (arXiv, 2024)
- Hsieh et al., RULER: What's the Real Context Size of Your Long-Context Language Models? (arXiv, 2024)
- Miller, Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (arXiv, 2024)
- OpenAI, Optimizing LLM accuracy
- Anthropic, Define success criteria and build evaluations
- Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv, 2023)
- Wang et al., Large Language Models are not Fair Evaluators (arXiv, 2023)
- Ragas documentation, Faithfulness
- Ragas documentation, Context Recall
- He and Thinking Machines Lab, Defeating Nondeterminism in LLM Inference (September 2025)
- Kurtic et al., "Give Me BF16 or Give Me Death"? Accuracy-Performance Trade-Offs in LLM Quantization (arXiv, 2024)
- EleutherAI, lm-evaluation-harness (GitHub)
Frequently asked questions
How many test questions do I need to evaluate an LLM?
About 20 questions with known answers are enough for a first baseline and to spot gross failures. To compare two good models you need far more: with 100 questions and a score of 80%, the 95% confidence interval is about plus or minus 8 points. One statistical worked example needs about 1,000 questions to detect a 3-point difference.
Can I use an LLM to grade the answers?
Yes, for free-text answers, but only after checking it. In published tests, strong LLM judges agreed with human preferences over 80% of the time, yet showed position, verbosity and self-enhancement biases. Have engineers grade a sample, compare the judge with them, and keep exact checks in code for values, units and codes.
Should the test set include questions the documents cannot answer?
Yes. A plant assistant must say when the documents do not contain the answer instead of inventing one. Include questions whose answer is in no document and count how often the system correctly replies that it found nothing.
Can we run the whole evaluation on-premise?
Yes. Open-source tools such as EleutherAI's lm-evaluation-harness can run models served locally, and the judge model can be an open-weight model on your own servers. If the judge is a foreign cloud API, your documents and answers leave your network with every grading call.
How often should we re-run the evaluation?
After every change that can affect answers: a new model or model version, a different quantization, a new prompt or chat template, new documents or revisions, or changes to OCR, chunking and retrieval. Keep the results of each run so you can see regressions over time.



