Table of Contents
To measure hallucination in an industrial assistant, split each answer into its factual claims and check every claim against the passage it cites. Count unsupported claims, citations that do not back their sentence, and wrong answers to questions the documents cannot answer. Check critical numbers in code, and keep measuring after go-live.
Key takeaways
- NIST defines confabulation, colloquially "hallucination", as confidently stated but erroneous or false content that can mislead users (NIST AI 600-1).
- Retrieval reduces hallucination but does not remove it: legal research tools built on retrieval each hallucinated between 17% and 33% of the time in a preregistered test (Magesh et al., 2024).
- Citations need checking too. In an audit of four generative search engines, only 51.5% of sentences were fully supported by their citations (Liu et al., 2023).
- Score "not found" separately from wrong answers. Plain accuracy rewards guessing over admitting uncertainty (Kalai et al., 2025).
- Measure at claim level, separate hallucination from retrieval failure, and treat any wrong critical value as blocking.
What counts as a hallucination in a plant assistant?
NIST's Generative AI Profile uses the term confabulation: "the production of confidently stated but erroneous or false content (known colloquially as 'hallucinations' or 'fabrications') by which users may be misled or deceived" (NIST AI 600-1).
A survey of hallucination in large language models splits the problem in two (Huang et al., 2023). A factuality hallucination conflicts with verifiable real-world facts. A faithfulness hallucination departs from the user's instruction or the supplied context, or contradicts itself. For an assistant that answers from your documents, faithfulness is what you measure first: the documents are the reference, even where the model's general knowledge disagrees.
In a plant, it usually looks like one of these:
- A value that is in no document, such as a plausible clearance or torque.
- A real value with the wrong unit, equipment tag or operating condition.
- A correct statement with a citation to a page that does not contain it.
- Steps merged from two procedures, or from a superseded revision.
- An answer to a question the documents cannot answer, instead of "not found".
Keep this apart from a retrieval error, where the right passage never reached the model. The two need different fixes, as our industrial AI glossary notes, and the RAG guide for maintenance manuals covers the retrieval side. Revision mix-ups are often a retrieval and metadata problem rather than a model one.
Does retrieval (RAG) stop hallucination?
No, and published audits show by how much it falls short.
Magesh et al. (2024) ran a preregistered evaluation of AI legal research tools whose providers had marketed retrieval as avoiding hallucination. On 202 queries, Lexis+ AI, Westlaw AI-Assisted Research and Ask Practical Law AI each hallucinated between 17% and 33% of the time. That was less than GPT-4 used as a general-purpose chatbot, but far from zero. Their definition is useful for a plant: a response counts as hallucinated if it is either incorrect or misgrounded, meaning it cites a source that does not support the claim. Their set also included false-premise questions, which have a direct plant equivalent: a question that assumes a procedure step or a part that does not exist.
Citations themselves can be wrong. Liu et al. (2023) audited Bing Chat, NeevaAI, perplexity.ai and YouChat and found that only 51.5% of generated sentences were fully supported by citations, and only 74.5% of citations supported their associated sentence. On the ALCE benchmark, even the best models lacked complete citation support 50% of the time on one dataset (Gao et al., 2023).
A citation shown next to an answer makes it checkable. It does not make it correct. Your measurement has to open the cited passage.
Which hallucination metrics should you track?
No single number covers it. The table lists the measures that matter for a document assistant and where each comes from.
| Metric | What it counts | How to compute it | Origin |
|---|---|---|---|
| Faithfulness | Share of an answer's claims supported by the retrieved passages | Split the answer into claims, check each against the context, divide supported by total (0 to 1) | Ragas |
| Citation support | Share of cited sentences that the cited passage actually supports | Check each sentence against its own citation, not the whole context | Liu et al., 2023; Gao et al., 2023 |
| Factual precision | Share of atomic facts that a reference source supports | Break long answers into atomic facts and verify each | Min et al., 2023 |
| Wrong-answer rate on unanswerable questions | How often the assistant invents an answer when the documents are silent | Grade each answer as correct, incorrect or not attempted | Wei et al., 2024 |
| Critical-value error rate | Wrong setpoints, torques, limits, units or tags | Exact match or numeric tolerance in code against the reference answer | Your own test set |
| Answer consistency | Disagreement between several samples of the same question | Sample the answer several times and compare | Manakul et al., 2023 |
Claim-level scoring matters because long answers mix right and wrong. FActScore was built for this: it breaks a generation into atomic facts and computes the percentage supported by a reliable knowledge source. On biographies, ChatGPT reached only 58% (Min et al., 2023). A whole-answer pass or fail grade would hide that one wrong fact among five right ones.
Consistency is a cheap warning signal, not a proof. SelfCheckGPT rests on the idea that if a model knows a fact, its sampled answers tend to agree, while invented facts vary between samples (Manakul et al., 2023).
Why does a single accuracy score hide hallucination?
Because it treats "I don't know" as a failure, it pays a model to guess. Kalai and colleagues argue that language models hallucinate because training and evaluation procedures reward guessing over acknowledging uncertainty, and that the fix is to change how existing benchmarks are scored (Kalai et al., 2025).
The SimpleQA benchmark shows the alternative: each answer is graded as correct, incorrect or not attempted (Wei et al., 2024). Use the same three outcomes on your plant test set:
| Outcome | Answerable question | Unanswerable question |
|---|---|---|
| Correct | Right value, right source | Says the documents do not contain it |
| Not attempted | Says "not found" although the answer exists | (not applicable) |
| Incorrect | Wrong value or unsupported claim | Invents an answer |
Report the incorrect rate on its own. For a technician, a "not found" costs a look in the manual; an invented isolation step can cost much more. Decide with the plant which trade-off is acceptable before you compare models.
How do you build a hallucination test set?
Start from the general method in our guide to evaluating an LLM on your own documents, then add what hallucination needs:
- Label claims, not just answers. For a sample of answers, engineers mark each factual claim as supported, unsupported or contradicted by the cited passage. The RAGTruth corpus shows the format: nearly 18,000 RAG responses annotated by hand at word level (Niu et al., 2023).
- Add unanswerable and false-premise questions. Ask about a value that is in no document, or about a pump that the unit does not have.
- Mark critical questions. Setpoints, trip values, torques, chemical compatibilities and isolation steps. One wrong answer here blocks go-live, whatever the average.
- Test with the correct passage supplied as well as end to end. If the model invents details with the right passage in front of it, that is a model or prompt problem; if it only fails end to end, look at retrieval first.
- Run each question more than once. Answers vary between runs; resampling answers is one of the variance-reduction steps Miller (2024) recommends.
- Report error margins and compare models question by question. Miller recommends standard errors, clustered errors when questions come in related groups, and paired differences when two models answer the same questions.
Who grades the answers: code, a judge model or engineers?
All three, each where it is strongest.
Code checks what can be checked exactly: every number in the answer must appear in the cited passage with the same unit, and every cited document and page must exist and be current.
A judge model scores free-text claims at scale. Ragas computes faithfulness with an LLM, and its documentation also offers Vectara's HHEM-2.1-Open, a small open T5 classifier trained to detect hallucinations, for the claim-checking step (Ragas). The same model family drives Vectara's public leaderboard, which measures how often models add unsupported content when asked to summarise a passage using only that passage (Vectara). That ranking is a summarisation test, not a test of your documents. RAGTruth's authors also found that a relatively small fine-tuned model can reach competitive hallucination detection compared with prompting GPT-4 (Niu et al., 2023), which matters when the checker must run on-premise.
Engineers grade critical questions and a sample of the rest, and calibrate the judge. Strong LLM judges agree with human preferences over 80% of the time but show position, verbosity and self-enhancement biases (Zheng et al., 2023). Measure how often the judge and the engineers agree before trusting it.
How do you keep measuring after go-live?
A test set measures the system you built; production shows the questions people actually ask.
- Run the code checks on every answer: numbers, units, tags and citations against the cited passages. Flag failures instead of showing them silently.
- Have engineers review a sample of real answers each week, using the same claim labels as the test set.
- Let users report a wrong answer with one click, and route reports to the document owner.
- Re-run the full test set after every change of model, quantization, prompt, documents or index, as described in keeping an on-premise LLM up to date. Track the trend alongside other model metrics; see model drift monitoring.
Measurement tells you where to act. Vendor guidance on reducing hallucinations lists allowing the model to say it does not know, grounding answers in direct quotes, and having it retract any claim it cannot support with a quote, and notes that these techniques reduce hallucinations without eliminating them (Anthropic documentation). Fine-tuning is not a fix for missing knowledge; see fine-tuning vs RAG. When the assistant drafts actions, a person should approve them: see human-in-the-loop AI agents.
What we have seen at KABAS AI
DALILE runs fully on-premise and answers with exact sources. That makes the method above practical: every claim can be checked against its cited passage, and the test set, the judge model and the logs stay on the same servers as the documents.
When we scope an assistant with a client, our advice is:
- Agree with the engineers who will use it what counts as a hallucination and which questions are critical, before choosing a model.
- Score "not found" and "wrong" separately, and agree the acceptable number of wrong answers on critical questions, which should normally be zero.
- Keep hallucination checks in code wherever the answer is a number, a unit or a document reference.
- Keep the test set, the labels and the results on the client's infrastructure and under the client's ownership.
If your assistant will also query or act on plant systems, our agentic AI services cover design and deployment.
Sources
- NIST, AI 600-1: Artificial Intelligence Risk Management Framework: Generative Artificial Intelligence Profile (July 2024)
- Huang et al., A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questions (arXiv, 2023)
- Magesh et al., Hallucination-Free? Assessing the Reliability of Leading AI Legal Research Tools (arXiv, 2024)
- Liu, Zhang and Liang, Evaluating Verifiability in Generative Search Engines (arXiv, 2023)
- Gao et al., Enabling Large Language Models to Generate Text with Citations (arXiv, 2023)
- Min et al., FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation (arXiv, 2023)
- Wei et al., Measuring short-form factuality in large language models (arXiv, 2024)
- Kalai et al., Why Language Models Hallucinate (arXiv, 2025)
- Manakul, Liusie and Gales, SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models (arXiv, 2023)
- Niu et al., RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models (arXiv, 2023)
- Ragas documentation, Faithfulness
- Vectara, Hallucination Leaderboard (GitHub)
- Zheng et al., Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena (arXiv, 2023)
- Miller, Adding Error Bars to Evals: A Statistical Approach to Language Model Evaluations (arXiv, 2024)
- Anthropic documentation, Reduce hallucinations
Frequently asked questions
What is a hallucination in an industrial AI assistant?
It is a confidently stated answer that is false or not supported by the plant's documents, such as an invented torque value, a wrong unit, or a citation to a page that does not say what the answer claims. NIST calls this confabulation. It is different from a retrieval error, where the right passage was never found.
Does RAG stop an assistant from hallucinating?
It reduces hallucination but does not remove it. In a preregistered study of legal research tools built on retrieval, each tool still hallucinated between 17% and 33% of the time. Grounding answers in documents makes hallucinations easier to detect, because each claim can be checked against its cited passage.
What is the most useful single metric for hallucination?
There is no single one. Track the share of answer claims supported by the retrieved passages (faithfulness), the share of citations that actually support their sentence, and the rate of wrong answers on questions the documents cannot answer. On critical values such as setpoints and torques, check every number in code.
Can an LLM judge detect hallucinations reliably?
It can help, but it must be checked against engineers first. Strong LLM judges agree with human preferences over 80% of the time in published tests, yet show position, verbosity and self-enhancement biases. Small open classifiers such as HHEM-2.1-Open can also score claims and run on your own servers.
How often should hallucination be measured?
Before go-live on a fixed test set, after every change of model, quantization, prompt, documents or index, and continuously in production on a sample of real answers reviewed by engineers. Keep the results of each run so a regression is visible.



