Table of Contents
A RAG assistant for maintenance manuals retrieves the relevant passages from your own documents and has a language model answer from them, citing the source page. Most of the work is in the documents, not the model: OCR of scans, structure-aware chunking, hybrid search for part numbers, revision control, access control and a test set built from real technician questions.
Key takeaways
- Retrieval quality sets the ceiling on answer quality. In Anthropic's published tests, combining contextual embeddings, keyword (BM25) search and reranking cut the top-20 retrieval failure rate from 5.7% to 1.9% (Anthropic).
- Vector search alone misses exact strings such as part numbers and fault codes. Pair it with keyword search.
- Scanned Arabic documents are still hard to read automatically: on the KITAB-Bench benchmark, the best model tested reached only 65% accuracy converting Arabic PDFs to Markdown (Heakl et al., 2025).
- Superseded revisions are a safety problem, not only a search problem. Index revision status and filter obsolete documents out by default.
- Access control has to be applied at retrieval time, before any text reaches the model (OWASP LLM08:2025).
What is RAG, in one paragraph?
Retrieval-augmented generation (RAG) combines a language model with a searchable index of documents. The term comes from a 2020 paper by Lewis et al., which paired a text generator with a dense vector index of Wikipedia. In a plant, the index holds manuals, procedures, work-order histories and inspection reports. When a technician asks a question, the system retrieves the most relevant passages and instructs the model to answer only from them, with citations. The model is not retrained; it reads your documents at question time, so updates mean re-indexing and answers can be checked against the page.
What does the pipeline look like?
A production pipeline has eight stages. Build and test each one separately.
| Stage | What it does | Typical failure on maintenance documents |
|---|---|---|
| Ingestion | Collects files from the document management system, file shares and CMMS exports | Duplicate copies, missing revision metadata |
| OCR | Turns scanned pages into text | Arabic script and numerals misread, tables flattened |
| Parsing and chunking | Splits documents into passages along their structure | A torque table cut in half, a warning separated from its step |
| Embeddings | Converts each chunk into a vector | Model weak in Arabic, French or technical vocabulary |
| Retrieval | Finds candidate chunks for a question | Exact part numbers and fault codes missed |
| Reranking | Reorders candidates by relevance | Skipped to save latency, so noise reaches the model |
| Generation | Writes the answer from the retrieved chunks | Mixes sources or fills gaps from the model's memory |
| Citations | Links each statement to document, revision and page | Points to a whole document instead of the passage |
Ingestion: start with an inventory
List the sources and their owners: OEM manuals, internal procedures, P&IDs, maintenance reports, HSE instructions. For each, record the format (native PDF, scan, Word, spreadsheet), the language, and who decides which revision is current.
OCR: treat scans as their own project
Many plants hold manuals as scanned PDFs. Open-source engines such as Tesseract ship trained data for Arabic, French and English (Tesseract documentation), but Arabic remains difficult. The KITAB-Bench authors point to Arabic's cursive script, right-to-left flow and complex typography, and list numeral recognition errors and table structure detection among the recurring failures (Heakl et al., 2025). Numerals and tables are where maintenance values live, so check OCR samples by hand and link every chunk to its page image.
Chunking: follow the document's structure
Fixed-size chunks cut through tables and procedures. Structure-aware parsers do better. Docling, started at IBM Research, recovers page layout, reading order and table structure, offers OCR for scans, can run locally including in air-gapped environments, and is MIT-licensed. Its hybrid chunker splits along document elements, merges small neighbouring pieces that share the same headings, and by default repeats the table header in each piece when a table has to be split (Docling chunking documentation).
Also keep each warning in the same chunk as its step, and store the heading path (for example "Pump P-101 / Bearing replacement") as metadata. Anthropic suggests chunks of "usually no more than a few hundred tokens" and notes that chunk size, boundaries and overlap all affect retrieval (Anthropic). Treat chunk size as a parameter you tune against your test set.
Embeddings and retrieval: use hybrid search
Embeddings capture meaning ("bearing overheating" finds "high bearing temperature") but are weaker on exact strings. Anthropic gives the example of a query for the error code "TS-999": embeddings may return general text about error codes, while BM25 keyword search finds the exact match (Anthropic). Part numbers, equipment tags and alarm codes behave the same way, so run both searches and merge the results.
For a multilingual corpus, choose an embedding model that covers your languages. The open BGE-M3 model states support for more than 100 languages, produces dense, sparse (lexical) and multi-vector representations from one model, accepts inputs of up to 8,192 tokens, and its authors recommend hybrid retrieval followed by a reranker (BAAI model card). Test candidates on your own Arabic and French questions, not general leaderboards.
Reranking: filter before you generate
A reranker, typically a cross-encoder that reads the question and each candidate together, reorders the retrieved candidates so only the best few reach the model. In Anthropic's experiments, adding reranking on top of contextual embeddings and contextual BM25 took the reduction in retrieval failures from 49% to 67% (Anthropic). Sending fewer, better passages also helps generation: research on long contexts found that models use information at the beginning or end of the input more reliably than information in the middle (Liu et al., "Lost in the Middle").
Generation and citations: make every answer checkable
Instruct the model to answer only from the retrieved passages, to say plainly when the documents do not contain the answer, and to cite document number, revision and page for every value. Show the cited passage next to the answer. A torque value with its manual, revision and page can be verified in seconds; a bare number cannot be trusted.
Why do maintenance documents break a basic RAG setup?
Tables and diagrams. Clearances, torque values and lubrication schedules sit in tables. If parsing flattens a table into a line of numbers, the model cannot tell which value belongs to which bolt. Keep tables as tables and index the caption with them. For exploded views, index at least the figure caption and the adjacent parts list, and link to the page image.
Part numbers and codes. Beyond keyword search, normalise variants (with and without dashes, leading zeros) and extract identifiers into a metadata field so users can filter by equipment or part.
Mixed Arabic, French and English. An OEM manual may be in English, the site procedure in French and the shift report in Arabic. A question asked in Arabic must still find the French procedure, which calls for a cross-lingual embedding model, keyword search that handles both scripts, and answers in the user's language citing the original source. More on the language side in building LLMs that understand Arabic industrial terminology.
Revisions and superseded documents. This is the pitfall with real safety consequences. If revision B and revision D of the same procedure are both indexed, the retriever can return the obsolete one. Store each document's number, revision, effective date and status (current, superseded, draft) as metadata, filter to current documents by default, and when a new revision is approved, mark the old one superseded in the same step. Show superseded revisions only on explicit request, for example in an incident investigation, and label them clearly.
How do you evaluate the answers?
Build the test set before you build the system. Collect real questions from technicians, planners and engineers, record the correct answer and the passage that contains it, and include questions whose answer is not in the documents at all. Then measure retrieval and generation separately.
| What to measure | Question it answers | How |
|---|---|---|
| Retrieval recall | Did the right passage appear in the top results? | Share of test questions whose source passage is in the top k, the metric Anthropic reports as recall at 20 |
| Faithfulness | Is every claim in the answer supported by the retrieved text? | Ragas scores it from 0 to 1 as supported claims divided by total claims |
| Context precision and recall | Were the retrieved passages relevant and complete? | Ragas metrics of the same names |
| Refusal behaviour | Does the system say "not found" when the documents are silent? | Unanswerable questions in the test set |
| Expert review | Are safety-relevant values right? | A maintenance engineer reads a sample of answers |
The Ragas framework was designed to run without ground-truth human annotations, and its documentation defines faithfulness as the proportion of an answer's claims that the retrieved context supports. Automated scores help compare versions; they do not replace an engineer checking safety-relevant values. Re-run the test set after every change to OCR, chunking, models or prompts.
How do you control who sees what?
OWASP lists vector and embedding weaknesses as LLM08 in its 2025 Top 10 for LLM applications. It warns that inadequate or misaligned access controls can expose embeddings containing sensitive information, and recommends fine-grained access controls, permission-aware vector stores and strict partitioning of datasets (OWASP LLM08:2025).
The common pattern: at ingestion, copy the list of groups allowed to read each document from the source system onto every chunk; at query time, filter retrieval by the authenticated user's groups. Microsoft documents this "security filter" pattern for Azure AI Search and is explicit that the principal is just a string used in a filter, with no authentication or authorisation behind it (Microsoft Learn). The same holds for any vector database: identity must come from server-side authentication, never from the client, and permissions must be re-synced when they change at the source.
Question logs and maintenance reports also contain personal data, such as the names of technicians. In Algeria, Law 18-07 of 10 June 2018, amended by Law 25-11 of 24 July 2025, governs the processing of personal data. According to law firm Gide, processing must be notified to the national authority (ANPDP) in advance, and transfers of personal data abroad generally require its prior authorisation (Gide). This is one argument for keeping the whole pipeline, including logs, on your own infrastructure. Check how these rules apply to your case with legal counsel. Our guide to sovereign, on-premise AI covers the deployment side.
Implementation checklist
- Inventory document sources, owners, formats, languages and who approves revisions.
- Pick one equipment family or one unit for the pilot, not the whole plant.
- Write the test set from real questions before building anything.
- Run OCR on a sample of scans and check numerals and tables by hand.
- Parse with a structure-aware tool; keep tables and warnings intact.
- Tag every chunk with document number, revision, status, language, equipment tags, page and access groups.
- Index with both vector and keyword search; test cross-language questions.
- Add a reranker and send only the top few passages to the model.
- Require citations and a "not found" answer when the documents are silent.
- Filter by user permissions at retrieval time, using server-side identity.
- Exclude superseded revisions by default and tie re-indexing to document approval.
- Re-run the test set on every change and have an engineer review a sample of answers regularly.
Where does DALILE fit?
DALILE is KABAS AI's on-premise large language model for industrial enterprises. It runs fully on-site and supports Arabic, French and English, so documents, embeddings and logs stay on your infrastructure. ADFC (Sonatrach Group) and ENAD Shymeca, two clients that use Virtual Lab, are exploring integration with DALILE for knowledge discovery and technical documentation. To see how a document assistant differs from an agent that acts on systems, read chatbot vs copilot vs AI agent; for the product overview, see the DALILE guide. If you run oil and gas or manufacturing sites and want to scope a pilot on your own manuals, start from the DALILE page.
Sources
- Lewis et al., Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks (arXiv, 2020)
- Anthropic, Introducing Contextual Retrieval (September 2024)
- Heakl et al., KITAB-Bench: A Comprehensive Multi-Domain Benchmark for Arabic OCR and Document Understanding (arXiv, 2025)
- Tesseract documentation, traineddata files for different versions
- Docling on GitHub
- Docling documentation, Chunking
- BAAI, BGE-M3 model card (Hugging Face)
- Liu et al., Lost in the Middle: How Language Models Use Long Contexts (arXiv, 2023)
- Es et al., Ragas: Automated Evaluation of Retrieval Augmented Generation (arXiv, 2023)
- Ragas documentation, Faithfulness
- OWASP, LLM08:2025 Vector and Embedding Weaknesses
- Microsoft Learn, Security filter pattern in Azure AI Search
- Gide, Protection of personal data: focus on Algerian regulations
Frequently asked questions
What is RAG for technical documentation?
Retrieval-augmented generation (RAG) is a setup where a language model answers questions from passages retrieved from your own documents, rather than from what it memorised during training. For technical documentation, the index holds manuals, procedures and maintenance reports, and each answer cites the document, revision and page it came from.
Do I need to fine-tune a model on my maintenance manuals?
Usually not as a first step. RAG reads your documents at question time, so updating knowledge means re-indexing a new revision rather than retraining a model. Fine-tuning can help later with vocabulary or answer style, but it does not give you citations or revision control.
Why does my RAG system miss part numbers and fault codes?
Embedding-based search captures meaning but is weak on exact strings such as part numbers, tag numbers and error codes. Add keyword search (BM25) alongside vector search and merge the results, and extract identifiers into a metadata field you can filter on.
Can RAG work with scanned manuals in Arabic and French?
Yes, but OCR quality decides the outcome. Arabic OCR remains difficult, especially for numerals and tables, so sample the OCR output by hand before indexing and keep a link from every chunk to the original page image so users can check values.
How do I stop the assistant from quoting an obsolete procedure?
Store each document's number, revision, effective date and status as metadata, and filter retrieval to current documents by default. When a new revision is approved, index it and mark the old one as superseded in the same step.
Should a RAG system for maintenance documents run on-premise?
It depends on how sensitive the documents and logs are and on the rules that apply to you. Keeping the whole pipeline on your own infrastructure means documents, embeddings, prompts and logs stay under your control, which simplifies access control and data protection questions.



