Table of Contents
When maintenance records are poor, treat the CMMS as a list of candidate events, not as ground truth. Rebuild each failure from several sources (work orders, shift logs, alarm journals, sensor trends), have engineers mark its onset and end on the data, record how confident each label is, and evaluate models per event, not per point.
Key takeaways
- Historical work orders frequently have missing or incorrect functional location, failure mode, maintenance action and status fields, but free text and knowledge of maintenance practice can repair many of them (Hodkiewicz and Ho, 2016).
- A work order date is not a failure onset. In ISO 14224 the failure date is the date of detection (Selvik, 2014), and initial degradation and initial detection "rarely coincide" (NASA, 2008).
- A period with no recorded failure is unlabelled, not healthy.
- Label errors change which model looks best: ten widely used benchmark datasets averaged at least 3.3% test label errors (Northcutt et al., 2021).
- Score models per failure event, with warning time and false alerts. Under point adjustment, even a random score can look state of the art (Kim et al., 2022).
Why are failure labels hard to get from plant records?
A failure model needs examples of failures, placed correctly in time on the sensor data. The usual record is the CMMS, designed to plan and track work, not to train models. Three problems recur:
- Missing or wrong fields. Studying heavy mobile equipment work orders from several organisations, Hodkiewicz and Ho found that historical records frequently contain missing or incorrect functional location, failure mode, maintenance action and status fields. Free text, knowledge of maintenance practice and the asset's life-cycle position made them fit for purpose, through a repeatable, transparent rule-based cleaning process (Hodkiewicz and Ho, 2016).
- Free text. NIST built its Nestor toolkit because maintenance work orders use specialised vocabulary and jargon that off-the-shelf language tools struggle with (NIST, Nestor). In Algerian plants the same note may mix French, Arabic and abbreviations; see documents that mix Arabic, French and English.
- Wrong time. A work order is dated when someone raised it, not when the fault started.
Failures are also rare, so every misplaced or missing one counts.
Why is the work order date not the failure time?
NASA's reliability-centred maintenance guide states that "the point of initial degradation and point of initial detection rarely coincide" (NASA, 2008). A bearing can degrade for weeks before anyone notices, and the work order may follow the alarm by days.
ISO 14224, the reliability and maintenance data standard for the petroleum, petrochemical and natural gas industries, reflects this. As presented by the technical editor of the ISO working group, the failure date is the "date of failure detection (year/month/day)", the impact on equipment function is classed as critical, degraded or incipient, and a separate field records the detection method (Selvik, 2014). A detection date suits reliability statistics, but a model trained on minute-level sensor data needs to know when the signal changed.
A time-series failure label therefore needs more than one timestamp:
| Label field | What it means | Typical source |
|---|---|---|
| Asset and tags | The equipment and the sensor tags that see the failure | CMMS functional location, tag list |
| Failure mode | What was observed, from a fixed list | ISO 14224-style code, or rebuilt from the text |
| Onset | First visible change in the sensor data | Engineer's review of the trend |
| Detection | When people noticed: alarm, round, inspection | Alarm journal, shift log, work order |
| Functional failure or trip | When the equipment stopped doing its job | Trip log, event journal, production records |
| Repair and return to service | Period to exclude from normal data | Work order dates, permits |
| Detection method | Alarm, condition monitoring, operator round, breakdown | Work order, inspection report |
| Confidence and source | How sure the reviewer is, and which records support the label | Reviewer's note |
The gap between onset and detection is what a predictive model tries to use, so it is the field to get right.
Where can labels come from when the CMMS is weak?
| Source | What it gives | Main weakness |
|---|---|---|
| CMMS work orders | Asset, dates, free text, sometimes a failure code | Codes missing or generic; paperwork dates |
| Shift logs and handover notes | What operators saw and when | Free text, often on paper or in spreadsheets |
| Alarm and event journal | Exact times of alarms and trips | Most alarms are not failures |
| Trip and incident reports | Confirmed cause of major events | Only for the largest events |
| Inspection and condition monitoring reports | Vibration, oil and thermography findings with dates | Periodic; stored as documents |
| The sensor data itself | The onset, as seen by an engineer | Needs expert time; judgement varies |
| Interviews with maintenance staff | Events never written down | Memory fades; dates are approximate |
A label is strong when two independent sources agree, such as a work order and a trip in the event journal, or a shift note and a visible change on the trend.
How do you rebuild failure labels, step by step?
Run the checks in industrial data quality problems first: a frozen transmitter or a loop test can look exactly like a failure. Then:
- Choose one equipment class and two or three failure modes that matter, such as seal leaks and bearing failures on a pump fleet. Write a one-line definition of each mode.
- Extract every work order on those assets for the period, including cancelled ones, with their free text.
- Classify the text with simple rules first: keyword lists per mode, such as "garniture", "fuite", "roulement" or "vibration", reviewed by a maintenance engineer. Nestor, for example, groups similar terms and spelling variants for quick review (NIST, Nestor).
- Match each candidate to other sources: alarms, trips and shift notes in a window around the work order date. Our guide to process historian data for machine learning covers the extraction.
- Mark onset and end on the trend. The open-source tool Label Studio, for example, has a template for labelling spans of multichannel time-series data loaded from CSV files (Label Studio).
- Record confidence and sources for every label, and keep rejected candidates with the reason.
- Have a second reviewer label a sample independently. Cohen's kappa measures agreement between two annotators; 0 means no agreement beyond chance and 1 complete agreement (scikit-learn). Low agreement usually means a failure mode definition is unclear.
- Write the rules down so the same process runs on next year's records.
Can weak supervision and active learning reduce the effort?
Both reduce the hours experts spend.
Weak supervision. Instead of labelling each example by hand, experts write labelling functions: heuristics such as "a pump trip within 48 hours of a work order mentioning 'garniture' is a seal failure". Snorkel combines such noisy rules without ground truth. In its user study, subject matter experts built models 2.8 times faster than with seven hours of hand labelling, with an average 45.5% better predictive performance (Ratner et al., 2017). Those results come from benchmark tasks, not plant data.
Active learning. The model chooses which examples the expert labels next, usually those it is least sure about. The key idea is that a learning algorithm can achieve greater accuracy with fewer labelled examples if it may choose its training data (Settles, 2009). In a plant, the engineer reviews the most ambiguous periods each week, not a random sample.
Unlabelled is not healthy. If some failures were never recorded, the "no failure" periods hide positives. Elkan and Noto showed that if the labelled positives are selected at random from all positives, a classifier trained on positive and unlabelled examples predicts probabilities that differ from the true ones by only a constant factor (Elkan and Noto, 2008). In plants, serious failures are recorded more often than minor ones, so the assumption rarely holds fully, but the lesson stands: an unrecorded period is not a negative.
What happens when some labels are wrong?
Label errors damage evaluation as much as training. Northcutt, Athalye and Mueller estimated an average of at least 3.3% label errors in the test sets of ten widely used image, text and audio datasets. On ImageNet, a 6% increase in the share of mislabelled test examples was enough for ResNet-18 to outperform ResNet-50 (Northcutt et al., 2021).
Time-series benchmarks are no exception. Wu and Keogh write that all the popular anomaly benchmarks they examined "appear to have mislabeled data, both false positives and false negatives" (Wu and Keogh, 2022).
For plant data, spend expert time on the test set first, with every test label double-reviewed, even if training labels stay rough.
How should you score a model on failure events?
Failures last for a period, so point-by-point accuracy misleads. Tatbul et al. note that many real-world anomalies are range-based, and extend precision and recall to ranges (Tatbul et al., 2018). The opposite mistake is common in papers: under point adjustment, if one moment in an anomaly segment is detected, the whole segment counts as detected. Kim et al. showed that with this protocol even a random anomaly score can look like a state-of-the-art method (Kim et al., 2022).
Score what maintenance planners care about:
| Measure | Question it answers |
|---|---|
| Events detected | Of the confirmed failures in the test period, how many raised an alert before functional failure? |
| Warning time | How long before the trip, or before people noticed, did the alert come? |
| False alerts per week | How many alerts matched no confirmed event? |
| Results by label confidence | Does performance hold on high-confidence labels alone? |
Watch for run-to-failure bias. Wu and Keogh found that many benchmark anomalies sit near the end of the series, so a naive method that flags the last point has an excellent chance of being correct. If your histories stop at the failure, check that the model has not simply learned "late in the record", and split by time: train on older periods, test on later ones.
What if there are almost no failures to label?
Do not start with a supervised failure classifier. Train a detector on normal operation and use the few labelled events to test it, as our anomaly detection methods guide describes, and pool identical equipment under similar conditions. Labels become essential later, for the fault classification and remaining-life models described in AI predictive maintenance in oil and gas, and for checking a live model, as covered in model drift monitoring.
How do you get better records from now on?
Labels are cheapest when captured at close-out, so change the work order:
- Make failure mode, detection method and failure impact pick lists, mapped to ISO 14224 where your codes allow.
- Add a field for when the problem was first noticed, with date and time.
- Link each work order to the tag, and to the alarm or trip when there is one.
- Review a sample of closed work orders each month with the planners.
An AI agent can flag orders closed without failure codes; see connecting an AI agent to a CMMS. Our oil and gas data readiness checklist includes a quick work order test.
What we have seen at KABAS AI
Our data science service covers data preparation, predictive models that anticipate failures, and MLOps pipelines that monitor them in production. Our oil and gas page lists predictive maintenance among its use cases, with DALILE connecting production data, maintenance records and equipment histories. We have not published a predictive maintenance project, so this is guidance, not a reported result.
At ADFC (Sonatrach Group), formulation experience was "dispersed in various formats and locations" before Virtual Lab structured past tests and validated recipes as reusable data. Failure history is usually scattered the same way. Our advice:
- Budget engineer time for labels in the project plan; it is the scarcest input.
- Build the test set first, double-reviewed, before any model.
- Keep records, labels and labelling tools inside your network, since work orders describe your assets and people.
Sources
- Hodkiewicz and Ho, Cleaning historical maintenance work order data for reliability analysis, Journal of Quality in Maintenance Engineering 22(2), 2016
- NIST, Nestor: toolkit for maintenance work order text
- Selvik, Collection and exchange of reliability and maintenance data for equipment (ISO 14224), ISO/TC 67/WG4, Rio de Janeiro, 2014
- NASA, Reliability-Centered Maintenance Guide for Facilities and Collateral Equipment, September 2008
- Label Studio, Time Series Labeling template
- scikit-learn, cohen_kappa_score
- Ratner et al., Snorkel: Rapid Training Data Creation with Weak Supervision, PVLDB 11(3), 2017 (arXiv 1711.10160)
- Settles, Active Learning Literature Survey, University of Wisconsin-Madison, 2009
- Elkan and Noto, Learning Classifiers from Only Positive and Unlabeled Data, KDD 2008
- Northcutt, Athalye and Mueller, Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks, NeurIPS Datasets and Benchmarks 2021 (arXiv 2103.14749)
- Wu and Keogh, Current Time Series Anomaly Detection Benchmarks are Flawed and are Creating the Illusion of Progress, ICDE 2022 (arXiv 2009.13807)
- Tatbul et al., Precision and Recall for Time Series, NeurIPS 2018 (arXiv 1803.03639)
- Kim et al., Towards a Rigorous Evaluation of Time-Series Anomaly Detection, AAAI 2022 (arXiv 2109.05257)
Frequently asked questions
How many failure examples do we need to train a failure prediction model?
There is no fixed number, and it depends on the failure mode and how varied the operating conditions are. With only a handful of confirmed events per mode, train an anomaly detector on normal operation and keep the events to test it. A supervised classifier for a specific failure mode needs enough reviewed examples of that mode across the conditions it will meet in service.
Can a language model read our work orders and assign failure codes?
It can propose codes from free text, including notes that mix French, Arabic and abbreviations, and that saves review time. Treat its output as candidate labels: compare it with a sample reviewed by engineers before trusting it, and keep the source text next to every proposed code. Work orders describe your assets and people, so run the model inside your own network.
How long should the pre-failure window be?
Set it per failure mode from the onsets engineers mark on the data, not from a rule of thumb. The window should start where the signal really changes and leave enough time for the maintenance team to act. Testing a few window lengths and comparing warning time and false alerts shows which one is useful.
Should periods without a recorded failure be labelled as healthy?
Not automatically. Some failures were never recorded, and repair periods, start-ups and tests also sit in those periods. Label as normal only the periods that operators and engineers have reviewed, exclude repairs and tests, and treat the rest as unlabelled.
Who should do the labelling?
Maintenance or reliability engineers and experienced operators decide what a failure is and when it started; a data scientist prepares the data, the tools and the checks. Have a second reviewer label a sample independently, and use their disagreements to sharpen the failure mode definitions.
Do we need to adopt ISO 14224 before we start labelling?
No. Its failure mode, failure cause, detection method and failure impact fields are a useful reference, and mapping your existing CMMS codes to them makes labels easier to reuse across sites. A first project can start from your own codes, as long as each failure mode is defined in writing.



