Table of Contents
For most industrial sensor data, start with multivariate statistical monitoring (PCA with Hotelling's T² and SPE) on top of the univariate limits you already have. Isolation forest and local outlier factor suit data with many features; autoencoders and LSTM models suit complex time patterns, but cost more and in a large 2022 comparison were not more accurate.
Key takeaways
- Industrial detectors are usually trained on normal operation and flag what differs, which scikit-learn calls novelty detection. You need labelled events to test a detector, not to build it.
- Limits on single tags miss unusual combinations. A PCA model watches all variables with two statistics, and a high SPE means an observation breaks the correlation structure.
- In a 2022 evaluation of 71 algorithms on 976 time series, no family was a clear winner and deep learning methods were not yet competitive.
- False alarms decide whether operators keep using a detector. A 3-sigma chart raises a false alarm every 371 points on average, per variable.
- Model each operating mode separately, and keep the detector advisory, with no write access to the control system.
What counts as an anomaly in plant data?
An anomaly is data that does not look like normal operation. In a plant, it comes from the instrument (a stuck or drifting transmitter), the process (an upset, a feed change) or the equipment (a bearing starting to fail). The detector cannot tell which; it only says that something has changed.
The 2022 evaluation by Schmidl, Wenig and Papenbrock illustrates three shapes of anomaly (Schmidl et al., 2022):
- Point anomaly: a single extreme value, such as a spike.
- Subsequence anomaly: a stretch of data with an unusual shape, even if each value is in range, such as an oscillation appearing in a level loop.
- Correlation anomaly: variables that stop moving together, such as an outlet temperature that no longer follows the steam flow that usually drives it.
No single-tag limit can see a correlation anomaly. The same study concludes that a multivariate algorithm is necessary to detect anomalies in the correlation between series.
What data do you train on: outliers or normal operation?
The scikit-learn documentation separates two settings (scikit-learn). In outlier detection, the training data contains outliers, and the method looks for the dense region while ignoring them. In novelty detection, the training data is "not polluted by outliers", and the question is whether a new observation differs from it.
Plant monitoring is usually novelty detection. Select a period of normal operation, check it with operators and the shift log, train on it, then score new data as it arrives. No labelled failures are needed to build the model, which helps, because most plants have few well-documented ones. Supervised methods are rare: only 7 of the 158 methods collected by Schmidl et al. were supervised, and in their tests supervised algorithms did not achieve better results than semi-supervised or unsupervised ones.
Which methods are used, and how do they compare?
| Family | Examples | Labels needed to train | What it catches | Main weakness |
|---|---|---|---|---|
| Limits and univariate control charts | Alarm limits, Shewhart, EWMA, CUSUM | None | One variable leaving its normal range or drifting | Misses unusual combinations; many charts mean many false alarms |
| Multivariate statistical | PCA with T² and SPE; PLS | Normal data only | Unusual combinations and broken correlations; contribution plots point to the variables | Linear; usually one model per operating mode |
| Machine learning outlier detectors | Isolation forest, local outlier factor (LOF), one-class SVM | Normal data, or data with few outliers | Unusual points in data with many features; nonlinear shapes | Scores each row on its own, so time patterns must be built as features; harder to explain |
| Prediction and reconstruction models | Regression residuals, autoencoders, LSTM encoder-decoder | Normal data only | Complex time patterns and nonlinear relations | More data, computing and tuning; harder to explain |
| Supervised fault classifiers | Gradient boosting, neural network classifiers | Labelled examples of each fault | Known fault types, by name | Cannot recognise a fault absent from its training data |
Limits and univariate control charts
Every DCS already has high and low alarm limits. Statistical process control adds charts with limits set from the variable's own normal variation. A Shewhart chart signals when a point falls outside ±3 standard deviations. An EWMA chart gives less and less weight to older data and can be made sensitive to a small or gradual drift. A CUSUM chart accumulates deviations from the target and is more efficient at detecting small shifts in the mean, of 2 standard deviations or less.
Slow drift deserves this attention: in the Schmidl evaluation, trends were the hardest anomaly type to detect and extreme values the easiest. For a single critical variable, such as a bearing temperature, an EWMA or CUSUM chart remains the right tool.
PCA with Hotelling's T² and SPE
MacGregor and Kourti describe how Shewhart, CUSUM and EWMA charts extend to many variables through Hotelling's T² and through PCA and PLS, with examples from a mineral processing plant and a continuous polymerization process (MacGregor and Kourti, 1995). PCA compresses dozens of correlated tags into a few components that describe normal operation. Two statistics are then charted:
- T² is the distance from the centre of the model plane to the new observation's projection onto it. A high T² means the variables have moved further than usual, but still together, as in a throughput increase.
- SPE (squared prediction error, also called Q) is the distance off the plane. A large SPE means the observation is inconsistent with the model and breaks the correlation structure, as a failing transmitter does when it no longer agrees with its neighbours.
When either chart signals, a contribution plot shows which variables drive the deviation. Our post on golden batch analysis applies the same family of methods to batch processes.
Isolation forest, LOF and one-class SVM
These come from machine learning and are available in common libraries (scikit-learn):
- Isolation forest isolates observations with random splits. Anomalies are isolated in fewer splits, so a short average path marks an anomaly. The documentation describes it as an efficient method for high-dimensional data.
- Local outlier factor flags samples whose density is substantially lower than that of their neighbours, taking both local and global properties of the data into account.
- One-class SVM is sensitive to outliers in its training data and needs careful tuning of its
nuparameter.
All three score each row on its own, ignoring sample order, so dynamics must be added as features: lagged values, rolling means, rates of change. Explaining which tag caused a high score also takes extra work, whereas a PCA contribution plot gives it directly.
Autoencoders and LSTM models
These neural networks learn to reproduce normal data and use the error as the anomaly score. The LSTM encoder-decoder of Malhotra et al. learns to reconstruct "normal" time-series behaviour, uses reconstruction error to detect anomalies, and was tested on two real-world engine datasets (Malhotra et al., 2016).
They are worth trying when relations are clearly nonlinear and histories are long and clean. They cost more: in the Schmidl evaluation, LSTM-AD and EncDec-AD exceeded the study's memory limit in 50% and 26% of experiments.
Do deep learning methods beat simpler ones?
Not on current evidence. Schmidl et al. found that every family can be effective, that there is no clear winner, and that deep learning approaches are "not (yet) competitive" despite their higher training effort (Schmidl et al., 2022). They also stress that every practical deployment needs careful testing.
Published benchmarks are part of the problem. Wu and Keogh argue that most examples in widely used time-series anomaly benchmarks have at least one of four flaws, so many published comparisons may be unreliable and much of the apparent progress "may be illusionary" (Wu and Keogh, 2022). A benchmark score says little about your plant data.
How many false alarms can operators absorb?
This decides adoption more than accuracy does. NIST states that a standard Shewhart chart, signalling only beyond ±3 standard deviations, gives a false alarm every 371 points on average.
The arithmetic is unforgiving. Sampled once a minute, one variable gives a false alarm about every six hours. Chart 50 variables separately and they raise about eight false alarms an hour, assuming normal, independent data. The ISA-18.2 alarm management guidance, as summarised by a Yokogawa architect in Control Engineering, lists about 6 alarms per hour on average as very likely to be acceptable and about 12 as the maximum manageable for an alarm system. A detector's alerts come on top of the alarms the plant already has.
Three habits keep the rate down:
- Use fewer, combined statistics: one T² and one SPE chart instead of 50 single-tag charts.
- Require persistence, such as several consecutive points beyond the limit, before alerting.
- At first, send results to an engineering display or a daily review list, not the operator's alarm summary.
How do you handle start-ups, grade changes and operating modes?
A model trained on one operating mode will flag every other mode as abnormal. For multimode processes, the usual practice is to build local monitoring models for each mode and to switch between them. In practice:
- Use the mode, product grade or load signal from the control system to select the model.
- Exclude start-ups, shutdowns and planned tests from training data, using the event log.
- Rebuild the reference data after equipment changes or a new feed, as with a soft sensor that drifts.
How do you test a detector on your own data, step by step?
- Choose one unit and one purpose, such as early warning of compressor upsets, with a named owner.
- Extract a year of history with the shift log, trips and maintenance records. Our guide on integrating AI with SCADA and OT systems covers access through the historian or OPC UA.
- Label known events with start and end times: upsets, trips, instrument faults.
- Split by time. Train on earlier normal operation; test on a later period containing the labelled events.
- Start simple. Fit EWMA charts on key tags and a PCA model with T² and SPE; try isolation forest or an autoencoder only if they miss events that matter.
- Measure what operators feel: events detected, minutes of warning before the existing alarm, false alarms per day.
- Run in parallel for several weeks, review every alert with operators, then decide.
Before using plant data, a team can practise on public sets: the Tennessee Eastman simulation data of Rieth et al., with 52 process variables sampled every 3 minutes, 20 fault types and 500 simulation runs each, and the Skoltech Anomaly Benchmark (SKAB), recorded on a water circulation testbed with a pump and eight measured variables. Both come from a simulation or a testbed, not a running plant.
What we have seen at KABAS AI
Our manufacturing page lists quality trend analysis and anomaly detection among DALILE's capabilities for production, and predictive maintenance scheduling among those of Virtual Lab. Our data science team covers data preparation, trend detection, predictive models and the pipelines that keep models monitored in production.
Our practical advice follows from the evidence above. Most of the work is in the data, not the algorithm: joining historian tags, event logs and maintenance records with consistent timestamps. Agree with operators on what a useful alert is before choosing a method, and count false alarms from the first day. Keep the detector inside your network with read-only access to plant data; our guide to air-gapped AI deployment covers plants with no internet connection. Before you start, check the data with an AI readiness assessment and read why industrial AI projects fail.
Sources
- Schmidl, Wenig and Papenbrock, Anomaly Detection in Time Series: A Comprehensive Evaluation, PVLDB 15(9), 2022
- Wu and Keogh, Current Time Series Anomaly Detection Benchmarks are Flawed and are Creating the Illusion of Progress, ICDE 2022 (arXiv 2009.13807)
- scikit-learn, Novelty and Outlier Detection, user guide
- NIST/SEMATECH e-Handbook of Statistical Methods, What are Variables Control Charts?
- NIST/SEMATECH e-Handbook of Statistical Methods, EWMA Control Charts
- NIST/SEMATECH e-Handbook of Statistical Methods, CUSUM Control Charts
- MacGregor and Kourti, Statistical process control of multivariate processes, Control Engineering Practice 3, 1995
- Dunn, Process Improvement Using Data, Hotelling's T² statistic
- Dunn, Process Improvement Using Data, Interpreting the residuals
- Malhotra et al., LSTM-based Encoder-Decoder for Multi-sensor Anomaly Detection, ICML 2016 Anomaly Detection Workshop (arXiv 1607.00148)
- Zhang, Zhou and Chen, Monitoring multimode processes: a modified PCA algorithm with continual learning ability, Journal of Process Control (arXiv 2012.07044)
- Tennant, Implementing Alarm Management per the ANSI/ISA-18.2 Standard, Control Engineering, 2013
- Rieth et al., Additional Tennessee Eastman Process Simulation Data for Anomaly Detection Evaluation, Harvard Dataverse, 2017
- Skoltech Anomaly Benchmark (SKAB), GitHub repository
Frequently asked questions
What is the difference between anomaly detection and fault diagnosis?
Anomaly detection says that the data no longer looks like normal operation. Fault diagnosis says which fault is present. Contribution plots from a PCA model point to the variables behind an alarm, which helps an engineer diagnose, but naming the fault automatically requires a classifier trained on labelled examples of that fault.
Do we need labelled failure data to start?
Not to build the detector. Most industrial detectors are trained on a period of normal operation and flag what differs from it. You do need some labelled events, such as known upsets, trips and instrument faults from the shift log, to test whether the detector finds them and how often it raises false alarms.
Is anomaly detection the same as predictive maintenance?
They overlap. Anomaly detection is often the first layer of a predictive maintenance system, flagging equipment that behaves differently from its normal pattern. Predictive maintenance adds diagnosis of the fault and an estimate of how long the equipment can keep running, which need failure history that anomaly detection alone does not.
How often should an anomaly detection model be retrained?
There is no fixed interval. Retrain or rebuild the reference data after equipment changes, catalyst changes, new products or a new operating mode, and whenever the false alarm rate starts to rise. Reviewing alarms with operators every few weeks shows when the model no longer reflects normal operation.
Should the anomaly detector write to the control system?
No. Run it as an advisory layer that reads data from the historian or an OPC UA server and shows its results to operators and engineers. Operators act through their existing procedures, and the detector has no write access to any control system variable.
Can these models run on-premise without internet access?
Yes. Control charts, PCA models, isolation forests and local outlier factor are small and run on ordinary servers inside the plant network. Deep learning models need more memory and computing power, but they can also be trained and run on-premise.



