Table of Contents
The most common data quality problems in industrial machine learning are missing values hidden as zeros or constants, frozen sensors, outliers, sensor drift, historian compression, misaligned timestamps and sampling rates, and periods that are not normal operation, such as shutdowns and loop tests. Most look plausible on a trend, so each needs an explicit check before modelling.
Key takeaways
- A review of process industry soft sensors lists noise, missing values, outliers, co-linear features and varying sampling rates as common in plant data, and notes that handling them takes much manual work (Kadlec et al., 2009).
- Missing values often appear as zero or another constant, not a blank, so a model can learn them as real measurements.
- Historian compression changes the mean and variance of archived data; Thornhill et al. proposed using compressed data, with caution, only when the compression factor is 3 or less (Thornhill et al., 2004).
- OPC UA marks results Good, Uncertain or Bad, and results with a bad status "shall never be used" (OPC Foundation). Keep that status with every extracted value.
Why is plant data harder than it looks?
Years of historian data suggest plenty of training data, but much of it describes the instruments and the data system as well as the process.
Kadlec, Gabrys and Strandt reviewed data-driven soft sensors across the chemical, bioprocess and steel industries and found that many unresolved issues in their development and maintenance originate in the process data (Kadlec et al., 2009). The same problems affect soft sensors, predictive quality models and anomaly detectors alike.
Data problems also surface late. Interviewing 53 AI practitioners in India, East and West African countries and the USA, in fields such as health and conservation, Sambasivan et al. found that "data cascades", compounding downstream problems caused by data issues, had 92% prevalence (Sambasivan et al., 2021). The study was not about plants, but the lesson carries over.
What are the common problems?
| Problem | How it shows in the data | Typical cause | First check |
|---|---|---|---|
| Missing or placeholder values | Blanks, zeros or a constant value | Sensor failure, maintenance or removal; lost link | Count blanks, zeros and repeated values per tag |
| Frozen values | Same value for hours while related tags move | Stuck transmitter, failed interface | Longest run of identical values |
| Outliers | Spikes, physically impossible values | Noise, transmission errors, or real upsets | Physical limits; median-based limits |
| Sensor drift | Slow offset from redundant instruments or the lab | Fouling, ageing, lost calibration | Compare with redundant tags, lab and balances |
| Bad or uncertain status | Flagged values exported without the flag | Device fault, loop test, communication loss | Extract the status with each value |
| Historian compression | Straight segments between archived points; lower variance | Wide compression or exception settings | Archived points per day |
| Timestamp and rate mismatches | Lab results at entry time; repeated or missing hours | Manual entry, clock settings, process delays | Compare sample, analysis and entry times |
| Non-process periods | Start-ups, shutdowns and tests inside "normal" data | Event log not joined | Join shift, maintenance and alarm logs |
| Label problems | Few labels; inputs that use future information | Low sampling frequency, data leakage | Check what was known at prediction time |
Missing values: why are blanks the easy case?
Kadlec et al. define missing data in process plants as values that do not reflect the real state of the measured quantity, and note that affected variables usually show zero or another constant. Common causes are sensor failure, maintenance or removal, and transmission or database problems (Kadlec et al., 2009).
A blank is easy to detect. A zero is not: a model treats a flow of zero as a real stop. Compare runs of zero or of one value with maintenance records and communication alarms.
The same review calls filling gaps with the mean "very primitive and not recommended", though still common, and points to methods that use the other variables. scikit-learn distinguishes univariate imputation, which uses only the tag itself, from multivariate imputation, which uses all available features, and notes that keeping track of which values were missing "can be informative" (scikit-learn). Fill only short gaps, flag every filled value, and leave long gaps out.
Frozen values and outliers: how do you spot them?
A frozen value stops changing while the process around it moves. Kadlec et al. list "locked variables having constant value" among the obvious problems to find at first inspection. Look for long runs of identical values or a rolling standard deviation of zero, and compare with tags that should move with it. Setpoints and on/off states are constant by design.
Outliers come in two kinds (Kadlec et al., 2009):
- Obvious outliers break physical or technological limits, such as a negative absolute pressure. Engineers must supply those limits.
- Non-obvious outliers stay within limits but outside the typical range. The 3-sigma rule relies on the mean and standard deviation, which outliers distort; the Hampel identifier uses the median and median absolute deviation, which is more robust.
Do not delete every unusual value: an upset or trip is real data. Remove instrument errors; label process events.
Is it the sensor or the process that is drifting?
Kadlec et al. separate two kinds of drift. Process drift comes from wear, catalyst deactivation, feed purity or weather, and changes the process. Sensor drift comes from the measuring device and does not reflect any process change; the answer is to recalibrate the instrument or adapt the model, not to act on the process.
Compare the tag with redundant transmitters, lab results and mass or energy balances, and pull calibration dates from the maintenance system. A model trained across an undetected sensor drift learns it as process behaviour.
Do you keep the quality status with each value?
Devices and data servers already flag many bad values. NAMUR recommendation NE 107 defines four device status signals; "function check" marks a temporarily invalid signal "such as forced outputs during loop tests", and "failure" an invalid signal caused by a malfunction (Endress+Hauser). In OPC UA, Uncertain results "might not be suitable for some purposes", and results with a bad status "shall never be used" (OPC Foundation).
In a values-only export, a loop-test reading looks like any other. Extract the status too, and drop or flag values that are not Good.
How does historian compression change the data?
Historians often store only some of the values they receive. Kadlec et al. describe systems that record a sample only when a variable changes by more than a threshold: too high a threshold misses important process changes.
Thornhill, Shoukat Choudhury and Shah studied the swinging door algorithm, which OSIsoft stated its PI historian used, and similar piecewise linear methods (Thornhill et al., 2004). Compression changed the mean and reduced the variance of reconstructed data, and altered control loop performance indices. They found:
- Data with a compression factor of 3 or less "may be used with caution" for data-driven analyses.
- In one industrial data set, compression factors reached about 94 and only five tags were at 3 or below; the authors judged it unsuitable for data-driven analysis.
- Averaging, data reconciliation and mass balancing should use original data, not reconstructed values.
A 2025 preprint by Saudi Aramco engineers, using synthetic signals, reported that overly aggressive swinging door compression missed 35% of transient anomalies (Faqehi et al., 2025). It is a simulation, not a plant study, but it points the same way.
To check a tag, compare archived points with the scan rate, or, as Thornhill et al. do, reconstruct it at the original interval and count zero second differences. Our guide to historian data for machine learning covers extraction settings.
Are your timestamps telling the truth?
- Lab results. Use the time the sample was drawn, not when it was analysed or entered in the LIMS, as our soft sensor guide explains.
- Process delays. Material takes time to pass through a reactor or column, so inputs must be shifted by the residence or dead time, which needs process knowledge (Kadlec et al.).
- Different rates. For each lab sample, take the last process value at or before the shifted sample time, with a maximum gap so an old value is not used silently. pandas
merge_asofdoes this with a "backward" search and atolerance(pandas). - Clocks. When clocks go back for daylight saving time, local times repeat; when they go forward, some do not exist (PEP 495). Algeria has used UTC+1 without daylight saving since 1981, according to the IANA time zone database, but a server set to another zone, or data merged from sites that change clocks, can still create duplicate or missing hours. Store and join data in UTC, and check that controller, historian and LIMS clocks agree.
Which periods should stay out of the training data?
A model trained on "all data" learns start-ups, shutdowns, loop tests and calibrations as normal operation. Mark them from the shift, maintenance and event logs, then exclude or label them. Label operating modes too; our anomaly detection guide explains why each mode usually needs its own model. Batch plants must also match batch IDs across systems, as covered in golden batch analysis.
Are the labels good enough?
- How many are there? A year of once-per-shift lab samples gives about a thousand points, and defects may be a small fraction of parts, as our predictive quality guide shows.
- Are they right? Lab methods have their own repeatability. Screen lab values before training.
- Were they known at prediction time? scikit-learn defines data leakage as using "information that would not be available at prediction time" to build the model, which gives overly optimistic estimates and poorer performance in production (scikit-learn). A rework flag left among the inputs is a typical case.
How do you check data quality, step by step?
- Write the question first: model, target, tags. A tag good enough for a daily report may be too compressed for a soft sensor.
- Get the tag list with metadata: units, ranges, scan rate, compression settings, instrument replacement dates.
- Extract raw values with their status, at archived resolution, not interpolated, with lab, event and maintenance data.
- Profile each tag: share of missing, zero and constant values, longest frozen run, values outside physical limits.
- Check compression and timestamps, including lab sample versus entry times and clock settings.
- Mark non-process periods with operators.
- Check for drift against redundant instruments, the lab and balances.
- Write every rule down so it runs the same way on live data.
Should you clean the data or fix the source?
Both, but source fixes last longer and help every later model. Repair or recalibrate the instruments behind the worst tags, review compression settings for tags that feed models with your historian vendor's guidance, record lab sample times, and keep event logs complete. Our posts on industrial data governance and the oil and gas data readiness checklist cover ownership and the questions to ask before a project.
The same checks must also run, with alerts, on the live data feeding the model.
What we have seen at KABAS AI
Our data science service starts with data collection and preparation (gathering, cleaning and structuring data for analysis and AI) and ends with MLOps pipelines that monitor models in production. On our manufacturing page, DALILE offers quality trend analysis and anomaly detection on production data, which depend on the checks described here.
Our practical advice: profile the data before agreeing on a model's scope, and ask the instrument technicians and process engineers who know which transmitters are unreliable to turn that knowledge into written rules. Keep raw extracts inside your own network. On staffing, see building a first industrial data team.
Sources
- Kadlec, Gabrys and Strandt, Data-driven Soft Sensors in the Process Industry, Computers and Chemical Engineering 33(4), 2009
- Thornhill, Shoukat Choudhury and Shah, The impact of compression on data-driven process analyses, Journal of Process Control 14(4), 2004
- Faqehi et al., The Impact of Data Compression in Real-Time and Historical Data Acquisition Systems on the Accuracy of Analytical Solutions, arXiv 2510.26868, 2025
- Sambasivan et al., "Everyone wants to do the model work, not the data work": Data Cascades in High-Stakes AI, CHI 2021
- OPC Foundation, OPC 10000-4 Part 4: Services, 7.38.1 StatusCode General
- Endress+Hauser, NAMUR NE 107
- scikit-learn, Imputation of missing values
- scikit-learn, Common pitfalls and recommended practices
- pandas, pandas.merge_asof
- Python, PEP 495: Local Time Disambiguation
- IANA time zone database, africa file (Africa/Algiers)
Frequently asked questions
Should we fill missing sensor values or remove them?
Fill only short gaps, and flag every filled value so the model and its users know it was estimated. Long gaps, and zeros or constant values left by a failed sensor, are better removed or marked as missing. Methods that estimate a missing value from related tags are usually better than filling it with the tag's average.
What historian compression setting is safe for machine learning?
There is no universal setting. Thornhill and co-authors proposed that archived data with a compression factor of 3 or less may be used with caution for data-driven analyses. For tags that feed a model, review the compression and exception settings with your historian administrator, and consider keeping a less compressed copy.
Can a machine learning model cope with bad data by itself?
Only partly. Robust methods reduce the effect of a few outliers, but a frozen transmitter, a placeholder zero or a drifting sensor looks like valid data to the model. These have to be found with explicit checks and process knowledge, both in the training data and on live data.
How do we tell sensor drift from a real change in the process?
Compare the tag with independent information: a redundant transmitter, lab results, or a mass or energy balance. If one instrument moves while the others agree, suspect the sensor and check its calibration history. If several related measurements move together, the process has more likely changed.
Who should own data quality in a plant?
Data quality needs the instrument team, process engineers and the people who run the historian and LIMS, not only data scientists. A practical arrangement is a named owner for each critical tag list, a documented set of checks, and a routine for fixing the source of recurring problems.
Do we need perfect data before starting a project?
No. What matters is knowing which problems are present, how they affect the specific model, and whether they can be fixed or excluded. A short profiling exercise on the relevant tags answers that before modelling starts.



