Table of Contents
To get process historian data out for machine learning, query raw recorded values through a read-only path that crosses the DMZ, keep each value's quality flag, store timestamps in UTC, then resample to a common grid with time-weighted averages. First check each tag's compression: archives compressed more than about threefold can mislead data-driven models.
Key takeaways
- A historian does not keep every reading. Exception and compression filters drop values, and AVEVA describes compression as removing data that can be recreated with interpolation.
- Swinging door compression altered the mean and standard deviation of plant data, and a peer-reviewed study proposed that only data with a compression factor of 3 or less may be used with caution for data-driven analysis.
- Pull raw recorded values with their status. Interpolated values are calculated, and a simple average of irregular points is not the same as a time-weighted average.
- Keep timestamps in UTC, join lab results and events by time with an as-of join, and split training and test data by time.
- Move data out of OT through the DMZ with read-only, push-based patterns, as current joint guidance recommends.
What does a process historian actually store?
NIST defines a data historian as "a centralized database supporting data analysis using statistical process control techniques" (NIST CSRC). In the Purdue model it sits at Level 3, above SCADA and the DCS (Purdue model and AI). It stores one time series per tag: a timestamp, a value and a status.
What it does not store is every scan. In the AVEVA PI System, two filters run before archiving (AVEVA, 2023):
- Exception reporting is a simple deadband applied by the interface. A new value is sent only if it moves more than the exception deviation, or if a maximum time has passed.
- Compression uses the swinging door algorithm to "eliminate data that can safely be recreated with interpolation". The stated goal is to "filter out instrument and process noise and still record significant process changes".
The same AVEVA presentation suggests setting the compression deviation no larger than the instrument's precision, and the exception deviation at half of it. In an offshore pilot it describes, exception and compression together cut the data stored onshore by about 94%. Other historians use other algorithms, but the result is similar: archived points are irregular, dense around changes and sparse during steady operation.
How much compression is too much for machine learning?
Thornhill, Choudhury and Shah studied this directly (Thornhill et al., 2004). They define the compression factor as the ratio between the storage needed for the original data and for the archived data. Their findings:
- Swinging door compression "altered key statistical features of the data set such as the average value and standard deviation", and other data-driven analyses changed too.
- For two process data sets, results at a compression factor of 3 were close to the uncompressed case, and no nonlinearity was induced at 3 or below. Hence their rule: data with a factor of 3 or less "may be used with caution".
- In one industrial data set, factors reached 93.8 and only five tags were at 3 or below. The original data "were lost forever" once compressed.
The tension is worth knowing. The AVEVA presentation, written for historian performance, mentions a "suggested minimum compression ratio of 3:1". The same ratio that is a floor for performance is a ceiling for analysis.
You can estimate the factor per tag before modelling. Count the recorded values in a period and compare them with the number of scans. A tag scanned every 10 seconds produces 8,640 scans a day; if the historian kept 300 values, the factor is about 29. Thornhill et al. also describe how to detect compression from the data itself, by counting zero second derivatives in the reconstructed trend.
Raw, interpolated or aggregated: which retrieval mode should you use?
OPC UA Historical Access defines three kinds of read for historical data (OPC 10000-11). Vendor interfaces use their own names for similar options.
| Retrieval mode | OPC UA term | What you get | Use for ML | Main risk |
|---|---|---|---|---|
| Raw (recorded) | Read raw | The archived values at their own timestamps, with status | Default for training data; you control resampling and gaps | Large volumes; irregular spacing |
| At time (interpolated) | Read at time | A value at each requested timestamp, interpolated between archived points when none exists | Quick look at evenly spaced data | Hides compression; can draw lines across outages |
| Processed (aggregates) | Read processed | One value per interval: time average, minimum, maximum, count, percent good and others | Features at the model's resolution, computed by the server | The wrong aggregate gives biased values |
In an at-time read, OPC UA marks each returned value as raw or interpolated. In a raw read, a period with no data comes back as Good_NoData. Keep those flags in your extract.
The aggregate choice matters more than it looks. The OPC UA Average aggregate "adds up the values of all Good Raw data for each interval, and divides the sum by the number of Good values" (OPC 10000-13, Average). TimeAverage draws sloped lines between points and divides the area under them by the interval length (OPC 10000-13, TimeAverage). On compressed data, archived points cluster around changes, so a simple average over-weights transients. Use time-weighted averages for analog tags, and stepped values for discrete tags such as pump status: the TimeAverage2 variant follows the tag's Stepped property.
Volume is the practical limit. AVEVA lists slow responses, query timeouts and hitting the archive's collection limit (ArcMaxCollect) among common query performance problems (AVEVA, 2023). Pull raw data in chunks, for example one month per tag group, outside peak hours.
How do you handle quality flags and bad values?
Every OPC UA value carries a status code whose severity is Good, Uncertain or Bad. Good means the result "may be used", Uncertain that it "might not be suitable for some purposes", and Bad that it "cannot be used" (OPC 10000-4). Depending on the historian, a failed reading may also be stored as a text marker rather than a number.
Rules that avoid most surprises:
- Keep the status column next to every value through the whole pipeline.
- Treat Bad as missing, never as zero. A zero flow during a communication failure teaches the model a shutdown that never happened.
- Do not interpolate across long gaps. Set a maximum gap per tag, based on its dynamics, and drop or flag the windows beyond it.
- Look for flat lines with Good status. A stuck transmitter can report a constant value with no error.
The TimeAverage aggregate itself omits Bad values and marks the result Uncertain_DataSubNormal, so check the status of aggregates too. Our post on industrial data quality problems covers the wider list.
How do you handle timestamps and time zones?
In OPC UA, "all time values conveyed between OPC UA Servers and Clients are UTC values" and clients convert to local time. The same section warns that the source and server timestamps "may originate from different clocks that have no synchronization" (OPC 10000-3).
Algeria makes this simpler than Europe: the time zone database lists Africa/Algiers on CET (UTC+1) since May 1981, with no daylight saving time since 1980 (IANA tz database). The traps are elsewhere: a laboratory spreadsheet in local time, a vendor server set to a European zone, or a lab result stamped with its analysis time instead of its sampling time. Store everything in UTC and convert only for display. Our soft sensor guide explains why the sampling time is the one that matters.
How do you join lab results and events to process data?
Lab results, grade changes, trips and maintenance work arrive at irregular times. An as-of join matches each lab sample to the nearest process row in time. In pandas, merge_asof is "similar to a left-join except that we match on nearest key rather than equal keys", with a direction of backward, forward or nearest and an optional tolerance (pandas).
Use direction="backward" after shifting the sampling time by the dead time between the process and the sample point. The model then sees only process values that existed before the sample, which is what it will see in service. Set a tolerance so a sample with no process data nearby gets no match rather than an old one. Use the event log to mask start-ups, shutdowns and tests, as in our guide to anomaly detection methods.
How should you split and store the extract?
Split by time, never at random. scikit-learn notes that ordinary cross-validation on time-ordered data "would lead to training on future data and evaluating on past data". Its TimeSeriesSplit has a gap parameter that drops samples between training and test sets (scikit-learn), useful when features use rolling windows.
Store the result in a columnar format such as Apache Parquet, "an open source, column-oriented data file format designed for efficient data storage and retrieval" (Apache Parquet). Save a manifest beside it: tag list, retrieval mode, aggregate and interval, time range, compression settings at extraction time and the date of the query. Without it, nobody can rebuild the training set when the model needs retraining.
Where should the extraction run?
Not on the control network. NIST SP 800-82r3 says any communications between the enterprise level and the operations management level "are required to go through services within the DMZ" (NIST SP 800-82r3, section 5.4.1). The joint guidance led by CISA prefers "push-based or brokered architectures that move required features or summaries out of OT without granting persistent inbound access", with one-way transfer patterns and audited staging buffers (CISA et al., 2025).
In practice: a read-only account on a historian copy in the DMZ, or a scheduled export to a staging area. Our guides on integrating AI with SCADA and OT systems and data diodes and one-way transfer cover the architecture.
Step by step: from historian to training set
- Fix the use case and the tag list with the process engineer. Record each tag's units, span, scan rate and exception and compression settings.
- Estimate the compression factor per tag over a sample month, and flag tags far above 3.
- Pull raw recorded values with status in chunks, through the read-only path.
- Convert to UTC, sort and remove duplicates.
- Mark Bad and Uncertain values and long gaps. Never fill them with zeros.
- Resample to one grid with time-weighted averages for analog tags and stepped values for discrete ones.
- Join lab results and events with an as-of join, shifted by the dead time.
- Split by time, save as Parquet with a manifest, and script every step so it can be rerun.
- Ask the OT team to lower compression on the tags the model depends on, from now on.
What we have seen at KABAS AI
Our data science service covers data collection and preparation, predictive models and the MLOps pipelines that keep them running, and our information systems service covers integration of existing systems. The guidance here follows from that kind of work, not from a published benchmark.
Three points are worth planning early. Budget for data access before modelling, because the extract can take longer than the model. Script the extract from the first day, because a model that cannot be retrained on the same pipeline will drift without a fix. And compute features the same way in production as in training: if the model learned from 1-minute time-weighted averages, the live feed must deliver exactly that. For staffing, see your first industrial data team, and check the historian with an AI readiness assessment before you start.
Sources
- NIST CSRC Glossary, Data historian
- NIST SP 800-82r3, Guide to Operational Technology (OT) Security, 2023
- CISA et al., Principles for the Secure Integration of Artificial Intelligence in Operational Technology, December 2025
- Bregenzer, Exception, Compression, and their Impacts on PI System Performance, AVEVA, 2023
- Thornhill, Choudhury and Shah, The impact of compression on data-driven process analyses, Journal of Process Control 14, 2004
- OPC 10000-11, OPC UA Part 11: Historical Access, Read raw, processed and at time
- OPC 10000-13, OPC UA Part 13: Aggregates, Average
- OPC 10000-13, OPC UA Part 13: Aggregates, TimeAverage
- OPC 10000-4, OPC UA Part 4: Services, StatusCode
- OPC 10000-3, OPC UA Part 3: Address Space Model, UtcTime
- IANA time zone database, africa file (Africa/Algiers)
- pandas documentation, pandas.merge_asof
- scikit-learn documentation, TimeSeriesSplit
- Apache Parquet, Overview
Frequently asked questions
Can I train a model on interpolated values from the historian?
You can, but it is risky as a default. Interpolated values are calculated between archived points, so they hide how much the tag was compressed and can draw straight lines across outages. Pull raw recorded values with their status, then resample them yourself so you control how gaps and bad values are handled.
What is a compression factor and how do I estimate it?
It is the ratio between the number of values the instrument produced and the number the historian kept. A practical estimate is the number of scans in a period divided by the number of recorded values in the same period: a tag scanned every 10 seconds has 8,640 scans a day, so 300 recorded values means a factor of about 29. Thornhill and colleagues proposed that data with a factor of 3 or less may be used with caution for data-driven analysis.
Should we turn off compression in the historian?
Not across the board. Compression keeps the historian fast and storage manageable. A better approach is to agree with the OT team on the tags a model needs and lower their compression and exception settings from now on, after checking the effect on storage and query performance. Data already compressed cannot be recovered.
What sampling interval should the training data have?
It depends on the process dynamics and the use case, not on the historian. A soft sensor for a quality variable sampled every few hours may only need 1 to 5 minute averages, while a vibration or fast pressure problem needs much finer data. Choose the interval with the process engineer, and use time-weighted averages to get there.
Can the data science server query the historian directly?
Only through the path your OT security rules allow. NIST SP 800-82r3 says communications between the enterprise level and the operations level must go through services in the DMZ, and joint guidance led by CISA prefers push-based or brokered architectures. In practice that means a read-only account on a historian copy or a scheduled extract delivered to a staging area, not a live connection into the control network.



