Table of Contents
Golden batch analysis compares each new batch with a reference built from past good batches, so deviations appear while the batch is still running. Batch quality prediction goes one step further: a model trained on the trajectories and lab results of past batches estimates the final quality before the batch ends, leaving time to act.
Key takeaways
- A single "best" batch is a weak reference. Comparing against it ignores interactions between variables and normal variation; a profile built from many acceptable batches does better (Chemical Processing).
- Multivariate batch monitoring, developed by Nomikos and MacGregor in the 1990s, needs only a historical database of past successful batches.
- Batches rarely run at the same pace, so they must be aligned by phase, by a maturity variable or by dynamic time warping before they are compared.
- Multiway PLS can give on-line predictions of final product quality with approximate confidence intervals. The lab still decides release.
What is a golden batch?
In batch production, such as resins, lubricants, specialty chemicals, detergents, food or pharmaceuticals, each batch follows a recipe of charges, heating, reaction, holding and discharge. ISA-88, the batch control standard series, gives these plants a common structure, including a model for batch production records (Part 4).
The "golden batch" is the simplest way to use those records. Engineers pick a batch that went well, overlay its temperature, pressure and dosing trends on each new batch, and investigate where the new batch departs from it.
A common improvement is an envelope. For each point in time, take the mean of several good batches and draw limits at plus or minus three standard deviations. A 2024 study on a simulated penicillin fermentation built its golden profile this way, aggregating data from multiple high-quality batches and scoring each new batch by its distance from the profile.
Why is a single golden batch a weak reference?
An article co-written by a Lubrizol engineer and an Emerson manager states the problem directly. Comparing with a golden batch is "a univariate approach to what in reality is a multivariate problem". It ignores interactions between inputs, process variables and quality, and it has no notion of acceptable versus unacceptable variation. The authors call trying to copy it chasing "fool's gold".
Three practical reasons sit behind that verdict.
Raw materials change. The same setpoints applied to a different lot can give a different result. The FDA's PAT guidance notes that time-defined end points, such as "blend for 10 minutes", in some cases do not consider the effects of physical differences in raw materials. Within its framework, an end point is the achievement of the desired material attribute, not a fixed time.
One batch is one sample. The best batch may have been good partly by luck. It says nothing about how much variation is normal.
Variables move together. A batch can keep every trend inside its own envelope and still be abnormal, because the combination is unusual: for example, a normal temperature with a normal coolant flow that, together, imply less heat released than usual.
| Reference | Built from | What it catches | What it misses |
|---|---|---|---|
| Single golden batch | One past batch | Large departures from that batch | Normal variation; interactions between variables |
| Univariate envelope (mean ± 3 standard deviations) | Several good batches, one variable at a time | A variable outside its normal range at a given time | Unusual combinations of variables that are each in range |
| Multivariate golden profile (multiway PCA or PLS) | Many good batches, all variables together | Out-of-range values and unusual combinations; can be linked to final quality | Faults never seen in training are flagged but not identified; needs enough representative batches |
What data do you need?
Batch data has three dimensions: batches, variables and time. Researchers describe it as a three-way array that must be unfolded into a two-way table before standard PCA or PLS can use it.
In practice, gather four kinds of data per batch:
- Trajectories from the historian for every relevant variable, at a resolution finer than the process dynamics.
- Initial conditions: raw material lots and their certificate or incoming test values, charge quantities, equipment used.
- Phase markers from the batch system: start and end of each phase, so batches can be segmented.
- Final quality from the LIMS, linked to the batch number, including batches that failed.
The historian timestamp, the batch ID in the control system and the lot number in the LIMS rarely match without work. Joining them is usually the first and longest task.
How do you align batches that run at different speeds?
No two batches take exactly the same time. A heat-up phase may last longer on a cold day, and a reaction may run faster with a more reactive lot. Comparing minute 90 of one batch with minute 90 of another then compares different stages of the process.
Four approaches are common:
| Approach | How it works | When it fits |
|---|---|---|
| Phase segmentation | Model each phase separately, using the batch system's phase markers | Recipes with clear phases; the Lubrizol project built a separate model for each process stage |
| Maturity variable | Replace clock time with a variable that rises steadily, such as cumulative feed or conversion | When such a variable exists; Westerhuis and colleagues propose methods to predict the per cent completion of batches of unequal duration |
| Model-based maturity | A PLS model predicts batch maturity from all variables, and limits are set along that axis | When no single variable tracks progress; Eigenvector's batch maturity method is one implementation |
| Dynamic time warping | Stretches and compresses each trajectory to match a reference with minimal process knowledge | Irregular timing without a natural maturity variable |
Do not throw away the timing information. In a troubleshooting study of an industrial batch dryer, the timing of batch stages was one of the groups of variables linked to poor product quality.
How does a multivariate model monitor a running batch?
The method published by Nomikos and MacGregor uses multiway principal component analysis (PCA) to compress all trajectories of all good batches into a few scores. A new batch is projected onto the same model as it runs, and multivariate control charts show whether it still follows the normal pattern. Control limits come from the reference distribution of past successful batches, and the method was applied to an industrial batch polymerization reactor. Because the model looks at all variables together, it can flag a batch whose variables are each in range but whose combination is unusual.
When a chart signals, contribution plots show which variables drive the deviation. In the batch dryer study, contribution plots helped identify the features that separated good from bad batches.
The Lubrizol field trial at its Rouen plant, which used 18 input, 38 process and 4 output variables sampled every minute, reports problems found this way, including a previously undetected problem with a mass flow meter for a key component and a component being introduced too slowly. Treat these as a vendor-associated case report, not an independent evaluation.
How do you predict final batch quality before the batch ends?
Monitoring says "this batch is unusual". Quality prediction says "this batch will finish at a viscosity of X". Multiway partial least squares (PLS) relates the trajectories to the final lab results and extracts the parts of the trajectories most related to quality. In Nomikos and MacGregor's formulation, the monitoring charts also provide on-line predictions of the final product qualities, with approximate confidence intervals.
Early in a batch, much of the trajectory is still unknown, so early predictions carry more uncertainty than late ones. Show the interval, not only the number. In the Lubrizol trial, operators could view the predicted end-of-batch quality with confidence limits on a separate trend; the article gives no accuracy figures.
Intermediate lab samples help. In a simulation study of emulsion copolymerization, Duchesne and MacGregor report that intermediate quality measurements can significantly reduce the number of batch runs needed for the analysis, and that the same models point to trajectory changes that improve final quality.
The idea is close to a soft sensor, which estimates quality continuously in a continuous process. The same rules apply: time-split validation, comparison with the lab, and a named owner. For process models that go beyond statistics, see surrogate models for chemical processes.
How do you start, step by step?
- Choose one product and one quality problem, such as viscosity outside specification or batch-to-batch variation in reaction time, with a cost you can state.
- Assemble the batch records: trajectories, phase markers, raw material lots and final lab results, including failed and reworked batches.
- Label each batch as good or bad against the specification, and note known causes for the bad ones.
- Segment and align the batches by phase, then by maturity or time warping where needed.
- Build the monitoring model on good batches only, then test it on held-out good batches and on the known bad ones. It should flag the bad batches without flooding operators with false alarms.
- Build the quality prediction model and check it on later batches it has not seen, comparing predictions at several points in the batch with the final lab result.
- Run it in parallel with normal operation. Operators see the charts and predictions but act only through existing procedures. Our guide on integrating AI with SCADA and OT systems covers the connection.
- Maintain it. Rebuild the reference set after equipment changes, new suppliers or recipe changes, and review alarms with operators.
Before using plant data, a team can practise on the public IndPenSim benchmark: 100 simulated batches of a 100,000-litre penicillin fermentation, run under recipe control, operator control and advanced process control, with ten batches containing faults. Remember that it is simulated data.
What do regulators expect?
In pharmaceutical manufacturing, US FDA guidance supports this approach. The PAT guidance says multivariate statistical process control can be feasible and valuable for real-time measurements. The process validation guidance asks for data to be statistically trended and states that scrutiny of intra-batch as well as inter-batch variation is part of continued process verification. It also warns against overreacting to individual events.
These are US guidances. For the requirements that apply to your plant in Algeria, check the relevant regulator's texts and confirm with counsel.
What we have seen at KABAS AI
Our published projects are in formulation R&D, not production batch monitoring, but they meet the same underlying problems.
At ADFC (Sonatrach Group), raw material variability between batches and suppliers was one of the stated challenges, and comparisons between simulated and actual results guide refinements. At ENAD Shymeca, the case study cites documentation and traceability demands for compliance, and the need to capture knowledge from experienced researchers approaching retirement. In both cases the first work was structuring scattered records into a reusable base, which is also the first and longest step of any golden batch project.
Virtual Lab lets teams test dosages, sequences, temperatures and other process parameters virtually before lab or pilot runs, keeps a complete test history with pass/fail rules, and connects to LIMS and ERP systems. Our manufacturing page lists batch production, such as lubricants and fine chemicals, among the production types we support, and the chemistry page covers quality control and process optimisation. For the wider approach, read our pillar on Virtual Lab for industrial R&D, and for the recipe side, AI formulation optimisation. Before you start, the checklist in why industrial AI projects fail is worth a read.
Sources
- Wojewodka and Marruchella, Online Analytics Improves Batch Operations, Chemical Processing, 2011
- Nomikos and MacGregor, Monitoring batch processes using multiway principal component analysis, AIChE Journal 40, 1994
- Nomikos and MacGregor, Multivariate SPC charts for monitoring batch processes, Technometrics 37, 1995
- Nomikos and MacGregor, Multi-way partial least squares in monitoring batch processes, Chemometrics and Intelligent Laboratory Systems 30, 1995
- Kassidas, MacGregor and Taylor, Synchronization of batch trajectories using dynamic time warping, AIChE Journal 44, 1998
- Westerhuis, Kourti and MacGregor, Comparing alternative approaches for multivariate statistical analysis of batch process data, Journal of Chemometrics 13, 1999
- Duchesne and MacGregor, Multivariate analysis and optimization of process variable trajectories for batch processes, Chemometrics and Intelligent Laboratory Systems 51, 2000
- García-Muñoz et al., Troubleshooting of an industrial batch process using multivariate methods, Industrial and Engineering Chemistry Research 42, 2003
- Eigenvector Research, Batch Maturity documentation
- Luo et al., The golden batch-driven root cause analysis for anomalies in bioreactor fermentation process, Frontiers in Manufacturing Technology, 2024
- Goldrick et al., IndPenSim data: Modern day monitoring and control challenges outlined on an industrial-scale benchmark fermentation process, Mendeley Data, 2019
- ISA, ISA-88 Standards
- US FDA, PAT: A Framework for Innovative Pharmaceutical Development, Manufacturing, and Quality Assurance, 2004
- US FDA, Process Validation: General Principles and Practices, 2011
Frequently asked questions
What is the difference between a golden batch and a golden profile?
A golden batch is one past batch, usually the best one, used as the reference for all others. A golden profile is built from many acceptable batches, so it describes both the typical trajectory and the normal spread around it. Multivariate golden profiles also capture how variables move together, which a single batch cannot show.
How many past batches do we need to build a model?
There is no universal number. The methods described here only need a historical database of past good batches, but that history must cover the normal variation in raw materials, equipment and operators that new batches will meet. Keep the known bad batches too, because they are the best test of whether the model detects problems.
Can batch quality prediction replace the final lab release test?
In most plants, no. The prediction tells operators early that a batch is heading off target, while the lab result still decides release. In pharmaceutical manufacturing, the US FDA describes real time release based on process data, but says the agency's approval should be obtained before using it for products covered by a marketing application.
What if our batches do not all take the same time?
That is the normal case, and it is why batches must be aligned before they are compared. Common approaches are to split the batch by phase, to replace clock time with a maturity variable such as percent conversion, or to use dynamic time warping. The time each phase takes is often useful information in itself.
Is batch quality prediction the same as a soft sensor?
They are close relatives. A soft sensor usually estimates a quality variable continuously in a continuous process. Batch quality prediction estimates the quality the batch will have when it finishes, using the part of the batch trajectory measured so far.
Can these models run on our own servers?
Yes. Multivariate monitoring and quality prediction models such as PCA and PLS are small and run on ordinary servers inside the plant network. Batch records and recipes are often sensitive, so many companies prefer to keep both the data and the models on-premise.



