Table of Contents
AI formulation optimisation is a loop. You write the specification as measurable targets, define which recipes are allowed, train a model on your past lab results, let an optimiser propose a small batch of recipes, test them and feed the results back. Each round improves the model, and only a lab result confirms that a recipe meets specification.
Key takeaways
- Most of the work happens before any model: a numerical specification, a design space with its constraints, and clean lab history.
- Mixture proportions sum to one, so changing one component changes the others. The model and the optimiser must both respect this (NIST/SEMATECH).
- Use a model that reports its own uncertainty and test it on recipe families it has not seen, not on a random split.
- Several targets can be combined into one desirability score or kept separate as a set of trade-offs. Cost and other values derived from the recipe should be calculated, not predicted.
- Agree the stopping rule in advance, and treat every prediction as a hypothesis until the lab confirms it.
What does AI formulation optimisation involve?
Our guide to the virtual lab explains the building blocks: a surrogate model that predicts test results from a recipe, an optimiser that chooses the next experiments, and a record of every trial. Design of experiments vs Bayesian optimisation compares fixed and sequential plans. This article covers the order of work on a real project.
The approach suits slow or costly tests. Peter Frazier's tutorial describes Bayesian optimisation as built for objectives that take minutes or hours to evaluate, tolerant of noisy results, and best suited to continuous problems of less than 20 dimensions. Eight components and two process settings fit; fifty candidate raw materials need narrowing first.
Step 1: How do you turn a product specification into numbers?
An optimiser cannot work from "cleans well and stays stable". Every target needs a test method, a unit and an acceptable range. Classify each target before touching any data:
| Kind of target | Drilling fluid example | Detergent example | How the optimiser treats it |
|---|---|---|---|
| Hit a window | Rheology within the range set for the well section | Viscosity within the filling line's range | Target is best |
| Maximise | Shale recovery in an inhibition test | Cleaning performance | Maximise |
| Minimise | Fluid loss | Raw material cost | Minimise |
| Pass or fail | Stable after ageing at temperature | No phase separation after storage | Constraint, or a separate classifier |
The first three kinds match the three forms of the desirability approach in the NIST/SEMATECH e-Handbook. Pass or fail results need different handling: Cao and colleagues coupled a classifier with a multi-objective optimiser so that discrete and continuous outputs were optimised together on a formulated liquid product. Also write down the hard rules: banned components, regulatory limits and maximum cost.
Step 2: How do you define the design space and its constraints?
The design space is every recipe the optimiser may propose. Define it with the formulators, not from the data alone. Formulations carry a constraint that most optimisation problems do not: proportions must sum to one. The NIST/SEMATECH handbook notes that lower bounds on components keep the feasible region a simplex, and recommends rescaling to pseudo-components because constrained mixture regions tend to show high multicollinearity.
| Constraint | Example | How to encode it |
|---|---|---|
| Proportions sum to 100% | All components of a liquid detergent | Equality constraint, or one balance component calculated from the others |
| Minimum and maximum per component | A thickener allowed only within a set range | Bounds on that variable; pseudo-components for the model |
| Ratio between components | Water-to-binder ratio in a cement mix | Linear inequality constraint |
| Choice between options | Which supplier's clay, which surfactant | Categorical variable, or fixed per study |
| Cost ceiling | Maximum raw material cost per tonne | Linear constraint calculated from the price list |
| Banned or restricted component | A substance excluded by a customer or a regulation | Remove it from the design space |
Tools differ in what they accept. BoTorch's optimize_acqf takes both linear inequality and equality constraints. Where a tool does not support equality constraints, the balance-component approach in the table above avoids the problem.
Categorical choices are harder than dosages. One option is to describe each choice by measured properties: the Gryffin study showed that physicochemical descriptors can substantially speed up the search over options such as catalysts or solvents. The simpler start is to fix the choice, optimise dosages, then compare options.
Step 3: How do you prepare the lab history?
Our virtual lab guide lists what records to collect. Preparing them for a model means:
- One basis for every recipe. Convert grams per batch, kilograms per cubic metre or parts per hundred into one basis, such as mass fraction of the final product.
- One row per test. Keep replicate results as separate rows, not averages, so the model can estimate test noise.
- Test method with every result. Flag results produced by an older or modified method and decide with the lab whether they are comparable.
- Failures included. Recipes that separated, gelled or missed specification show the model where the boundaries are.
- A family label. Tag each recipe with its project or base recipe. Step 5 uses this label.
Step 4: When do you need new experiments before modelling?
If the history is thin or clustered around one reference recipe, the model cannot learn the effect of variables that never changed. Run a small initial set first.
For mixtures with only the sum constraint, the NIST/SEMATECH handbook describes simplex-lattice and simplex-centroid designs. When components have their own lower and upper limits, it points to constrained designs such as extreme-vertices or D-optimal designs. For process settings such as temperature or mixing time, a space-filling sample is a common choice. Latin hypercube sampling splits each variable's range into equal intervals and places exactly one point in each. It does not respect the sum constraint by itself, so sample the free components and calculate the balance.
Include two or three replicates of your current reference recipe. Their spread gives the test repeatability, which you need in steps 5 and 8.
Step 5: How do you build and check the model?
Start with a simple baseline. For mixtures, the handbook's canonical polynomial has no intercept because the components sum to one, and its quadratic form adds a cross-product term for each pair of components. If a more complex model cannot beat it on held-out data, keep the polynomial.
A Gaussian process is the usual next step. The scikit-learn documentation notes that it returns a mean and a standard deviation for each prediction, that a WhiteKernel term can estimate the noise level from the data, and that the method loses efficiency when the number of features exceeds a few dozen. Time can also be an input: in a study on concrete, Ament and colleagues modelled strength jointly over composition and the logarithm of curing time.
Validate carefully. A random split puts near-copies of the same base recipe in both training and test sets, so the error looks smaller than it is. Use the family label from step 3 with group-based cross-validation: with scikit-learn's GroupKFold, each group appears exactly once in the test set across all folds. Then compare the prediction error with the test repeatability from step 4; a model cannot be shown to be more accurate than the test itself.
Step 6: How do you handle several targets at once?
There are two common approaches.
| Desirability score | Set of trade-offs (Pareto front) | |
|---|---|---|
| Idea | Map each result to a score from 0 to 1 and combine the scores with a geometric mean | Keep targets separate and find recipes where no target improves without another getting worse |
| Result | One top-ranked recipe | A set of options to choose from |
| Suits | A fixed specification with agreed priorities | Trade-offs that are a business decision, such as cost against performance |
| Watch out for | Limits and weights drive the answer; one score of zero makes the total zero | Mapping the trade-offs can take more experiments |
The desirability approach goes back to Derringer and Suich (1980), and the handbook explains how one unacceptable response sets the overall score to zero. For trade-offs, BoTorch provides multi-objective acquisition functions such as qLogNEHVI that search for the Pareto front.
Some targets should not be modelled at all. Raw material cost, active matter content and carbon footprint are arithmetic on the recipe. In the concrete study, global warming potential was computed as a deterministic linear function of the mix, and only strength was learned from tests; the authors report that the AI-proposed batches pushed the measured trade-off outward. Never ask a model to learn what a spreadsheet can calculate.
Step 7: How does the optimiser propose the next batch?
The optimiser scores candidate recipes with an acquisition function, such as expected improvement over the best result so far (Frazier). This balances recipes that look promising against recipes in regions the model knows little about. Labs work in batches, so ask for several candidates at once. BoTorch's optimize_acqf generates a batch of candidates jointly while respecting the constraints from step 2. Size the batch to what the lab can run in one cycle.
Before the batch goes to the bench, a formulator reviews it:
- Can each recipe be made, given the order of addition and solubility?
- Is any recipe a near-copy of one already tested?
- Does it pass the safety review?
The review is about feasibility and safety, not about overruling the ranking out of habit. In a Nature study, Bayesian optimisation outperformed human decision-making in reaction optimisation, in both the average number of experiments and consistency.
Step 8: How do you test, update and decide when to stop?
Each round follows the same sequence:
- Run the batch with the standard test methods, with the reference recipe as a control.
- Record the predicted value and its uncertainty for each recipe before the result is known.
- Add all results, including failures, to the dataset and refit the model.
- Check whether measured results fell inside the predicted uncertainty, and whether the best result improved.
- Decide whether to continue, narrow or widen the design space, or stop.
Agree the stopping rule in step 1: for example, a recipe meets the full specification in replicates, the experiment budget is spent, or two rounds in a row improve the best result by less than the test repeatability.
Speed depends on the lab. With robotic platforms, Cao and colleagues found nine suitable recipes within 15 working days. Most industrial labs test by hand, so each round takes longer, but the loop is the same. The chosen recipe still goes through scale-up and qualification testing; a model prediction is never a qualification result.
What we have seen at KABAS AI
Our Virtual Lab organises this loop in four phases: Think (targets, constraints, initial simulations), Test (what-if scenarios, multi-criteria optimisation, a laboratory test playbook), Industrialize (visual comparisons, standardised reports, guided substitutions) and Capitalize (structured data, reusable reference recipes).
At ADFC (Sonatrach Group), the optimisation balances rheological performance, fluid loss control, wellbore stability, environmental compliance and cost. Engineers use what-if scenarios for component substitutions and dosage changes, which matters because raw materials vary between batches and suppliers. Comparisons between simulated and actual results guide refinements.
At ENAD Shymeca, historical formulation data was migrated into the platform before it was used for prediction, and the optimisation balances objectives such as cleaning efficacy against cost and foam characteristics against environmental impact. The case study reports a 40% reduction in physical testing cycles.
In both projects, structuring past formulation records came before any optimisation, which is why steps 1 to 3 come first. To check whether your records are ready, start with a short AI readiness assessment; why industrial AI projects fail covers the traps. Recipes are core intellectual property, so decide early where data and models will run, as discussed in our guide to sovereign, on-premise AI. The chemistry solutions page shows how Virtual Lab fits formulation work.
Sources
- Frazier, A Tutorial on Bayesian Optimization, arXiv 1807.02811, 2018
- NIST/SEMATECH e-Handbook of Statistical Methods: Multiple responses, the desirability approach
- NIST/SEMATECH e-Handbook of Statistical Methods: Constrained mixture designs
- NIST/SEMATECH e-Handbook of Statistical Methods: Simplex-lattice designs
- Cao et al., Optimization of Formulations Using Robotic Experiments Driven by Machine Learning DoE, Cell Reports Physical Science, 2021
- BoTorch documentation: optimisation of acquisition functions (optimize_acqf)
- BoTorch documentation: multi-objective Bayesian optimisation
- Häse et al., Gryffin: An algorithm for Bayesian optimization of categorical variables informed by expert knowledge, arXiv 2003.12127
- SciPy documentation: scipy.stats.qmc.LatinHypercube
- scikit-learn documentation: Gaussian Processes
- scikit-learn documentation: GroupKFold
- Ament et al., Sustainable Concrete via Bayesian Optimization, NeurIPS 2023 workshop, arXiv 2310.18288
- Shields et al., Bayesian reaction optimization as a tool for chemical synthesis, Nature, 2021
Frequently asked questions
How many experiments does AI formulation optimisation need?
There is no fixed number. It depends on how many components and settings you vary, how noisy your test methods are and how far you move from recipes you already know. Agree a stopping rule and an experiment budget before the first round, and count every test, including failures and replicates.
Can we use AI if our past recipes were never part of a designed experiment?
Yes. Historical recipes and results can train a first model if they are recorded on a consistent basis with their test methods. Check the model on recipe families it has not seen, and if the history is thin or clustered around one reference recipe, add a small designed set of experiments before optimising.
How do we handle the fact that ingredients must add up to 100%?
Either give the optimiser an equality constraint so the proportions always sum to one, or leave one component, often water or a solvent, as the balance that is calculated from the others. Lower and upper limits on each component are added as bounds. The model should be fitted on the same basis, such as mass fractions of the final product.
Which model should we start with?
Start with a simple baseline such as a canonical mixture polynomial, then try a Gaussian process, which also reports how uncertain each prediction is. Keep the more complex model only if it predicts held-out recipe families better than the baseline.
Does the optimiser replace the formulator?
No. Formulators define the targets and the allowed ranges, review every proposed batch for feasibility and safety, and decide when to stop. The optimiser only ranks candidate recipes, and the lab result, not the prediction, decides whether a recipe meets the specification.
Can our formulation data stay on our own servers?
Yes. The models used for formulation optimisation are small and can run on your own infrastructure. KABAS AI's Virtual Lab can be deployed on-premise or on a local PC without an internet connection, which matters when recipes are core intellectual property.



