Table of Contents
AI pilots usually stall after they work, not before. A pilot proves a model on one line, with a motivated team and hand-built data feeds. A rollout needs the same result on every line and site: automated data pipelines, validation on each line, change control, trained shifts, a named owner and a running budget the pilot never had.
Key takeaways
- In a Gartner survey of 644 organisations, fewer than half of AI projects reached production, and going from prototype to production took about eight months on average (CIO Dive).
- The World Economic Forum calls the slowdown after early success the "Scaling Slump": the data, technology, talent and organisational solutions at the factory level "are not the same at the network level" (WEF, 2023).
- A model that works on line 1 can lose accuracy on line 2. Distribution shift "can substantially degrade the accuracy" of deployed machine learning systems (Koh et al., 2021).
- Running many similar models is a problem of its own: Google engineers name "updating many configurations for many similar models safely and automatically" as a source of technical debt (Sculley et al., 2015).
- Plan the rollout before the pilot starts: per-line pass criteria, staged deployment, change control, and a contract and budget that cover the phases after the pilot.
Why doesn't a successful pilot scale on its own?
A pilot runs on the best-instrumented line, with a data scientist on call and engineers who want it to work. Those conditions do not travel.
In a Gartner survey of organisations in the US, Germany and the UK, fewer than half of AI projects reached production, and the top barrier was the inability to estimate and demonstrate the value of AI projects (CIO Dive).
The World Economic Forum's Global Lighthouse Network, a group of factories recognised for advanced manufacturing, describes a second stall after pilot purgatory, the "Scaling Slump", when some companies "balk at the potential costs of replicating single-site approaches across entire networks". Even these leading factories took 10 to 20 months to implement their first few use cases (WEF, 2023).
This post covers that second stall. For problems that stop a project earlier, such as a vague goal or unready data, see why industrial AI projects fail.
What changes between a pilot and a plant-wide rollout?
| Dimension | Pilot | Plant-wide rollout |
|---|---|---|
| Data | One-off exports cleaned by hand | Automated pipelines for every line, with consistent tag names |
| Model | One model tuned on one line | One model per line, or a shared model validated on each line |
| Integration | A dashboard or a spreadsheet | Interfaces to the historian, MES, DCS or ERP at every site, through the security zones |
| Users | A few volunteers | Every shift, at every site |
| Support | The data scientist who built it | Written procedures, a named owner, monitoring and a fallback |
| Change control | Often informal | Formal review, updated operating procedures and training |
| Budget | A project budget | An annual running budget |
Each row is work the pilot did not need. None of it is model development.
Why does a model that works on line 1 fail on line 2?
Two lines that make the same product are rarely identical. Instruments differ in make, range and calibration, tag names differ, and one line is older, runs a different product mix or uses another supplier's raw material.
Machine learning is sensitive to these differences. The WILDS benchmark shows that distribution shift, where the training data differs from the data seen in use, "can substantially degrade the accuracy" of deployed systems. One of its examples is shifts across hospitals for tumour identification, the medical equivalent of moving a model to a new site. On each dataset, standard training gave "substantially lower out-of-distribution than in-distribution performance" (Koh et al., 2021).
A second risk is training-serving skew, a difference between performance in training and in service. Google's Rules of Machine Learning list among its causes a discrepancy between how data is handled in the training and serving pipelines, and recommend logging the features used at serving time to train on them (Google). On a new line, a different historian interpolation setting or unit of measure is enough to create one.
Common naming reduces this work. ISA-95 aims to "establish common terminology" and provides an equipment hierarchy that applies across discrete, continuous and logistics industries (ISA). A shared structure for site, area, line and equipment lets the same pipeline find the same signals everywhere. Our posts on the unified namespace and OPC UA for data scientists cover two ways to build one.
Why don't hand-built pilots survive replication?
Google engineers studying technical debt in machine learning systems show that the model code is a small part of the whole. They warn about "glue code", "pipeline jungles" of scrapes and joins, and configuration that teams may treat as an afterthought, and note that mature systems "may have dozens or hundreds of models running simultaneously", each with configurations to update safely (Sculley et al., 2015).
A pilot often sits at the bottom of Microsoft's MLOps maturity model. At level 0, builds, deployments and training are manual, and the result is "typically a single model file" handed off manually. Higher levels automate training and deployment, trace each deployment back to its data, and at level 4 let drift signals trigger retraining (Microsoft).
A rollout to five lines does not need level 4. It does need a scripted pipeline, version control for code, models and per-line configuration, and accuracy tracking in one place. Otherwise every new line is a new pilot.
Should you roll out line by line or all at once?
The WEF describes three rollout strategies used by Lighthouse companies (WEF, 2023):
| Approach | How it works | Suits |
|---|---|---|
| Technology-led | One use case deployed at many sites at once, run from a central technical organisation | Strong central IT and similar production lines |
| Capability-led | A centre of excellence packages use cases and pushes them to sites; one example covered energy optimisation at 32 sites | Sites that pull support and customisations from a central team |
| Build and replicate | One site is transformed first, then copied to the others one at a time | Large plants that operate similarly |
Whichever you choose, deploy in stages. Google's site reliability engineers define canarying as "a partial and time-limited deployment of a change in a service and its evaluation", which requires the ability to "compare the canary signals to the control signals" (Google SRE). In a plant, the subset is the next line, the control is that line's results without the model, and shadow mode is the first stage: the model runs and is scored, but nobody acts on it yet.
The WEF also describes "assetization", packaging a use case for fast deployment. Agilent bundled computer vision tools with plug-in connectors to MES and test software, SOPs, user documentation and training guides for operators, and deployed five new applications across 57 work centres and 16 product lines in less than four months (WEF, 2023). Package the use case, not only the model.
Who changes the procedures, and who signs off?
A model that changes how operators run the process is a process change. In US process safety regulation, employers must have written procedures to manage changes to "process chemicals, technology, equipment, and procedures", and affected employees must be informed and trained before start-up. The non-mandatory appendix lists "computer program revisions and changes in alarms and interlocks" among the changes covered (29 CFR 1910.119). Algerian plants follow their own regulations and company procedures, but the logic holds: run each new line through your management of change process and check the requirements with your HSE department.
Joint guidance led by CISA asks plant owners to take inventory of AI components, log their inputs and outputs, and "incorporate new AI system failure states, including how to bypass or replace an AI system" into functional safety and incident response processes (CISA et al., 2025). With ten lines, that is a real list to maintain. Our post on OT cybersecurity when you add AI covers the network side.
Someone has to drive all this. Nearly 70% of Lighthouses cite a transformation office as the most critical of the six enablers of scale the WEF identifies (WEF, 2023).
How do you budget and contract for the rollout?
A pilot is funded as a project. A rollout needs a running budget for monitoring, retraining, support and the owner's time on every line. Lighthouses typically anticipate a payback period of around two and a half years (WEF, 2023). Our guide to industrial AI project cost lists the cost lines.
Measure value the same way on every line. The WEF framework includes validating impact "by tracking key metrics in standardized format" (WEF, 2023), which addresses the barrier in the Gartner survey.
For Algerian public buyers, Law 23-12 of 5 August 2023 shapes the contract:
- Article 16 requires needs to be determined before any procedure starts and forbids splitting them to avoid tender procedures.
- Article 30 allows a contract with a firm tranche and one or more conditional tranches when economic or financial conditions justify it. A pilot as the firm tranche and a rollout as a conditional tranche fits this structure.
- Article 13 exempts public economic enterprises from the law's award rules, but requires them to adopt their own procedures based on its principles.
This is a summary, not legal advice. Confirm your case with your procurement department and counsel.
What should a rollout plan include?
- Scope the rollout before the pilot starts. List the lines and sites, and pick a pilot line that resembles them rather than the best-equipped one.
- Write a pass criterion per line. Metric, baseline, test period and pass mark, agreed with the line's production and quality managers.
- Inventory line-to-line differences. Instruments, tag names, sampling rates, products, raw materials and operating modes.
- Automate the data path. Scripted extraction, consistent naming, and version control for code, models and per-line configuration.
- Package the use case. Model, connectors, configuration, SOPs, training material and the fallback procedure.
- Deploy in stages. Shadow mode on the next line, compare with its own results, then switch on.
- Run change control and train every shift before each go-live.
- Name owners. One per site and one central, with monitoring and retraining triggers. See model drift monitoring.
- Fund the running costs and contract the rollout as a phase, not an afterthought.
What we have seen at KABAS AI
Our two published projects are R&D deployments of Virtual Lab, not production-line rollouts, but both case studies describe expansion after the first deployment. ENAD Shymeca followed five steps: discovery, platform configuration, data integration with migration of historical formulation data, training and adoption, and continuous optimisation. The case study reports a 40% reduction in physical testing cycles and says ENAD Shymeca continues to expand its use of Virtual Lab across additional product lines. ADFC is expanding its use across additional drilling fluid applications and project types, and validated formulations are structured as reusable data for future projects.
The practical lesson: what carries over to the next product line or site is structured data and a repeatable method, more than the first model. Decide early where the system will run, because DALILE and Virtual Lab are deployed on the customer's own infrastructure, so every new site raises the same hosting and integration questions. And keep the client's own experts at the centre of each deployment, as discussed in building a first industrial data team.
Where should you start?
If a pilot has worked and the rollout has not started, write the plan above before funding anything else. An AI readiness assessment of each target line shows where the data and connectivity gaps are. Our manufacturing page lists the use cases we address, and our consulting team runs readiness assessments, use case identification and AI roadmaps. Contact us to discuss your case.
Sources
- Wilkinson, Enterprises struggle to show the value of AI projects (Gartner survey), CIO Dive, 17 May 2024
- World Economic Forum, Global Lighthouse Network: Adopting AI at Speed and Scale, December 2023
- Koh et al., WILDS: A Benchmark of in-the-Wild Distribution Shifts, arXiv 2012.07421, 2021
- Google for Developers, Rules of Machine Learning
- ISA, ISA-95 Standard
- Sculley et al., Hidden Technical Debt in Machine Learning Systems, NeurIPS 2015
- Microsoft, MLOps maturity model, Azure Architecture Center
- Google, Site Reliability Workbook, Canarying Releases
- 29 CFR 1910.119, Process safety management of highly hazardous chemicals (GovInfo, 2025 edition)
- CISA, ASD's ACSC et al., Principles for the Secure Integration of Artificial Intelligence in Operational Technology, 3 December 2025
- Law 23-12 of 5 August 2023 on public procurement, Journal officiel no. 51 (FAOLEX copy)
Frequently asked questions
What is the difference between an AI pilot and a rollout?
A pilot answers one question: does the model deliver value on one line, with one team, under close supervision? A rollout makes that result repeatable on every line and site, with automated data pipelines, integration with plant systems, trained operators on every shift, change control, monitoring and a running budget. Most of the work in a rollout is outside the model.
How long does it take to move an AI project from prototype to production?
A Gartner survey of 644 organisations, reported by CIO Dive in May 2024, found that the prototype-to-production cycle takes about eight months on average and that fewer than half of AI projects reach production. In the World Economic Forum's 2023 Global Lighthouse Network report, the leading factories took 10 to 20 months for their first use cases, and 75% now say they can deploy a new one in less than six months.
Can we reuse the same model on every production line?
Sometimes, but do not assume it. Lines differ in instruments, tag names, equipment age, raw materials and product mix, and machine learning models lose accuracy when the data they see differs from their training data. Run the model in shadow mode on each new line, compare it with the line's own reference measurements, and recalibrate or retrain it where needed.
Should we roll out to all sites at once or one site at a time?
It depends on how similar the sites are and where the skills sit. The World Economic Forum describes three approaches: one use case deployed at many sites at once by a central team, a centre of excellence that packages use cases and pushes them to sites, and a build-and-replicate model that transforms one site at a time. Whatever the approach, deploy in stages and check each stage before the next.
Does an AI tool need a management of change review?
If its output changes how operators run the process, treat it as a change. In US process safety regulation, management of change covers changes to process technology, equipment and procedures, and the non-mandatory guidance lists computer program revisions and changes in alarms and interlocks among the examples. Follow your company's own procedure and Algerian requirements, and confirm with your HSE department.
How can an Algerian public buyer contract a pilot followed by a rollout?
Law 23-12 of 5 August 2023 on public procurement requires needs to be determined before the procedure starts and forbids splitting them to avoid tender procedures. Article 30 allows contracts with a firm tranche and one or more conditional tranches, which fits a pilot followed by a rollout. Public economic enterprises apply their own procedures based on the law's principles. Confirm your case with counsel.



