Table of Contents
Edge AI on the factory floor means running a trained model on hardware next to the machine, such as a PLC module, smart camera or industrial PC, instead of a central server. Use it when a decision must come within milliseconds, when data such as video is too heavy to send, or when the line must run without a network.
Key takeaways
- Joint guidance led by CISA notes that some modern PLCs or edge controllers already run lightweight, pre-trained models for tasks such as local anomaly detection (CISA et al., 2025).
- Hardware ranges from PLC modules to embedded GPUs. NVIDIA rates its Jetson AGX Orin Industrial module for -40 °C to 85 °C and a 10-year operating lifetime (NVIDIA).
- Quantization makes models fit small devices: full integer quantization makes a model 4 times smaller and over 3 times faster, but can reduce accuracy, particularly for smaller networks (Google LiteRT).
- Train on servers, run at the edge, and manage models as versioned packages that keep working when the network is down.
- A model that acts on the machine is part of the control system: it needs a fallback, change management and drift monitoring.
What counts as "edge" in a plant?
NIST describes edge computing as the layer of end devices that provides "local computing capability on a sensor, metering or some other devices that are network-accessible" (NIST SP 500-325). On a factory floor, that means a module in the PLC rack, a smart camera above a conveyor, or a fanless industrial PC in the machine cabinet.
Edge is not the same as on-premise. A GPU server in the site data room is on-premise, but it is not at the edge: it sits at Level 3 or 4 of the Purdue model, away from the machine. The CISA-led guidance lists edge models under Level 1, where "some modern PLCs or edge controllers execute lightweight, pre-trained predictive models" for local anomaly detection, load balancing and maintaining a known safe state (CISA et al., 2025).
When should inference run at the edge?
Four reasons justify putting a model next to the machine.
Timing. A reject gate, a robot pick or a stop signal must act within the machine's cycle. The CISA-led guidance warns that AI systems "may not meet the strict timing requirements of OT environments", and that this matters more when the AI system is involved in control. A model in the controller or beside it avoids the network round trip.
Data volume. Cameras and vibration sensors produce far more data than the decision needs. One uncompressed 1080p colour camera at 30 frames per second produces 1,920 × 1,080 × 3 bytes × 30, about 187 MB/s or 1.5 Gbit/s by our arithmetic, more than a gigabit link carries. Real cameras often use monochrome sensors, lower frame rates or compression, but sending a verdict of a few bytes instead of the images is still the simpler design.
Availability. A line that depends on a central server stops when the network does. A model on the device keeps inspecting during a switch failure or a server update. For sites where the link to headquarters is itself unreliable, see AI at remote sites with poor connectivity.
Data containment. Images of products, operators' stations or process data never need to leave the cell; only results and summaries do.
When none of these apply, a server is usually easier to run. A soft sensor updated every minute, a model that needs data from the whole plant, or an assistant that answers questions from documents belongs on a server.
| Where the model runs | Example hardware | Keeps running if the plant network fails | Typical models | Who owns changes |
|---|---|---|---|---|
| In the controller | PLC AI module, inference engine in the PLC runtime | Yes | Small classifiers, anomaly scores on machine signals | Control engineers, under control logic change procedures |
| Beside the machine | Smart camera, embedded GPU box, industrial PC | Yes | Vision inspection, vibration analysis, small language models | Automation and data teams together |
| Site server room | GPU servers at Level 3 or 4 | No, for users on the floor | Training, plant-wide predictive models, LLM assistants | IT and data teams |
| Cloud | Provider data centres | No | Training and analytics, if data may leave the site | IT and the provider |
What hardware runs edge AI?
Inside the controller. In 2018 Siemens launched the TM NPU module for the S7-1500 and ET 200MP, with an Intel Movidius Myriad X vision processing unit, USB 3.1 and Gigabit Ethernet for cameras or microphones, and a trained neural network supplied on an SD card; the results are evaluated in the CPU program (Siemens). Beckhoff takes a software route: its TwinCAT inference engines execute models exported as ONNX files "directly in the real-time context of the control system", and a separate server option uses GPUs for "near real-time" processing of larger models (Beckhoff).
Embedded GPU modules. NVIDIA's Jetson Orin family is a common choice for vision at the edge. Figures from NVIDIA's specifications:
| Module | AI performance (sparse INT8) | Memory | Power range |
|---|---|---|---|
| Jetson Orin Nano 8GB (Super) | 67 TOPS | 8 GB | 7 to 25 W |
| Jetson Orin NX 16GB | 157 TOPS | 16 GB | 10 to 40 W |
| Jetson AGX Orin 64GB | 275 TOPS | 64 GB | 15 to 60 W |
| Jetson AGX Orin Industrial | 248 TOPS | 64 GB with ECC | 15 to 75 W |
Read TOPS as a peak, not as the speed of your model. For the Orin Nano 8GB in Super mode, NVIDIA gives 67 TOPS sparse but 33 TOPS dense (NVIDIA, 2025). The industrial module also lists inline ECC, shock and vibration ratings and a 10-year operating lifetime, against 5 years for the commercial version.
Industrial PCs. Many lines already have x86 industrial PCs. Intel's open-source OpenVINO toolkit runs inference on CPUs, Intel GPUs and Intel NPUs, and accepts models from PyTorch, TensorFlow, ONNX and other frameworks (OpenVINO).
Can a language model run at the edge?
Small ones can. NVIDIA measured these generation speeds on a Jetson Orin Nano 8GB in Super mode, with 4-bit quantization (NVIDIA, 2025):
| Model | Tokens per second |
|---|---|
| Llama 3.2 3B | 43.1 |
| Qwen 2.5 7B | 21.8 |
| Llama 3.1 8B | 19.1 |
That is enough for one operator asking a question at a station. It is not enough for a site-wide assistant that answers many users from long documents in Arabic, French and English; that needs a server sized for concurrent requests, as our GPU sizing guide explains. Our guide to small language models covers which tasks a 3B to 8B model handles well, and 4-bit quantization makes testing on your own questions even more important.
How do you fit a model onto edge hardware?
Use an open exchange format. ONNX is "an open format built to represent machine learning models", so a model trained in one framework can run in another inference engine (ONNX). Beckhoff's engines and OpenVINO both accept it.
Quantize, then measure. Google's LiteRT documentation lists dynamic range quantization at 4 times smaller with a 2 to 3 times speedup, and full integer quantization at 4 times smaller with over 3 times speedup, the latter also suited to microcontrollers and Edge TPUs (Google LiteRT). ONNX Runtime warns that quantization "is not a loss-less transformation", recommends static quantization with calibration data for CNNs, the architecture behind many vision models, and dynamic quantization for transformers (ONNX Runtime).
Run the quantized and full-precision models on the same held-out test set, including the rare defects and unusual operating conditions, before you accept the smaller one.
How do you deploy edge AI, step by step?
- Define the decision and its deadline. What the model decides, how fast, and what the machine does if the model gives no answer. The CISA-led guidance asks for failsafe mechanisms that revert to traditional automation or manual operation.
- Collect data at the station. Use the real camera, lighting and sensors, across shifts, products and seasons. Our guide to computer vision for defect detection covers images and labels.
- Train on servers you control, where the data history and the compute are.
- Export, quantize and check accuracy against the full-precision model, as above.
- Benchmark on the target device in the power mode, temperature and software load it will have in service.
- Run in shadow mode. The model logs its verdicts while the line runs as before, and engineers compare them with reality.
- Go live under change management, with the fallback tested and the previous version kept for rollback.
- Monitor and update. Track accuracy and input data over time, as described in our guide to model drift monitoring.
How do you manage models on many devices?
Treat each model as a versioned software package. In Azure IoT Edge, for example, modules are container images, and devices "can keep operating indefinitely" offline after one initial sync with the cloud service. Messages waiting to go out expire after a time-to-live that defaults to 7,200 seconds (two hours), so raise it if outages can last longer (Microsoft). Plants that do not want a cloud dependency can run the control plane on their own servers: KubeEdge, a graduated Cloud Native Computing Foundation project, supports "autonomous operation of edge even during disconnection from cloud" (KubeEdge).
Whatever the tool, four rules apply:
- Push results out; do not open paths in. The CISA-led guidance prefers push-based or brokered architectures that move features or summaries out of OT without persistent inbound access. Publishing verdicts to a broker, as in a unified namespace, fits this pattern.
- Secure the devices as OT components. ISA/IEC 62443-4-2 sets technical security requirements for IACS components (ISA). Our article on OT cybersecurity when adding AI covers the rest.
- Keep a record of what runs where: device, model version, training data and date of deployment.
- Update and validate regularly. The same guidance asks owners to regularly update and validate AI models for accuracy and effectiveness. For isolated sites, plan how new versions get in; see air-gapped AI deployment.
What we have seen at KABAS AI
DALILE runs completely offline on the client's own servers and connects to systems such as SAP, Oracle, LIMS, document management systems and SQL databases. It is a server-side system, not an edge device, and that split is the practical guidance we give: put narrow, fast decisions on edge devices next to the machine, and keep training, fleet management and language-model assistants on servers you control.
The two sides meet through data. When edge devices write their results, such as defect counts or anomaly flags, to a database the assistant can reach, engineers can query those results alongside manuals and quality records instead of in yet another separate tool. The figures in this article come from the public sources cited, not from KABAS deployments. Our information systems team maps and integrates these systems, and our manufacturing page describes how we support production and quality teams.
Sources
- CISA, ASD's ACSC and partners, Principles for the Secure Integration of Artificial Intelligence in Operational Technology, December 2025
- NIST SP 500-325, Fog Computing Conceptual Model, 2018
- Siemens, Artificial Intelligence for Simatic, press release PR2018110017DFEN, November 2018
- Beckhoff, TwinCAT 3 Machine Learning
- NVIDIA, Jetson Orin technical specifications
- NVIDIA, JetPack 6.2 Brings Super Mode to NVIDIA Jetson Orin Nano and Jetson Orin NX Modules, January 2025
- OpenVINO toolkit, GitHub repository
- ONNX, Open Neural Network Exchange
- Google, LiteRT post-training quantization
- ONNX Runtime, Quantize ONNX models
- Microsoft, Operate Azure IoT Edge devices offline
- Microsoft, How Azure IoT Edge modules run logic on devices
- KubeEdge project website
- ISA, ISA/IEC 62443 series of standards
Frequently asked questions
What is the difference between edge AI and on-premise AI?
Both keep data on your site. On-premise AI usually runs on servers in a data room, serving many users or lines from one place. Edge AI runs on small devices next to the machine, such as a PLC module, a smart camera or an industrial PC, so it can answer within the machine's timing and keep working if the plant network fails. Most plants need both: edge devices for fast local decisions and servers for training, fleet management and language models.
Do I need a GPU for edge AI?
Not always. Small classification or anomaly models can run on a CPU, especially after 8-bit quantization, and some controllers run them inside the PLC runtime. Accelerators such as Intel NPUs or a vision processing unit on a PLC module also exist. Embedded GPUs such as NVIDIA Jetson modules make sense for camera inspection at high frame rates or for several models on one device.
Can edge AI keep working without an internet connection?
Yes. Inference runs on the device, so it does not need the internet. Management tools are built for this too: Microsoft documents that Azure IoT Edge devices keep operating indefinitely offline after one initial sync, and KubeEdge supports autonomous operation during disconnection from the cloud. What you lose while disconnected is remote updates and central monitoring, so plan how new model versions reach the site.
Should an edge model act directly on the machine?
It can, for example by triggering a reject gate on a conveyor, but then it is part of the control system and must follow the same safety review and change management as control logic. Joint guidance led by CISA advises limiting active control by AI without a human in the loop and having failsafe mechanisms that revert to traditional automation or manual operation. Never use an AI model as a safety function.
Can a language model run on an edge device?
Small ones can. NVIDIA reports about 19 tokens per second for Llama 3.1 8B on a Jetson Orin Nano 8GB in its Super mode, using 4-bit quantization. That suits one operator at a station, not a site-wide assistant answering many users from long documents, which belongs on a server.
How often should edge models be updated?
There is no fixed interval. Update when monitoring shows that accuracy is falling, after a change to the product, the camera, the lighting or the process, or when a better model passes your tests. Every update should go through testing, shadow running and change management like any other change to the line.



