Table of Contents
To start a computer vision defect detection project, pick one station and a few defects that cost money, set up lighting and a camera that make those defects clearly visible, and collect images from real production. Then train a simple model, compare it with your inspectors on parts it has never seen, and only then connect it to the line.
Key takeaways
- Lighting and resolution decide what a camera can see. A defect should cover at least 3 to 4 pixels for clean detection (Vision Doctor).
- Anomaly detection models learn from defect-free images only, which suits lines where defects are rare. PatchCore reached an image-level AUROC of up to 99.6% on the MVTec AD benchmark (Roth et al., 2022).
- Supervised models can work with few defects: one segmentation method learned surface-crack detection from about 25 to 30 defective samples (Tabernik et al., 2019).
- Human inspection is the baseline to beat, and it is not perfect. In a Sandia study, inspectors rejected 85% of defective parts and 35% of good ones (See et al., 2015).
- Validate on parts, lots and days the model has never seen, and set the alarm threshold from the cost of a false reject and an escape.
What can computer vision do for inspection?
A camera, a light and a model look at each part, or each section of a continuous strip, and output a decision. The output can take four forms, and the choice shapes how much labelling you need.
| Approach | Output | Training labels | Suits | Watch out for |
|---|---|---|---|---|
| Image classification | OK or not OK, or a defect class, per image | One label per image | One part per image, fixed view | Says nothing about where the defect is |
| Object detection | A box and a class for each defect | A box around each defect | Locating and counting discrete defects | Needs labelled examples of every defect type |
| Segmentation | A pixel mask of each defect | Pixel-level masks | Measuring defect size or area against a specification | Labelling is slow |
| Anomaly detection | An anomaly score and a heat map | Defect-free images only | Rare or unknown defects, new lines | Flags any change, and does not name the defect |
The first three are supervised: the model learns from images that a person has labelled. Anomaly detection learns what a good part looks like and scores how far each new image is from it, similar to the anomaly detection methods used on sensor data.
Vision inspection looks at the finished surface. If a defect is decided earlier in the process and is not visible, predictive quality on process data is the better tool, and the two can be combined.
Why not leave it to human inspectors?
Visual inspection is hard work for people. In a Sandia National Laboratories study, 82 inspectors examined 140 parts for eight defect types. They correctly rejected 85% of defective items and incorrectly rejected 35% of acceptable parts. The authors note that this hit rate was not vastly superior to an industry average of 80%, and describe inspection as a workload-intensive task dominated by mental demand and effort (See et al., 2015).
Two lessons follow. A vision system does not need to be perfect to be useful; it needs to beat your current inspection on your parts. And your inspectors' decisions are not a perfect reference either, so the labels you train on need checking.
Start with the defect catalogue, not the model
Before buying a camera, write down:
- Which defects matter: name, photo, and the cost of each one reaching the customer.
- Acceptance limits: the size, depth or position at which a scratch becomes a reject. Borderline cases cause most of the disagreements later.
- Where the defect is visible: which face of the part, at which station, before or after which operation.
- Line speed and handling: parts per minute, whether the part stops, how it is presented to the camera.
MVTec separates two kinds of defects. Structural anomalies "appear as scratches, dents, or contaminations"; logical anomalies "violate underlying constraints", such as a permissible object in the wrong place or a required part that is missing (MVTec LOCO AD). Check which kind your catalogue contains: a missing clip and a scratched surface may need different methods, or a simple rule-based check for the first.
Why do lighting and the camera matter more than the model?
A model cannot find a defect that the image does not show. Lighting is chosen for the specific surface features that must be highlighted (Opto Engineering).
| Lighting technique | How it works | What it shows |
|---|---|---|
| Bright field (front light) | Light reflected by a flat surface enters the lens | Scratches and defects appear dark on a bright background |
| Dark field (low angle) | Only light scattered by the surface reaches the lens | Surface irregularities appear bright on a dark background |
| Backlight | Light source behind the part | The outline of opaque parts; inclusions such as air bubbles in transparent plastic |
| Dome (diffuse) | Uniform light from many angles, like a cloudy day | Curved or complex parts without disturbing reflections |
| Coaxial | Light along the camera axis, perpendicular to the part | Defects on highly reflective surfaces |
Source: Opto Engineering.
Then size the resolution. Vision Doctor states that a spot of at least 3 to 4 pixels is required for clean optical detection, and smaller only with good contrast. Its example: a 64 mm wide image on 640 pixels gives 0.1 mm per pixel, so the smallest reliably visible feature is about 0.3 mm (Vision Doctor). Work backwards from your smallest reject-size defect and the field of view you need.
Enclose the station against daylight and shop lighting. Changes in lighting, lens focus or camera position also change what the model sees.
Anomaly detection or a supervised model?
Anomaly detection fits when defects are rare or varied. The MVTec AD benchmark, with over 5,000 images in 15 object and texture categories, provides only defect-free images for training (MVTec AD). PatchCore, built for this "cold-start problem", keeps a memory bank of features from normal images and reached an image-level AUROC of up to 99.6% on it (Roth et al., 2022). The open-source Anomalib library, under the Apache-2.0 licence, includes PatchCore and other methods. The limit: the model flags anything new, including a new supplier's surface finish, and it does not tell you which defect it found.
Supervised models fit when you have, or can collect, examples of each defect type and need the defect named or measured. Tabernik et al. showed a segmentation-based model learning surface-crack detection from about 25 to 30 defective training samples, "instead of hundreds or thousands" (Tabernik et al., 2019). A follow-up showed that mostly image-level labels plus a small number of pixel-level masks can approach fully supervised performance at lower annotation cost (Božič et al., 2021).
A practical sequence is to start with anomaly detection to sort normal parts from suspect ones, review the suspect ones with inspectors, and use those reviewed images to train a supervised model later.
How many images and labels do you need?
Real defect datasets are imbalanced. KolektorSDD2, collected for an industrial inspection problem, has 3,335 images, of which 356 are defective (ViCoS Lab). Your line will likely have fewer defects than that, so collect normal images continuously and save every rejected part.
Cover the variation you will meet in service: every product variant, material lot, shift and camera position. Label inside your network with a self-hosted tool such as CVAT, whose community edition is open source and supports boxes, polygons and masks.
Check the labels. A study of ten of the most commonly used computer vision, language and audio datasets estimated an average of at least 3.3% label errors in their test sets (Northcutt et al., 2021). Have two inspectors label the same sample of borderline images and resolve disagreements against the written acceptance limits.
Do not train a production model on public datasets. MVTec AD and KolektorSDD2 are released under a non-commercial licence, and Ultralytics states that an Enterprise License is needed to use its YOLO models without open-sourcing your project (Ultralytics). Check the licence of every dataset, library and pretrained model before deployment.
How do you validate before trusting it?
Split by group, not by image. Images of the same part, lot or day look alike, and a random split puts near-copies in both training and test sets. scikit-learn recommends group-wise cross-validation when data comes from different "subjects, experiments, measurement devices", so that test samples come from groups not seen in training (scikit-learn). Hold out whole lots or days, and test on a later period.
Count what matters on the line: defects caught, good parts falsely rejected, and escapes, for each defect type.
Set the threshold from costs. The default 0.5 rule is "most certainly not ideal for most use cases", and the threshold should never be tuned on the training data (scikit-learn). A missed crack in a pressure part and a cosmetic stain do not deserve the same threshold.
Run in shadow mode. Let the system score every part for several weeks while inspectors keep deciding, then compare. Our guide to moving from pilot to plant rollout covers the sign-off.
Where should the model run?
Inspection decisions are needed within the cycle time, so models often run next to the camera on an industrial PC; our post on edge AI on the factory floor covers the hardware choices. Anomalib states that most of its models can be exported to OpenVINO for accelerated inference on Intel hardware (Anomalib).
The result then needs a destination: a reject actuator, the PLC, or the MES record of the part. Our guide to who does what between MES, ERP and AI explains where inspection results belong.
Once live, the model will drift when a supplier, tool, lens or light changes. Keep logging images and inspector decisions, and follow the routine in model drift monitoring.
How do you start, step by step?
- Pick one station and two or three defects with a cost you can state.
- Write the defect catalogue with photos and acceptance limits.
- Test lighting and resolution on known good and bad parts until every defect is clearly visible.
- Collect production images for several weeks, across variants, lots and shifts.
- Start with anomaly detection if defects are rare; add a supervised model when you have enough reviewed defects.
- Validate on held-out lots and days and report catches, false rejects and escapes.
- Set the threshold with quality and production managers.
- Run in shadow mode, then connect to the line, with a named owner who reviews performance every month.
What we have seen at KABAS AI
Our manufacturing solutions page lists quality control, automated inspection and defect detection among the use cases we address, and our consulting service lists computer vision among our areas of expertise. The data science team covers the path from data collection and preparation to predictive models and MLOps pipelines that keep models monitored in production.
Our practical advice: spend the first weeks on the defect catalogue, the lighting and the image collection, not on choosing a neural network. Agree the cost of a false reject and of an escape before training anything. Keep images and models on your own servers, since images of your parts and defects show how your product is made. Before you start, an AI readiness assessment helps check whether the data and the people are in place.
Sources
- See et al., Visual inspection reliability for precision manufactured parts, Human Factors, 2015 (Sandia National Laboratories)
- Roth et al., Towards Total Recall in Industrial Anomaly Detection, CVPR 2022 (arXiv 2106.08265)
- MVTec, MVTec AD dataset
- MVTec, MVTec LOCO AD dataset
- Tabernik et al., Segmentation-Based Deep-Learning Approach for Surface-Defect Detection, Journal of Intelligent Manufacturing, 2019 (arXiv 1903.08536)
- Božič, Tabernik and Skočaj, Mixed supervision for surface-defect detection: from weakly to fully supervised learning, Computers in Industry, 2021 (arXiv 2104.06064)
- ViCoS Lab, Kolektor Surface-Defect Dataset 2 (KolektorSDD2)
- Northcutt, Athalye and Mueller, Pervasive Label Errors in Test Sets Destabilize Machine Learning Benchmarks, NeurIPS 2021 (arXiv 2103.14749)
- Opto Engineering, Illumination geometries and techniques
- Vision Doctor, Resolution of sensors
- Anomalib, open-source anomaly detection library (GitHub)
- CVAT, Computer Vision Annotation Tool
- Ultralytics, Licensing
- scikit-learn, Cross-validation: evaluating estimator performance
- scikit-learn, Tuning the decision threshold for class prediction
Frequently asked questions
Can computer vision replace human visual inspectors?
Not on day one. A vision system first runs alongside inspectors, who keep deciding pass or fail while the model's results are compared with theirs. Inspectors also define the defect catalogue, label the images and review the parts the model flags, so their knowledge remains central even after the system goes live.
Do we need deep learning, or is rule-based machine vision enough?
If the defect can be described by a measurement, such as a missing hole, a wrong dimension or an absent label, rule-based tools that measure edges, areas and contrast may be enough and are easier to validate. Deep learning helps when defects vary in shape and appearance, such as scratches, cracks, stains or texture faults on natural or cast surfaces.
How many defect images do we need to start?
It depends on the method. Anomaly detection models are trained on defect-free images and need defective images only to test them. Supervised models need labelled examples of each defect type: one published method learned surface-crack detection from about 25 to 30 defective samples, but most lines need more to cover all defect types, part variants and lighting conditions.
Can we use public datasets such as MVTec AD to train our model?
Use them to learn and to compare methods, not to train a production model. MVTec AD and KolektorSDD2 are released under a non-commercial licence, and their parts, cameras and lighting differ from yours. A model for your line has to be trained and validated on images from your own station.
Does the vision system need a GPU on the line?
Not necessarily. Some inspection models can run on an industrial PC without a dedicated GPU, and libraries such as Anomalib export models to OpenVINO for Intel hardware. The deciding factors are line speed, image size and the number of cameras, so measure the inference time per image on the target hardware during the pilot.



