Table of Contents
Oil and gas data is ready for AI when you can name the decision the model supports, extract the historian, well test, lab and maintenance records it needs with correct timestamps and tag mapping, count enough labelled events, and confirm who owns the data and where it may be processed. The checklist below tests each point.
Key takeaways
- Volume is rarely the problem. Process data is "often abundant but poor in information" (Lawrence et al., 2024), and in fields without continuous flow measurement, well tests provide new reference data at most 1-2 times per month (Grimstad et al., 2021).
- A historian archive is not raw data. Compression is designed to discard values that can be recreated by interpolation, and default settings are not zero (AVEVA, 2023).
- Maintenance records are useful only when failures are coded consistently. ISO 14224 defines minimum equipment, failure and maintenance data for the petroleum, petrochemical and natural gas industries.
- In Algeria, upstream data are the exclusive property of the State and managed by ALNAFT (Law 19-13, Article 18). Check with counsel before using them in an AI project.
Why does data readiness decide an oil and gas AI project?
Most oil and gas companies have years of data. The question is whether it contains the information a model needs.
A survey of machine learning in process industries notes that process data contains "significant disturbances, outliers, and missing values", and that the lack of labelled data is the main challenge for soft sensors (Lawrence et al., 2024). Joint guidance led by CISA puts it plainly: AI models "can only be as effective as the quality of their training data", and collecting normalised sensor data is difficult in distributed OT environments (CISA et al., 2025).
Production data has a further limit: low variety. Operators keep a well's production fairly constant during plateau, then open chokes gradually during decline, which creates correlations that mislead data-driven models (Grimstad et al., 2021). Years of steady operation can teach a model less than a few months of varied operation.
Data that is not ready is one of the usual reasons projects stall; see why industrial AI projects fail.
What data does each oil and gas use case need?
Readiness depends on the use case. A drilling model and a document assistant need different data. The table maps common use cases, described in our overview of AI use cases in North African oil and gas, to their data and the gaps to look for.
| Use case | Main data | Reference or labels | Gaps to check |
|---|---|---|---|
| Drilling optimisation | Real-time rig and downhole data, often exchanged in WITSML | Drilling reports, recorded non-productive time | Data held on service company systems; rig-to-rig differences in sensors |
| Well log interpretation | Logs, cores, seismic, studies | Core analyses, interpreted logs | Gaps in logs; upstream data ownership |
| Production and virtual flow metering | Wellhead pressures, temperatures, choke opening | Well tests, multiphase meters | Few tests per well; low variety |
| Predictive maintenance | Vibration, process data, CMMS work orders | Coded failure records | Free-text work orders; few failures |
| Process quality in gas plants and refineries | Historian tags, LIMS | Lab results | Sample time vs entry time; historian compression |
| Document assistants | Procedures, manuals, reports in Arabic, French and English | Questions with checked answers | Scanned pages; old versions |
Gaps in logs are common. A study of 1,026 North Sea wells found 650 gaps in neutron porosity logs across 323 wells, caused by tools that fail, recording switched off at the wrong time, or the wrong interval logged (Lopes and Jorge, 2017). For document assistants, see our guides to RAG on maintenance manuals and OCR for scanned technical documents.
The checklist: twelve checks before the pilot
Run these for one use case, not for the whole company. Each check has a simple test.
- The decision is written down. Name the decision the model will change, who makes it, and the KPI. Test: the asset team agrees on one sentence.
- Every source is listed with an owner. Historian, LIMS, CMMS, well test database, drilling data store, document management. Test: for each source, a named owner and an access route.
- The history covers the operating modes. Plateau and decline, feed changes, turnarounds, seasonal conditions. Test: plot the target and key inputs over the full period and mark the modes.
- You know what the historian kept. Compression uses a "swinging door" algorithm to drop values that can be recreated by interpolation (AVEVA, 2023). Test: export compression and exception settings for the tags you need. Our guide to process historian data for machine learning goes further.
- Timestamps are aligned. Lab results carry the sampling time, not the entry time. Clocks and time zones match across systems. Inputs sampled faster than the target are synchronised (Lawrence et al., 2024). Test: line up ten lab samples with the process trend by hand.
- Tags map to equipment and wells. Consistent names, units and an asset hierarchy. Test: pick twenty tags at random and trace each to its equipment or well.
- Labels are counted. Well tests, lab results, coded failures. Test: count them per well or per equipment item over the period. If there are only a handful, plan for it.
- Failures and maintenance are coded. Failure mode, cause and downtime, not only free text. ISO 14224 is a ready-made reference. Test: read twenty recent work orders and see whether the failure can be identified.
- Events are logged. Shutdowns, workovers, calibrations, instrument replacements. Test: check that last year's events can be found with their dates.
- Ownership and permitted use are confirmed. Company data, partner data, service company data and upstream data held for the State. Test: legal confirms in writing what may be used and by whom.
- The processing location is decided. On site, in a national data centre, or elsewhere, and how data leaves the OT network. Test: the security team approves the data path.
- The dataset is documented. Why it was built, what it contains, how it was collected and its recommended uses, as proposed in datasheets for datasets. Test: a new engineer can understand it without asking.
Score each check as ready, fixable during the pilot, or a blocker. Missing labels and unconfirmed ownership are usually blockers; inconsistent tag names can often be fixed during the pilot. Our broader AI readiness assessment adds connectivity, organisation and budget. Typical data defects are covered in industrial data quality problems.
Which standards help structure oil and gas data?
You do not need to adopt a standard before a first project. But the standards below give names and structures that make data easier to map, share and reuse.
| Standard | What it covers | Use in an AI project |
|---|---|---|
| WITSML (Energistics) | Drilling, completions and interventions data, transmitted from the rig site to service companies and operators, and shared with partners | A common format for drilling data from several rigs and contractors |
| ISO 14224 | Equipment taxonomy, failure and maintenance data for petroleum, petrochemical and natural gas equipment | Consistent failure modes and causes to train and evaluate maintenance models |
| ISA-95 / IEC 62264 | Models and terminology for the interface between enterprise and control functions | A shared vocabulary linking plant data to production and business data |
| OPC UA | Platform-independent data exchange with an information modelling framework and built-in security | Reading control system data with its context |
| OSDU (The Open Group) | A data model and API specification, with an open-source reference implementation | A target structure for subsurface and well data |
Who owns oil and gas data in Algeria?
Law 19-13 of 11 December 2019 on hydrocarbon activities defines "data" broadly: results, information, cores, studies, and geological, geophysical, hydrogeological and geochemical information acquired during upstream activities, including 2D, 3D and 4D seismic (Law 19-13, Article 2).
Article 18 states that these data are the exclusive property of the State. ALNAFT manages, preserves, uses, makes available and distributes them, and the national company, contracting parties and prospectors must transmit them to ALNAFT under its procedures. Article 19 gives ALNAFT the management of the data bank for the hydrocarbon mining domain and the power to set conditions for access and distribution.
For an AI project, this means upstream data is not simply the operator's to share with a vendor or send to an external service. Downstream plant data and maintenance records fall under other rules and your contracts. This is a summary of the law's text, not legal advice: confirm the conditions that apply to your project with your legal counsel and, where relevant, with ALNAFT. Our post on industrial data governance covers internal rules for data ownership.
The CISA-led guidance adds a security angle: know where OT data used for training is stored, keep it within the organisation's control, and be aware that companies based abroad are subject to foreign laws (CISA et al., 2025). It recommends push-based architectures that move data out of the OT network without persistent inbound access. See data diodes and one-way transfer and integrating AI with SCADA and OT systems.
Who should do the readiness work?
Data readiness is not a task for the data scientist alone. The CISA-led guidance notes that capturing safety-related edge cases needs "OT operator expertise" (CISA et al., 2025).
A workable team has a production or process engineer who knows the asset, an OT or IT engineer who knows the systems, and a data scientist. Give each dataset a named owner who stays responsible after go-live, when new wells, new tags and replaced instruments appear. Our article on building a first industrial data team describes the roles.
What we have seen at KABAS AI
Our published oil and gas project is at ADFC (Sonatrach Group), where Virtual Lab supports drilling fluid formulation. The case study describes formulation experience "dispersed in various formats and locations" before the project. Virtual Lab now holds formulation data, test results and validated recipes in one structured repository, with traceability for audits, and the deployment sits within Sonatrach's IT environment.
Three lessons from that project apply to any oil and gas use case:
- Structure the records you already have before collecting new ones. Past tests and reports are the labels most projects lack.
- Keep the physical measurement as the reference. At ADFC, comparisons between simulated and actual results guide refinements.
- Decide the hosting early. When data must stay inside the company, hosting shapes integration and cost. DALILE and Virtual Lab can both run on the customer's own infrastructure.
Where should you start?
Pick one use case with a clear decision and a known reference measurement, such as well tests or lab results, and run the twelve checks on it. The blockers you find are the first work package, before any model.
Our consulting team can run this review with your production, OT and data teams. See our oil and gas solutions, or contact us to discuss a specific asset.
Sources
- Lawrence et al., Machine learning for industrial sensing and control: A survey and practical perspective, Control Engineering Practice, 2024 (arXiv 2401.13836)
- Grimstad et al., Bayesian Neural Networks for Virtual Flow Metering: An Empirical Study, Applied Soft Computing 112, 2021 (arXiv 2102.01391)
- Lopes and Jorge, Mind the Gap: a Well Log Data Analysis, Workshop on Data Mining for Oil and Gas, 2017 (arXiv 1705.03669)
- CISA, ASD's ACSC et al., Principles for the Secure Integration of Artificial Intelligence in Operational Technology, 3 December 2025
- Bregenzer, Exception, Compression, and their Impacts on PI System Performance, AVEVA, 2023
- EVS-EN ISO 14224:2016, Petroleum, petrochemical and natural gas industries: Collection and exchange of reliability and maintenance data for equipment (catalogue description)
- Energistics, WITSML data standards
- ISA, ISA-95 Standard: Enterprise-Control System Integration
- OPC Foundation, Unified Architecture
- The Open Group, OSDU Forum
- Gebru et al., Datasheets for Datasets, Communications of the ACM, 2021 (arXiv 1803.09010)
- Loi n° 19-13 du 11 décembre 2019 régissant les activités d'hydrocarbures, Journal officiel n° 79 du 22 décembre 2019 (FAOLEX)
Frequently asked questions
How much historical data does an oil and gas AI project need?
There is no fixed number of years. What matters is the number of labelled examples, such as well tests, lab results or coded failures, and whether the history covers the operating modes the model will meet. In fields without continuous flow measurement, well tests are done at most once or twice a month, so one well yields at most about 24 reference points per year.
Do we need to clean all our data before starting an AI project?
No. Run the checklist for one use case and fix the data that use case needs. Cleaning every tag and record in a plant or field first is slow, and much of it may never be used. The work done for the first use case, such as tag mapping and event logs, is reused by the next ones.
Do we need to adopt OSDU, WITSML or ISO 14224 before starting?
No. These standards are useful references for naming, failure coding and data exchange, and following them makes data easier to reuse. But a first project can start from the systems you already have, as long as the data for its use case is mapped, documented and consistent.
Can we use drilling or well data held by service companies to train a model?
Check your contracts first, because they define who holds and may use the data. In Algeria, Law 19-13 states that upstream data such as seismic, logs, cores and studies are the exclusive property of the State and are managed by ALNAFT. Confirm with your legal counsel how that applies to your project before data is shared or used for training.
Who should own data readiness in an oil and gas company?
A small team: a production or process engineer who knows the asset, an OT or IT engineer who knows the systems, and a data scientist. One named person should own each dataset after go-live, so that new tags, new wells and instrument changes are recorded.
Can the data stay on site during an AI project?
Yes. Models and data can run on the company's own infrastructure. Joint guidance led by CISA advises operators to know where OT data used for training is stored, to keep it within the organisation's control and to understand how vendors access it.



