The Data Deluge Problem Nobody Planned For

A modern offshore platform generates sensor readings, alarm logs, equipment health signals, and process historian entries at a rate that would have been unimaginable when the facility was designed. The distributed control system (DCS) and safety instrumented system (SIS) were engineered to act on that data in real time, but nobody fully specified what to do with the accumulated archive. The result is familiar to most operations teams: terabytes of historian data sitting on servers, occasionally queried after an incident, otherwise ignored. Meanwhile, .

This article addresses how operations, maintenance, and engineering teams can build a practical big data capability without overpromising to management or underdelivering to the field.

What "Big Data" Actually Means in This Context

The term is overused, but the engineering definition is precise enough to be useful: big data in oil and gas refers to datasets that are too large, too fast-arriving, or too structurally varied to be handled by conventional relational databases and manual analysis workflows.

In practice this means:

  • Volume: years of high-frequency process historian data across hundreds of tags per asset
  • Velocity: real-time streaming from condition monitoring systems, multiphase flow meters, and subsea control modules
  • Variety: structured (SCADA tags), semi-structured (maintenance work orders in CMMS), and unstructured (inspection reports, well logs, free-text operator logs)

The operational value is not in the data itself. It is in the decisions the data enables — earlier detection of degradation, better scheduling of planned maintenance, and reduced false-alarm rates that cause alarm fatigue and desensitise control room operators.

Standards and Regulatory Context

Before deploying any analytics layer that touches safety-critical decisions, the team must understand where the boundary lies between advisory systems and safety functions.

IEC 61511 (Functional Safety — Safety Instrumented Systems for the Process Industry Sector) is unambiguous: a safety instrumented function (SIF) must be designed, validated, and maintained within a defined safety lifecycle. An analytics model running on historian data is not a SIF. If the output of a machine learning model is used to trigger a process action, that action pathway must be assessed under the IEC 61511 safety lifecycle — it cannot bypass it.

ISA-TR84.00.02 provides supplemental guidance on SIL assignment and management. When teams are debating whether a predictive alert should be wired into the SIS logic or handled as a separate advisory layer, the decision must be grounded in IEC 61511 functional safety lifecycle assessment, not ISA-TR84.00.02 alone. The general answer is: keep them separate unless a formal safety lifecycle assessment has been completed.

For rotating equipment, API 670 (Machinery Protection Systems) provides recommended practices for vibration, position, and temperature monitoring on critical machines. Any big data initiative that ingests vibration data from API 670-compliant systems should respect the primacy of the hardwired protection system — analytics is a complement, not a replacement. Any big data initiative that ingests vibration data from API 670-compliant systems must respect the primacy of the hardwired protection system — analytics is a complement, not a replacement.

Where pipeline integrity monitoring is in scope, ASME B31.8S (Managing System Integrity of Gas Pipelines) provides a framework for risk-based assessment and integrity management that can inform the design of data collection and analysis strategies for pipeline big data programmes.

Practical Architecture for Field Deployment

Data Acquisition Layer

The foundation is reliable tag collection. Before investing in cloud infrastructure or advanced analytics platforms, audit the existing historian — OSIsoft PI and similar systems are common — for tag completeness, scan rate consistency, and compression settings. Aggressive exception reporting compression can destroy the signal quality needed for vibration trending or early fouling detection. Verify that critical tags are being stored at a scan rate appropriate to the physical process being monitored.

For assets without existing instrumentation, wireless sensor networks (using ISA-100.11a or WirelessHART protocols) can extend coverage to locations where cabling is impractical. Confirm intrinsic safety certification and hazardous area classification before installation, in line with IEC 60079 requirements.

Data Contextualization

Raw tag values without context produce misleading models. A pump discharge pressure reading means different things during startup, steady-state operation, and a planned rate change. At minimum, each dataset should be tagged with:

  • Operating mode (from DCS state or manual annotation)
  • Maintenance events (from CMMS, including work order type and date)
  • Production rate and fluid composition where available

Without this context, a machine learning model trained on mixed operating modes will generate false positives during every rate change — exactly the kind of alarm fatigue that makes operators distrust the system.

Analytics Tiers

A practical programme works in tiers, from simple to complex:

Tier Method Typical Application Data Requirement
1 Statistical process control, trending Fouling detection, baseline drift Moderate historian depth
2 Physics-based models Compressor performance curves, heat exchanger efficiency Process design data + historian
3 Machine learning (supervised) Failure mode classification Labelled historical failure data
4 Machine learning (unsupervised) Anomaly detection without labelled failures Large historian, good context tagging

Most facilities should start at Tier 1 and Tier 2. The reason is not technical conservatism — it is data quality. Deploying Tier 4 anomaly detection on poorly contextualised data produces noise, not insight.

Rotating Equipment: A Worked Example (Illustrative)

The following scenario is illustrative and does not represent a specific facility or documented incident.

Consider a gas compression train on an offshore platform. The API 670-compliant protection system handles hardwired trips on high vibration and high bearing temperature. The operations team wants earlier warning than the trip setpoints provide, to allow planned intervention during a scheduled maintenance window rather than an emergency shutdown.

The team extracts two years of historian data: vibration (overall and spectral where available), bearing temperatures, suction and discharge pressures, flow, and lube oil differential pressure. They annotate the dataset with CMMS records identifying bearing replacements and seal changes.

A physics-based performance model is built first: using the OEM-supplied head-flow curve, actual operating points are compared against the design curve at each timestep. Sustained deviation from the expected curve — after correcting for gas composition and inlet conditions — is used as an early indicator of internal wear or fouling.

In parallel, a simple trend of lube oil differential pressure against a commissioning baseline is established. The engineering team agrees that any sustained upward drift from the commissioning baseline, persisting across multiple operating shifts and not explained by a filter change, warrants a work order review. No specific threshold number is embedded in the alert logic; the alert is advisory and reviewed by the rotating equipment engineer before action is taken.

This two-layer approach — physics-based performance monitoring plus simple trend monitoring — is achievable with standard historian tools, does not require a data scientist, and keeps the safety boundary with the hardwired protection system intact.

Implementation Checklist

Use this checklist before committing budget to a big data programme:

Data Readiness

  • [ ] Historian tag list audited for completeness against P&IDs
  • [ ] Compression and scan rate settings verified for critical tags
  • [ ] CMMS data accessible and linkable to historian timestamps
  • [ ] Operating mode states captured in historian or derivable from tag logic

Governance and Safety Boundary

  • [ ] Analytics outputs classified as advisory (not SIF) unless formal IEC 61511 safety lifecycle has been completed
  • [ ] Clear escalation path defined: who acts on an alert, within what timeframe, with what authority
  • [ ] Alarm management review completed to avoid adding to existing alarm load (EEMUA Publication 191 is the recognised industry guidance for alarm rationalisation)

Model Validation

  • [ ] Physics-based models validated against commissioning data or OEM performance curves
  • [ ] Machine learning models (if used) tested on a held-out dataset before live deployment
  • [ ] False positive rate assessed during a defined trial period before operational rollout

Field and Maintenance Integration

  • [ ] Maintenance leads trained on how to interpret and act on alerts
  • [ ] Feedback loop established: field findings from work orders fed back to improve models
  • [ ] Review cycle scheduled (minimum annually) to retrain or recalibrate models as equipment ages

Conclusion and Next Steps

The barrier to extracting value from big data in oil and gas is rarely technology. It is data quality, context, and the discipline to start simple.

The recommended sequence for most operations teams is:

  1. Audit and remediate historian data quality before any analytics investment
  2. Deploy Tier 1 and Tier 2 analytics on the two or three equipment classes responsible for the majority of unplanned downtime at the facility
  3. Establish a formal feedback loop between analytics alerts and CMMS work order outcomes
  4. Only progress to machine learning methods once sufficient labelled failure data exists and the simpler tiers have demonstrated value

Keep the safety boundary clear: analytics is advisory. Any pathway from an analytics output to a process action must go through a human decision or a formally assessed safety lifecycle. That discipline protects both the facility and the credibility of the programme.