Crunching data to increase plant safety and reliability

In ecology, the boundaries of habitats are where the action is. One can typically observe an increased variety of plants and animals at the interface, i.e. life happens at the edge. In a process control facility, there is action at the edges between systems like the distributed control system (DCS), safety instrumented system (SIS) and operator interface. The action at a system edge can help or harm the facility’s safe operation and reliability, i.e. risk happens at the edge.

A typical industrial plant will have a safety system, a process control system, an enterprise resource planning system, years of historical operating data, a plethora of design and as-built drawings, and numerous other repositories of plant data. Major equipment, instrumentation and other assets are usually represented in many of these data repositories rather than just one. The more places the same data is held, the higher the risk the data will differ rather than match.

Plant safety and reliability are maximized when the plant is built in accordance with a well-engineered design and the various operating, management and documentation systems reflect the true as-built state of the plant and are all accurate and synchronized. As with all dreams of utopia, this perfect synchronicity is rarely achieved. Competing business priorities, the difficulty of change management in complicated facilities and human factors conspire to break the alignment of design, build, as-built documentation and operating systems. The misalignments contribute in varying degrees to inefficiencies, decreased safety of operations and decreased operating availability. Worst case, misalignments between systems or between systems and documentation are root causes of incidents that harm people, environment or facilities. 

Due to the frustration of dealing with operational and maintenance issues caused by misalignments, an oil refinery hired Cybertech Automation Inc. to execute a project to identify high-risk misalignments and provide sufficient guidance to each system owner to resolve material issues. The project development phase considered one-off evaluation methods such as sole-purpose spreadsheets and brute force analysis to identify deficiencies, but instead recognized an opportunity to build an evergreen solution to audit systems. To meet both the project need of issue identification and to enable system alignment validation on an ongoing basis, Cybertech’s project team created the Process Control and Safety Instrumented System Auditor ("the Auditor"). 

The Auditor accepts native data from plant systems, standardizes the data with purpose-built C# data mining applications and SQL data manipulation, and applies tiered SQL views to analyze the data. Views are used to ensure that data is not duplicated and to automatically update when the base native data is changed. This architecture provides the means to both easily perform repeat checks after each system owner advises they have resolved identified issues and to monitor systems alignment throughout the facility’s life cycle.

The project faced a number of challenges in meeting the requirements. First, and typical of almost all big data analytics projects, was the challenge of low-quality data. Example quality issues were: incomplete data, nonstandard data, obsolete data and incorrect data. Information from one system that completely disagreed with another system was reviewed to see which system contained the most reliable as-built or design data. Where the integrity of all conflicting systems was suspect, field verification or subject-matter-experts determined which of the conflicting data was correct.

The project also faced the challenge of acquiring data from numerous systems/owners and making the data comparable to that from other systems. The project team accepted data from each system owner in a format they could easily provide since the project had no authority over the system
owners. Due to accepting native format data, the project needed to create data loading and standardization applications and queries to standardize data types and formats.

To allow analysis, the project performed semiautomated and manual work to map instrument tags between systems. Instrument tag numbers were the most common key field in the data, enabling cross-referencing of related data. For example, the safety requirements specification (SRS) has a final element compressor motor that shuts down on a vibration initiator. The SIS references the output relay tag name for the shutdown, not the compressor motor equipment number. The motor identifier and relay tag need to be associated to allow comparison between the SRS document and the SIS. In addition, if there is a vibration monitoring system in place, the multiple vibration tags need to be associated to the corresponding trip tag wired to the SIS. The system retains the tag mapping work, making it a one-time process for each plant.

Resisting the Siren Song of Analytics was the final system design and build challenge. This is a human psyche problem rather than a data or technology problem. Once the team had loaded almost 10 million records of data from 14 data sources, the data comparison and analytics possibilities seemed infinite. The temptation to get lost in analysis was high, but the project team managed to retain focus on high-priority misalignments, meeting the project requirements within budget.

The project proposal and planning phases identified only the systems and data sources for comparison and the high level issues the plant was experiencing. Due to the lack of detailed specifications, the project team utilized agile development methods for the system design and implementation phases. Agile development allowed for flexible responses to frequent stakeholder interaction in order to keep the end system and its deliverables in line with stakeholder requirements that both changed and matured as the project progressed.

The team started by building SQL views to identify and report the high-risk issues defined during project planning. True to the principles of agile development, newly identified high-value targets of analysis were substituted for lower value ones as the specific tests for each issue were designed and implemented. Benefits that were unplanned and of little-to-no additional cost also resulted as many of the continuity checks built to maximize the quality and completeness of the analysis results turned out to be valuable on their own. For example, identifying system instrument tags that were in one source but not in another identified tag naming issues between the field instruments and the
as-built documentation systems.

In the end, more than 100 SQL views were developed by the project for reporting. Each provided either identification of specific risk or reliability issues (i.e. primary reports) or a continuity check within or between the data sources (secondary reports). Since the system is dynamic, all primary and secondary reports refresh as soon as any base data is reloaded, the analytics are fully automatic.