In ecology, the boundaries of habitats are where the action is. One can typically observe an increased variety of plants and animals at the interface, i.e. life happens at the edge. In a process control facility, there is action at the edges between systems like the distributed control system (DCS), safety instrumented system (SIS) and operator interface. The action at a system edge can help or harm the facility’s safe operation and reliability, i.e. risk happens at the edge.

A typical industrial plant will have a safety system, a process control system, an enterprise resource planning system, years of historical operating data, a plethora of design and as-built drawings, and numerous other repositories of plant data. Major equipment, instrumentation and other assets are usually represented in many of these data repositories rather than just one. The more places the same data is held, the higher the risk the data will differ rather than match.

Plant safety and reliability are maximized when the plant is built in accordance with a well-engineered design and the various operating, management and documentation systems reflect the true as-built state of the plant and are all accurate and synchronized. As with all dreams of utopia, this perfect synchronicity is rarely achieved. Competing business priorities, the difficulty of change management in complicated facilities and human factors conspire to break the alignment of design, build, as-built documentation and operating systems. The misalignments contribute in varying degrees to inefficiencies, decreased safety of operations and decreased operating availability. Worst case, misalignments between systems or between systems and documentation are root causes of incidents that harm people, environment or facilities. 

Due to the frustration of dealing with operational and maintenance issues caused by misalignments, an oil refinery hired Cybertech Automation Inc. to execute a project to identify high-risk misalignments and provide sufficient guidance to each system owner to resolve material issues. The project development phase considered one-off evaluation methods such as sole-purpose spreadsheets and brute force analysis to identify deficiencies, but instead recognized an opportunity to build an evergreen solution to audit systems. To meet both the project need of issue identification and to enable system alignment validation on an ongoing basis, Cybertech’s project team created the Process Control and Safety Instrumented System Auditor (“the Auditor”). 

The Auditor accepts native data from plant systems, standardizes the data with purpose-built C# data mining applications and SQL data manipulation, and applies tiered SQL views to analyze the data. Views are used to ensure that data is not duplicated and to automatically update when the base native data is changed. This architecture provides the means to both easily perform repeat checks after each system owner advises they have resolved identified issues and to monitor systems alignment throughout the facility’s life cycle.

The project faced a number of challenges in meeting the requirements. First, and typical of almost all big data analytics projects, was the challenge of low-quality data. Example quality issues were: incomplete data, nonstandard data, obsolete data and incorrect data. Information from one system that completely disagreed with another system was reviewed to see which system contained the most reliable as-built or design data. Where the integrity of all conflicting systems was suspect, field verification or subject-matter-experts determined which of the conflicting data was correct.

The project also faced the challenge of acquiring data from numerous systems/owners and making the data comparable to that from other systems. The project team accepted data from each system owner in a format they could easily provide since the project had no authority over the system
owners. Due to accepting native format data, the project needed to create data loading and standardization applications and queries to standardize data types and formats.

To allow analysis, the project performed semiautomated and manual work to map instrument tags between systems. Instrument tag numbers were the most common key field in the data, enabling cross-referencing of related data. For example, the safety requirements specification (SRS) has a final element compressor motor that shuts down on a vibration initiator. The SIS references the output relay tag name for the shutdown, not the compressor motor equipment number. The motor identifier and relay tag need to be associated to allow comparison between the SRS document and the SIS. In addition, if there is a vibration monitoring system in place, the multiple vibration tags need to be associated to the corresponding trip tag wired to the SIS. The system retains the tag mapping work, making it a one-time process for each plant.

Resisting the Siren Song of Analytics was the final system design and build challenge. This is a human psyche problem rather than a data or technology problem. Once the team had loaded almost 10 million records of data from 14 data sources, the data comparison and analytics possibilities seemed infinite. The temptation to get lost in analysis was high, but the project team managed to retain focus on high-priority misalignments, meeting the project requirements within budget.

The project proposal and planning phases identified only the systems and data sources for comparison and the high level issues the plant was experiencing. Due to the lack of detailed specifications, the project team utilized agile development methods for the system design and implementation phases. Agile development allowed for flexible responses to frequent stakeholder interaction in order to keep the end system and its deliverables in line with stakeholder requirements that both changed and matured as the project progressed.

The team started by building SQL views to identify and report the high-risk issues defined during project planning. True to the principles of agile development, newly identified high-value targets of analysis were substituted for lower value ones as the specific tests for each issue were designed and implemented. Benefits that were unplanned and of little-to-no additional cost also resulted as many of the continuity checks built to maximize the quality and completeness of the analysis results turned out to be valuable on their own. For example, identifying system instrument tags that were in one source but not in another identified tag naming issues between the field instruments and the
as-built documentation systems.

In the end, more than 100 SQL views were developed by the project for reporting. Each provided either identification of specific risk or reliability issues (i.e. primary reports) or a continuity check within or between the data sources (secondary reports). Since the system is dynamic, all primary and secondary reports refresh as soon as any base data is reloaded, the analytics are fully automatic.

Auditor database server, Cybertech Automation

Auditor database server, Image courtesy of Cybertech Automation Inc.

Safety issues were the main driver for the project’s acceptance and funding. The following are examples of how the Auditor analyzes data to identify the diverse issues that can create safety risks. 

Smart instrumentation performs self-diagnosis and sends a specific mA output signal when the self-diagnosis identifies a bad state. The SIS configuration must recognize this mA signal as indicating a bad instrument rather than a typical reading or over-range reading. Older facilities will have a mix of smart and legacy instrumentation, so the mA values may differ for legitimate process readings versus bad instrument indication. This creates opportunity for configuration error. Misalignment results, worst case, in the SIS continuing to operate with a bad instrument or mistakenly bypassing a still-operating instrument. If the process being monitored goes to a dangerous state, the SIS will then fail to perform the correct action to take the process to a safe state. The system checks these and related issues, verifying that the instrument alarm and system fault settings are aligned.

The system performs many of the SIS verification and validation steps required by IEC 61511. Checks include confirming that the initiators and final
elements specified in the SRS are implemented, the process safety times required by the SRS are not exceeded, and that the SIS configuration meets the initiator voting required by the SRS. Failure of the SIS configuration to meet the SRS can result in identified hazards not being properly protected against — putting people, facility and the environment at risk.

Instrument range and trip setpoints are another area where issues can cause many different safety and reliability risks. When an instrument or system is not calibrated/configured to the design range, operators and the SIS may be blind to the process entering an unsafe state. If the misalignment does not result in unsafe operation, it can cause spurious trips if the SIS trip setpoint ends up acting in the normal operating range of the process. There are numerous places where range issues can appear, thus these areas are high-priority for the system to check. Specific tests performed by the system include: instrument configured to incorrect range, control/safety system configured to incorrect range, errors in units of measurement, errors in engineering unit conversion between systems, and as-built documentation systems containing incorrect ranges. 

The above scenarios are a small sample of the checks performed by the system. For each check, issues identified were submitted to the responsible system owner and tracked for resolution. Once a system owner reported that issues were resolved, the data for their system was easily reloaded. The dynamic reports immediately validated whether the issues were resolved or not.

In developing and using the system, the project met its goal of identifying and initiating resolution of process control, SIS, and documentation safety and reliability issues. Errors in any human endeavor are inevitable, more so in managing change in complex process facilities. By refreshing the base data on a regular basis, the client can quickly act upon any new system configuration, database, and documentation misalignments that arise.  

Dean Whitford is a PMI-certified project manager, program manager and business analyst with 19 years of software development and data analytics management experience. He has been a project manager for Cybertech Automation Inc., a North American controls and automation multidisciplinary engineering service provider (EPC and MAC), for four years and two months. He may be contacted at https://www.linkedin.com/in/deanwhitford.