By Bill Ray, Eric Kauffman and Craig Nicholson
This is the second in our multi part series on Industrial Control Systems.
The old way to look at ICS upgrades
Power Plant Industrial Control Systems (ICS), in the traditional sense, served two primary purposes, 1) basic sequencing of valves and other physical controls and, 2) protection against inadmissible operating conditions. Typically, the trigger for an upgrade is the inflection point between decreasing system reliability and the challenges and expense of the upgrade. When that occurs, instinctual response is to get the system back to the way it ran before. Since the way it ran before is the way the operators are used to. Short of a costly revenue impacting failure, ICS upgrades tended to be viewed as an elective “want” versus a plant “need”. Without a clear path to improve plant performance, proposed upgrades are often dropped from plant budgets.
A New Technology Era – Using Data!
A lot has changed in the last decade, technology advancements in the field of machine learning, advanced analytics and artificial intelligence are adding new capabilities to the purpose of the ICS. These advancements are opening up new opportunities to improve productivity and power plant performance, thus increasing competitiveness and extending economic life.
The changes are often in the form of increased automation, faster adjustments of operational parameters, and potentially expanding the safe operating envelope of the plant. This inherently takes actions as well as many operational decisions out of the hands of the operators. More than a simple upgrade, this is a change-management situation and thus calls for a larger strategic mindset.
Often this is grouped into what is referred to as a “digital strategy”. The foundation of the digital strategy is the data and capabilities of the industrial control system but it does not end there. Upgrading a control system can vastly improve access to process data and increase insights that can be drawn from daily operations. Further, a more modern control system will have improved communication security while avoiding legacy system bandwidth limitations and the cyber security risks associated with aging communication protocols.
To position a plant for future success, how does a plant owner best manage its ICS life cycle? How do you mitigate known upgrade challenges associated with learning new systems, change management and potential short term system reliability risks?
“The foundation of a power plant’s digital strategy begins with process level data from the ICS“
Taking a Structured Approach
Implementing a holistic ICS retrofit plan can avoid many of the costly pitfalls that keep plant managers awake at night.
Here are some of the key areas that should be addressed.
Start With Your Business Needs
a controls upgrade should dovetail with the business strategy in terms of current and anticipated performance, capability and lifecycle of the plant as well as support OpEx and CapEx requirements. This will serve as a basis to frame what success looks like in terms of controls life cycle and how this fits into the overall operation strategy for the plant. Too often, the selection of a controls upgrade supplier is based on the vendor’s system features or marketing jargon, instead of what is the best fit for the plant business strategy. Like in many other strategic decision efforts, start with the desired outcome and work back to the requirements of the controls.
Plant Assessment
A system evaluation shall identify current infrastructure as well as existing pinch points and anticipated future issues. Typically, a plant has multiple systems that all should be part of the evaluation with the goal of consolidating and simplifying wherever possible. Such an assessment helps prioritize upgrade needs, strategic spares for legacy systems and critical system maintenance activities. Vulnerabilities are mapped out and identified such as obsolete Human Machine Interfaces (HMIs) or Windows systems, with expired service support which ends the development of system patches, exacerbating cyber intrusion risk. From an operational standpoint, the plant assessment can also include key processes such as startups that could benefit from process reengineering and automation.
Retrofit Plan and Upgrade Strategy
After completion of a plant and business needs assessment, the owner is well positioned to further develop a retrofit plan and upgrade strategy. The retrofit plan is the basis for a request for proposal (RFP), which may be tied into a service agreement or as a standalone transaction. The RFP will serve to clearly articulate the plant’s wants and needs thus providing clarity to bidders allowing for comparison of proposals. This is important as not all upgrades are the same. For example, without clarity on the ask, one vendor may offer a partial upgrade utilizing existing infrastructure (box change) while others offer full system replacement (rip and replace). Knowing which path is optimal for the plant is determined by the prior assessment and goal setting phase.
It is common to stage or phase in upgrades such that one unit is upgraded before the other. In such cases, additional interim steps and interface considerations to legacy systems may be required all while ensuring safe and reliable plant operations. How bidders respond to such plant specific requests will serve as a key part of the bidder evaluation.
Development & Implementation
Upgrades are engineering intensive projects. In fact, the split in cost between engineering and hardware is typically 70/30. As such, it’s key to get value for your dollar on the engineering scope.
Generally, controls projects can be broken down into distinct phases of hardware design, procurement, software design and testing. In order to ensure a quality outcome, the prudent approach with a vendor is to ‘trust but verify’ at each phase of the project. Once a vendor has frozen design and started purchasing hardware, your system is ‘locked in’. Prior to this point, technical assessment of the system’s architecture is required to ensure consistency with prudent industry practice as well as conforming to contractual requirements.
The assessment of the software design and methodologies are just as important verifying conformity in HMI design and control philosophy congruent with current plant operations. Typically, poorly engineered systems are a result of poor communication either during handover from commercial to execution or within the execution team followed by a lack of process experience. A seasoned controls engineer with good process knowledge will have the ability to understand the potential failure modes that the system will be subject to and test the robustness of the design to such failures. A key success factor in any controls system can be attributed to testing methodology – don’t just test for working features, test for failures. With the right expertise and due diligence, latencies in system design can easily be identified and corrected during testing sparing control room operators from spurious alarms and system reliability issues for years to come.
Training Ahead of the Upgrade
Once an upgraded system is deployed, it can be a shock for plant personnel. A well designed system will mimic desired functionality and minimize disruptions to operations, however, inevitably, there is a learning curve until operations and maintenance personnel are fully versed in the new system. Training programs from vendors are typically an afterthought resulting in a few days training at the end of commissioning. Personnel training focusing on plant operations and cyber security should be a key part of the controls retrofit plan and not left until post upgrade. A training plan may go so far as to include the purchase of a plant simulator, but, at a minimum, should include classroom and/or onsite training.
Unlocking Insights From Your Data Is Your Intelligence
Digital technologies are in a nascent phase and the potential gains for power plant owners will continue to rapidly evolve. With a structured approach to controls life cycle management, owners can have confidence in better utilizing their data and realizing gains in these rapidly evolving technologies while maintaining and upgrading their power plant industrial control systems with minimal disruption to their operations.
For additional discussion
bill.ray@aimpowerconsulting.com
craig.nicholson@aimpowerconsuting.com
www.aimpowerconsulting.com