Power plant personnel: Best practices

The real dollar value of skilled operators.
By Brian Wodka, PE, CEM, LEED AP RMF, Engineering, Inc. December 7, 2016

Brian Wodka, PE, CEM, LEED AP RMF, Engineering, Inc.Over the years, electronics, software and technology in general have become less expensive for power plants, but operators’ wages continue to rise-especially as the labor pool diminishes. As equipment is becoming engineered for more reliability and controllability, power plants are becoming more automated. This has become the justification for fewer power plant operators on staff, with some facilities attempting to become or successfully becoming completely unmanned.

While this seems to make sense on a financial report, the reality is that technology is not advanced enough for complete automation of essential occupations-and for critical facilities, it can never fully be automated.

Reliability

Reliability is essential to the successful operation of a power plant. By definition, a power plant generates some form or forms of needed power (heating, cooling, electricity, etc.). The idea of "reliability" has a pretty standard definition and is a part of common vernacular.

With reliability being a top priority, maintenance needs to be regularly performed to sustain the reliability of the power plant equipment. In order for equipment to remain reliable, the proper maintenance has to be performed, and plant operators have to be fully prepared for the unexpected.

For example, the worst time to have a heating plant outage would be the coldest day of the year. However, on that day, the power plant equipment will be operating at its greatest capacity, potentially higher than ever before. The coldest day of the year is also the day of the highest natural gas demand, so plant owners can expect to be curtailed and have to run on back-up fuel oil. The equipment that typically runs less than one week a year is now supporting the entire plant at full load and needs to sustain the entire campus. If the power plant is designed to operate in an automated state for the majority of its life, what do you think happens in these critical, high-risk situations? More than likely, the plant (or some portion of the plant) fails-and that’s when you realize the value of having skilled plant operators on staff.

In situations like these, reliability engineering and statistics are paramount. Knowing that with enough time, regardless of the design, a failure is inevitable; owners should assume that a failure (or failures) will occur and prepare accordingly. By assuming that a failure will definitely occur, the element of probability is removed. With probability removed and certainty present, owners can evaluate the impact of emergency plant loss situations. You will find that as facilities become more automated, they are intrinsically accepting much more risk than they realize; not from the loss of reliability, but from the loss of maintainability.

Maintainability

"Maintainability" can be defined as the probability that the maintenance can be fully performed in the allocated time period. In other words, what are the chances that the maintenance can be completed on time?

Considering the forecast of technology in the foreseeable future, the following axiom holds a very powerful argument:

"Proper maintenance and repair of a power plant cannot be fully automated."

This includes maintaining the operations of the plant during a high-risk scenario. Automated systems all too often default to shutdown when something minor could be done to prevent it. Few people will argue with the ability of an automated system to instantaneously identify an issue or alarm, and then proceed to carefully and systematically shut the system or the entire plant down.

However, plants are also automatically shut down during cases of false alarms.

Due to this fact, properly trained, human, power plant operators will continue to be one of the most valuable and cost-effective assets of a power plant simply due to their ability to best evaluate and resolve an emergency situation consisting of multiple system complications or failures.

In addition, as power plant equipment becomes more automated, the controls and complexity tend to increase. Reparability tends to decrease as issues with electronics, software and custom/proprietary components require outsourcing the work to specialists or replacing the entire component. There is a direct correlation between more automation and reduced maintainability. This means that not if, but when a system fails in a highly automated plant, it typically takes longer to repair.

As equipment-automation increases, the importance of minimizing downtime increases. If increasing the automation of a power plant causes the duration of downtime (maintainability) to increase more than the increase in uptime (reliability), then the net availability of the power plant actually goes down.

Planned outages

A "planned outage" is a controlled, scheduled shutdown of the power plant. Depending upon the size, complexity and criticality of the power plant, the amount of effort associated with a planned outage can vary tremendously. For a facility that only has one planned outage every 18 months, the process of planning starts months in advance.

If the outage is only for one month, there tends to be a tremendous amount of coordination, material and labor procurement and a detailed examination of logistics. It is a fixed timeline with exhaustive preparation. Operators have 30 days to make sure that the power plant operates at maximum availability for the next 540 days. If the outage is not planned properly and extends beyond the allocated time, there is usually a direct calculable value associated with the extended outage.

The same planning importance is applicable with smaller facilities with shorter planned outages, since the relative proportions tend to be the same. The outages and costs may be smaller, but owners are working with a tighter budget.

Either way, the necessity of properly planning an outage improves the chances of ensuring that all the necessary work is performed within the allocated time frame (effective maintainability). Planning outages can be greatly assisted by the use of software, but this is another process that cannot be fully automated.

Unplanned outages

An "unplanned outage" is an unscheduled shutdown of a power plant. Something went wrong, and you "lost the plant." People are fired over unplanned outages as they can easily cost millions of dollars in damages, liabilities and premium fees for emergency work. Other than an explosion, an unplanned outage can be one of the most stressful events in a power plant. With safety, millions of dollars of equipment and jobs all on the line, operators are scrambling and stressed out, knowing that upper management will scrutinize everything they do-and don’t do. These are the instances that can get coverage in the local, national or even international news. With operators acting frantically, the potential for error increases along with the probability of practicing unsafe procedures. Those types of mistakes could easily delay the plant recovery or even make matters worse.

Unplanned outages can be triggered by relatively small issues due to human error or computer error. Those small issues can compound and result in the plant being shut down. Computers are excellent at monitoring, computing and processing algorithms, but they are also prone to unexplained "glitches" and freezing, and they are only as good as the communication signals on which they operate and the quality of their (human) programming. Even though humans make mistakes, humans can also compensate with physical, in-the-field action.

Humans are also able to assess conditions to which a computer is blind. This is a critical advantage of humans versus automation, especially during an unplanned outage. We’ve seen that the number of unplanned outages and the duration of those outages are not necessarily reduced with increased automation. In fact, in many instances automation can increase outage issues.

Prepared outages

If minimizing scheduled downtime is the practice of maintainability, then the art of maintainability is achieved when plant operators proactively prepare for unplanned outages. By definition, you cannot plan an unplanned outage, but you can prepare for common unplanned outages. This can be done by first anticipating the most likely modes of an unplanned outage. There are certain techniques that can be done that have proven effective in preparing for unplanned outages:

  1. Hold a lessons-learned meeting after each outage to get all operators’ views and opinions on what could have been done to prevent the event and what could be done to prevent future repeat events.
  2. Ensure proper depth of inventory and supply chain management. Operators should make sure you have backup inventory should any equipment fail.
  3. Practice emergency procedures. Owners should quiz operators on "What do you do if … ?" situations. It is a great exercise to make sure everyone knows the proper procedures, in order to avoid disagreements or confusion during a crisis.
  4. Owners should make sure they have the right tools on-hand to address common repairs to critical components. This can save time and ensure the repair is more than just temporary.

As owners practice maintainability, it helps to envision how multiple complications can hit you all at once and anticipate these problems. These exercises provide valuable training and build operator confidence. Plant operators are an owner’s biggest resource-they are often very familiar with handling high-risk situations and can provide foresight and advice during similar situations.

In an industry that is enamored by reliability, it is easy to underestimate and underappreciate the value of skilled plant operators and the value of maintainability. There is true elegance in the ability of a seasoned power plant operator to acknowledge, identify, assess, adapt and resolve a perfect storm from being a multi-million dollar publicized excursion into only a minor blip on the trending historian of the power plant parameters. This is something that cannot be automated; it is something that is not necessarily intuitive; it is an expertise that power plant operators hone over years.

During a critical situation, a power plant operator’s skill is judged by their speed of resolution. While the power plant is in crisis, owners are not concerned about operators’ ability to maximize uptime, but by their ability to minimize downtime. With an infinite number of ways things can go wrong, calm, level-headed, decision-making in time of crisis, along with proper preparation, are skills only available from an experienced operator-which can never be fully automated.

-Brian Wodka is a mechanical engineer and partner at RMF Engineering. He leads the power plant assessment and reliability team at RMF Engineering and has performed power plant assessments and boiler inspections for the past 15 years.