Managing chaos

When too many alarms turn our control rooms chaotic, the problem is often of our own making, but so is the solution. Here are practical tips on launching an alarm management program.

08/19/2013


In our daily lives we are sometimes called upon to manage chaotic situations, as anyone with children can attest. The more children one has the more frequent and chaotic the situations are likely to be. Also the more likely that most of the screaming isn’t even real. Soon you treat the chaos like car alarms in California; they’re always going off so everybody ignores them. Of course that means that if someone really is breaking into a car no one tries to stop the thief. I even heard one talking alarm that warned, “Step away from the car,” if you got too close. Of course, that just caused people get close to the car to hear it talk.

Operators’ lives are all about waiting for the chaos to start. Of course that assumes they can tell the real alarms from the warnings, that is. In many of our facilities we have many more things that fall into the warning or even event log categories that flash and sound the horn than we have real alarms, so after a while the operator doesn’t even look at the screen to see if it’s important. They just hit the acknowledge button to silence the horn much like we do with our car alarms.

So if that’s the case why do we bother to create so many alarms? If you understand the history of DCSs, you realize that early in their history the only way to get a record of an event was to alarm it. This was a pretty good advance over older analog systems that typically had no way to print an event. In some industries this led to the use of the first-out annunciators to aid in determining what caused a trip. As control systems progressed so did their capability to host alarms and to create alarm floods because the extra alarm capability didn’t include ways to suppress alarms based on conditions. The best we could do was to assign priorities based on criticality.

But the beast had been unleashed and only recently have there been moves to rethink alarms and how we use them. Unfortunately we have become addicted to the bloody things in ways that I suspect we never anticipated. In the days of light-box annunciators, the cost per alarm and the real estate involved made us think carefully about things to be alarmed. Once alarms effectively became free we used them for everything. Our biggest addiction is to information alarms, those that tell the operator something is going on that someone else thinks he should know about but typically isn’t required to do anything or can’t do anything about. Next in line are alarms caused by an action the operator took such as a low flow alarm because he stopped a pump. During operation the low flow alarm might be a critical alarm but the operator knows he caused a low flow condition so why tell him he did. Yes we had some of those kinds of alarms in the light box days but nothing like the quantity we have now.

For a number of years now we’ve had the tools to change how we use alarms but our biggest hurdles are our addictions. Supervisors still want operators to be aware of what’s going on in their areas by pointing it out to them rather than fixing the display issues that make it hard for an operator to see what’s going on. Projects typically don’t include the time and money, mainly for ROI reasons, to do a proper alarm rationalization. The result is we’re still stuck with those because-I-did-something alarms and alarm floods when a trip occurs. Exacerbating this is the fact that our larger plants are effectively multiple plants with managers that have different operating philosophies driving different implementations of what should be a unified strategy.

So how do we fix this mess we’ve gotten ourselves into?

  • Start by acknowledging the problems it’s causing and recognizing that it can be fixed. Most companies are using some combination of HAZOP, LOPA, and PHA to identify what’s really critical to safety. That can be a starting point for implementing a plant-wide alarm philosophy once one has been developed.
  • Identify the bad actors and target them before anything else. Alarms that chatter because of too little deadband or from being set too close to a normal operating condition on a noisy process or even on a badly tuned loop. Look at normal shutdown periods for alarms that are caused by shutting down.
  • Develop an alarm management strategy and make it someone’s job to ensure that every control system gets reviewed to comply with it. Make sure that this person will have the clout to ensure people participate and the passion to evangelize for it.
  • Understand what tools for implementing the policy are available in your control system. Can it handle conditional alarming? Can it provide alarm help? Can it do alarm shelving? How easy is it to prioritize alarms? How does it capture and present non-alarm events?
  • Piggyback onto projects. Have your champion ensure that any new projects have a budget for implementing the alarm rationalization. This ensures a finite scope to attack rather than just trying to do the whole plant at once.
  • Don’t get discouraged if the first one is a struggle. Even when people have bought into the concept the first one or two projects will be a struggle for people because it will in some cases threaten long held concepts and in others people’s desire for control. If they have not bought in then they may be upset when you question the rationale behind an alarm that they created.
  • Sell the concept that if there is no operator action required, it isn’t an alarm. This will be an ongoing battle with those addicted to informational alarms. An extension of this suggests that even if there is an operator action required, it may still be a low priority alarm based on time to respond and/or potential to do damage.
  • Keep referring back to the criteria you’ve set to make sure that you’re being consistent in your alarm rankings. This is particularly important as you move from project to project or plant area to plant area because as the players change, so does their understanding of what’s written.
  • Don’t be afraid to refine the philosophy. ISA 18.2 is a great starting point for working through the implementation, but you still have to address the specific peculiarities of your plant and your processes.

The good news is you can do this. The bad news is it’s a lot of work and involves a lot of people’s time. So what’s stopping you from taking these steps? Why are you still living with chaos?

This post was written by Bruce Brandt. Bruce is the Technology Leader at MAVERICK Technologies, a leading automation solutions provider offering industrial automation, strategic manufacturing, and enterprise integration services for the process industries. MAVERICK delivers expertise and consulting in a wide variety of areas including industrial automation controls, distributed control systems, manufacturing execution systems, operational strategy, business process optimization and more. 



No comments
The Top Plant program honors outstanding manufacturing facilities in North America. View the 2013 Top Plant.
The Product of the Year program recognizes products newly released in the manufacturing industries.
The Engineering Leaders Under 40 program identifies and gives recognition to young engineers who...
The true cost of lubrication: Three keys to consider when evaluating oils; Plant Engineering Lubrication Guide; 11 ways to protect bearing assets; Is lubrication part of your KPIs?
Contract maintenance: 5 ways to keep things humming while keeping an eye on costs; Pneumatic systems; Energy monitoring; The sixth 'S' is safety
Transport your data: Supply chain information critical to operational excellence; High-voltage faults; Portable cooling; Safety automation isn't automatic
Case Study Database

Case Study Database

Get more exposure for your case study by uploading it to the Plant Engineering case study database, where end-users can identify relevant solutions and explore what the experts are doing to effectively implement a variety of technology and productivity related projects.

These case studies provide examples of how knowledgeable solution providers have used technology, processes and people to create effective and successful implementations in real-world situations. Case studies can be completed by filling out a simple online form where you can outline the project title, abstract, and full story in 1500 words or less; upload photos, videos and a logo.

Click here to visit the Case Study Database and upload your case study.

Maintaining low data center PUE; Using eco mode in UPS systems; Commissioning electrical and power systems; Exploring dc power distribution alternatives
Synchronizing industrial Ethernet networks; Selecting protocol conversion gateways; Integrating HMIs with PLCs and PACs
Why manufacturers need to see energy in a different light: Current approaches to energy management yield quick savings, but leave plant managers searching for ways of improving on those early gains.

Annual Salary Survey

Participate in the 2013 Salary Survey

In a year when manufacturing continued to lead the economic rebound, it makes sense that plant manager bonuses rebounded. Plant Engineering’s annual Salary Survey shows both wages and bonuses rose in 2012 after a retreat the year before.

Average salary across all job titles for plant floor management rose 3.5% to $95,446, and bonus compensation jumped to $15,162, a 4.2% increase from the 2010 level and double the 2011 total, which showed a sharp drop in bonus.

2012 Salary Survey Analysis

2012 Salary Survey Results

Maintenance and reliability tips and best practices from the maintenance and reliability coaches at Allied Reliability Group.
The One Voice for Manufacturing blog reports on federal public policy issues impacting the manufacturing sector. One Voice is a joint effort by the National Tooling and Machining...
The Society for Maintenance and Reliability Professionals an organization devoted...
Join this ongoing discussion of machine guarding topics, including solutions assessments, regulatory compliance, gap analysis...
IMS Research, recently acquired by IHS Inc., is a leading independent supplier of market research and consultancy to the global electronics industry.
Maintenance is not optional in manufacturing. It’s a profit center, driving productivity and uptime while reducing overall repair costs.
The Lachance on CMMS blog is about current maintenance topics. Blogger Paul Lachance is president and chief technology officer for Smartware Group.