Water/wastewater: achieving the three levels of redundancy

To provide reliable water and wastewater treatment, municipalities must provide a 24-hour operation on a shoestring budget while battling constantly changing regulations, influent criteria, new security concerns and aging infrastructures. To address these challenges, many facilities are either considering an upgrade of their current automation system or installing automation for the first time.


To provide reliable water and wastewater treatment, municipalities must provide a 24-hour operation on a shoestring budget while battling constantly changing regulations, influent criteria, new security concerns and aging infrastructures. To address these challenges, many facilities are either considering an upgrade of their current automation system or installing automation for the first time. When examining an automation system’s design, the question of redundancy often arises. But what is redundancy, and does it equate to a hot standby system?

What is redundancy?

Redundancy comes in many forms and is inherent at some level in any plant design. The most basic form of redundancy requires the inclusion of a hand-off-auto switch for each component. In the automatic mode, the plant or system controller runs the process. In the hand mode this step is bypassed, resulting in continued treatment but a loss of efficiency and/or quality.

For example, in an oxidation ditch scenario, the automatic operation may stage aerators based on dissolved oxygen levels. In hand mode, it would be best to turn on all the aerators. This would ensure adequate levels of air without having to continuously monitor process conditions. However, it would also waste power.

As an alternative, sometimes a process can be bypassed. This is common in headworks, in which a bypass channel around the screen can be found. Using this bypass channel can ensure the plant operates, but it may allow particles to accumulate in the downstream clarifiers and basins.

Another form of redundancy exists when more equipment is installed than is required. For example, three pumps may be provided when only two are needed. This type of redundancy is quite common. Typically the third pump still has its own dedicated starter, variable frequency drive and control components. This extends some redundancy to the control and automation system as well.

Redundancy is also gained with multiple process trains. Sometimes each train has its own automation system, or trains have been grouped into multiple control panels. This also represents a level of redundancy.

Finally, redundancy can be designed into an automation system.

Three types of automation redundancy

Automation redundancy can be accomplished via three methods: cold, warm and hot.

Cold redundancy %%MDASSML%% Cold redundancy is for those processes where response time is of minimal concern and may require operator intervention. For example, consider two belt presses, each with a dedicated control panel. If one belt press becomes inoperable, the operator can resume operation by simply starting the other press.

In this example, the loss of one belt press may result in an accumulation of un-pressed solids, but will not likely impact the ability to meet permit. Thus, operator intervention is acceptable, and a cold redundancy design may be a good solution. However, when time is more critical, warm or hot redundancy are better approaches.

Warm redundancy %%MDASSML%% Warm redundancy is used where time is somewhat critical but a momentary outage is still acceptable. In this scenario, a momentary bump can be expected. During this bump, the valves, motors and other devices might shutdown temporarily, and the sensors may not report back to the PLC system during the bump.

An example of a warm redundancy application may be exemplified on an ATAD system. In an ATAD process, biological decomposition creates heat. Mixers, aerators and foam cutters are frequently used to distribute the heat, introduce air and reduce foam accumulation. During a momentary changeover, the mixers, aerators and foam cutters could stop working. During the changeover, the heat in the system would remain constant due to the nature of the process, and progression towards Class A would not be compromised.

However, cessation of the mixers, aerators and foam cutters for more than a few minutes would eventually cause an undesired foam build up and heat stratification, causing the process to transition from aerobic to anaerobic. Thus, in this system, the process can tolerate having the mixers, aerators and foam cutters off for a few seconds, with the understanding that they must be restored quickly and automatically to avoid risking the process integrity.

Warm redundancy systems typically have two processors connected in a primary and standby configuration. The primary processor controls the system’s inputs and outputs (I/O) while the standby processor is powered up and waits for the primary processor to stop controlling the process. When this occurs, the standby processor assumes control of the I/O and takes the designation of primary processor, allowing the offline processor to become the secondary processor, which can then be maintained without sacrificing process control.

During normal operation, the primary processor provides periodic updates to the standby processor. These updates usually occur at the end of each program scan and may only involve a portion of the data at any time. Therefore, when a changeover occurs, the standby processor can work off of incomplete data since it may take the standby processor a few program scans to catch up to where the primary was before the changeover. This can contribute to a bump in the process during the changeover.

From a hardware perspective, warm and hot redundancy systems are almost identical, and can be easily confused when looking at suppliers’ product data. Care should be taken to carefully examine the different types of systems.

Hot redundancy %%MDASSML%% Hot Redundancy is used when the process must not go down for even a brief moment under any circumstance. An example of a hot redundancy system application is a membrane or BAF application. In both instances, the process may not require the hot redundancy capabilities, but during a backwash, redundancy could be critical.

During a backwash, a valve reversal, a motor halt or re-sequencing of the valves may cause an incomplete backwash, allow contaminants to enter the clean water or may even contribute to inoperable equipment. Since none of these scenarios are tolerable, a hot redundancy configuration would be the preferred approach in this case.

Again, the hardware layout of a hot redundancy system is almost identical to a warm redundancy system. However, hot redundancy systems provide bumpless transfer of the I/O during a changeover from primary to standby. To accomplish this, communication messaging, updating the secondary processor and program execution must be properly managed.

To ensure that a hot redundancy system operates correctly, data must be transferred from the primary processor to the secondary processor with every logic cycle to ensure data integrity. Two methods exist for accomplishing the needed data transfer.

The first is to perform the transfer at the end of the program scan. Only upon completion of the transfer will the scan resume. This approach can be called “scan-and-transfer.” This technique was first employed by Modicon when it created the first redundant PLC, and it continues to be an approach taken by many suppliers today. As real world experience has shown, this method can provide accurate updating of data and good control.

However, there are criteria that should be considered when using a scan-and-transfer system. First, the true scan time of the program will be a combination of the program scan and the transfer update. Since scan time can be critical in certain applications, suppliers that support this type of transfer will indicate in their manual that the program should be designed to minimize scanning. They will offer suggestions on how to limit rung executions to only those instances when conditional logic has changed. If these suggestions are not incorporated properly, a situation could arise where a bump is experienced on the output. Any bump possibility would create a warm redundancy situation.

While the scan and transfer technique was used for years, and is a reliable and accurate method when properly applied, technological advances have made a new technique possible %%MDASSML%% a method that ensures the transfer does not depend on scan time.

This new method is referred to as “asynchronous transfer.” In an asynchronous transfer, the primary processor has two separate microprocessors embedded in its circuitry. The first microprocessor executes the program. At the end of the execution, all data is passed to the second microprocessor. This second microprocessor handles all transfer tasks while the first microprocessor executes the next program scan. Thus, one microprocessor is executing while the other is transferring data to the standby processor.

As this transfer of data from primary processor to secondary processor is asynchronous to the program scan, it now becomes possible to transfer the entire data table without affecting program execution. This eliminates any need to design the program for an optimized scan. In many cases, this benefit makes asynchronous transfer a better fit for water and wastewater treatment.

Redundancy in water and wastewater treatment is critical. However, redundancy is a philosophy, not a set solution.

Fig. 1. Cold redundancy is for those processes where response time is of minimal concern and may require operator intervention. If cessation can be tolerated until the operator can intervene, then cold is best.

Fig. 2. Warm redundancy can be used when a process bump can be tolerated. When the system can handle a slight cessation but automatic intervention is necessary, warm redundancy is the solution.

Fig. 3. For those operations that cannot experience even a momentary cessation, hot redundancy is the only option.

Author Information
Grant Van Hemert, P.E. is an application engineer at Schneider Electric’s Water and Wastewater Competency Center in Nashville, TN.

Top Plant
The Top Plant program honors outstanding manufacturing facilities in North America.
Product of the Year
The Product of the Year program recognizes products newly released in the manufacturing industries.
System Integrator of the Year
Each year, a panel of Control Engineering and Plant Engineering editors and industry expert judges select the System Integrator of the Year Award winners in three categories.
October 2018
Tools vs. sensors, functional safety, compressor rental, an operational network of maintenance and safety
September 2018
2018 Engineering Leaders under 40, Women in Engineering, Six ways to reduce waste in manufacturing, and Four robot implementation challenges.
GAMS preview, 2018 Mid-Year Report, EAM and Safety
October 2018
2018 Product of the Year; Subsurface data methodologies; Digital twins; Well lifecycle data
August 2018
SCADA standardization, capital expenditures, data-driven drilling and execution
June 2018
Machine learning, produced water benefits, programming cavity pumps
Spring 2018
Burners for heat-treating furnaces, CHP, dryers, gas humidification, and more
October 2018
Complex upgrades for system integrators; Process control safety and compliance
September 2018
Effective process analytics; Four reasons why LTE networks are not IIoT ready

Annual Salary Survey

After two years of economic concerns, manufacturing leaders once again have homed in on the single biggest issue facing their operations:

It's the workers—or more specifically, the lack of workers.

The 2017 Plant Engineering Salary Survey looks at not just what plant managers make, but what they think. As they look across their plants today, plant managers say they don’t have the operational depth to take on the new technologies and new challenges of global manufacturing.

Read more: 2017 Salary Survey

The Maintenance and Reliability Coach's blog
Maintenance and reliability tips and best practices from the maintenance and reliability coaches at Allied Reliability Group.
One Voice for Manufacturing
The One Voice for Manufacturing blog reports on federal public policy issues impacting the manufacturing sector. One Voice is a joint effort by the National Tooling and Machining...
The Maintenance and Reliability Professionals Blog
The Society for Maintenance and Reliability Professionals an organization devoted...
Machine Safety
Join this ongoing discussion of machine guarding topics, including solutions assessments, regulatory compliance, gap analysis...
Research Analyst Blog
IMS Research, recently acquired by IHS Inc., is a leading independent supplier of market research and consultancy to the global electronics industry.
Marshall on Maintenance
Maintenance is not optional in manufacturing. It’s a profit center, driving productivity and uptime while reducing overall repair costs.
Lachance on CMMS
The Lachance on CMMS blog is about current maintenance topics. Blogger Paul Lachance is president and chief technology officer for Smartware Group.
Material Handling
This digital report explains how everything from conveyors and robots to automatic picking systems and digital orders have evolved to keep pace with the speed of change in the supply chain.
Electrical Safety Update
This digital report explains how plant engineers need to take greater care when it comes to electrical safety incidents on the plant floor.
IIoT: Machines, Equipment, & Asset Management
Articles in this digital report highlight technologies that enable Industrial Internet of Things, IIoT-related products and strategies.
Randy Steele
Maintenance Manager; California Oils Corp.
Matthew J. Woo, PE, RCDD, LEED AP BD+C
Associate, Electrical Engineering; Wood Harbinger
Randy Oliver
Control Systems Engineer; Robert Bosch Corp.
Data Centers: Impacts of Climate and Cooling Technology
This course focuses on climate analysis, appropriateness of cooling system selection, and combining cooling systems.
Safety First: Arc Flash 101
This course will help identify and reveal electrical hazards and identify the solutions to implementing and maintaining a safe work environment.
Critical Power: Hospital Electrical Systems
This course explains how maintaining power and communication systems through emergency power-generation systems is critical.
Design of Safe and Reliable Hydraulic Systems for Subsea Applications
This eGuide explains how the operation of hydraulic systems for subsea applications requires the user to consider additional aspects because of the unique conditions that apply to the setting
click me