Water/wastewater: achieving the three levels of redundancy

To provide reliable water and wastewater treatment, municipalities must provide a 24-hour operation on a shoestring budget while battling constantly changing regulations, influent criteria, new security concerns and aging infrastructures. To address these challenges, many facilities are either considering an upgrade of their current automation system or installing automation for the first time.


To provide reliable water and wastewater treatment, municipalities must provide a 24-hour operation on a shoestring budget while battling constantly changing regulations, influent criteria, new security concerns and aging infrastructures. To address these challenges, many facilities are either considering an upgrade of their current automation system or installing automation for the first time. When examining an automation system’s design, the question of redundancy often arises. But what is redundancy, and does it equate to a hot standby system?

What is redundancy?

Redundancy comes in many forms and is inherent at some level in any plant design. The most basic form of redundancy requires the inclusion of a hand-off-auto switch for each component. In the automatic mode, the plant or system controller runs the process. In the hand mode this step is bypassed, resulting in continued treatment but a loss of efficiency and/or quality.

For example, in an oxidation ditch scenario, the automatic operation may stage aerators based on dissolved oxygen levels. In hand mode, it would be best to turn on all the aerators. This would ensure adequate levels of air without having to continuously monitor process conditions. However, it would also waste power.

As an alternative, sometimes a process can be bypassed. This is common in headworks, in which a bypass channel around the screen can be found. Using this bypass channel can ensure the plant operates, but it may allow particles to accumulate in the downstream clarifiers and basins.

Another form of redundancy exists when more equipment is installed than is required. For example, three pumps may be provided when only two are needed. This type of redundancy is quite common. Typically the third pump still has its own dedicated starter, variable frequency drive and control components. This extends some redundancy to the control and automation system as well.

Redundancy is also gained with multiple process trains. Sometimes each train has its own automation system, or trains have been grouped into multiple control panels. This also represents a level of redundancy.

Finally, redundancy can be designed into an automation system.

Three types of automation redundancy

Automation redundancy can be accomplished via three methods: cold, warm and hot.

Cold redundancy %%MDASSML%% Cold redundancy is for those processes where response time is of minimal concern and may require operator intervention. For example, consider two belt presses, each with a dedicated control panel. If one belt press becomes inoperable, the operator can resume operation by simply starting the other press.

In this example, the loss of one belt press may result in an accumulation of un-pressed solids, but will not likely impact the ability to meet permit. Thus, operator intervention is acceptable, and a cold redundancy design may be a good solution. However, when time is more critical, warm or hot redundancy are better approaches.

Warm redundancy %%MDASSML%% Warm redundancy is used where time is somewhat critical but a momentary outage is still acceptable. In this scenario, a momentary bump can be expected. During this bump, the valves, motors and other devices might shutdown temporarily, and the sensors may not report back to the PLC system during the bump.

An example of a warm redundancy application may be exemplified on an ATAD system. In an ATAD process, biological decomposition creates heat. Mixers, aerators and foam cutters are frequently used to distribute the heat, introduce air and reduce foam accumulation. During a momentary changeover, the mixers, aerators and foam cutters could stop working. During the changeover, the heat in the system would remain constant due to the nature of the process, and progression towards Class A would not be compromised.

However, cessation of the mixers, aerators and foam cutters for more than a few minutes would eventually cause an undesired foam build up and heat stratification, causing the process to transition from aerobic to anaerobic. Thus, in this system, the process can tolerate having the mixers, aerators and foam cutters off for a few seconds, with the understanding that they must be restored quickly and automatically to avoid risking the process integrity.

Warm redundancy systems typically have two processors connected in a primary and standby configuration. The primary processor controls the system’s inputs and outputs (I/O) while the standby processor is powered up and waits for the primary processor to stop controlling the process. When this occurs, the standby processor assumes control of the I/O and takes the designation of primary processor, allowing the offline processor to become the secondary processor, which can then be maintained without sacrificing process control.

During normal operation, the primary processor provides periodic updates to the standby processor. These updates usually occur at the end of each program scan and may only involve a portion of the data at any time. Therefore, when a changeover occurs, the standby processor can work off of incomplete data since it may take the standby processor a few program scans to catch up to where the primary was before the changeover. This can contribute to a bump in the process during the changeover.

From a hardware perspective, warm and hot redundancy systems are almost identical, and can be easily confused when looking at suppliers’ product data. Care should be taken to carefully examine the different types of systems.

Hot redundancy %%MDASSML%% Hot Redundancy is used when the process must not go down for even a brief moment under any circumstance. An example of a hot redundancy system application is a membrane or BAF application. In both instances, the process may not require the hot redundancy capabilities, but during a backwash, redundancy could be critical.

During a backwash, a valve reversal, a motor halt or re-sequencing of the valves may cause an incomplete backwash, allow contaminants to enter the clean water or may even contribute to inoperable equipment. Since none of these scenarios are tolerable, a hot redundancy configuration would be the preferred approach in this case.

Again, the hardware layout of a hot redundancy system is almost identical to a warm redundancy system. However, hot redundancy systems provide bumpless transfer of the I/O during a changeover from primary to standby. To accomplish this, communication messaging, updating the secondary processor and program execution must be properly managed.

To ensure that a hot redundancy system operates correctly, data must be transferred from the primary processor to the secondary processor with every logic cycle to ensure data integrity. Two methods exist for accomplishing the needed data transfer.

The first is to perform the transfer at the end of the program scan. Only upon completion of the transfer will the scan resume. This approach can be called “scan-and-transfer.” This technique was first employed by Modicon when it created the first redundant PLC, and it continues to be an approach taken by many suppliers today. As real world experience has shown, this method can provide accurate updating of data and good control.

However, there are criteria that should be considered when using a scan-and-transfer system. First, the true scan time of the program will be a combination of the program scan and the transfer update. Since scan time can be critical in certain applications, suppliers that support this type of transfer will indicate in their manual that the program should be designed to minimize scanning. They will offer suggestions on how to limit rung executions to only those instances when conditional logic has changed. If these suggestions are not incorporated properly, a situation could arise where a bump is experienced on the output. Any bump possibility would create a warm redundancy situation.

While the scan and transfer technique was used for years, and is a reliable and accurate method when properly applied, technological advances have made a new technique possible %%MDASSML%% a method that ensures the transfer does not depend on scan time.

This new method is referred to as “asynchronous transfer.” In an asynchronous transfer, the primary processor has two separate microprocessors embedded in its circuitry. The first microprocessor executes the program. At the end of the execution, all data is passed to the second microprocessor. This second microprocessor handles all transfer tasks while the first microprocessor executes the next program scan. Thus, one microprocessor is executing while the other is transferring data to the standby processor.

As this transfer of data from primary processor to secondary processor is asynchronous to the program scan, it now becomes possible to transfer the entire data table without affecting program execution. This eliminates any need to design the program for an optimized scan. In many cases, this benefit makes asynchronous transfer a better fit for water and wastewater treatment.

Redundancy in water and wastewater treatment is critical. However, redundancy is a philosophy, not a set solution.

Fig. 1. Cold redundancy is for those processes where response time is of minimal concern and may require operator intervention. If cessation can be tolerated until the operator can intervene, then cold is best.

Fig. 2. Warm redundancy can be used when a process bump can be tolerated. When the system can handle a slight cessation but automatic intervention is necessary, warm redundancy is the solution.

Fig. 3. For those operations that cannot experience even a momentary cessation, hot redundancy is the only option.

Author Information

Grant Van Hemert, P.E. is an application engineer at Schneider Electric’s Water and Wastewater Competency Center in Nashville, TN.

No comments
The Top Plant program honors outstanding manufacturing facilities in North America. View the 2013 Top Plant.
The Product of the Year program recognizes products newly released in the manufacturing industries.
The Engineering Leaders Under 40 program identifies and gives recognition to young engineers who...
A cool solution: Collaboration, chemistry leads to foundry coat product development; See the 2015 Product of the Year Finalists
Raising the standard: What's new with NFPA 70E; A global view of manufacturing; Maintenance data; Fit bearings properly
Sister act: Building on their father's legacy, a new generation moves Bales Metal Surface Solutions forward; Meet the 2015 Engineering Leaders Under 40
Cyber security cost-efficient for industrial control systems; Extracting full value from operational data; Managing cyber security risks
Drilling for Big Data: Managing the flow of information; Big data drilldown series: Challenge and opportunity; OT to IT: Creating a circle of improvement; Industry loses best workers, again
Pipeline vulnerabilities? Securing hydrocarbon transit; Predictive analytics hit the mainstream; Dirty pipelines decrease flow, production—pig your line; Ensuring pipeline physical and cyber security
Upgrading secondary control systems; Keeping enclosures conditioned; Diagnostics increase equipment uptime; Mechatronics simplifies machine design
Designing positive-energy buildings; Ensuring power quality; Complying with NFPA 110; Minimizing arc flash hazards
Building high availability into industrial computers; Of key metrics and myth busting; The truth about five common VFD myths

Annual Salary Survey

After almost a decade of uncertainty, the confidence of plant floor managers is soaring. Even with a number of challenges and while implementing new technologies, there is a renewed sense of optimism among plant managers about their business and their future.

The respondents to the 2014 Plant Engineering Salary Survey come from throughout the U.S. and serve a variety of industries, but they are uniform in their optimism about manufacturing. This year’s survey found 79% consider manufacturing a secure career. That’s up from 75% in 2013 and significantly higher than the 63% figure when Plant Engineering first started asking that question a decade ago.

Read more: 2014 Salary Survey: Confidence rises amid the challenges

Maintenance and reliability tips and best practices from the maintenance and reliability coaches at Allied Reliability Group.
The One Voice for Manufacturing blog reports on federal public policy issues impacting the manufacturing sector. One Voice is a joint effort by the National Tooling and Machining...
The Society for Maintenance and Reliability Professionals an organization devoted...
Join this ongoing discussion of machine guarding topics, including solutions assessments, regulatory compliance, gap analysis...
IMS Research, recently acquired by IHS Inc., is a leading independent supplier of market research and consultancy to the global electronics industry.
Maintenance is not optional in manufacturing. It’s a profit center, driving productivity and uptime while reducing overall repair costs.
The Lachance on CMMS blog is about current maintenance topics. Blogger Paul Lachance is president and chief technology officer for Smartware Group.