Sustaining SI support for water/wastewater facilities

As water/wastewater municipalities are struggling to replace retiring technical personnel and support increasingly sophisticated automation technologies, they are finding well-staffed system integrators can provide the electrical, instrumentation and controls support they need.

By Matt Boyer December 12, 2022
Courtesy: Tesco Controls

 

Learning Objectives

  • Municipalities are facing a labor shortage and lack qualified personnel. Systems integrators can help fill the gap.
  • Integrators can provide contractual services for municipalities and utilities like water and wastewater plants and develop a long-term relationship.
  • Integrators also can help with automation and process instrumentation support for end users.

System integration insights

  • Water and wastewater facilities have many challenges for their day-to-day operations, and system integrators are helping resolve challenges, many increasing with new regulations and complex treatment processes.
  • System integrators (SIs) can help these facilities by providing a neutral perspective on what they need and don’t need and giving an assessment on the physical health of the machines.
  • SIs also can provide audits on the hardware and software tying machines together and give them options to prevent a potential cyberattack like Oldsmar, which identified water and wastewater industry cybersecurity vulnerabilities in 2021.

Municipalities are facing a multitude of challenges on how they operate their water and wastewater systems. These challenges include new regulatory standards, complex treatment processes, environmental scarcity and population-driven increases in system demand. Other compounding issues to contend with include the need to incorporate continuing advancements in technology like smart instrumentation and final control elements as well as the struggle to replace retiring operators and control technicians. These aspects exacerbate already complex requirements water and wastewater facilities face for calibrating, maintaining and troubleshooting their equipment (Figure 1).

Figure 1: Water/wastewater municipalities, and processing operations of all types, face many challenges with adopting innovative technologies, supporting increased digitalization, and replacing retiring workers. A full-service SI can provide the sustaining support they need. Courtesy: Tesco Controls

Figure 1: Water/wastewater municipalities, and processing operations of all types, face many challenges with adopting innovative technologies, supporting increased digitalization, and replacing retiring workers. A full-service SI can provide the sustaining support they need. Courtesy: Tesco Controls

However, these challenges can be addressed by systems integrators (SIs), who can help by providing contractual preventive maintenance, calibration and troubleshooting services. SIs help relieve the burden to keep production systems online by complementing the municipality’s staff capabilities while filling gaps in staff numbers and/or expertise.

SI support system

For many years, municipality staffing levels and skill sets – especially for simpler system designs – were sufficient to handle most issues. Basic operations used pressure, level, flow and other process instrumentation to control pumps, valves and other automated equipment. Today’s systems and technology are increasingly digitized and more sophisticated to deliver improved performance, but often with greater complexity to design, deploy and operate (Figure 2).

Municipalities benefit from hiring and retaining staff who are multidisciplined. Technical staff must understand smart instrumentation, networking, radio telemetry, supervisory control and data acquisition (SCADA) application development, human-machine interface (HMI) configuration, programmable logic controller (PLC) programming and electrical control components setup. This is on top of the basic electrical, mechanical and safety skills that were always required. Finding and keeping these workers has become more difficult, as the existing workforce is “graying,” and new workers often prefer remote work options.

Figure 2: Industrial automation has advanced far beyond traditional methods in both capability and complexity. An SI is in the business of assembling an entire team of multidisciplined staff to support all aspects of designing and maintaining today’s electrical, instrumentation, HMI/SCADA, and PLC systems. Courtesy: Tesco Controls

Figure 2: Industrial automation has advanced far beyond traditional methods in both capability and complexity. An SI is in the business of assembling an entire team of multidisciplined staff to support all aspects of designing and maintaining today’s electrical, instrumentation, HMI/SCADA, and PLC systems. Courtesy: Tesco Controls

Many SIs offer post-startup services on the specific systems and equipment they were involved with. Responses typically were on a case-by-case basis after an issue arose, requiring troubleshooting and repair at a level beyond what onsite technical staff could handle. This work was often invoked at a moment’s notice and billed at time and material (T&M) rates.

Acknowledging the flaws of this approach, some larger SI firms have developed contractual solutions—usually in the form of annual, bi-annual, or multi-annual contracts—based around preventive and extended maintenance solutions. This approach enables SI staff to work on control panels, motor control centers (MCCs), variable frequency drives (VFDs) and other automation to monitor system health, identify potential problems and points of failure, and proactively mitigate issues before they cause shutdowns.

Controls, automation service contract options

Complete service contracts, which include provisions for SIs to investigate control and electrical panels, can yield significant savings by reducing outages. For example, these investigations frequently identify failed uninterruptable power supplies (UPSs), discover dead PLC backup batteries, and find broken cooling fan and thermostat devices needing replacement to protect equipment from hotter summertime temperatures (Figure 3).

Figure 3: A complete service contract offered by an SI like Tesco Controls incorporates trouble response services, but also provides for periodic inspections so problems can be detected—and resolved—proactively, to avoid downtime and other consequences. Courtesy: Tesco Controls

Figure 3: A complete service contract offered by an SI like Tesco Controls incorporates trouble response services, but also provides for periodic inspections so problems can be detected—and resolved—proactively, to avoid downtime and other consequences. Courtesy: Tesco Controls

These issues do not always annunciate conditions in ways operators notice ahead of time. However, operators would certainly become aware of the situation in the event of a power failure or heat wave, which can take down system devices and equipment, potentially resulting in downtime, delayed restart time, and program and memory loss.

By engaging a well-versed system integrator from outside the municipality to routinely inspect control panels, facilities can significantly mitigate potential points of failure inhouse operational staff may not notice while focusing on day-to-day issues.

Beyond control panels, some clients choose to have SI support teams inspect electrical equipment, as well. MCCs are industrial-grade electrical hardware, but they are often overlooked by operators who believe that if equipment is running, then all must be well. A prepared SI can scrutinize operational MCCs with voltage and amperage checks, and in some cases use power quality analyzers to identify possible utility power issues (Figure 4).

Figure 4: End users should look for SIs with the training and skillset to support all aspects of electrical instrumentation and controls. Courtesy: Tesco Controls

Figure 4: End users should look for SIs with the training and skillset to support all aspects of electrical instrumentation and controls. Courtesy: Tesco Controls

A thermal imager is another helpful diagnostic device that a municipality might not have on-hand. Thermal imagers are used to perform infrared (IR) scans of electrical components and terminations – MCCs, VFDs, soft starters, contactors, and transformers – while in operation to identify loose connections or failing components. A simple retorquing activity can prevent a lost-time failure event from loose connections.

Infrared scan finds loose electrical connection

For example, an SI was performing IR scans at a municipality as part of its preventative maintenance program, and they found a loose connector on the breaker feeding a VFD. While the VFD was still in operation and appeared to be working normally, the scan showed one of the three conductors on the load side of the breaker was 100 °F hotter than the other two.

The staff was able to safely power down, lockout/tagout (LOTO) the drive, and properly remedy the connection. This solved the temperature issue, preventing operational and safety problems that could have occurred later if not addressed.

System integrator process instrumentation support

Only some SIs are equipped to perform electrical support services, but a greater number can provide instrument calibration and verification support (Figure 5). Many sites have hundreds or thousands of field instruments of varying vintage, make and model. These often require annual attention to confirm and document proper operation, communication, scaling and interaction with PLCs and SCADA.

While instruments usually are dialed in during startup and commissioning, process and environmental conditions change over time and instrument drift occurs causing errors. If flowmeter was replaced but configured with incorrect scaling, the SCADA system could under-report total process flow.

Figure 5: Considering that a typical municipality or processing plant uses hundreds or thousands of field instruments requiring periodic calibration and inspection, many end users rely on the support of a full-service SI like Tesco Controls to supplement their in-house staffing. Courtesy: Tesco Controls

Figure 5: Considering that a typical municipality or processing plant uses hundreds or thousands of field instruments requiring periodic calibration and inspection, many end users rely on the support of a full-service SI like Tesco Controls to supplement their in-house staffing. Courtesy: Tesco Controls

Help with instrument calibration

Many end users would be overwhelmed by ongoing instrument calibration efforts, which is where an SI can provide verification services. This generally consists of tying into each 4-20mA signal loop (or using a HART or Foundation Fieldbus communicator), and simulating the instrument through its scaled range. This requires specially-trained personnel, test equipment and calibration standards.

Trained specialists can work with basic measurements like flow, level, pressure and temperature, as well as advanced analytical technologies. Although water and wastewater staff are often required to be certified in performing analytical calibrations, some regulatory agencies also recommend or require a third-party verification. During these visits, the SI can help identify issues that operators may have missed, such as analyzer tubing degradation, expired reagents and obsolete devices.

Automation hardware and software maintenance services

There is a general focus on contractual support services around preventive maintenance services only, and specifically for hardware-based equipment like MCCs, instruments, and control panels. This is an important starting point, but by structuring these services around technical electrical, instrumentation and controls services, an SI can provide more comprehensive options and help municipalities handle entire systems with access to the SI’s full knowledge base.

For example, an expanded service contract supporting SCADA maintenance can help identify potential issues with an end user’s computer hardware and software. While a user may believe their system is working well, a qualified SI can scrutinize the hardware’s health in addition to software versions and security vulnerabilities. These proactive efforts may uncover impending hardware failures and potential software issues, giving the client a possible solution prior to catastrophic failure. A reactive response, by contrast would likely involve an expensive, time-consuming and stressful scramble to replace failed elements and restore system software and settings.

SCADA maintenance tasks

A SCADA maintenance service has other benefits. Often, district staff will bring up small issues they have noticed with their SCADA system over the years, and the SI can usually address some of these issues during the normal contractual work (Figure 6).

Figure 6: As part of a complete service contract, an SI can perform HMI/SCADA maintenance and resolve a wide range of issues during normal visits, without requiring special projects to be initiated. Courtesy: Tesco Controls

Figure 6: As part of a complete service contract, an SI can perform HMI/SCADA maintenance and resolve a wide range of issues during normal visits, without requiring special projects to be initiated. Courtesy: Tesco Controls

Perhaps a tag on a display has a maximum value that is clamped too low, below a flowmeter’s actual process value. Or another operator’s phone number needs to be added to an alarm callout roster. These minor issues will not generally warrant a full quote for work to resolve them, but they can be addressed in the scope of preventive maintenance service visits.

SI visits also can act as operational training sessions for operators and other end users. This includes ad-hoc lessons on adding a new trend to a graph, filtering historical alarms and many other navigational and operational details. While SCADA operation instructions are covered during startup and in documentation – such as an operations and maintenance manual – a contracted periodic annual, bi-annual, or even quarterly service visit is an excellent opportunity to refresh operator knowledge, assist newer users and help municipality staff get the most out of their system.

Sustaining SI help keeps automation running

The challenges facing municipalities continue to grow in number and complexity. Well-staffed SI firms possess the right mix of multidisciplinary personnel and extensive knowledgebases to provide sustaining support to end user clients (Figure 7).

Figure 7: Even once a municipal or processing plant is up and running, there are many reasons for end users to remain engaged with a full-service SI like Tesco Controls to provide ongoing support for increasingly complex electrical, instrumentation, and control systems. Courtesy: Tesco Controls

Figure 7: Even once a municipal or processing plant is up and running, there are many reasons for end users to remain engaged with a full-service SI like Tesco Controls to provide ongoing support for increasingly complex electrical, instrumentation, and control systems. Courtesy: Tesco Controls

Some SIs are leveraging these abilities to work beyond the traditional design, develop and deploy model. They are building client relationships through extended contractual maintenance, service, and support agreements. This helps municipal end users keep crucial water and wastewater systems online and running at peak performance.

Matt Boyer is the extended maintenance and system services (EMASS) supervisor at Tesco Controls. Edited by Chris Vavra, web content manager, CFE Media and Technology, cvavra@cfemedia.com.

MORE ANSWERS

Keywords: system integration, wastewater facility

ONLINE

See additional system integration stories at https://www.controleng.com/system-integration/

CONSIDER THIS

What ways can systems integrators help keep your facility running?

Original content can be found at Control Engineering.


Author Bio: Matt Boyer is the extended maintenance and system services (EMASS) supervisor at Tesco Controls. Boyer has extensive experience as a field service engineer in the water/wastewater and automotive racing industries, and holds ISA CCST Level 1 and Inductive Automation Ignition SCADA platform certifications, and has a BA in Political Science with a minor in History, and an AA in Automotive Technology.