Advice for a manufacturing pandemic response plan

The year 2020 started like most years, but by the time March ended, it was clear the challenges and issues our society faced were unprecedented. The country went into a lockdown state by state and travel was restricted to essential activities. Most of our manufacturing customers were deemed essential operations and so production needed to continue.

While our project work went on with our engineers and computer-aided design (CAD) designers at home, our ability to provide services at many of our client sites was initially restricted while manufacturers figured out how to restructure and secure their operations. These restrictions varied from manufacturer to manufacturer. In the months that followed, some of the restrictions have loosened up, but, with good cause, they have not entirely been lifted. By all indications, it may be quite a while before they are.

The COVID-19 pandemic is often compared to the 1918 flu pandemic and many may take comfort in this being a once-in-a-lifetime event. Consider, though, that in the last 20 years, we have also faced SARS (2004), H1N1 (2009) and Ebola (2014) and it is hard not to imagine the next threat could be much closer than we would like to think (see Figure 1).

Now that we are several months into this pandemic, it may be a good time to codify some of what has been learned into a new pandemic response plan. For manufacturers, the goals are the same as a disaster recovery plan: to protect people and to minimize interruptions in operations.

To that end, based on our recent experiences, we submit the following to get the conversation started.

Virtual site visits

The most obvious response to the increased pandemic restrictions is to limit activities by outside contractors onsite. For system integrators and original equipment manufacturers (OEMs), this often means accessing control systems remotely, such as through a virtual private network (VPN). For technical, safety and security reasons, many of our customers do not allow offsite access directly to their control systems. When the initial lockdown orders went into effect, many manufacturers put restrictions in place that prohibited site visits from contractors, as well.

One of our customers got creative when service to one of their systems became necessary. They established a temporary procedure to virtually escort the engineer into the system. The customer’s information technology (IT) group first had to establish a temporary path to access the engineering workstation, then grant remote access to that path to their process engineer.

Once remote access was established, the customer shared their screen on a video conference and our engineer verbally directed them through the troubleshooting process (see Figure 2). With the customer’s help, our engineer was able to help troubleshoot the system and point onsite maintenance to a field instrument that was not providing the feedback it should have been.

If a pandemic response includes different levels of lockdown, the question of how manufacturing systems get the on-demand service they need at each of those levels must be addressed. In this case, there is an argument for making an allowance for some level of remote support in a pandemic.

Team approach

For manufacturers, a large focus of the pandemic response has been keeping operation staff isolated from day-to-day interactions — not just with contractors but also internal support personnel, such as technicians and process engineers. One of our customers established a team approach for engineering and technical support. Support personnel have been divided up and assigned to teams. The teams do not interact with each other. This limits the impact and exposure a single case would have. A team could be quarantined, and the plant could continue running with the remaining teams.

For support personnel and operations staff, it was a matter of more strictly enforcing the separation of shifts. Taking that a step further, this manufacturer also extended the team approach to the engineering and plant management level where two teams have been established and work at the site on opposite days. When a team is off, they work remotely to support the plant where they can.

The tradeoff in this approach is the strain it puts on resources. The rigid approach is understandable, but it makes things like the occasional day off harder to cover and slows down internal project work because resources are less mobile and more confined to their teams.

In this case, the customer looked to outside contractors to help round out their support teams and complete work that would have normally been handled internally.

Remote visualization

Many manufacturers have already invested in a portal that would allow staff engineers who are not at the site the ability to view the process just as operations is seeing it. In a manufacturing environment, these portals should generally be set up as read-only so someone who is remote cannot affect a running process. Even with this restriction, a process engineer can view the graphics, examine the trends and sort through current and historical alarms to help operations work through an issue.

Many of our customers mandated that process engineers work from home when COVID-19 struck. It was at that time when many of the supervisory control and data acquisition (SCADA) and distributed control system (DCS) vendors provided extended trial licenses and discounting on their portal products. Leveraging these offers, we helped several customers set up portals so remote process engineers were able to be more effective in their interactions with operations.

The utility value of products like this goes beyond usage in a pandemic. A tool like this also could mean the difference between needing to go into the plant to help with a 2 A.M. emergency and delay corrective actions versus being able to simply open the laptop, decide on a course of action and go back to bed (see Figure 3).

Regardless, a pandemic response plan should include tools like this. The return comes in the form of reduced downtime and other gained efficiencies that when a process engineer can be more engaged with operations from a distance.

Backup control room

In plants with a centralized control room, the concept of a backup control room has picked up momentum in recent years. Prior to COVID-19, the discussion of a backup control room mostly rose out of disaster recovery requirements developed for fire and flood.

About four years ago, I was attending a meeting at a central utility plant when the building was evacuated. The firefighters who showed up to deal with the emergency could not even enter the building because they were not equipped for the situation. They had to wait for trained personnel to arrive. During this time, the process continued doing what it was doing before, and site operations could neither see nor direct the process to do anything different.

A backup control room, potentially outside the utility plant, would have allowed them to monitor equipment and, if need be, control and shutdown portions of the system.

During a pandemic, a backup control room would allow one control room to be deep cleaned in case of possible contamination while the other is in use. This concept also could be coupled with the team approach discussed above where each operations team would take over operations from alternating control rooms while the other is cleaned in preparation for the next shift.

The wider adoption of thin client technology in the industrial space has helped make the setup of a backup control room easier and more cost effective than before. A set of thin clients in a backup control room can serve as operator workstations using the same set of servers and terminal sessions the primary control room uses. There are ways to architect the additions to the system where sessions can be shared between backup and primary control rooms, saving the cost of additional licenses.

The backup control room also can be smaller. Since one thin client can host multiple sessions, a redundant setup may not need to have as many stations as the primary control room.

Offsite plant management and engineering

There are tools and technologies that can facilitate remotely managing a plant and engineering from a distance including:

• Asset management software: When plant engineering is forced to socially distance with technicians and contract support, it is harder to rely on direct communication about what is happening with maintenance and troubleshooting efforts. An asset management system provides methods for automatically tracking changes to programmable logic controllers (PLCs), human-machine interfaces (HMIs) and drives. Since the software stores a master copy of the configurations, it also can provide notifications when unexpected changes occur.
• Manufacturing execution systems (MES): Running a plant from a distance is hard and knowing how it is performing in real time is even more difficult if you are trying to minimize interactions with operators on the plant floor. A well-thought-out MES can help by automatically collecting the data from the plant floor and input from operations and making that available to plant managers and engineers in real time.
• Network monitoring: There is a selection of products available for monitoring process networks and the devices on them. Some of these are software and others are network appliances. Regardless of the form factor, these products are designed to act as early warning systems for problems that have not impacted operation yet or as a historical record that allows an engineer to go back and see what was happening on the network during a past event.

These are a few of the many of tools that can help engineering and management stay in touch with the plant floor even if they cannot be there in person. 

Looking forward

Obviously, the COVID-19 pandemic is not over, but the conversations about what we can do to prepare for the next event should happen while all of this is fresh in our collective memories.

Given that the development of a pandemic plan begins by asking the “What if…” questions, it is worth noting that many of the questions that manufacturing must ask now are not necessarily the same as those that would have been asked before.

With technology constantly evolving, a manufacturing pandemic response plan should be reviewed and tested on a regular basis so it is always ready to put into action. After all, there will be another pandemic or epidemic event in the future, it is only a question of when.

Applied Control Engineering is a member of the Control System Integrators Association (CSIA).

Original content can be found at Control Engineering.

Do you have experience and expertise with the topics mentioned in this content? You should consider contributing to our WTWH Media editorial team and getting the recognition you and your company deserve. Click here to start this process.