Building a patch management program

Companies that move from legacy proprietary networks to open platforms often discover those new systems need more attention than they thought.


It truly is a virtual world. The quest for improving plant efficiency through automation has placed an increasingly greater importance on software applications over the last few decades. As a result, the trend in the automation industry has been a move towards open systems platforms to support the growing number of applications and the increased need for connectivity between automation and business systems. With this move towards open systems come the same challenges that are faced by the IT industry. One of those challenges is the need to keep the systems up to date with respect to security updates. Without these updates, systems become more vulnerable to cyber attacks and incidents. This requires implementing a software patch management strategy for the automation systems. While your IT department undoubtedly has a patch management strategy for its own systems, that strategy must be adjusted to accommodate today’s modern automation systems. Without a good automation system patch management strategy, keeping systems up to date can become a very daunting task.

Just as industrial plants have worked for many years to make safety a core part of company culture, proper IT practices, such as software patch management, are becoming an integral part of operational success. Today’s industrial automation and control system requires a patch management program as part of the operational plans for the process and the plant.

A good patch management strategy will improve the robustness and reliability of an industrial automation and control system, but where does one start when it comes to ensuring the basic, best patch management practices? A patch management program should be built upon what corporate IT should already have in place for patch management of critical servers. Engaging automation and control system vendors is also a vital part of a patch management strategy. With these items in place, plants can minimize the impact of unplanned outages related to consequences of not patching systems and keeping them up to date.

The patch management programs that are used for companies’ critical business servers are very similar to the ones that must be employed for industrial automation systems. Plants must start with the IT program and add the necessary elements required for industrial automation. The changes may include specific steps required for compliance with regulations such as NERC-CIP.

Basic requirements

At a minimum, a patch management program should include:

Configuration management—This includes things such as an inventory of all hardware and software used in the industrial automation and control environment, including the versions of hardware and software. The software list should include not only automation software, but all of the software in the overall systems, such as MS Windows, SQL, Adobe Reader, and so on. In many cases, this list can be provided by the automation vendor. Additional information in configuration management should include information about the ports and services enabled on each computer. This information is needed to determine if a patch is necessary to install on the automation system. Many times, a patch is released and the software is disabled on the control system; thus, the patch does not need to be installed. A listing of ports and services will help to determine if patch installation is necessary. Again, vendors can provide this information.

Backup/archive—There will be times when it may be necessary to uninstall a patch because it impacted the system in some way. Many times, the only way to uninstall a patch is to restore the system to the state it was prior to applying the patch. It is therefore a very good practice to back up the original system before any patch is installed.

Incident response and disaster recovery plans—The role of the incident response team in a patch management program is two-fold: First, the team needs to be prepared for any incidents that result from the installation of the patch. Second, a dedicated team is necessary when a system vulnerability is exploited. If a patch is not yet available, the incident response team will need to work with all the teams to determine how best to contain the exploit and mitigate the vulnerability until a patch is available. This will involve working with the appropriate industrial automation system vendor, as well as internal teams.

Unit patching operations—Patching an industrial control system needs to be coordinated with the operations team responsible for operating the process. Patching a system, in most cases, will result in a reboot of the system being patched, and if not done properly can result in a loss of view to the process.  The operations team can determine the best time when a patch can be applied to the system and may be able to assist with its installation. The operations team should determine the order in which the computers are patched and have the right to stop the patching process at any time due to the plant’s operational requirements. 

The actual patch management plan—Each patch to be implemented on the system should have a deployment plan associated with it. This plan should include a schedule for when the patch will be applied across all automation and control systems, as well as deployment instructions, measures for progress of deployment, and back-out plans in the event that a patch causes an unexpected system failure. The plan should also have a defined workflow that includes items such as:

  • Vulnerability monitoring
  • Vendor patch monitoring
  • Risk assessment
  • Vulnerability mitigation planning
  • Mitigation deployment
  • Patch testing
  • Patch release for deployment
  • Patch scheduling
  • Patch deployment
  • Patch validation and monitoring
  • Patch removal 
  • Patch program tracking and auditing.


In keeping track of all known vulnerabilities and determining system risks, plants must have an analysis and thorough understanding of mitigations already in place for those vulnerabilities. When patches are made available, an analysis of the risk to the system for applying those patches will also be included. The team analyzing these risks may determine that some patches introduce too much risk. In those cases, they will define different mitigations for specific vulnerabilities.

Additionally, patch testing and deployment release may involve the automation and control system vendor. Many of these vendors test and accredit commercial-off-the-shelf (COTS) patches associated with their systems. Even when the vendor has tested a patch, it is still good practice to test it in an actual plant environment to ensure it does not negatively impact the system it’s meant to protect and maintain. A lab system, if available, is the best place to first test the patch. If not, plants should deploy the patch on noncritical parts of their systems prior to mass deployment. This approach assures there will be no surprises. Part of patch testing should also include testing to assure that a patch can be uninstalled or rolled back.


Patch scheduling and deployment can be done once a patch has been tested and is deemed ready to deploy on the industrial automation and control system. This scheduling should always include the approval of the production managers. Following a common best practice, where possible, patches should be deployed through patch servers, which should be located in a DMZ between the process control network and the enterprise network.

Patch servers should be configured so that they are only reachable by nodes on the process control network, and they should not be able to reach the Internet. Automation and control system patch servers should be configured to get their patches from enterprise patch servers. Once deployment is scheduled, it is then critical to monitor the deployment to assure that all systems are properly patched within the scheduled time frame. In some critical infrastructure sectors, it is necessary to track patch installation for regulatory purposes.

As stated earlier, one partner to consider in a patch management program is the control system vendor. A reliable partner vendor should accredit all COTS patches that its system depends upon. At a minimum, this should include all Microsoft Security Updates. Additionally, plants must ensure they are made aware of all COTS software installed on their control systems, as well as expect the control system vendors to tell them what software is being used. Once the plant knows what software is installed, it should also know what services are enabled in its COTS platforms and what network ports are used by the control system. At the very least, this information should be included in system documentation from the control system vendor.

Control system design

So how can a manufacturer design a plant control system that will be best suited to a patch management program?

In order for plants to assure they can patch their industrial automation and control systems with minimum impact to operations, it is necessary to design the system in a manner that will support patch management. One critical example of such a design is redundancy that allows seamless operation during the patching process. This is achieved by allowing the backup component to be patched first and then switched over before the primary component is patched. Where redundancy is not possible, the system must be designed in a manner that minimizes the impact to operations while that component is off line for the patch process. And of course, implementing patch servers should be a natural part of system design.

Finally, because a patch results in a change to the operating control system, it’s important to incorporate a patch management program into a change-management program. In this case, key stakeholders in this program could include the owners of the manufacturing process, IT leadership, engineering, plant operations, and other relevant support teams.

Many people associate patch management with the required installation of items such as Microsoft Security Updates. This is only one aspect of patch management. Many patches are issued by control system vendors to correct reliability issues. Microsoft, Cisco, and Adobe, to name a few, will issue security patches on a regular cycle. Microsoft will also issue reliability and bug fixes through its Hotfix system.  While the focus is on security patches, many patches are issued to correct some form of a vulnerability in the system. In an industrial automation and control system, vulnerabilities can lead to serious issues such as system crashes, slower-than-normal system performance, and outside interference. For these reasons a patch management program is an important part of the overall management of any industrial automation and control system.

Kevin Staggs, CISSP, is an engineering fellow with Honeywell Automation and Control Solutions. For more information, visit:

Top Plant
The Top Plant program honors outstanding manufacturing facilities in North America.
Product of the Year
The Product of the Year program recognizes products newly released in the manufacturing industries.
System Integrator of the Year
Each year, a panel of Control Engineering and Plant Engineering editors and industry expert judges select the System Integrator of the Year Award winners in three categories.
September 2018
2018 Engineering Leaders under 40, Women in Engineering, Six ways to reduce waste in manufacturing, and Four robot implementation challenges.
GAMS preview, 2018 Mid-Year Report, EAM and Safety
June 2018
2018 Lubrication Guide, Motor and maintenance management, Control system migration
August 2018
SCADA standardization, capital expenditures, data-driven drilling and execution
June 2018
Machine learning, produced water benefits, programming cavity pumps
April 2018
ROVs, rigs, and the real time; wellsite valve manifolds; AI on a chip; analytics use for pipelines
Spring 2018
Burners for heat-treating furnaces, CHP, dryers, gas humidification, and more
August 2018
Choosing an automation controller, Lean manufacturing
September 2018
Effective process analytics; Four reasons why LTE networks are not IIoT ready

Annual Salary Survey

After two years of economic concerns, manufacturing leaders once again have homed in on the single biggest issue facing their operations:

It's the workers—or more specifically, the lack of workers.

The 2017 Plant Engineering Salary Survey looks at not just what plant managers make, but what they think. As they look across their plants today, plant managers say they don’t have the operational depth to take on the new technologies and new challenges of global manufacturing.

Read more: 2017 Salary Survey

The Maintenance and Reliability Coach's blog
Maintenance and reliability tips and best practices from the maintenance and reliability coaches at Allied Reliability Group.
One Voice for Manufacturing
The One Voice for Manufacturing blog reports on federal public policy issues impacting the manufacturing sector. One Voice is a joint effort by the National Tooling and Machining...
The Maintenance and Reliability Professionals Blog
The Society for Maintenance and Reliability Professionals an organization devoted...
Machine Safety
Join this ongoing discussion of machine guarding topics, including solutions assessments, regulatory compliance, gap analysis...
Research Analyst Blog
IMS Research, recently acquired by IHS Inc., is a leading independent supplier of market research and consultancy to the global electronics industry.
Marshall on Maintenance
Maintenance is not optional in manufacturing. It’s a profit center, driving productivity and uptime while reducing overall repair costs.
Lachance on CMMS
The Lachance on CMMS blog is about current maintenance topics. Blogger Paul Lachance is president and chief technology officer for Smartware Group.
Material Handling
This digital report explains how everything from conveyors and robots to automatic picking systems and digital orders have evolved to keep pace with the speed of change in the supply chain.
Electrical Safety Update
This digital report explains how plant engineers need to take greater care when it comes to electrical safety incidents on the plant floor.
IIoT: Machines, Equipment, & Asset Management
Articles in this digital report highlight technologies that enable Industrial Internet of Things, IIoT-related products and strategies.
Randy Steele
Maintenance Manager; California Oils Corp.
Matthew J. Woo, PE, RCDD, LEED AP BD+C
Associate, Electrical Engineering; Wood Harbinger
Randy Oliver
Control Systems Engineer; Robert Bosch Corp.
Data Centers: Impacts of Climate and Cooling Technology
This course focuses on climate analysis, appropriateness of cooling system selection, and combining cooling systems.
Safety First: Arc Flash 101
This course will help identify and reveal electrical hazards and identify the solutions to implementing and maintaining a safe work environment.
Critical Power: Hospital Electrical Systems
This course explains how maintaining power and communication systems through emergency power-generation systems is critical.
Design of Safe and Reliable Hydraulic Systems for Subsea Applications
This eGuide explains how the operation of hydraulic systems for subsea applications requires the user to consider additional aspects because of the unique conditions that apply to the setting
click me