Building a patch management program
Companies that move from legacy proprietary networks to open platforms often discover those new systems need more attention than they thought.
Kevin Staggs, Honeywell Automation and Control Solutions
It truly is a virtual world. The quest for improving plant efficiency through automation has placed an increasingly greater importance on software applications over the last few decades. As a result, the trend in the automation industry has been a move towards open systems platforms to support the growing number of applications and the increased need for connectivity between automation and business systems. With this move towards open systems come the same challenges that are faced by the IT industry. One of those challenges is the need to keep the systems up to date with respect to security updates. Without these updates, systems become more vulnerable to cyber attacks and incidents. This requires implementing a software patch management strategy for the automation systems. While your IT department undoubtedly has a patch management strategy for its own systems, that strategy must be adjusted to accommodate today’s modern automation systems. Without a good automation system patch management strategy, keeping systems up to date can become a very daunting task.
Just as industrial plants have worked for many years to make safety a core part of company culture, proper IT practices, such as software patch management, are becoming an integral part of operational success. Today’s industrial automation and control system requires a patch management program as part of the operational plans for the process and the plant.
A good patch management strategy will improve the robustness and reliability of an industrial automation and control system, but where does one start when it comes to ensuring the basic, best patch management practices? A patch management program should be built upon what corporate IT should already have in place for patch management of critical servers. Engaging automation and control system vendors is also a vital part of a patch management strategy. With these items in place, plants can minimize the impact of unplanned outages related to consequences of not patching systems and keeping them up to date.
The patch management programs that are used for companies’ critical business servers are very similar to the ones that must be employed for industrial automation systems. Plants must start with the IT program and add the necessary elements required for industrial automation. The changes may include specific steps required for compliance with regulations such as NERC-CIP.
At a minimum, a patch management program should include:
Configuration management—This includes things such as an inventory of all hardware and software used in the industrial automation and control environment, including the versions of hardware and software. The software list should include not only automation software, but all of the software in the overall systems, such as MS Windows, SQL, Adobe Reader, and so on. In many cases, this list can be provided by the automation vendor. Additional information in configuration management should include information about the ports and services enabled on each computer. This information is needed to determine if a patch is necessary to install on the automation system. Many times, a patch is released and the software is disabled on the control system; thus, the patch does not need to be installed. A listing of ports and services will help to determine if patch installation is necessary. Again, vendors can provide this information.
Backup/archive—There will be times when it may be necessary to uninstall a patch because it impacted the system in some way. Many times, the only way to uninstall a patch is to restore the system to the state it was prior to applying the patch. It is therefore a very good practice to back up the original system before any patch is installed.
Incident response and disaster recovery plans—The role of the incident response team in a patch management program is two-fold: First, the team needs to be prepared for any incidents that result from the installation of the patch. Second, a dedicated team is necessary when a system vulnerability is exploited. If a patch is not yet available, the incident response team will need to work with all the teams to determine how best to contain the exploit and mitigate the vulnerability until a patch is available. This will involve working with the appropriate industrial automation system vendor, as well as internal teams.
Unit patching operations—Patching an industrial control system needs to be coordinated with the operations team responsible for operating the process. Patching a system, in most cases, will result in a reboot of the system being patched, and if not done properly can result in a loss of view to the process. The operations team can determine the best time when a patch can be applied to the system and may be able to assist with its installation. The operations team should determine the order in which the computers are patched and have the right to stop the patching process at any time due to the plant’s operational requirements.
The actual patch management plan—Each patch to be implemented on the system should have a deployment plan associated with it. This plan should include a schedule for when the patch will be applied across all automation and control systems, as well as deployment instructions, measures for progress of deployment, and back-out plans in the event that a patch causes an unexpected system failure. The plan should also have a defined workflow that includes items such as:
- Vulnerability monitoring
- Vendor patch monitoring
- Risk assessment
- Vulnerability mitigation planning
- Mitigation deployment
- Patch testing
- Patch release for deployment
- Patch scheduling
- Patch deployment
- Patch validation and monitoring
- Patch removal
- Patch program tracking and auditing.
In keeping track of all known vulnerabilities and determining system risks, plants must have an analysis and thorough understanding of mitigations already in place for those vulnerabilities. When patches are made available, an analysis of the risk to the system for applying those patches will also be included. The team analyzing these risks may determine that some patches introduce too much risk. In those cases, they will define different mitigations for specific vulnerabilities.
Additionally, patch testing and deployment release may involve the automation and control system vendor. Many of these vendors test and accredit commercial-off-the-shelf (COTS) patches associated with their systems. Even when the vendor has tested a patch, it is still good practice to test it in an actual plant environment to ensure it does not negatively impact the system it’s meant to protect and maintain. A lab system, if available, is the best place to first test the patch. If not, plants should deploy the patch on noncritical parts of their systems prior to mass deployment. This approach assures there will be no surprises. Part of patch testing should also include testing to assure that a patch can be uninstalled or rolled back.
Patch scheduling and deployment can be done once a patch has been tested and is deemed ready to deploy on the industrial automation and control system. This scheduling should always include the approval of the production managers. Following a common best practice, where possible, patches should be deployed through patch servers, which should be located in a DMZ between the process control network and the enterprise network.
Patch servers should be configured so that they are only reachable by nodes on the process control network, and they should not be able to reach the Internet. Automation and control system patch servers should be configured to get their patches from enterprise patch servers. Once deployment is scheduled, it is then critical to monitor the deployment to assure that all systems are properly patched within the scheduled time frame. In some critical infrastructure sectors, it is necessary to track patch installation for regulatory purposes.
As stated earlier, one partner to consider in a patch management program is the control system vendor. A reliable partner vendor should accredit all COTS patches that its system depends upon. At a minimum, this should include all Microsoft Security Updates. Additionally, plants must ensure they are made aware of all COTS software installed on their control systems, as well as expect the control system vendors to tell them what software is being used. Once the plant knows what software is installed, it should also know what services are enabled in its COTS platforms and what network ports are used by the control system. At the very least, this information should be included in system documentation from the control system vendor.
Control system design
So how can a manufacturer design a plant control system that will be best suited to a patch management program?
In order for plants to assure they can patch their industrial automation and control systems with minimum impact to operations, it is necessary to design the system in a manner that will support patch management. One critical example of such a design is redundancy that allows seamless operation during the patching process. This is achieved by allowing the backup component to be patched first and then switched over before the primary component is patched. Where redundancy is not possible, the system must be designed in a manner that minimizes the impact to operations while that component is off line for the patch process. And of course, implementing patch servers should be a natural part of system design.
Finally, because a patch results in a change to the operating control system, it’s important to incorporate a patch management program into a change-management program. In this case, key stakeholders in this program could include the owners of the manufacturing process, IT leadership, engineering, plant operations, and other relevant support teams.
Many people associate patch management with the required installation of items such as Microsoft Security Updates. This is only one aspect of patch management. Many patches are issued by control system vendors to correct reliability issues. Microsoft, Cisco, and Adobe, to name a few, will issue security patches on a regular cycle. Microsoft will also issue reliability and bug fixes through its Hotfix system. While the focus is on security patches, many patches are issued to correct some form of a vulnerability in the system. In an industrial automation and control system, vulnerabilities can lead to serious issues such as system crashes, slower-than-normal system performance, and outside interference. For these reasons a patch management program is an important part of the overall management of any industrial automation and control system.
Kevin Staggs, CISSP, is an engineering fellow with Honeywell Automation and Control Solutions. For more information, visit: www.honeywellprocess.com.