Software patching is vital to secure operations, but introduces more risks

End of Microsoft Windows XP support raises concerns about industrial networks, connections to PC-based assets, and software patching. Assess software patching risks with 11 critical questions. In manufacturing plant floor applications, security and safety are an integrated concern. Control system cyber security is not the same as desktop PC security.

04/30/2014


GE Measurement & Control’s Cyber Asset Protection (CAP) Testing Lab helps assess the risks and priorities of industrial software patching. Courtesy: GE Measurement & ControlIt's common to think of security updates as self-contained packages, as if the latest anti-virus or Microsoft Windows update was simply a new feature that gets added to the security stack, keeping trouble that much farther away. Yet, when it comes to patching cyber assets on industrial control systems (ICS), one needs to take a little more care than for an office or home PC.

When the office or home PC gets updated (automatically of course), it's understood that there's a possibility of unexpected consequences. Unless there is a major glitch like a lock-up, blue screen, or a primary application's malfunction, the assumption is that everything will work out for the better. In the worst case, the PC gets a reboot, and the expectation is that the next set of updates will correct the inconvenience.

In an industrial plant setting this kind of thinking and lack of awareness begs for disaster. The continuity of operations is critical. Even a minor communication hiccup or loss of view can have undesired results such as interruption of operations, or even catastrophic damage to major equipment [which can increase risk for personnel as well as production].

Regularly applying tested and validated software patches helps maintain access to plant infrastructure and provides critical cyber protection and reliability for daily operations. When operators/owners take a do-it-yourself approach to patching, they often experience unanticipated challenges and risks because of the bandwidth and resources required to properly identify and test software updates before uploading them onto the cyber assets. Manufacturer-provided patching is an excellent starting point for operators to safely execute updates and maintain operational conditions in the plant. 

Is the patch needed?

Do we really need this patch on the PCs?

Maybe! Software manufacturers continuously update, test, and retest their products to improve security and operational efficiency. Hackers continually attempt to find vulnerabilities. This combination leads to the release of updates more frequently than many operators would like to see. Yet, are all of the updates really needed by the plant? Just because a company like Microsoft, which has numerous users operating across a broad range of environments, says that a particular update is critical, it may not be the case for an individual plant's operations. In fact, while some updates may be critical for millions of users, they may be irrelevant for many others. On the other hand, a critical and timely update, for an application such as .NET, could be overlooked by a plant operator due to the lack of knowledge of the internal software functions. This is why it is beneficial for plant operators to ask their equipment manufacturers for help to identify, test, and upload patches following a systematic process. 

Assess patch risk: 11 critical questions

Assessing the relevance of a given patch can be a complex exercise. Knowledge base articles from software manufacturers that provide details on updates are generally comprehensive, and quite detailed. Questions to ask include:

  1. Are the operating systems it affects in use in your operation?
  2. If so, are the vulnerabilities it addresses active on your machines?
  3. What antivirus signature update may detect and delete a .DDL from my SCADA application?
  4. Is the system using SQL server or Internet Explorer?
  5. What about Java or Adobe?
  6. What other third-party applications are in use? (The list of third-party applications on many PCs can be longer than expected.)
  7. Will the update affect my firewall settings or host intrusion detection application (HIDS)? You may find that a patch labeled "critical" protects Windows machines using a DVD authoring app from a possible Trojan horse infection. If DVD authoring is not installed on your systems, then this is one you can live without.
  8. What are patches? Gather all patches for the computer operating system, the application, and other third-party applications.
  9. Which patches are critical? Figure out which ones are critical.
  10. How should the patches be tested? Determine how to test these patches. The cycle starts over every 30 days.
  11. What are the risks and priorities? For that critical patch from Microsoft, should it go into the standard cycle or should you just install it? The patches that pass the relevance test are the ones that will not cause any noticeable changes to the work environment and continue to provide additional protection against security threats. [What are the related operational security and safety risks?]  

Troubleshoot control system interactions

The preferred way to validate patches is to run a set of controlled tests on a representative hardware/software platform. A maintenance system or simulator typically provides an environment where a bad patch result will not interrupt plant operations. Once the patch set has passed this series of tests, the manufacturer begins an incremental installation on the actual plant control systems. This can be a tall order with many different testing environments required, depending on the heterogeneity of the installed base cyber assets.

A secure lab environment with a variety of representative equipment, various operating systems, and typical configurations provides the ideal conditions for testing patches to ensure an error-free update. For most companies, the problem of comprehensive testing before installation is the most challenging step. Securely updating a plant's software is time consuming and requires a significant level of continuous expertise. [subhead]

Selection, validation testing

Mark Hammer is a product line manager at GE Measurement & Control, responsible for developing and creating implementation procedures for control system cyber security programs in the power generation and oil and gas industries. Courtesy: GE Measurement &Many operators are required to keep systems with the most current patches and updates by regulation or company policy. For others, it is an industry best practice that is highly recommend. A good process of gathering, selection, and validation testing should be used to avoid the nightmare scenarios and even minor disruptions to plant operations. Thoroughness is the key, and patching is an essential part of ongoing maintenance to keep plant assets reliable and safe.

- Mark Hammer is a product line manager at GE Measurement & Control, responsible for developing and creating implementation procedures for control system cyber security programs in the power generation and oil and gas industries. Edited by Mark T. Hoske, content manager, CFE Media, Control Engineering, mhoske@cfemedia.com

ONLINE

www.controleng.com/archives May, under this headline, find additional advice, links, and resources about the end of Microsoft Windows XP support.

Control Engineering has an online cyber security training series of videos

Key concepts 

  • Company policies, regulations, and best practices can guide best practices.
  • Gathering, selection, and validation testing should be used to lower risks
  • Thorough patching process is an essential part of ongoing maintenance to keep plant assets reliable and safe.

Consider this

Price of poor patching could include unplanned outages, risk to safety, or loss of critical company assets and information.

ONLINE extra 

More about the author: Mark Hammer is a product line manager at GE Measurement & Control. He is responsible for developing and creating implementation procedures for control system cyber security programs within the power generation and oil and gas industries. He has more than 25 years of experience in the controls and automation industry with a number of leading automation and safety system vendors. He holds both a bachelor's degree in mechanical engineering and master's in business.

- See related articles below.



Anonymous , 06/05/14 12:47 PM:

Excellent article and right on point! However, there is no way that manufacturers can keep up with Microsoft. The whole thing is extremely complex. You will never be bet your life sure that a patch will not break something!
The Top Plant program honors outstanding manufacturing facilities in North America. View the 2015 Top Plant.
The Product of the Year program recognizes products newly released in the manufacturing industries.
The Engineering Leaders Under 40 program identifies and gives recognition to young engineers who...
Hannover Messe 2016: Taking hold of the future - Partner Country status spotlights U.S. manufacturing; Honoring manufacturing excellence: The 2015 Product of the Year Winners
Inside IIoT: How technology, strategy can improve your operation; Dry media or web scrubber?; Six steps to design a PM program
World-class manufacturing: A recipe for success: Finding the right mix for a salad dressing line; 2015 Salary Survey: Manufacturing slump dims enthusiasm
Getting to the bottom of subsea repairs: Older pipelines need more attention, and operators need a repair strategy; OTC preview; Offshore production difficult - and crucial
Digital oilfields: Integrated HMI/SCADA systems enable smarter data acquisition; Real-world impact of simulation; Electric actuator technology prospers in production fields
Special report: U.S. natural gas; LNG transport technologies evolve to meet market demand; Understanding new methane regulations; Predictive maintenance for gas pipeline compressors
Warehouse winter comfort: The HTHV solution; Cooling with natural gas; Plastics industry booming
Managing automation upgrades, retrofits; Making technical, business sense; Ensuring network cyber security
Designing generator systems; Using online commissioning tools; Selective coordination best practices

Annual Salary Survey

Before the calendar turned, 2016 already had the makings of a pivotal year for manufacturing, and for the world.

There were the big events for the year, including the United States as Partner Country at Hannover Messe in April and the 2016 International Manufacturing Technology Show in Chicago in September. There's also the matter of the U.S. presidential elections in November, which promise to shape policy in manufacturing for years to come.

But the year started with global economic turmoil, as a slowdown in Chinese manufacturing triggered a worldwide stock hiccup that sent values plummeting. The continued plunge in world oil prices has resulted in a slowdown in exploration and, by extension, the manufacture of exploration equipment.

Read more: 2015 Salary Survey

Maintenance and reliability tips and best practices from the maintenance and reliability coaches at Allied Reliability Group.
The One Voice for Manufacturing blog reports on federal public policy issues impacting the manufacturing sector. One Voice is a joint effort by the National Tooling and Machining...
The Society for Maintenance and Reliability Professionals an organization devoted...
Join this ongoing discussion of machine guarding topics, including solutions assessments, regulatory compliance, gap analysis...
IMS Research, recently acquired by IHS Inc., is a leading independent supplier of market research and consultancy to the global electronics industry.
Maintenance is not optional in manufacturing. It’s a profit center, driving productivity and uptime while reducing overall repair costs.
The Lachance on CMMS blog is about current maintenance topics. Blogger Paul Lachance is president and chief technology officer for Smartware Group.
This article collection contains several articles on the vital role that compressed air plays in manufacturing plants.
This article collection contains several articles on the Industrial Internet of Things (IIoT) and how it is transforming manufacturing.
click me