Learning tough lessons from ICS attacks

A security incident can cause damage to brand reputation, loss of competitive advantage, legal or regulatory non-compliance issues, considerable financial damage, and harm to the environment and community.


Industrial control systems (ICS), much to the industry’s chagrin, are not immune to data breaches, ransomware attacks, viruses, malware, insider attacks or any other form of assault. On top of that, there are some experts that say they need tangible proof attacks really happen. Well, there is.

Just take a look at the Verizon 2016 Data Breach Investigations Report (DBIR). The report details 16 common breach scenarios and the cases are each told from the perspective of the stakeholders involved, such as corporate communications, legal counsel, or the human resources professional.

In one poignant scenario, a manufacturer fell victim and had to get their house in order fairly quickly. Here is what the report said about this one incident:

“A company, we’ll call Gator-Grasp Fasteners, retained the Verizon RISK Team to perform a health check of their industrial environment. This particular customer was in the business of fabricating specialized fasteners, which were required to pass very specific engineering requirements, such as meeting or surpassing certain strength, tensile stress, mechanical properties and material content thresholds.

“At the onset of the health check, Gator-Grasp Fasteners’ automation engineers expressed skepticism and mild dissent, arguing that a “health check” was not necessary. In their many years of being on the job, the “patient” had always functioned well and had shown no signs of being “unhealthy.” So why mess with things? They assured their management that the Operational Technology (OT) environment was secure and that they expected there would be no significant findings. After all, the automation engineers were experts and they knew what they were doing. Nonetheless, management insisted and the automation engineers reluctantly agreed to work with the RISK Team.

Not in my house

“As with any engagement, there was a kick-off meeting, which was used to introduce everyone, set initial expectations, discuss the in-scope environment, request additional information and schedule the onsite visit.

“The requested information included a list of network segments, IP address ranges, IP address assignments, and an asset inventory. “The Gator-Grasp Fasteners Team was instructed not to create any new documentation in order to avoid a situation where the creation of new documentation would potentially mask a procedural deficiency. In assembling the requested documentation, Gator-Grasp Fasteners quickly realized that what it did have was inadequate.

“During the on-site visit, the automation engineers, the RISK Team’s critical infrastructure protection/cybersecurity (CIP/CS) specialists and other subject matter experts (SMEs) discussed the various OT systems, in-place security measures and other operational procedures. This included processes and practices (aka “institutional knowledge”) that are followed, but were not necessarily documented. These discussions revealed that over the past few months, the network seemed “sluggish,” which the automation engineers and SMEs attributed to older, legacy equipment. With an understanding of the situation in mind, we visited various locations where we walked the manufacturing floor and made additional observations.

“One of the first things we noticed was some OT systems had anti-virus protection while others didn’t. For those that didn’t, we were told that, since they were isolated, they didn’t need protection. Incredibly, when we looked at the anti-virus logs on the OT systems that had malware protection, we found them replete with malware detections, deletions, and quarantine alerts. Of the 57 systems in total, 33 systems had at least one malware alert, and many had multiple alerts.

“When we inquired about these alerts, we found that the automation engineers and operators were well aware. They reasoned that since the malware protection was correcting and “repairing” the problems, everything was acceptable. We explained that there was clearly an underlying problem leading to the repeat infections and recommended a more detailed review to identify the root cause."

Need logging

“Gator-Grasp Fasteners had no documented Incident Response (IR) process for investigating incidents, so we took the lead. The company did not have a centralized logging solution and what devices did log did not provide insight into how the malware was getting into the network. The problem? We needed more visibility.

“With the cooperation of Gator-Grasp Fasteners, we set up a Switched Port Analyzer (SPAN) port and deployed a passive network analyzer to collect and analyze the traffic. Using indicators related to the identified malware, we reviewed network traffic and quickly identified multiple potentially infected systems. As we expected, the network traffic revealed malware infections associated with the legacy OT systems that did not have anti-virus protection. Further analysis revealed that a number of misconfigurations existed – which had allowed unauthorized network communication.

“The infected systems, many of which were very actively searching for new systems, were a good candidate for the “slow network” problems identified during earlier interviews. Using the collected network traffic, we ran statistics on data transfer rates and quickly realized that the scanning attempts were saturating legacy network connections with probes. With a concrete list of infected systems, we targeted the population of compromised endpoints.

“Despite the widespread infection, Gator-Grasp Fasteners had been fortunate. Review of the malware resident on each system revealed common drive-by infections, all targeted at stealing banking credentials. As none of the infected OT systems were utilized for anything other than process management, it was unlikely that further damage had occurred. The network trouble was an unintended side effect of the malware’s attempts to find new systems compounded with overly permissive firewall rules.

“We provided a list of known infected systems to Gator-Grasp Fasteners, which quickly began rebuilding them from known good images. To keep remediated systems remediated during this process, we continued network traffic monitoring for known indicators and behaviors associated with the identified malware. With the current issue well on the path to being resolved, we turned our attention to the uninfected, but still “troubled” OT systems.”

Verizon RISK team broke down some recommendations into three categories covering the company’s entire OT environment:

  1. Unnecessary legacy systems in unmanned locations. These systems ended up removed from the network and decommissioned. These were difficult to track down as they were not documented, making them hard to find, which ultimately delayed the containment and eradication activities.
  2. Necessary legacy systems unable to end up protected by an anti-virus solution. The RISK team manually removed the existing malware and the systems ended up hardened from a best practices standpoint. Stringent firewall rules ended up deployed to prevent access to and from these systems, designed to limit the reach of any future compromises.
  3. New systems not patched or protected by an anti-virus solution. These computer systems ended up patched and malware protection installed.

Best practices

Just as in the non-ICS world, a security incident can cause damage to brand reputation, loss of competitive advantage, legal or regulatory non-compliance issues, considerable financial damage, and harm to the environment and community.

The biggest lessons learned could be summed up in one automation engineer’s comments: "…Well, being here for over 25 years, I thought I knew all the ins and outs. I didn’t consider documentation very important, but in the end, I realized, there was a lot that I didn’t know, and what I didn’t know ended up being a big part of the problem."

The RISK team found there were multiple corrective actions that Gator-Grasp Fasteners needed to take to shore up their detection, mitigation, and response efforts, including:

  • Perform IR planning. An IR Plan is critical to resolving security issues by providing direction and guidance to responders.
  • Conduct first responder training. Train those most likely to identify security issues about the IR Plan; educate them to collect information and triage immediately.
  • Harden OT systems. Devices with overly permissive default configurations should end up reviewed and unneeded configuration options disabled, to reduce the risk of misuse.
  • Patch and patch often. Develop a patch management program to properly secure assets and networks. Security patches fix known vulnerabilities and mitigate the spread of malware.
  • Utilize anti-virus/intrusion detection system (IDS) protection. Install a host-based anti-virus solution or intrusion detection system on all IT/OT systems and keep the definitions up-to-date.
  • Configure logging, monitoring and alerting. Centralize logging from all devices into a single location and periodically review logs for signs of suspicious activity such as anti-virus alerts, failed log-in attempts, or network communications involving external systems.
  • Maintain IR/disaster recovery plans. It is essential to have well-documented and run-tested IR and disaster recovery (DR) Plans. If not, the response and recovery process will be disorganized, potentially incomplete, and take much longer.

Gregory Hale is the editor and founder of Industrial Safety and Security Source (ISSSource.com), a news and information Website covering safety and security issues in the manufacturing automation sector. This content originally appeared on ISSSource.com. ISSSource is a CFE Media content partner. Edited by Joy Chang, CFE Media, jchang@cfemedia.com.

ONLINE extra

See related stories from ISSSourced linked below.

The Top Plant program honors outstanding manufacturing facilities in North America. View the 2015 Top Plant.
The Product of the Year program recognizes products newly released in the manufacturing industries.
Each year, a panel of Control Engineering and Plant Engineering editors and industry expert judges select the System Integrator of the Year Award winners in three categories.
Doubling down on digital manufacturing; Data driving predictive maintenance; Electric motors and generators; Rewarding operational improvement
2017 Lubrication Guide; Software tools; Microgrids and energy strategies; Use robots effectively
Prescriptive maintenance; Hannover Messe 2017 recap; Reduce welding errors
The cloud, mobility, and remote operations; SCADA and contextual mobility; Custom UPS empowering a secure pipeline
Infrastructure for natural gas expansion; Artificial lift methods; Disruptive technology and fugitive gas emissions
Mobility as the means to offshore innovation; Preventing another Deepwater Horizon; ROVs as subsea robots; SCADA and the radio spectrum
Research team developing Tesla coil designs; Implementing wireless process sensing
Commissioning electrical systems; Designing emergency and standby generator systems; Paralleling switchgear generator systems
Natural gas engines; New applications for fuel cells; Large engines become more efficient; Extending boiler life

Annual Salary Survey

Before the calendar turned, 2016 already had the makings of a pivotal year for manufacturing, and for the world.

There were the big events for the year, including the United States as Partner Country at Hannover Messe in April and the 2016 International Manufacturing Technology Show in Chicago in September. There's also the matter of the U.S. presidential elections in November, which promise to shape policy in manufacturing for years to come.

But the year started with global economic turmoil, as a slowdown in Chinese manufacturing triggered a worldwide stock hiccup that sent values plummeting. The continued plunge in world oil prices has resulted in a slowdown in exploration and, by extension, the manufacture of exploration equipment.

Read more: 2015 Salary Survey

Maintenance and reliability tips and best practices from the maintenance and reliability coaches at Allied Reliability Group.
The One Voice for Manufacturing blog reports on federal public policy issues impacting the manufacturing sector. One Voice is a joint effort by the National Tooling and Machining...
The Society for Maintenance and Reliability Professionals an organization devoted...
Join this ongoing discussion of machine guarding topics, including solutions assessments, regulatory compliance, gap analysis...
IMS Research, recently acquired by IHS Inc., is a leading independent supplier of market research and consultancy to the global electronics industry.
Maintenance is not optional in manufacturing. It’s a profit center, driving productivity and uptime while reducing overall repair costs.
The Lachance on CMMS blog is about current maintenance topics. Blogger Paul Lachance is president and chief technology officer for Smartware Group.
The maintenance journey has been a long, slow trek for most manufacturers and has gone from preventive maintenance to predictive maintenance.
Featured articles highlight technologies that enable the Industrial Internet of Things, IIoT-related products and strategies to get data more easily to the user.
This digital report will explore several aspects of how IIoT will transform manufacturing in the coming years.
Maintenance Manager; California Oils Corp.
Associate, Electrical Engineering; Wood Harbinger
Control Systems Engineer; Robert Bosch Corp.
This course focuses on climate analysis, appropriateness of cooling system selection, and combining cooling systems.
This course will help identify and reveal electrical hazards and identify the solutions to implementing and maintaining a safe work environment.
This course explains how maintaining power and communication systems through emergency power-generation systems is critical.
click me