The active cyber defense cycle: Part 3 of 5: incident response

In the process of performing asset identification and network security monitoring, analysts eventually find threats in industrial control system networks.

By Robert M. Lee August 24, 2015

In the process of performing asset identification and network security monitoring, analysts eventually find threats in industrial control system networks. At this point, the team must be ready to execute incident response procedures. While there are many models, such as the National Institute for Standards and Technology incident response methodology, it is vital to prevent incident responders from operating in a silo. Security analysts should continue monitoring networks for threats to help incident responders determine how widespread threats are and if they are spreading on the network. Incident responders should move quickly and methodically to gather forensic evidence and identify instances of malware. This can be incredibly difficult across distributed oil and gas networks. Sensitive systems and large distributed networks make incident response a challenge. For this reason, the majority of the work required in incident response should be done prior to the incident.

Every organization should have an incident response plan. The plan should incorporate points of contact within operations technology (OT) and information technology (IT). The combination of IT and OT personnel with a common understanding of the environment and its needs-as well as a crossover of the skillsets required to work in each field-ensures success during incident response. The plan should contain points of contact, checklists for the procedures that responders must follow, and should fit into the overall incident response plan of the organization to include events that are not cyber related. The plan should be rehearsed and available in multiple locations including inside a jump kit. A jump kit is a bag that contains the tools, checklists, and procedures that responders will need. For example, the jump kit should include extra hard drives and CDs for data storage, converters and connectors for various interfaces, emergency contact phone numbers, and anything that the responders will find valuable during an incident. In operations environments it’s a good idea to ensure that a checklist reminds responders to have access to personal protective equipment, such as hard hats and steel-toed boots. There are plenty of guides for creating an incident response plan.

Sampling the pre-incident environment

Security personnel should gather forensic evidence prior to an incident. Even if an organization does not have the personnel, time, or skills to analyze collected evidence for indications of malicious activity, the organization should have pre-incident samples. During an incident, responders can use the pre-incident samples to identify abnormalities between that evidence and post-incident evidence, allowing them to quickly rule out benign processes and files. Data collected as a baseline must never be assumed to be completely clean, but it is a starting point for incident responders that will save time. Time and money are directly related during incidents; saving time equates to saving money and resources. A tool that has been used extensively on the Microsoft Windows OT systems in oil and gas networks is the cyber security company Mandiant’s Redline. There are other tools in the community that work similarly but Redline will be used for the purpose of having an example to point to in this article.

Redline is a free tool that can be installed on a nonproduction Windows system. After installation, personnel will be prompted with two options: Analyze data or collect data. The analyze data option allows security personnel to review evidence collected in a graphical user interface to identify potentially malicious processes and files on the system. However, this requires the allocation of resources such as analyst time and training. For such training, consider taking research and education organization SANS ICS 515: Active Defense and Incident Response class. The easy win is the collect data option. Using this option, personnel can create a collector that can be saved to a universal serial bus (USB). This USB can then be taken to a Windows-based system such as a human machine interface and used to collect data.

Using the collector is as simple as putting the USB into the system and executing the RunAudit script. It will then collect the appropriate data including system memory and system timeline information. Personnel can take samples from the environment using this method and then securely store them. Even without analyzing the data, it will be invaluable to incident responders during an incident.

This baseline information helps responders quickly rule out files that they should not analyze. Furthermore, it helps to identify abnormalities, and determine a window of time that the adversary has been present in the environment. It is important to note, however, that the collector should always be used on test systems first before using in the production environment. Additionally, use in the production environment should take place at safe times such as scheduled downtimes. USBs also must be maintained and cleaned to ensure that any existing malware is not spread from system to system.

Consider this sample process:

  • Install Redline on a nonproduction Windows system
  • Create a collector on a forensically clean USB.
  • Test the collector on a test environment to ensure it does not impact systems.
  • Insert the USB into a Windows OT system during a safe period such as scheduled downtime.
  • Execute the RunAudit script to collect data such as system memory and disk information.
  • Store the USB in a secure location with documentation that identifies for which system it was used.

It is preferable for analysts to review this data. In the active cyber defense cycle (ACDC), incident responders should sample the environment and look for threats-even before the network security monitor analysts alert them to an incident. This process of sampling and analyzing can uncover threats not easily observable in the network. However, given real world constraints, personnel should at least sample the environment and store the data for incident responders to use in the future. It is important to fully test any security measure, such as sampling the environment, before implementing it on production systems.

Preparing for the worst

After sampling the environment, it is useful to create a collector for remote sites. For example, in oil and gas networks, there are often remote sites that are difficult to access during incident response. By creating a collector and pre-positioning it at those remote locations, even untrained personnel can insert the USB into a potentially-infected system during an incident and run the script when directed. The USBs can then be taken or shipped to a central point where incident responders are conducting analyses. In this case, the serial number of the USB should be tracked and the USBs should be securely stored and clearly marked that they are for incident response scenarios only. Operators and engineers should not use the USBs outside of that situation, otherwise, they may accidentally spread malware in the network via the USB by accessing an infected system. The incident response plan should also call for the approval to use USBs that are identified, tracked, and cleaned during an incident. Management approval may be required ahead of time.

Redline is not the only tool that is free to use. Another option is FTK Imager. It is ok to use other tools, but the most important thing is being able to have tools ready for quick and easy use in the event of an incident. For the purpose of the ACDC, incident responders should take care to identify samples of the threat, such as malware, that can be passed to the threat and environment manipulation team. They will be able to analyze the threat and make recommendations for needed security changes.

Part 4 coming in December: Threat and environment manipulation

– Robert M. Lee is the cofounder of the critical infrastructure cyber security company Dragos Security LLC, which developed a passive asset discovery and visualization software tool. Lee is a PhD candidate at Kings College, London researching control system cyber security. He is the course author of SANS ICS 515: Active Defense and Incident Response, the author of the book SCADA and Me, and a U.S. Air Force cyber warfare operations officer. Edited by Eric R. Eissler, editor-in-chief, Oil & Gas Engineering,

Original content can be found at Oil and Gas Engineering.