Bridging the gap between IT, operations

Understanding the shared goals can bring peace – and value to manufacturers.

By Mark Wylie, Cisco May 24, 2012

We have already seen a paradigm shift in manufacturing operations in terms of opening up networks. For years manufacturers have purchased or constructed purpose-built production lines based on their specific requirements. They controlled what machines went into the plants and the level of networking needed. Often a specification would read “provide an Ethernet port for communications” without any indication of how those communications might be used. They just knew it was going to be needed one day.

At the same time, the need for information visibility throughout the organization increased, bringing corporate IT into the manufacturing operations (some for the first time) as companies found they needed to build new competencies in efficiency through their supply chain.

The convergence of these two inseparable trends (the merging of different mindsets) has created new challenges for businesses, leading to new misunderstandings as well as new efficiencies.

There has already been a physical shift in network architecture, but a new paradigm is entering. The new shift is now coupled with hyper-growth in terms of applications available for personal to use while on the manufacturing production line.

Technology convergence and business transformation

We have already seen the impact of technology convergence. Manufacturing companies have been moving away from legacy proprietary networks in control systems and instead are embracing open standard Ethernet, like Ethernet/IP. Meanwhile, corporate management has long desired a single network that can allow them to securely transfer data to where it needs to be. With timely information being a key asset as a competitive differentiator, companies are now realizing the importance of investing in networks that can better facilitate the business enterprise. The new paradigm therefore calls for increased understanding and collaboration between IT and operations.

Manufacturing companies are seeing dramatic changes to their business model, specifically in having to deal with an aging workforce as well as with more talent recruits not being based near operation facilities. Their supply chain has widened and deepened, requiring more attention from the corporate world to collaborate with outsourced suppliers and vendors.  

Product lifecycles are shortening in order to meet new global economy market pressures. All of this together requires manufacturers to react quicker, reduce costs and provide more services to successfully remedy manufacturing issues.

This has led to the notion of the ‘Internet of Things,’ in which devices collaborate with each other, as well as with their human counterparts, to drive more integration to the enterprise from factory operations.

IT readiness and operations

It is not a case of one versus the other. We can all play in this sandbox.

It is now more critical than ever to build “IT-readiness” into manufacturing operations as to create a more integrated, mixed environment in which both IT and operations can control it.

In this environment, manufacturing focuses on how to build the end product, while IT focuses on managing the networking, communications and security of the enterprise information such as maintaining call centers and sales portals. By playing nicely in the sandbox, manufacturing operations can occur seamlessly around the globe in which subject matter specialists can service and troubleshoot machines anytime, anywhere.

However in order to do so, it is critical that the operations technician and engineer understand how the machine they are working on integrates with both the network and the production line, and how the two functions communicate with each other. This means IT and operations must effectively collaborate with each other.

Integrating IT into a production environment requires a plant that:

  • Incorporates standard networking for internal to plant communications as well as to business wide communications.
  • Provides data that is relevant, timely and accurate so smart business decisions can be made.
  • Does not compromise network security or breach existing policies, including protection of the Intellectual Property regarding the product and the process.
  • Allows control traffic to co-exist with network traffic, ensuring there is no interference with machine operation. In the manufacturing process, control traffic needs priority over other types of traffic that is more business in nature. 

It is important to remember the fundamental purpose for the IT organization is to provide the availability and the protection of critical information. The manufacturing operations group on the other hand, needs to build a product to sell to customers for money. Sometimes, the two groups are at odds with each other over their respective priorities. It is possible, however, to reach a mutual understanding that can meet both groups’ priorities and goals.

An unfortunate shutdown

I am reminded of a time when I was helping a local distributor host an open house for their customers. They had recently remodeled their facility, including installing a control system that managed access to various parts of the building and lighting. At one point during the evening, the curious Mark in me wondered what a particular button on the Human Machine Interface (HMI) panel did and pressed it, automatically shutting off all the high-intensity lights in the warehouse.

All of the guests assumed the party was over and started to leave. We intercepted the departing guests, got the lights back on and had a good laugh. A couple weeks later, I was at the same distributor and noticed there was a new button on the HMI panel. They told me it was the “Anti-Mark” button.

A relatively simple example, but it illustrates what can happen if the wrong asset (me in this case) gains access to your production facility. Fortunately, in this case, I accidently turned off some lights and didn’t destroy a machine or a production process – but imagine if this had happened in the manufacturing operation area. What if personnel needed controlled access but couldn’t get in due to a password restriction because they didn’t have it?

The key difference from IT to operations

A co-existence must occur between IT and the manufacturing operations. Sometimes it is just a matter of understanding the respective groups’ priorities and making small compromises to accommodate each other. But we must not forget there are fundamental differences between the two groups. One of the most fundamental areas of difference between an IT and operations is in terms of updates and confidentiality requirements. The table here illustrates many of these at a high level.

IT and operations have different focuses, but a quality finished product is a common goal. Source: Cisco

IT Network

Operations Network

Focus

Protect intellectual property and company assets

24/7 operations, high availability

Priorities (in rank order)

1. Confidentiality

2. Integrity

3. Availability

1. Availability

2. Integrity

3. Confidentiality

Access Control

Strict network authentication and access policies

Strict physical access, simple network device access

Implications of a Device Failure

Continues to operate

Could stop operation

Upgrades

ASAP, during uptime

Scheduled, during downtime

We all understand the need for password protection and proper authentication, but in the manufacturing world the intent is different. In terms of information we are protecting in the plant it can be personal data (e.g. login information), but it can also include financial data or secret information related to the company intellectual properties. Consider this scenario, what if unauthorized personnel were allowed access control, while authorize personnel were kept off? Locking someone off of a control system that has been started and needs to be shut down could result in utter disaster. But what is the alternative – writing the machine password on a post-it note and tacking it to the machine?  As you can see, it is important to recognize what is being protected, why it is being protected and from whom it is being protected from.

At some point, you may have received a notice stating new updates are available for your computer which will require a restart of your system. If writing emails or chatting with co-workers on Instant Messenger, this can be a minor issue; seldom taking away from productivity. However, for someone in operations a forced update could be disastrous.

Imagine starting a process where a variety of components both solid and liquid are being mixed when IT sends a message down to the computer saying, “Please standby while we update your computer” – essentially taking away control from the person with direct line of responsibility for the process.

This could potentially result in the destruction of expensive machinery, the spillage of materials or even worse, bodily harm to someone on the plant floor due to the machine malfunctions.  This is one of the more critical differences between IT and operations, and one of the most important considerations in establishing an operating communication stream between both groups.

There is another key difference however which involves deciding the priority of information flow. In business systems it is extremely critical to maintain the Confidentiality of the Information, while maintaining the Integrity and Availability of the data. We generally refer to this prioritization as “CIA.” In manufacturing operations, it is often more important to prioritize the Availability of the data more so than the Confidentiality of it. Referring back to my earlier example of starting a mixing process and then losing visibility into to the process, operations cannot afford to lose view or control. Therefore, it might be necessary to sacrifice confidentiality to preserve availability. Hence, in the operations world the model shifts to an “AIC” prioritization.
Keep in mind that it is possible to have some situations in manufacturing operations where Confidentiality resumes the priority – imagine a design team collaborating via multiple locations around the world to develop a sophisticated new product. Here, Confidentiality resumes top priority and it may be the most important attribute.

Knowledge and understanding of these differences are key to joint amicable ownership by both the IT operations.

Adoption of solid designs

When designing a production facility that integrates in an “IT-ready” scheme, manufacturers need to be aware of some fundamental considerations:

  • Aligning the operation’s Industrial Ethernet configurations with existing IT policies, considering security, Quality of Service (QoS), access control and balancing those with the needs of the operations.
  • Using managed switches that provide network and security services that help maintain and diagnose the network at large.
  • Consistently using network services and follow IP addressing, sub-netting and default gateway settings conventions.
  • Utilizing existing management tools to help manage multi-cast traffic as well as priority uni-cast traffic. These tools include virtual LANs (VLANs), QoS and resiliency.
  • Aligning the manufacturing operations with emerging Industrial Control standards, including issues like Port security, access control lists and network access control.

Handling IP configurations, the unique address of a machine or device, partially tackles these considerations. It is also increasingly important to provide remote access while adhering to an existing security policy. This calls for creating a managed environment, where the data types are known and controlled, and critical, real-time information is prioritized over video monitoring or email traffic. Setting up a secure and accessible environment enables the right people to have access the right data at the right time, while blocking access to the wrong people.

This environment should also enable users to quickly isolate and fix problems related to the network as well as individual pieces of equipment. Finally, users should have the ability to scale to new production or reporting requirements as well as incorporate new technology when it makes sense. This provides enhancements over the plant’s life cycle while giving users new efficiencies.

As Ethernet expands to lower-level devices in the factory, companies often want to leverage existing security specialists who typically work in the company’s IT organization to help protect their factory floor networks.

For plant operations, IT can serve as a good resource in helping them to learn how to be productive and responsive while not physically being there. Imagine the power of securely accessing the operating parameters of the facility during the middle of the night without having to be there.

Yet, you can relax knowing that no unauthorized access can happen. This is the wave of the future – knowing what is going on in your operation using “IT capable” tools. You can be responsive, efficient productive AND have a life outside the factory.

The fusion of IT and operations allows production personnel and management to access, maintain, view and gather valuable data anytime, anywhere. This improved visibility enables manufacturing companies to quickly and more proactively respond to potential problems along the production line, helping to reduce downtime.

Furthermore, operations and IT save money by reducing the travel expenses needed for on-site services. In the final analysis, the benefits of working together provide significant efficiency benefits while preserving the autonomy of each group – a win for all!

Wylie is the manager of market management for Cisco.