Cover story: Troubleshooting Industrial Ethernet

It’s a problem when gremlins invade factory automation systems: Is the network at fault? As industrial automation systems gravitate more to industrial Ethernet, tools and techniques have improved to troubleshoot network problems. See the advice below; take an online network diagnostics survey. Hear podcast network troubleshooting interviews.

10/13/2010


Control Engineering October 2010 coverWhen John Lennon famously complained in song that “…it ain’t easy, you know how hard it can be,” he wasn’t talking about troubleshooting industrial Ethernet automation networks, but he could have been. As industrial automation systems gravitate more to industrial Ethernet, it is important to look at the tools and techniques available to troubleshoot problems on these networks.

The Control Engineering Network Diagnostics Survey asks readers about their experiences setting up and maintaining automation networks. What protocol standards do they use? How well do these networks hold up? What tools do they have available, and what tools are needed? If you buy or specify industrial network diagnostic devices or services, go to www.controleng.com under the events and awards menu and select Network Diagnostics Survey to make your opinions known. (Hurry, the survey will be open only through month’s end, or until we have a statistically valid sample.)

Cover Story summary: Industrial Network Diagnostics

-Industrial network troubleshooting help for Ethernet, others

 

- Take a network diagnostic survey-Survey is closed. For all surveys we have made public, and research from others, click here.

 

-Hardware, software, diagnostic tools

 

-Ethernet webcast; photo gallery

 

-Podcasts - additional network tips, advice

 

-More than a dozen links to additional resources, information

 

 

“Over 80% of the calls that come into our facility,” says Mara White, market development manager for Fluke Networks, “involve physical infrastructure problems, and I’ve heard the same thing from automation companies.”

Checklist

10 considerations for top network health and performance

 

1. Use the appropriate test equipment to collect information from the network

 

2. Check for return loss

 

3. Ensure the cable stays dry

 

4. Consider insertion losses

 

5. Higher temperature can decrease reliability

 

6. Installation materials may appear to affect cabling performance

 

7. Look at dc resistance

 

8. Check the fiber connectors

 

9. Clean carefully

 

10. Be careful not to add loss during network design

 

Source: 10 network troubleshooting tips from Fluke Networks,” Control Engineering, Mark T. Hoske, March 19, 2010.

 

Handheld portable network analyzers simplify troubleshooting automated testers. Source: Fluke NetworksMarty Jansons, network consultant at Siemens Industry, sees Internet protocol (IP) addressing faults as one of the biggest causes of network trouble. “If the IP addresses aren’t correct, or if someone puts in an IP address that may be already assigned to another node, those are the kinds of things that give you heartache.”

Michael Frayne, manager for network interfaces products at Molex, divides network infrastructure problems into three main issue categori

  • Media (and related connections)
  • Network configuration
  • Faulty and/or nonconforming devices at nodes and in links.

“Problems related to application/network configuration are often harder to find,” Frayne says, “and may require the use of both a protocol-layer (SNMP, etc.) as well as a device-specific tool to get to the root cause.

Hardware or software? It ain’t easy

“How do you know if you are chasing down a hardware or software fault?” Frayne continues. “For example, media problems may require the use of a physical layer diagnostic tool that measures aspects like the signal quality on the media. Protocol-layer and/or device-specific diagnostics tools can be used to determine if a device has faulted due to an application condition or a serious malfunction.”

 

Table of Critical Ethernet Characteristics

1

 

Network layout

 

Do not take office topologies to the plant floor, but implement plant/machine topologies with Ethernet. Office Ethernet infrastructures are typically based on commercial-grade products meant to be located in a temperature controlled environment, and switches designed around a large star topology. Industrial Ethernet architectures, on the other hand, take into account different conditions and additional feature requirements, such as high-speed redundancy.

 

2

 

Protocols

 

The critical thing to know about Ethernet is that it’s just the wire. You need an industrial protocol to run automation applications.

 

3

 

Throughput

 

It’s not how fast the network is, but how quickly and predictably the data gets to where it’s needed.

 

4

 

Network configuration

 

It’s not just how easy it is to setup, but how little programming you have to do to make it run.

 

5

 

Planning ahead

 

It’s not only important to support today’s applications. You need the same Ethernet to support all your applications in the future.

 

6

 

Legacy systems

 

The critical thing about Industrial Ethernet is not just if it talks Ethernet, but how it integrates with already implemented networks and machines from different vendors.

 

7

 

Cost

 

The critical thing about the cost of an Industrial Ethernet network is not the cost of the components, but the cost of engineering, installation, and maintenance.

 

Source: “7 critical things to know about Ethernet,” Jeremy Bryant, Siemens Energy & Automation Inc., Control Engineering, Jan. 4, 2008

 

This is a good waveform for a CAN-based network. Source: Molex“You may have to use a combination [of hardware and software tools],” Jansons points out. He recommends starting with a network analysis tool, which involves monitoring network traffic with an auxiliary PC.

“You see a troubleshooting sequence,” White concludes. “As soon as you can isolate where there may be a problem, and isolate it to a certain area, [and down] to a certain machine. Any of the physical cabling instruments is going to [finish the job].”

This CAN network waveform shows a short. Source: MolexMaintenance and post-failure diagnostic tools include handheld testers, in-line devices, laptop PC-based components, monitoring software, and other more application-specific accessories.

Rather than requiring an operator to interpret a signal based on experience, some newer devices extract key measurements from the complex network activity. This advanced capability is often accomplished by combining a protocol analysis and an electrical signal analyzer in one handheld tool.

Reflections of trouble

Software can monitor network faults such as loss of power, broken cables, or traffic overloads. Source: MoxaAnother tool for troubleshooting physical-layer components is the time domain reflectometer (TDR). TDRs send pulses over network media and measure reflections to pinpoint any problems, such as shorts, opens, mis-wired connectors, excessive cable lengths, or incorrect cable types.

TDRs use the media’s impedance characteristics to quantitatively determine the degree (via the strength of reflections) and position (via the time the reflection reaches the TDR) of any significant faults.

After an industrial network fault a message can be sent via text message, email, and / or a local alarm. Source: MoxaA fault, such as a crushed cable, will manifest itself as a localized change in the cable’s surge impedance. When an electrical pulse reaches such a surge-impedance discontinuity, it reflects some of its energy back the way it came—toward the TDR. The sharper the discontinuity, the more strongly the signal reflects. The farther the discontinuity is from the TDR, the longer it will take for the reflection to reach the TDR. Knowing the cable’s nominal surge impedance, and the speed at which signals traverse the cable, the TDR can calculate the seriousness of the fault as well as its exact position.

It is very common requirement for new Ethernet media installations to be 100% tested and certified by the installer using a TDR. Media testers, such as cable analyzers, are available for industrial control networks and have significant value when used during installation to verify correct installation.

Network interface cards

Network devices also work inline, designed to connect and monitor, sending alerts as needed. Source Molex.Besides the usual handheld devices, network diagnostics can also be done with PCs and laptop computers combined with network interface cards (NICs) featuring built-in diagnostic capabilities and related software. “Usually,” Jansons reports, “the switches themselves already have built-in features that allow you to not only recognize these problems, but also, using open protocols, such as SNMP (simple network management protocol), can be leveraged to help you with troubleshooting not only at the device level, but at the port level.”

So, troubleshooting industrial Ethernet automation networks is a complex task not easily boiled down to a step-by-step method you can apply indiscriminately. It is more like a toolbox, whose tools you have to select ad hoc depending on the particular situation. Control Engineering articles advise on what tools to put into the toolbox, and when to pull them out for use.

Design, operations, repair

“Network diagnostics is really one of three phases involved in successful deployment of industrial networks,” Nick Jones, Woodhead Connectivity (now part of Molex), said in “Network diagnostic tools aided by intelligent design, maintenance,” December 2002 issue. “The first phase is proper design and installation, including appropriate diagnostic capabilities. The second is network operations, which includes periodic check-ups and proactive maintenance to ensure network health. The third phase is post-failure repair, which is when most traditional diagnostics are performed.”

“Industrial network selection advice from Rockwell Automation,” an Aug. 4, 2009, article, listed 10 characteristics that engineers must keep in mind when using industrial Ethernet to control factory automation systems. Steve Zuponcic, manager of IA Solutions and commercial engineering at Rockwell Automation, provided the list; See the table, 10 considerations for top network health and performance.

For more info, see:

www.flukenetworks.com

www.molex.com

www.moxa.com

www.rockwellautomation.com

www.sea.siemens.com

More network diagnostic resources

Below is a list of references, articles, and resources to date from Control Engineering:

PHOTO GALLERY - Network diagnostics

Additional images related to network diagnostics follow.

Technician uses a handheld portable network analyzer to troubleshoot automated controls on a packaging line. Source: Fluke Networks

Cable analyzers are programmed to perform multiple tests needed to troubleshoot fiber optic and twisted-pair copper Ethernet links.  Source: Fluke Networks



No comments
The Top Plant program honors outstanding manufacturing facilities in North America. View the 2013 Top Plant.
The Product of the Year program recognizes products newly released in the manufacturing industries.
The Engineering Leaders Under 40 program identifies and gives recognition to young engineers who...
The true cost of lubrication: Three keys to consider when evaluating oils; Plant Engineering Lubrication Guide; 11 ways to protect bearing assets; Is lubrication part of your KPIs?
Contract maintenance: 5 ways to keep things humming while keeping an eye on costs; Pneumatic systems; Energy monitoring; The sixth 'S' is safety
Transport your data: Supply chain information critical to operational excellence; High-voltage faults; Portable cooling; Safety automation isn't automatic
Case Study Database

Case Study Database

Get more exposure for your case study by uploading it to the Plant Engineering case study database, where end-users can identify relevant solutions and explore what the experts are doing to effectively implement a variety of technology and productivity related projects.

These case studies provide examples of how knowledgeable solution providers have used technology, processes and people to create effective and successful implementations in real-world situations. Case studies can be completed by filling out a simple online form where you can outline the project title, abstract, and full story in 1500 words or less; upload photos, videos and a logo.

Click here to visit the Case Study Database and upload your case study.

Maintaining low data center PUE; Using eco mode in UPS systems; Commissioning electrical and power systems; Exploring dc power distribution alternatives
Synchronizing industrial Ethernet networks; Selecting protocol conversion gateways; Integrating HMIs with PLCs and PACs
Why manufacturers need to see energy in a different light: Current approaches to energy management yield quick savings, but leave plant managers searching for ways of improving on those early gains.

Annual Salary Survey

Participate in the 2013 Salary Survey

In a year when manufacturing continued to lead the economic rebound, it makes sense that plant manager bonuses rebounded. Plant Engineering’s annual Salary Survey shows both wages and bonuses rose in 2012 after a retreat the year before.

Average salary across all job titles for plant floor management rose 3.5% to $95,446, and bonus compensation jumped to $15,162, a 4.2% increase from the 2010 level and double the 2011 total, which showed a sharp drop in bonus.

2012 Salary Survey Analysis

2012 Salary Survey Results

Maintenance and reliability tips and best practices from the maintenance and reliability coaches at Allied Reliability Group.
The One Voice for Manufacturing blog reports on federal public policy issues impacting the manufacturing sector. One Voice is a joint effort by the National Tooling and Machining...
The Society for Maintenance and Reliability Professionals an organization devoted...
Join this ongoing discussion of machine guarding topics, including solutions assessments, regulatory compliance, gap analysis...
IMS Research, recently acquired by IHS Inc., is a leading independent supplier of market research and consultancy to the global electronics industry.
Maintenance is not optional in manufacturing. It’s a profit center, driving productivity and uptime while reducing overall repair costs.
The Lachance on CMMS blog is about current maintenance topics. Blogger Paul Lachance is president and chief technology officer for Smartware Group.