Tools troubleshoot industrial Ethernet problems

Technology ranges from the simple to the sophisticated

By Mark Mullins April 6, 2021

Due to its inherent reliability, performance and interoperability, Ethernet has rapidly become the communication protocol of choice for automation and control systems in the industrial environment. In fact, recent surveys found that 70% of all newly installed factory automation nodes employ Industrial Ethernet. This includes Industrial Ethernet applications like Modbus TCP/IP, EtherCat, EtherNet/IP and Profinet encapsulated with Ethernet frames to send and receive supervisory and control information between industrial devices and systems.

All of these applications are designed to work over Ethernet-based twisted-pair copper or fiber cables and connectors that have been hardened to stand up to factors such as vibration, dust and liquid ingress, chemicals and electromagnetic interference found in the industrial environment. The ability to maintain operations and productivity via Industrial Ethernet is only as good as this underlying cabling infrastructure, but even after that infrastructure has been installed, tested and apparently doing its job, problems can arise that bring industrial operations to a screeching halt.

Whether caused by accidental damage and contaminants to cables and connectors in the harsh industrial environment, interference from new machinery, or changes to infrastructure that went awry due to the wrong components or improper installation, plant managers and operations technology (OT) staff need to act fast to locate and fix the problem to keep machines in production. With more than half of Industrial Ethernet problems traced to the cabling infrastructure, knowing the most effective way to troubleshoot can mean the difference between an hour of unplanned downtime versus days that could wreak havoc on production standards and translate into millions of dollars of lost revenue.

The good news is that there are plenty of simple, inexpensive troubleshooting tools available that can help you quickly identify and locate copper and fiber cabling problems to expedite repair and reduce costly unplanned downtime.

Simple tools, simple problems

When machinery shuts down because it isn’t receiving Industrial Ethernet control signals over the cabling, the problem could be something as simple as a lack of continuity caused by a cut or break in the cable or a bad termination at the connecting interface.  Checking for continuity is easily accomplished with wiremap testing that looks for potential problems like opens caused by broken conductors, shorts from improperly terminated or damaged connectors, or miswirings that result in reversed pairs, cross pairs or split pairs, where conductors within the cabling have been terminated to the wrong pin position at a connector interface.

Rather than bringing in outside help with expensive testing equipment, you can use a simple, low-cost wiremap tool to quickly identify the problem. A good wiremapper will also indicate which pair is causing the problem, as well as verify the integrity of the shield on the cable — which is ideal for industrial environments where shielded twisted-pair cabling is the norm.

If the termination at the connector interface isn’t the problem and the fault exists somewhere along the length of the cable, downtime can be reduced by identifying the location with a tone and probe. Some wiremappers may even have tone generation built-in so all that’s needed is a low-cost probe.

A tone and probe works by injecting a high harmonic signal onto a conductor that generates an audible noise at the fault location. However, in some industrial environments, it can be difficult to access the entire length of the cable — especially if it is installed in protective conduit, located up near the ceiling, or traverses through other areas of the plant floor where production is still running. In this scenario, a somewhat more sophisticated time domain reflectometer (TDR) can be used to indicate the exact distance to a cabling fault. TDRs are available that can locate faults on any copper cabling and include an integrated toner to narrow in on the actual problem conductor pair within the cable.

When choosing one of these simple troubleshooting tools, it’s important to ensure that it supports the type of cabling and the distances of the cabling runs. To ease the process, consider a tool with a user-friendly graphical interface that simply and clearly identifies the problem. You may also want to look for a tool with ample battery life and rugged construction to stand up to the harsher industrial environment. When using a tone and probe, another consideration is the ability of the probe to filter out interference from common 50 or 60 Hz noise from nearby industrial machinery.

When the cabling is fiber

While fiber cabling is not as common on the factory floor, where lengths tend to be shorter and transmission speeds are typically less than 1000 Mb/s, some areas having longer links, higher working temperatures and immunity from electromagnet interference call for fiber optic cables. While fiber installation and initial testing requires more expertise and specialized tools, there are some inexpensive options available that you can use to troubleshoot.

There is a chance that the problem with fiber cabling is caused by a bend or break somewhere along the link. When fiber is bent beyond its recommended bend radius, which is typically 20 times the diameter of the cable, it can cause light signals to leak out of the fiber and prevent proper data transmission. These bends can happen during or after installation; small microbends that may have gone unnoticed during installation can cause the fiber glass to crack over time. Other poor installation practices like exceeding tensile load on the cable can also eventually lead to breaks.

One low-cost, simple troubleshooting tool for determining continuity in a fiber and finding a bend or break is a visual fault locator (VFL) that illuminates a fiber with a visible laser that will “leak” out where the fiber is broken or bent. However, a VFL only works well if you can visually inspect the entire fiber run. If the fiber is difficult to access or runs through conduit, you might need to invest in an optical fault finder. These more advanced tools send out a light pulse through the fiber and measures the power and the timing of the light reflected back from breaks, bends, and connections along the length of the cabling run. They are also ideal for locating the end of a fiber. By measuring the timing of the light reflected back, an optical fault finder can quickly indicate the distance to a problem.

In addition to a ruggedized design and use-friendly graphical interface, when choosing an optical fault finder, you want to make to use one that can detect live optical signals on the line before troubleshooting, as light from a fiber transceiver can be harmful to the human eye. You will also need it to work with your infrastructure’s type of fiber cabling and connectors. If you have multiple types of fiber connectors installed in your factory to accommodate various I/O interfaces, ideally you want an optical fault finder with interchangeable adapters.

Another notable feature of an optical fault finder is its ability to identify contaminated connectors. Dust or dirt on the endface of a fiber connector is the most common cause of fiber optic failures. Light transmission needs a clear path along the link, and even a single speck of dust can cause reflections and degrade a signal.

Industrial environments are inherently dusty so it’s no surprise that a dirty fiber endface could be the problem. If an optical fault finder identifies that the problem lies at a connector, another tool to have on hand is a fiber inspection camera or microscope. While using a microscope to look at a fiber connector endface can require the expertise to know what you’re looking at, there are fiber inspection tools available that have built-in intelligence to analyze the fiber endface for you.

They do this by analyzing the number and size of defects and scratches on the endface and comparing it to industry standards like IEC 61300-3-35 that specifies criteria for endface cleanliness. Of course, if you do discover a dirty fiber endface, you want to ensure that you have the onsite skillset and consumables required to clean the endface. If not, you may need to reach back out to the contractor that originally installed the fiber infrastructure in the first place.

Sometimes experts needed

When all else fails and the use of simple troubleshooting tools by your internal teams still doesn’t identify the problem with the network, it may be time to call in the experts with more sophisticated testing and troubleshooting equipment. They were likely the ones that installed and tested the infrastructure in the first place, so they should be well-equipped to find and fix the problem. But by all means, if you have the expertise on staff, use it and invest in more advanced tools that prevent you from having to wait and pay for outside help to get up and running.

For troubleshooting copper beyond what you can accomplish with a simple wiremap tool, tone and probe, and TDR, a good qualification tester will do the trick. Not only can a qualification tester do everything that a wiremap tool and TDR can do, it can also measure network speeds and identify less-obvious performance problems like crosstalk. Ultimately, a qualification tester may help you identify that the cabling infrastructure actually isn’t the problem.

When it comes to more advanced fiber troubleshooting, the next step up from an optical fault finder is an expensive optical time domain reflectometer (OTDR). While this more complex troubleshooting tool requires skill to operate, it has the capability to identify and locate events along the fiber that are causing reflections and measure their impact on the overall loss of signal, which may be what is causing the malfunction.

While calling in the experts may ultimately be inevitable, the important thing to remember is that there are simple, easy-to-use tools that you can use as your first line of defense to get your production back up and running sooner and cost you less in the long run.

Author Bio: Mark Mullins is one of the founding members of Fluke Networks, starting in 1993. He has been involved in all of the key areas of the business, including cable testing, network troubleshooting and analysis. He currently oversees the company’s global communications efforts, keeping customers and prospects up-to date on cable testing products and technologies. As a member of the Ethernet Alliance Marketing Committee, he is responsible for promoting the Gen 2 Power over Ethernet Certification program. He holds a B.S. in Computer Science and an MBA from the University of Washington.