# What happens when automation systems fail?

## Inside Machines: Do the math to ensure the statistically predicted system error rates are acceptable for the processes involved when looking at automation systems, such as machine vision. (Online extra: more about the math.)

11/16/2012

What happens when it fails? This simple question is often overlooked when automation systems are designed and implemented. Asking this question can provide another dimension to a solution, often creating extra work for a system integrator in the short term, but definitely has long-term benefits. Notice the question says “when” it fails—everything will fail eventually. What really matters is how often it fails, and what happens when it does. A few examples follow of actual vision systems where considering this question was critically important.

Vision-guided robotic bin picking

Vision-guided robotic (VGR) bin picking is a unique challenge. The intention is that product in bins is removed by robots and loaded into machines, onto conveyors, etc. As easy as this might sound, there are significant challenges in implementing it successfully, mainly based on the structure of the bin and parts inside. VGR itself is a whole other topic (vision—2D vs. 3D, bins—structured, layered, jumbled, and such), but no matter the technology, part presentation, and other factures, the success/failure rate “per pick” is a serious consideration.

This is best explained by example. Consider a bin that contains 100 parts. This is the “standard” bin, and whenever a bin is presented to a robot, it starts with 100 parts. Then consider the success rate on an individual part—this is the product of the vision success rate (How likely can parts be identified and located?) and the robot grip success rate (Once a part is located by vision, how likely is it that the part can be physically gripped?). In the example, if the vision success rate is 99.5% and grip success rate is 99.5%, then the per part success rate approximates 99% (99.5% x 99.5%).

That means for each part in the bin, the robot is 99% likely to pick it successfully. Sounds good, but consider that 99% over the 100 parts (an entire bin)—the “bin success rate.” Basic statistics tells us that the bin success rate is (0.99)100 = 0.366 or 36%. Suddenly 99% isn’t so good. This means that for a typical bin, there is only a 36% chance it will be emptied without issue or, in other words, a 64% chance there will be a failure at some point in that bin.

So what happens when it fails? This is an important question in this example, because it appears that in 64% of bins there will be an issue. Is it a big deal? This is application specific—perhaps the process is fine, and it will result in a couple “leftover” parts in the bin. The other extreme is that the process stops and requires significant intervention and downtime.

Automotive vision inspection

Another very common application of vision is inspection. Inspection systems typically perform measurement or part presence/absence or grading to provide some type of pass/fail result. On the simple end, an inspection system can consist of one camera looking at one feature. More complex systems often involve multiple cameras looking at multiple features of a part.

Consider a car coming off the end of an assembly line. A vision system mounted in a pit under the car with multiple cameras looks at 40 features, such as bolts, brackets, covers, and other attributes. Instead of looking at the vision success rate, the best way to look at this application is to consider the vision “false reject” rate. This is the likelihood that the vision system will give a “fail” result when it should have given a “pass” result.

In this example, consider customer requirements. The customer has 400 cars coming off of the production line in a shift, and when there is a false reject, an associate must drive the car to a manual inspection pit where another associate must inspect the car. This is a significant operation, so the customer only wants to manually inspect four cars per shift. Four cars out of 400 is 1%. That means on one car, the vision system must have a false reject rate of 1%. The easiest way to analyze this is to consider the total number of inspections over 400 cars; there are 40 inspections per car, which gives us 16,000 inspections. Of the 16,000, there can only be four false rejects.

Calculation: 4/16,000 = 0.00025, or a 0.025% false reject rate. In other words, 99.975% of inspections must not have a false reject. This is a stretch for a lot of simple vision inspections, and extremely difficult if not impossible for complex inspections.

Again the question—what happens when it fails? In this case the system required a look at feasibility. For the number of inspections involved in the system, the complexity of each inspection had to be very low for the project to be feasible. In addition, the integrator had to spend a very long time fine-tuning each individual inspection to achieve such high accuracy. This project likely would have been a failure from the integrator’s and customer’s point of view if “what happens when it fails” was not considered.

Individual vs. system fail rates

From these two examples it is easy to see how system failure rates can quickly get out of hand even if the individual failure rate looks good. Be realistic; expect that the system will fail, analyze how often it is likely and the impact it will have. It is very easy to overlook, but asking this question as early in the process as possible can mean the difference between a failed project and a great project!

- Kevin Ackerman, M.Sc, PEng, is a controls specialist with JMP Engineering Inc., a control system integrator in the Control Engineering System Integrator Hall of Fame. Edited by Mark T. Hoske, content manager, CFE Media, Control Engineering and Plant Engineering, mhoske@cfemedia.com.

JMP Engineering Inc is a CSIA member as of 3/5/2015

ONLINE extra: A few more words about the math....

The probability that a single part is picked successfully is (0.99)

The probability that 2 parts are both picked successfully is (0.99) X (0.99) or (0.99)2

By the same application of stats, 100 parts all picked successfully = 0.99 X 0.99 X 0.99 … 100 times or (0.99)100 = 0.366 or 36%.

The Top Plant program honors outstanding manufacturing facilities in North America. View the 2015 Top Plant.
The Product of the Year program recognizes products newly released in the manufacturing industries.
Each year, a panel of Control Engineering and Plant Engineering editors and industry expert judges select the System Integrator of the Year Award winners in three categories.
Pipe fabrication and IIoT; 2017 Product of the Year finalists
The future of electrical safety; Four keys to RPM success; Picking the right weld fume option
A new approach to the Skills Gap; Community colleges may hold the key for manufacturing; 2017 Engineering Leaders Under 40
Control room technology innovation; Practical approaches to corrosion protection; Pipeline regulator revises quality programs
The cloud, mobility, and remote operations; SCADA and contextual mobility; Custom UPS empowering a secure pipeline
Infrastructure for natural gas expansion; Artificial lift methods; Disruptive technology and fugitive gas emissions
Power system design for high-performance buildings; mitigating arc flash hazards
VFDs improving motion control applications; Powering automation and IIoT wirelessly; Connecting the dots
Natural gas engines; New applications for fuel cells; Large engines become more efficient; Extending boiler life

### Annual Salary Survey

Before the calendar turned, 2016 already had the makings of a pivotal year for manufacturing, and for the world.

There were the big events for the year, including the United States as Partner Country at Hannover Messe in April and the 2016 International Manufacturing Technology Show in Chicago in September. There's also the matter of the U.S. presidential elections in November, which promise to shape policy in manufacturing for years to come.

But the year started with global economic turmoil, as a slowdown in Chinese manufacturing triggered a worldwide stock hiccup that sent values plummeting. The continued plunge in world oil prices has resulted in a slowdown in exploration and, by extension, the manufacture of exploration equipment.

Maintenance and reliability tips and best practices from the maintenance and reliability coaches at Allied Reliability Group.
The One Voice for Manufacturing blog reports on federal public policy issues impacting the manufacturing sector. One Voice is a joint effort by the National Tooling and Machining...
The Society for Maintenance and Reliability Professionals an organization devoted...
Join this ongoing discussion of machine guarding topics, including solutions assessments, regulatory compliance, gap analysis...
IMS Research, recently acquired by IHS Inc., is a leading independent supplier of market research and consultancy to the global electronics industry.
Maintenance is not optional in manufacturing. It’s a profit center, driving productivity and uptime while reducing overall repair costs.
The Lachance on CMMS blog is about current maintenance topics. Blogger Paul Lachance is president and chief technology officer for Smartware Group.
The maintenance journey has been a long, slow trek for most manufacturers and has gone from preventive maintenance to predictive maintenance.
This digital report explains how plant engineers and subject matter experts (SME) need support for time series data and its many challenges.
This digital report will explore several aspects of how IIoT will transform manufacturing in the coming years.
Maintenance Manager; California Oils Corp.
Associate, Electrical Engineering; Wood Harbinger
Control Systems Engineer; Robert Bosch Corp.
This course focuses on climate analysis, appropriateness of cooling system selection, and combining cooling systems.
This course will help identify and reveal electrical hazards and identify the solutions to implementing and maintaining a safe work environment.
This course explains how maintaining power and communication systems through emergency power-generation systems is critical.