What happens when automation systems fail?

Inside Machines: Do the math to ensure the statistically predicted system error rates are acceptable for the processes involved when looking at automation systems, such as machine vision. (Online extra: more about the math.)

11/16/2012


A Fanuc M-710iB robot quickly picks brake rotors from a bin using vision guidance. Courtesy: JMP Engineering Inc.What happens when it fails? This simple question is often overlooked when automation systems are designed and implemented. Asking this question can provide another dimension to a solution, often creating extra work for a system integrator in the short term, but definitely has long-term benefits. Notice the question says “when” it fails—everything will fail eventually. What really matters is how often it fails, and what happens when it does. A few examples follow of actual vision systems where considering this question was critically important.

Vision-guided robotic bin picking

Vision-guided robotic (VGR) bin picking is a unique challenge. The intention is that product in bins is removed by robots and loaded into machines, onto conveyors, etc. As easy as this might sound, there are significant challenges in implementing it successfully, mainly based on the structure of the bin and parts inside. VGR itself is a whole other topic (vision—2D vs. 3D, bins—structured, layered, jumbled, and such), but no matter the technology, part presentation, and other factures, the success/failure rate “per pick” is a serious consideration.

This is best explained by example. Consider a bin that contains 100 parts. This is the “standard” bin, and whenever a bin is presented to a robot, it starts with 100 parts. Then consider the success rate on an individual part—this is the product of the vision success rate (How likely can parts be identified and located?) and the robot grip success rate (Once a part is located by vision, how likely is it that the part can be physically gripped?). In the example, if the vision success rate is 99.5% and grip success rate is 99.5%, then the per part success rate approximates 99% (99.5% x 99.5%).

That means for each part in the bin, the robot is 99% likely to pick it successfully. Sounds good, but consider that 99% over the 100 parts (an entire bin)—the “bin success rate.” Basic statistics tells us that the bin success rate is (0.99)100 = 0.366 or 36%. Suddenly 99% isn’t so good. This means that for a typical bin, there is only a 36% chance it will be emptied without issue or, in other words, a 64% chance there will be a failure at some point in that bin.

So what happens when it fails? This is an important question in this example, because it appears that in 64% of bins there will be an issue. Is it a big deal? This is application specific—perhaps the process is fine, and it will result in a couple “leftover” parts in the bin. The other extreme is that the process stops and requires significant intervention and downtime.

Automotive vision inspection

Another very common application of vision is inspection. Inspection systems typically perform measurement or part presence/absence or grading to provide some type of pass/fail result. On the simple end, an inspection system can consist of one camera looking at one feature. More complex systems often involve multiple cameras looking at multiple features of a part.

Consider a car coming off the end of an assembly line. A vision system mounted in a pit under the car with multiple cameras looks at 40 features, such as bolts, brackets, covers, and other attributes. Instead of looking at the vision success rate, the best way to look at this application is to consider the vision “false reject” rate. This is the likelihood that the vision system will give a “fail” result when it should have given a “pass” result.

In this example, consider customer requirements. The customer has 400 cars coming off of the production line in a shift, and when there is a false reject, an associate must drive the car to a manual inspection pit where another associate must inspect the car. This is a significant operation, so the customer only wants to manually inspect four cars per shift. Four cars out of 400 is 1%. That means on one car, the vision system must have a false reject rate of 1%. The easiest way to analyze this is to consider the total number of inspections over 400 cars; there are 40 inspections per car, which gives us 16,000 inspections. Of the 16,000, there can only be four false rejects.

Calculation: 4/16,000 = 0.00025, or a 0.025% false reject rate. In other words, 99.975% of inspections must not have a false reject. This is a stretch for a lot of simple vision inspections, and extremely difficult if not impossible for complex inspections.

Again the question—what happens when it fails? In this case the system required a look at feasibility. For the number of inspections involved in the system, the complexity of each inspection had to be very low for the project to be feasible. In addition, the integrator had to spend a very long time fine-tuning each individual inspection to achieve such high accuracy. This project likely would have been a failure from the integrator’s and customer’s point of view if “what happens when it fails” was not considered.

Individual vs. system fail rates

From these two examples it is easy to see how system failure rates can quickly get out of hand even if the individual failure rate looks good. Be realistic; expect that the system will fail, analyze how often it is likely and the impact it will have. It is very easy to overlook, but asking this question as early in the process as possible can mean the difference between a failed project and a great project!

- Kevin Ackerman, M.Sc, PEng, is a controls specialist with JMP Engineering Inc., a control system integrator in the Control Engineering System Integrator Hall of Fame. Edited by Mark T. Hoske, content manager, CFE Media, Control Engineering and Plant Engineering, mhoske(at)cfemedia.com.

www.jmpeng.com 

www.controleng.com/SIY

www.controleng.com/machinevision

ONLINE extra: A few more words about the math....

 

The probability that a single part is picked successfully is (0.99)

The probability that 2 parts are both picked successfully is (0.99) X (0.99) or (0.99)2

By the same application of stats, 100 parts all picked successfully = 0.99 X 0.99 X 0.99 … 100 times or (0.99)100 = 0.366 or 36%.



Anonymous , 01/18/13 11:07 PM:

What happens when an automation system fails? The customer should say, "either make it work or or take it away and give me my money back".
In the case of the bin picking vision system the odds of successed are improved by equiping the system with redundant methods of finding the target.
A redundant system might use both laser sensors and a vision system mounted on a robot arm.
Lets say that the vision system method is very fast but only has a success rate of 98%. If that method fails, laser sensors can be used which are also successful 98% but have a significantly longer search time. That raises the odds of one of the two systems working to 99.96%. The vision system and part gripper are most likely to work best if the exact distance to the target is verified by a distance measuring laser sensor. Simply getting precise target height data may improve the performance of the vision system and gripper. If the vision systems still fails to recognize the target with the camera at the correct distance from the target, the robot can be programed to use lasers to automatically detect the exact location and orentation of the target. This takes longer but is only required the low percentage of times when the vision system fails and allows the system to continue working without operator intervention. Once the position of the targer is verified by lasers, the camera is positioned in the correct location and takes another picture. If the system does not match the "taught" image to the current image, it can still assume that this is what the image looks like now and store the new image for future reference. Any time the vision system fails to match the current image to the original taught image, it can always compare it to other taught images to see if they match. It can also record statistics and make the decision to try the most reequently matching images first.

If all fails to correct the a problem with the vision system it's probably because the system requires maintenance or repair. After the problems are corrected the programs that use the lasers to find the part can be used to automatically retrain the Vision system.
The Top Plant program honors outstanding manufacturing facilities in North America. View the 2013 Top Plant.
The Product of the Year program recognizes products newly released in the manufacturing industries.
The Engineering Leaders Under 40 program identifies and gives recognition to young engineers who...
The true cost of lubrication: Three keys to consider when evaluating oils; Plant Engineering Lubrication Guide; 11 ways to protect bearing assets; Is lubrication part of your KPIs?
Contract maintenance: 5 ways to keep things humming while keeping an eye on costs; Pneumatic systems; Energy monitoring; The sixth 'S' is safety
Transport your data: Supply chain information critical to operational excellence; High-voltage faults; Portable cooling; Safety automation isn't automatic
Case Study Database

Case Study Database

Get more exposure for your case study by uploading it to the Plant Engineering case study database, where end-users can identify relevant solutions and explore what the experts are doing to effectively implement a variety of technology and productivity related projects.

These case studies provide examples of how knowledgeable solution providers have used technology, processes and people to create effective and successful implementations in real-world situations. Case studies can be completed by filling out a simple online form where you can outline the project title, abstract, and full story in 1500 words or less; upload photos, videos and a logo.

Click here to visit the Case Study Database and upload your case study.

Maintaining low data center PUE; Using eco mode in UPS systems; Commissioning electrical and power systems; Exploring dc power distribution alternatives
Synchronizing industrial Ethernet networks; Selecting protocol conversion gateways; Integrating HMIs with PLCs and PACs
Why manufacturers need to see energy in a different light: Current approaches to energy management yield quick savings, but leave plant managers searching for ways of improving on those early gains.

Annual Salary Survey

Participate in the 2013 Salary Survey

In a year when manufacturing continued to lead the economic rebound, it makes sense that plant manager bonuses rebounded. Plant Engineering’s annual Salary Survey shows both wages and bonuses rose in 2012 after a retreat the year before.

Average salary across all job titles for plant floor management rose 3.5% to $95,446, and bonus compensation jumped to $15,162, a 4.2% increase from the 2010 level and double the 2011 total, which showed a sharp drop in bonus.

2012 Salary Survey Analysis

2012 Salary Survey Results

Maintenance and reliability tips and best practices from the maintenance and reliability coaches at Allied Reliability Group.
The One Voice for Manufacturing blog reports on federal public policy issues impacting the manufacturing sector. One Voice is a joint effort by the National Tooling and Machining...
The Society for Maintenance and Reliability Professionals an organization devoted...
Join this ongoing discussion of machine guarding topics, including solutions assessments, regulatory compliance, gap analysis...
IMS Research, recently acquired by IHS Inc., is a leading independent supplier of market research and consultancy to the global electronics industry.
Maintenance is not optional in manufacturing. It’s a profit center, driving productivity and uptime while reducing overall repair costs.
The Lachance on CMMS blog is about current maintenance topics. Blogger Paul Lachance is president and chief technology officer for Smartware Group.