Vision Sensor Helps Robot Rock On

Vision sensor, edge tool software, and a PLC make robot "Roxanne" a Guitar Hero winner-and gives her student-inventor crucial vision system experience.


Visit to view a video of Roxanne's performance.

Machine vision sensors are often likened to human eyes: They scan and “see” the world around them, then transmit the data elsewhere for some action to occur. But can a vision sensor watch TV? Turns out it can. And when put together with mechanical actuators and a PLC brain, it can also play video games.

Casting about for a project for his robotics class at Minnesota West Community and Technical College, engineering student Pete Nikrin hit on the idea to design a robot to compete against a friend newly introduced to the Guitar Hero electronic game. Playing Guitar Hero requires fast button-pushing on a guitar-shaped input device in response to dots (notes) that move down a path on a video screen.

For the project, Nikrin decided to use a mannequin—complete with Minnesota West sweatshirt and painted fingernails—with a vision sensor in its left eye and mechanical actuators attached to its painted nails. And, he wanted to keep the video game and the robot two separate entities.


When Roxanne plays Guitar Hero, she can achieve up to 95% accuracy on Hard mode, besting creator, Pete Nikrin.
When Roxanne plays Guitar Hero, she can achieve up to 95% accuracy on Hard mode, besting creator, Pete Nikrin.

At first, he says, “I couldn't find anything online like it. Then, a month or two into it, a bunch of videos showed up [using other methods]. To the best of my knowledge, though, mine is the only one that has the robot linked only by vision to the game.” Other projects tap directly into the game's electronic signal, which Pete, as a robotics guy, thinks “is kind of cheating.”

Not unlike the process of problem-solving in an industrial application, Nikrin hit upon two frustrations early on: finding the right sensor and building a vision system with high enough speed for the application.

“We had to find a way for the vision system to pick up on the source of light coming from TV. [The first] type of sensor I tried would only see the back of the tube—which was pretty cool in itself, but didn't fit the application,” says Nikrin.

Bill Manor, robotics instructor at Minnesota West, suggested Nikrin incorporate a PresencePLUS P4 Omni vision sensor from Banner Engineering. “Students have used Banner vision sensors in many projects over the years—to inspect containers, for example, as they come down a conveyor,” Manor says. Minnesota West had purchased a vision system at a discount through Banner as a start-up education kit, he adds.

Nikrin and Manor used the 20-plus year experience of Jeff Curtis, senior applications engineer at Banner, to help troubleshoot the details of the application.

“Bill and I both thought that they [Banner] went above and beyond to help with a school project, which might seem trivial to some companies,” Nikrin added.

“Pete worked hard to do this,” says Curtis. “He did a lot of work, and all the programming. We're happy to help with vision applications, but we won't write logic for other people. It was personally satisfying to work on, though. Lots of us are geeks around here and have played Guitar Hero.”

Pete Nikrin designed the robot while he was a student at Minnesota West and now does vision projects for his employer, Meier Tool & Engineering.
Pete Nikrin designed the robot while he was a student at Minnesota West and now does vision projects for his employer, Meier Tool & Engineering.

Sensor selection

Nikrin installed the vision sensor as the robot's left eye and positioned it toward the video screen. Nikrin's team needed to ensure Roxanne could play within a range of lighting conditions—since she would be relocated from classrooms to gymnasiums for demonstrations—as well as confirm the robot was correctly oriented with the monitor displaying the video game.

On screen, Roxanne “sees” the little dots (notes) move down a path. The robot identified the notes to be played by using edge software, which is a simple linear tool used for locating edges. Edge tools help locate things in free space by detecting, counting and locating the transition between bright and dark pixels in an image area.

“When the robot was first set up, it would slouch over time,” says Curtis. “Using two edge tools, you can tell the imager where you should be vertically and horizontally. Then if the head moves slightly, you can find the correct scan area again.”

Curtis says other types of “locate tools are patterning tools. “Edge tools react in only a few milliseconds or fractions of milliseconds—much quicker than cumbersome area based tools,” he says. They find the absolute or relative position of the target in an image by finding its first edge.

To correctly orient the sensor with the monitor, “we honed a locate tool and gave it a fixed point—a piece of reflective tape on the PC monitor—to focus on,” Curtis explains. “This ensures the edge tools are in the proper location to detect each note as it comes along, and allows for any slight vibration in the application environment that could result in some deviation.”

Nikrin says the team set-up five edge tools that ran horizontally across the screen, one for every fret, and positioned the tools to focus on the notes at the bottom of each. “The Edge tools sent a constant signal as the five vertical fret lines progressed, and when a bright white dot appeared in the middle of a dark colored circle, the edge tool allowed the sensor to detect it.”

Curtis says vision sensors and edge tools are frequently used in industrial applications because they're very simple and very fast, good for verifying the simple presence or absence of parts. “They're used big time in packaging verification. You don't need optical character recognition system for that. We can do that with simple area or linear tool,” he says. “They're also good at checking a PC motherboard. With a basic green circuit board, you can check solder nodes and verify that all are there. With lasers and photo electric sensors, it's difficult.”

Banner's PresencePlus P4 Omni sensor is hidden behind Roxanne's left eye.

Processing speed

When it came to speed, Nikrin decided the camera's processor was too slow, so he chose a PLC to do the actuating. Also, the camera only had four I/O points and he needed five (one for each fret line). Once a note was identified, communicating the signal efficiently depended upon a heavy amount of programming, as well as Ethernet technology applied through a Modbus register, he says.

“Rather than use the processor in the camera, I hooked up a [Rockwell Automation] MicroLogix 1100 , which has Ethernet,” explains Nikrin. “Using the discrete I/O on the camera, the scans took 35 ms. By bitmapping the PLC and having the PLC do the processing, scans took 9 ms total—8 ms in the camera and 1 ms in the PLC.”

The MicroLogix PLC was programmed so that it constantly looked at the vision sensor's register. “Once the edge tool senses a note, the PLC notices the change in the register, and the logic in the PLC fires a solenoid that activates the robot's finger. Just as a human player would react, the robot's finger then presses down on the appropriate note on the guitar,” Nikrin explains.

Nikrin was surprised by “the simplicity of the project,” he says. “I was going for simplicity and elegance of execution.” His only disappointment? Lack of time meant he didn't get to work on the mechanics of the robot much. “The mechanics are lacking; it slows down the systems,” he says.

Regardless, after playing for two weeks, Roxanne had surpassed Nikrin in his ability with the game. On Medium mode, Roxanne hit 100% accuracy at times, and she averaged 98% accuracy during the remainder of Nikrin's tenure at Minnesota West. She could achieve up to 95% accuracy on Hard mode, and 80% accuracy on Expert mode (due to increased mechanical requirements on the robot's fingers).

Today, Roxanne still engages current and prospective Minnesota West engineering students, and Nikrin looks back on it with both a sense of accomplishment and a hefty dose of gratitude. He graduated from Minnesota West in 2008, and is now working as a manufacturing engineer at Meier Tool & Engineering. Not surprisingly, his first project there involved a vision system: “We're straightening, cutting, and forming a wire to a set length, and we're inspecting the finished product with a vision system,” he says. Soon, he'll be working on robotic guidance systems and robotic guides, he says—something Roxanne has prepared him for well.


Author Information
Renee Robbins is senior editor of Control Engineering. Reach her at .

The Top Plant program honors outstanding manufacturing facilities in North America. View the 2015 Top Plant.
The Product of the Year program recognizes products newly released in the manufacturing industries.
Each year, a panel of Control Engineering and Plant Engineering editors and industry expert judges select the System Integrator of the Year Award winners in three categories.
Pipe fabrication and IIoT; 2017 Product of the Year finalists
The future of electrical safety; Four keys to RPM success; Picking the right weld fume option
A new approach to the Skills Gap; Community colleges may hold the key for manufacturing; 2017 Engineering Leaders Under 40
Control room technology innovation; Practical approaches to corrosion protection; Pipeline regulator revises quality programs
The cloud, mobility, and remote operations; SCADA and contextual mobility; Custom UPS empowering a secure pipeline
Infrastructure for natural gas expansion; Artificial lift methods; Disruptive technology and fugitive gas emissions
Power system design for high-performance buildings; mitigating arc flash hazards
VFDs improving motion control applications; Powering automation and IIoT wirelessly; Connecting the dots
Natural gas engines; New applications for fuel cells; Large engines become more efficient; Extending boiler life

Annual Salary Survey

Before the calendar turned, 2016 already had the makings of a pivotal year for manufacturing, and for the world.

There were the big events for the year, including the United States as Partner Country at Hannover Messe in April and the 2016 International Manufacturing Technology Show in Chicago in September. There's also the matter of the U.S. presidential elections in November, which promise to shape policy in manufacturing for years to come.

But the year started with global economic turmoil, as a slowdown in Chinese manufacturing triggered a worldwide stock hiccup that sent values plummeting. The continued plunge in world oil prices has resulted in a slowdown in exploration and, by extension, the manufacture of exploration equipment.

Read more: 2015 Salary Survey

Maintenance and reliability tips and best practices from the maintenance and reliability coaches at Allied Reliability Group.
The One Voice for Manufacturing blog reports on federal public policy issues impacting the manufacturing sector. One Voice is a joint effort by the National Tooling and Machining...
The Society for Maintenance and Reliability Professionals an organization devoted...
Join this ongoing discussion of machine guarding topics, including solutions assessments, regulatory compliance, gap analysis...
IMS Research, recently acquired by IHS Inc., is a leading independent supplier of market research and consultancy to the global electronics industry.
Maintenance is not optional in manufacturing. It’s a profit center, driving productivity and uptime while reducing overall repair costs.
The Lachance on CMMS blog is about current maintenance topics. Blogger Paul Lachance is president and chief technology officer for Smartware Group.
The maintenance journey has been a long, slow trek for most manufacturers and has gone from preventive maintenance to predictive maintenance.
This digital report explains how plant engineers and subject matter experts (SME) need support for time series data and its many challenges.
This digital report will explore several aspects of how IIoT will transform manufacturing in the coming years.
Maintenance Manager; California Oils Corp.
Associate, Electrical Engineering; Wood Harbinger
Control Systems Engineer; Robert Bosch Corp.
This course focuses on climate analysis, appropriateness of cooling system selection, and combining cooling systems.
This course will help identify and reveal electrical hazards and identify the solutions to implementing and maintaining a safe work environment.
This course explains how maintaining power and communication systems through emergency power-generation systems is critical.
click me