Baseline your system or else

In this Automation System Integration blog post, Anthony Baker learns that the lack of data about how a system has been operating can create a moving target after an upgrade.

06/03/2013


Anthony’s customer required an architectural change to a control system to support some new functions, a conversion from a two PLC (programmable logic controller) system to a three PLC system. All the old interlocks were mapped and moved, and new interlocks added between PLCs. All the DA (data administrator) servers, Invensys Wonderware ArchestrA objects and Wonderware InTouch HMI software tags were reconfigured for the new setup. In short, all items in the checklist were complete.

The new architecture was installed Saturday morning, and a whole bunch of test cases were run over a three-day weekend, to make sure everything was still working, and to tweak all of the control timing for the new response time. Everything looked good, so Monday night, it’s time start-up production again. And everything looked good in production too; just some minor tweaks here and there. Not everything can be caught in testing, so this was expected.

Non-specific trend

On Tuesday evening the client appears:

Client: “We’re getting more products not sent to the right location.”

Immediately questions start running through Anthony’s head:

  • What could be causing this?
  • Are there mechanical issues?
  • Is the system not setup correctly?
  • Is there a communication or network issue? I think that has that happened before.
  • Why hasn’t this been brought up earlier? It looks like everything is running just fine.

After looking around for some possible causes, Anthony can’t really identify anything that is causing this issue.

Client: “We are still seeing more products showing up in incorrect locations.”

Anthony: “I can’t really find any issues, how much more product?”

Client: “More than before. We usually get 120/160/180 per shift, and now we’re seeing 160/180, but last shift we got 300.”

Anthony: “Well 300 seems like a lot more than usual. Something else must have gone wrong.”

Client: “Maybe it did. I don’t know really. I just know we’re seeing more than before these changes were put in.”

Moving target

By this point Anthony is tearing his hair out trying to find the problem.

Anthony’s manager: “You’re spending a lot of time on this. How are we going to know when we’re done?”

Anthony: “I don’t know. I’ve fixed a bunch of things that would have existed before the change, but they keep telling me that it’s more than before, or it’s too much in general.”

Anthony’s manager: “If now it’s too much, how much is right? 100? 50? 0? What’s the target here? Do we know for sure what they had before?”

Anthony: “I don’t know. They don’t have historical data. I heard one guy say six per shift was expected...”

This continued until all ideas were exhausted, along with Anthony. In the end, it got down to a count under 30 per shift, much lower than it was before, and the client was happy. Anthony fixed some things that had been in the code since nearly the beginning of the project, but hadn’t come up before.

Lessons learned

If you’re going to be making a change or improvement to a system make sure you have the baseline data on how it operated before, in production. Your test cases won’t always show all the problems on a complex system.

- Anthony Baker strikes again: Anthony Baker is a fictitious aggregation of experts from Callisto Integration, providing manufacturing consulting and systems integration. This blog provides integration advice in plant-floor controls, manufacturing execution systems (MES), and manufacturing consulting, from the factory floor through to the enterprise. Andrew Barker, P.Eng., Callisto Integration, compiled the advice. www.callistointegration.com

Callisto Integration is a CSIA member as of 3/1/2015



No comments
The Top Plant program honors outstanding manufacturing facilities in North America. View the 2015 Top Plant.
The Product of the Year program recognizes products newly released in the manufacturing industries.
The Engineering Leaders Under 40 program identifies and gives recognition to young engineers who...
Safer human-robot collaboration; 2017 Maintenance Survey; Digital Training; Converting your lighting system
IIoT grows up; Six ways to lower IIoT costs; Six mobile safety strategies; 2017 Salary Survey
2016 Top Plant; 2016 Best Practices on manufacturing progress, efficiency, safety
Mobility as the means to offshore innovation; Preventing another Deepwater Horizon; ROVs as subsea robots; SCADA and the radio spectrum
Future of oil and gas projects; Reservoir models; The importance of SCADA to oil and gas
Big Data and bigger solutions; Tablet technologies; SCADA developments
Automation modernization; Predictive analytics enable open connectivity; System integration success; Automation turns home brewer into brew house
Commissioning electrical systems; Designing emergency and standby generator systems; Paralleling switchgear generator systems
Natural gas for tomorrow's fleets; Colleges and universities moving to CHP; Power and steam and frozen foods

Annual Salary Survey

Before the calendar turned, 2016 already had the makings of a pivotal year for manufacturing, and for the world.

There were the big events for the year, including the United States as Partner Country at Hannover Messe in April and the 2016 International Manufacturing Technology Show in Chicago in September. There's also the matter of the U.S. presidential elections in November, which promise to shape policy in manufacturing for years to come.

But the year started with global economic turmoil, as a slowdown in Chinese manufacturing triggered a worldwide stock hiccup that sent values plummeting. The continued plunge in world oil prices has resulted in a slowdown in exploration and, by extension, the manufacture of exploration equipment.

Read more: 2015 Salary Survey

Maintenance and reliability tips and best practices from the maintenance and reliability coaches at Allied Reliability Group.
The One Voice for Manufacturing blog reports on federal public policy issues impacting the manufacturing sector. One Voice is a joint effort by the National Tooling and Machining...
The Society for Maintenance and Reliability Professionals an organization devoted...
Join this ongoing discussion of machine guarding topics, including solutions assessments, regulatory compliance, gap analysis...
IMS Research, recently acquired by IHS Inc., is a leading independent supplier of market research and consultancy to the global electronics industry.
Maintenance is not optional in manufacturing. It’s a profit center, driving productivity and uptime while reducing overall repair costs.
The Lachance on CMMS blog is about current maintenance topics. Blogger Paul Lachance is president and chief technology officer for Smartware Group.
This digital report will explore several aspects of how IIoT will transform manufacturing in the coming years.
Motion control advances and solutions can help with machine control, automated control on assembly lines, integration of robotics and automation, and machine safety.
Compressed air plays a vital role in most manufacturing plants, and availability of compressed air is crucial to a wide variety of operations.
Maintenance Manager; California Oils Corp.
Associate, Electrical Engineering; Wood Harbinger
Control Systems Engineer; Robert Bosch Corp.
click me