Nightmare recovery

How to avoid nightmare system integration projects: When an automation project is far astray, there may be a point where you must call in reinforcements to clean up rather than scrap it. Here’s how to recover from that nightmare project and what signs might point to a future nightmare.

03/07/2012


Here’s the situation: Your automation system integration client is staring down at a worst-case scenario, the prior partner fumbled the ball, and you have been called in to help. With pressure mounting from all sides to get the system up and running, it will take discipline to stabilize this situation and pull out a win for the client. Let’s look at how to best recover from this nightmare scenario. Recovery advice here can help end users and system integrators alike. 

Prioritize requirements

The start of any recovery plan is to evaluate where you are and sort out where you need to be. Begin with getting a good handle on the business priorities. Your client may have a large order in the balance, or millions of dollars in financing conditioned on packaging one case of finished product by a certain date. It is essential to separate out these unbendable needs, which may be a long cry from a fully operational system. In this way you can work with your client to realistically achieve the most important items first. Depending on circumstances, certain requirements may need to be shelved for a later date to ensure there is adequate time and resource to complete the primary objectives.

Assess the situation

Next you have to realistically assess the state of the project. Some parts of that assessment will be relatively easy to judge, such as what can be seen and touched. Is the building complete (walls, floors, ceiling, lighting, etc.)? Is all of the process and control equipment on-site? Are all required utilities operational and connected? Is everything rigged, assembled, piped, and wired?

Visualization screen before: “The information presented is lacking in detail with regard to the valve routing and flow paths. The screen also does not depict injection points of steam, water, and chemical.” Courtesy: TriCore Inc.

The harder questions to answer concern the level of readiness of the system software. Typically such software can be divided into three large categories: visualization, control, and manufacturing intelligence. While they all need to work together to properly provide the required functionality of the system, you will need to check each individually to see where it is at.

Visualization software depicts the status of the processing equipment and allows people to configure and interact with the processing equipment in a variety of ways. The equipment depictions are often graphical in nature, although not necessarily so. Things to consider in your assessment include: Is all of the equipment depicted? Does it appear to be animated properly? Are operator controls in place for each area? Are setpoints, manual control pop-ups, loop tuning parameters, and other configuration data in place and operational?

The control software is generally where all function control, device control, interlocks, sequencing, fault response, and other elements are programmed. While the particulars will vary greatly, there is a common assessment method that may be use. Develop a checklist of all common and functional operations of the control software. This should be easily obtained from the functional specification of the system. If no functional specification exists, create at least a summary version listing common operations, required functions, and a general description of how they should operate, inclusive of interlocks, and fault handling. You and your team will need to observe the operation of the control software and note on the checklist which functions are operating correctly and which are not.

Manufacturing intelligence is derived from all data that moves in and between systems from the plant floor to the boardroom. While this covers a wide gamut of information, it will generally have some common characteristics. Data may be pushed from other systems to the plant floor systems to configure things like product formulations and scheduling, equipment cleaning profiles, and other attributes. Common data pulled from the plant floor often includes what was made, how much was made, what raw material was consumed, and more. There are generally a suite of reports associated with each operational area of the system. When assessing the relative level of completion of these items, some questions to ask include: Are product formulation downloads operational? Are scheduling workflows updating appropriately? Is all relevant data being captured? Are event triggers in place? Are reports in each functional area operational? Are all queries and filters operational?

Develop your plan

Your value to this process will ultimately be determined by how well you manage to take the system from where it is to where it needs to be, as quickly and safely as possible. Solid and realistic planning is what is required at this point. Lack of planning likely contributed to the nightmare scenario in the first place.

Your planning may entail utilizing the original, existing suppliers, provided that it is in their skill set and capacity to do so. Such suppliers often provide great insight into how the nightmare scenario came to be, and have solid functional knowledge of the existing systems that you will likely find useful. Often such resources are buttressed by ones from your organization, the client’s organization, and occasionally even resources from your client’s customer’s organization. With your prioritized requirements in mind, set your schedule to ensure that the project top objectives are met. Be realistic in your calendar and resource planning to meet these requirements. At this point it is better to plan for the worst than hope for the best.

For any plan to be successful, it must have buy-in from all relevant stakeholders. The goal at this point is to get the situation under control, and quickly. To do that, everyone needs to understand the priorities, their role, their tasks, and the schedule. Schedule a standing daily meeting with all relevant stakeholders. This will be necessary to keep everything on track, keep all informed of daily progress, and, most importantly, allow you to apply all available assets to facilitate resolution of any obstacles that might present themselves.

Execution tips

With your plan in place and team assembled, commence to attack the problem with a reasonable expectation of success. There can be a lot of moving parts to manage during this process. You will need to stay disciplined to keep everything on track. Maintain your checklist and schedule of all systems operations required to execute your primary objectives.

Some system components may need to be brought online quickly to perform their basic operations. Depending on the severity of the scheduling requirements, it may be necessary to sacrifice robust testing of interlocks and fault responses in favor of basic operational functionality. In this event, once the primary objectives are achieved, it is essential to circle back on the list to these items to ensure all interlocks and fault responses are operating appropriately.

When pressed for time, it is often helpful to run multiple teams. More discipline and more coordination are required with this approach, but if done properly it can increase the project pace. There are a couple of approaches. One is to run multiple teams around the clock. This works best with three shifts, usually 10 hours a shift, to allow for adequate overlap on the head and tail of each shift. Each team will need a designated lead and clear objectives. If the process permits, multiple teams could also be run in the same shift, working on different functional areas of the facility.

Finish strong

Ensure as you approach the finish line that all of the teams’ efforts are adequately documented. You will need a punch list to keep track of open issues. Make sure all issues are tracked to resolution. The operation of all processes must be well documented down to the device level, reflecting any changes that might have occurred during this process. Make sure the plant staff is adequately trained in both the operation and the maintenance of the system. Once all the dust has settled and the situation is stabilized, have an open dialogue with all the stakeholders to expose the root causes of what got them into the nightmare scenario in the first place, to help them avoid such situations in the future.

Visualization screen after: The information presented in the after screen takes the assumptions and guesswork out of the equation. The screen clearly depicts valve routing, steam and water injection points, as well as chemical injection type and location.

While such experiences are never pleasant for those involved, you can survive a nightmare scenario and help your client avoid such pitfalls in the future.

Crystal ball to avoid nightmares

They say hindsight is 20/20, but with enough project experience, it’s possible to observe signs that a dream project might be heading toward nightmare mode. Here are a few things to watch for to keep you sleeping soundly at night:

  • Silence is not golden, but not when it comes to managing your project. “No news is good news” is not the path to a successful implementation. Regular status meetings with demonstrable progress reports will help keep your project on track.
  • No functional specification can turn sweet dreams sour. Many times the start down nightmare road begins with lack of a proper specification. Without this, there is nothing to validate against, and no good way to gain visibility into control software development status.
  • Who’s who matters. You have asked to see and vet the technical resources assigned to your project. In response you hear the sounds of crickets from your project manager. This is a pretty good indication that your project is being handled in a less-than-structured fashion, if it is being handled at all.
  • Beware of Groundhog Day. Are the last several progress reports giving you déjà vu? When things appear stalled, it is time to start looking deeper into the issues and challenges on your project, before things get out of hand.
  • If the stars need to align just right to meet the schedule, consider some solid contingency planning. Overly aggressive scheduling is another way many nightmare scenarios begin. Even with the best of teams, unexpected things can happen and progress slows. Make sure you have some cushion in your schedule, and a workable backup plan just in case.

McCarthy is president and chief executive officer of TriCore Inc. Edited by Mark T. Hoske, content manager, Control Engineering, CFE Media.

www.tricore.com 

http://controleng.com/integration has more information on system integration.



No comments
The Top Plant program honors outstanding manufacturing facilities in North America. View the 2013 Top Plant.
The Product of the Year program recognizes products newly released in the manufacturing industries.
The Leaders Under 40 program features outstanding young people who are making a difference in manufacturing. View the 2013 Leaders here.
The new control room: It's got all the bells and whistles - and alarms, too; Remote maintenance; Specifying VFDs
2014 forecast issue: To serve and to manufacture - Veterans will bring skill and discipline to the plant floor if we can find a way to get them there.
2013 Top Plant: Lincoln Electric Company, Cleveland, Ohio
Case Study Database

Case Study Database

Get more exposure for your case study by uploading it to the Plant Engineering case study database, where end-users can identify relevant solutions and explore what the experts are doing to effectively implement a variety of technology and productivity related projects.

These case studies provide examples of how knowledgeable solution providers have used technology, processes and people to create effective and successful implementations in real-world situations. Case studies can be completed by filling out a simple online form where you can outline the project title, abstract, and full story in 1500 words or less; upload photos, videos and a logo.

Click here to visit the Case Study Database and upload your case study.

Bring focus to PLC programming: 5 things to avoid in putting your system together; Managing the DCS upgrade; PLM upgrade: a step-by-step approach
Balancing the bagging triangle; PID tuning improves process efficiency; Standardizing control room HMIs
Commissioning electrical systems in mission critical facilities; Anticipating the Smart Grid; Mitigating arc flash hazards in medium-voltage switchgear; Comparing generator sizing software

Annual Salary Survey

Participate in the 2013 Salary Survey

In a year when manufacturing continued to lead the economic rebound, it makes sense that plant manager bonuses rebounded. Plant Engineering’s annual Salary Survey shows both wages and bonuses rose in 2012 after a retreat the year before.

Average salary across all job titles for plant floor management rose 3.5% to $95,446, and bonus compensation jumped to $15,162, a 4.2% increase from the 2010 level and double the 2011 total, which showed a sharp drop in bonus.

2012 Salary Survey Analysis

2012 Salary Survey Results

Maintenance and reliability tips and best practices from the maintenance and reliability coaches at Allied Reliability Group.
The One Voice for Manufacturing blog reports on federal public policy issues impacting the manufacturing sector. One Voice is a joint effort by the National Tooling and Machining...
The Society for Maintenance and Reliability Professionals an organization devoted...
Join this ongoing discussion of machine guarding topics, including solutions assessments, regulatory compliance, gap analysis...
IMS Research, recently acquired by IHS Inc., is a leading independent supplier of market research and consultancy to the global electronics industry.
Maintenance is not optional in manufacturing. It’s a profit center, driving productivity and uptime while reducing overall repair costs.
The Lachance on CMMS blog is about current maintenance topics. Blogger Paul Lachance is president and chief technology officer for Smartware Group.