Outsourcing control system integration makes sense

For manufacturers putting in a new line or in need of system upgrades, there are significant time and money advantages to outsourcing system integration.
By Kim Grant, Applied Sciences Group April 9, 2013

Like most business leaders these days, manufacturers are tasked with doing more with less. They have become expert in producing as much final product as possible, with as little waste as possible, with as few people as possible, all the while being energy efficient and sustainable. 

That’s a pretty tall order, keeping in mind that many elements are out of engineering and operation managers’ control. Enter the system integrator.

To maintain a Lean staff, manufacturers are outsourcing some of their engineering tasks, like project management, system design and implementation to system integrators. This is especially true for small- to-mid-sized manufacturers. For manufacturers putting in a new line or in need of system upgrades, there are significant time and money advantages to outsourcing.

But how do you know when to make a direct hire or outsource? Following are two examples of projects system integrators were called upon to work on and some of the considerations that were made before deciding to outsource. 

Building materials manufacturer requires one-day cutover

A large international building materials firm produces, processes and distributes materials worldwide. Their goal was to upgrade an antiquated and unsupported Allen-Bradley PLC-2 controller with a less-antiquated (but still legacy) PLC-5, for use on an acid treat system. The project had to be fast-tracked and the cutover had a one-day window during a planned summer shutdown. The ultimate goal was to provide graphics support, alarms and better communications back to the maintenance shed, something that the PLC-2 could not do.

The company contracted system integrator Applied Sciences Group, a member of the Control System Integrators Association, to perform the conversion. ASG’s engineers were able to effectively isolate the parts of the software that were a straightforward conversion between the PLC-2 and PLC-5 hardware platforms. ASG was successfully able to develop new control algorithms for the sections of the software that were not easily converted, and they met the cutover requirement. While internal company resources may have been competent to complete this work, it is questionable that they could have met either the fast-track schedule or the cutover constraint given their other plant responsibilities. Indeed, the original schedule had a six-week estimate; as an outside resource focused on the task, ASG completed the work in two.

First and foremost, it is expensive to find and hire the right person to bring into your organization for what could be a limited time opportunity. Hiring expenses include recruiting fees to find technically qualified people, background checks and pre-employment tests, basic salary, employment taxes, benefits, space, office equipment, training and supervision. According to a Studer Group survey of 610 CEOs by Harvard Business School, it takes 6.2 months to reach a break -even point on a new employee.

Secondly, small to mid-sized manufacturers performing upgrades to maintain their competitiveness and productivity may do so on a somewhat erratic timetable. Some choose to deploy new technologies that will make them more environmentally friendly; or in some cases upgrades are required to comply with regulatory guidance. In these scenarios, outsourcing to a system integrator makes great financial sense.

System integrators work with manufacturers to develop the scope of work for the project, establish milestones and completion due dates all the while keeping a watchful eye on costs and expenditures. The system integrator’s project manager will provide timely communication to all internal personnel relevant to the project as well as manage any external communication with vendors that could include equipment price negotiation, documentation, training, delivery or set-up.

A system integrator won’t take six months to learn an organization’s process and procedures for a project. They will be credentialed, certified and ready to work on day one. These highly trained individuals will typically have their own programming tools and necessary vendor relationships to make efficient use of a project’s budget and timeline.

Projects have a beginning, middle and end. If a manufacturer had made a full time hire to support a specific project, the end of that project could mean an employee termination or layoff.

In the case of outsourcing, project completion merely signals a contractual end to the client/system integrator relationship, and the system integrator moves on to their next project. System integrators are tasked with performing a variety of upgrades across a broad spectrum of industries. They will work on disparate equipment from multiple vendors and to further complicate this, the age at which these systems, pieces of equipment or versions were put into service can range from current technology to several decades old. 

In our scenario above, the project was to replace old PLC’s with slightly newer ones. The manufacturer had a narrow window of opportunity during a scheduled plant shutdown to avoid production downtime.  

Data tracking mandatory for medical manufacturer

There is also a continuing and still-accelerating trend in manufacturing to implement data tracking software like Historian or MES software to track productivity, materials, waste and machine downtime. Incremental process improvements are tied to the quality of the data that can be obtained from elements within the manufacturing process. In an industry that is carefully tracking capital expenditures and measuring return on investments, this type of software improves predictability, efficiency and control over a production line. 

A Fortune 100 medical manufacturer contracted ASG to do just that. Their production process includes applying a special coating to a substrate. Processes at the facility include formulation (batch processing), coating (continuous web processing), slitting and chopping (packaging). According to the scope of work, the entire process had to comply with FDA 21CFR Part 11 regulations so that any and all records could be stored electronically regardless of which product was being produced at the facility. This type of paperless traceability is highly regulated. Previous records had been stored electronically on disparate systems provided by the various equipment OEMs; however this facility upgrade was needed to comply with revised FDA regulations and current good manufacturing practices (cGMP).

The decision was made to outsource. The existing engineering and operations staff simply didn’t have the bandwidth or the expertise to do their own work and take on this year long project simultaneously, and since it was likely that limited process improvements would be made once production was restarted, hiring additional staff was not a desirable solution.

System integrator ASG was able to leverage a combination of off-the-shelf GE Intelligent Platforms software along with custom software development to meet the project’s challenges and requirements. The ultimate goal was to integrate a server-based system to support the needs of production, quality systems and engineering personnel.

The system comprised a high-availability computing environment with three servers. One is dedicated to a 3,500 tag GE Proficy Historian, the second is an unlimited tag GE Proficy iFIX SCADA, and the third a Proficy Real-Time Information Portal Server.

Working as a coordinated project team, the system integrator and the manufacturer implemented a progressive action plan to replace the disparate data collection systems (that were the result of several different OEM installations over time) by translating all old historical files to an updated usable format.

Specifically in the coating area there was an outdated Westinghouse distributed control system (DCS) that had been in place since the late 1970’s. The DCS’s data historian used a format that was not compliant with contemporary data collection historians (such as the OPC-compliant GE Proficy Historian.) To add to that, the old historian could not be modified to meet the regulatory requirements of 21 CFR Part 11. ASG reverse engineered the WDPF historian file formats, migrated the historical data and provided a connection to the Proficy Historian server. It was imperative that seven years of contiguous historical data be available to comply with 21 CFR Part 11.

Additional actions that were implemented over the year by the joint project team included:

  • Providing modifications to an existing SCADA, connection to the Proficy Historian, and a new National Instruments LabVIEW front end for high-speed data acquisition requirements on a slitter.
  • Upgrading a winder system to provide connection to the Proficy Historian.
  • A coating area machine that utilized a VAX VMS based recipe management system was upgraded to include a solution built around Microsoft SQL Server and a Visual C# GUI interface. Of particular importance was the manufacturer’s desire to have the recipe management system look as similar to the older VAX system as possible to make the re-education of the floor personnel as seamless as possible.
  • An Invensys Wonderware node with its own internal proprietary historical format was converted to the GE Proficy Historian. The system in the formulation area communicates with 10 Allen-Bradley PLCs.
  • Lastly, ASG upgraded the GE Proficy iFIX and Real-Time Information Portal Servers to take advantage of advanced new redundancy features of the latest versions of the software. The iFIX SCADA server now consists of redundant primary and secondary servers. The data were tightly synchronized between the two servers such that if the primary server had a failure, the secondary server would continue to run with no interruptions to the control or data collection process. This provided a high degree of fault tolerance and was designed to allow critical systems to continue running without loss of data. Additionally, this system now communicates to an off-the-shelf hardware/software solution from Emerson Process Management (Ovation Distributed Control System) to replace the obsolescent Westinghouse-distributed control hardware and software. 

Kim Grant is business development manager at Applied Sciences Group. For more information about ASG, visit www.asgrp.comASG is a CSIA member as of 2/26/2015. To learn about the Control System Integrators Association, visit www.controlsys.org