Commissioning control systems for MCFs

Controls success starts in design with clear requirements, and ends with thorough testing and complete turnover. Here are eight steps to follow.

12/31/2012


The phrase “integrated MEP control systems” generates images of holistic design where all components of the mechanical, electrical, plumbing (MEP), and fire protection systems work in harmony. Unfortunately, industry professionals see far too many mission critical facilities (MCFs) in which MEP systems are joined by a menagerie of control systems and protocols with varying degrees of success. Successful integration of MEP control systems depends as much on effective independent commissioning as it does on the right design and proper implementation.

Experienced commissioning agents (CxAs) do not claim to be design experts; however, they have seen many integrated MEP control systems that did not perform as designed during testing and commissioning. When commissioning begins at the planning phase of a project, the questions and insights gained from experience in commissioning MCFs contribute the greatest value to meeting project objectives and saving the owner and project team rework, change orders, and associated time and money. Here are eight key issues and insights at each phase of an MEP control integration project.

1. Develop a clear, written statement of operational and performance goals.

From commissioning experience, there are two critical success factors in the planning phase of an MEP control integration project: a clear, concise written definition of project goals in operational and performance terms; and the right decisions about multi- versus single-system automation and network architecture.

It is surprising how often project teams assume that the goals of the project are clear to all participants without clearly defining them in a written document. The CxA can help the team define the project goals in as much detail as possible, and put them in a concise, written owner’s project requirements (OPR) document.

The document should identify and define the owner’s operational and performance objectives in detail. For example, operational details might include the owner’s requirement for real-time communication of system operating data to staff of a remote, 24/7 network operations center. The owner’s performance goals might, for example, include key parameters such as refresh time for data reporting. Vetting the requirements through the client’s IT department for security requirements at this stage will save headaches in the future.

The document should be periodically reviewed over the course of the project, and key decisions that inevitably impact design and implementation of the project should also be documented.

A CxA can also help to get everyone literally on the same page by summarizing the typical 100-plus-page program document into a concise 10-page document that outlines the specific project requirements, including setpoints, required redundancy, and so on. The agent also uses this summary document to review design to ensure it remains true to the project goals.

2. Multisystem automation provides the most flexibility.

At the core, there are two ways to integrate multiple systems: select a single controls and automation vendor and make it responsible for all aspects of the integration, or use a third-party integration system to connect the various systems, process the data, and present it in a uniform manner. From experience, a CxA may urge use of a third-party integrator. Although the single-source solution has the advantage of ease of initial implementation and a single point of responsibility to resolve problems, the main drawback is that once a single vendor is in place, purchasing power is severely compromised. Switching controls vendors in an operating facility is costly and disruptive.

The third-party integrator introduces another vendor to the project, but provides the most flexibility for the owner’s future system modifications and upgrades. It also has the benefit of fully leveraging the controls and automation that are supplied by the manufacturers of most building equipment.

With the continuing development in automation protocols, the differences between the two approaches has diminished, but not disappeared. A CxA may also advise bringing to the planning table a third-party integration consultant. To ensure objectivity, specify that the consultant is ineligible to bid the implementation contract.

3. Network architecture: Two networks are better than one.

Historically, MEP control and automation networks were set up on a dedicated network infrastructure physically separated from the production IT network. As protocol translation technology has improved, the industry has increasingly relied on a shared Ethernet infrastructure for IT and MEP control systems. This saves money; however, the design of the active network components in this shared network must accommodate the protocols and data types being carried by all connected systems. Often, the standard Ethernet network hardware is inadequate for MEP control data, and this becomes obvious at testing and commissioning.

Over the past year, one CxA saw weeks of lost time in a number of projects due to the need to resolve issues related to single-system automation and single-network architecture. Time and money would have been saved had the potential drawbacks been anticipated by all members of the team in the planning phase.

4. Design/specification: Leave no room for multiple interpretations.

The proper design and specification is the most critical component in delivering a fully integrated MEP control system in a cost-effective manner. Having a “hands-on” experienced system integration consultant on the design team is usually money well spent. The participation of this profession will ensure that design and specifications are practical, economical, and, most importantly, achievable.

Equally important is development of a specification that leaves no room for interpretation. The CxA can be invaluable in helping to ensure that the specs are clear, concise, and detailed to avoid the misinterpretations that can lead to change orders at the testing and commissioning phase—or worse, disagreements that lead to litigation.

As the system design takes shape, the underlying concepts should be simple and easy to understand. If necessary, the concepts should be broken into smaller components until they can be easily explained to any programmer, system integrator, installer, or operator. Simplification of the concepts to their core elements ensures that the development of control sequences will be clear and easy to understand.

Similarly, the mark of high-quality MEP control system construction documents is a singular meaning. Leave no room for multiple interpretations. Ideally, invite both experienced industry professionals and stakeholders, such as the MEP operators, to review the documents for completeness and singular meaning.

5. Avoid software issues related to control code, graphics, and protocol mapping.

MEP control system vendors’ general approach to the development of control code is to have a single or small group of programmers prepare the code based upon the project documents and the programmers’ own approach. From experience as a CxA, this approach creates multiple opportunities for problems during implementation and throughout the entire lifecycle of the system. Ideally, criteria for control system vendor selection should include the creation and use of small, standard reusable blocks of code, which will reduce total programming time and troubleshooting requirements in the field.

No project can be delivered without some level of customization, but the standard blocks should be the starting point, with all modifications and additions tracked in a version control system that supports restoration of the stored code to various points in the development so the program can revert to a state when it was known to function. This functionality greatly reduces the level of effort to manage future updates and code enhancements.

Figure 2: As the system design takes shape, the underlying concepts should be simple and easy to understand. An integration map, such as this sample, clearly differentiates the integrated and non-integrated equipment controls, identifies the types of system communications, and highlights key revisions to the plan. Courtesy: Primary Integration Solutions

Project specifications should require the controls vendor to submit a complete control graphics screen map and mockups for the facility's staff to review and accept before code development begins in earnest. This enables the initial development of the code to more closely meet final requirements on the initial pass, saving rework and time later.

Similarly, specify that the controls contractor require from each manufacturer a detailed index of all their control points and how they are accessed via standard protocols. This information should be cataloged into a master points list that is used for network planning, as well as development of the control sequences.

The CxA will validate that the control graphics screen map and protocol map meet the client’s requirements.

6. Implementation: Factory testing saves time.

Some CxAs strongly recommend factory testing, which has gained some traction in the industry but is still underused. Requiring the controls and automation contractor to assemble its panels in the shop and complete programming and code development prior to deployment of equipment in the field can shave a great deal of time off the project delivery schedule when it is most valuable: at the end of the project just prior to turnover.

Typically, the engineer will specify that the test must be performed, and the manufacturer will write the test procedure. The engineer and CxA should review the procedure prior to testing. Schedule the factory acceptance test to allow the engineer, CxA, and facility's operations staff to assess the graphics and operator interface and review the modes of operation (including presentation of alarms). The test should be conducted to the greatest extent possible by using the actual hardware that will be deployed to the project. The CxA witnesses the testing process and validates the results. Identifying issues in the shop and getting user feedback before delivery to the site reduces field modifications and the potential errors that they introduce.

7. On-site testing and commissioning starts at the component level.

The process for testing and commissioning any system should start at the component level and expand in scope until the entire system, and ultimately the whole facility, is tested. By their nature the full implementation of integrated controls will be one of the last items on the schedule; however, this does not relieve the installer from completing proper testing of all installed components via point-to-point verification and loop checks.

Figure 3: Verification of all original equipment manufacturer (OEM) controls at each device is a prerequisite of any integration testing. Courtesy: Primary Integration SolutionsA key concept that is not well understood in the industry is the term "done." All too often a CxA receives a status report that something is "done,” only to find that several key components, such as power connections, are unfinished. This wastes time, and mission critical projects rarely have the luxury of wasted time. Before testing the integration of systems, the constituent systems themselves must be completed: that is, all open items are closed, and a complete, successful start-up has been documented. One way to streamline project delivery and avoid increasing project costs over time is for engineers to contractually require that contractors pay for the CxA’s additional time resulting from misreporting completion status.

Here is the recommended commissioning process from the component to system level:

• Original equipment manufacturer (OEM) controls: Verification of all OEM controls at each device is a prerequisite of any integration testing.

• Protocol transfer tests: Whichever communication protocol is being used, standard tests should be used to confirm proper configuration of the network infrastructure to support each of the protocols being used. Special care must be taken to test the robustness of the infrastructure to handle both normal and abnormal traffic levels.

• Silo system testing: Similarly, systems that have control systems installed by the controls contractor should have comprehensive testing done on all points to ensure proper connections and configuration within the control system. Some CxAs try to combine OEM and silo system testing; these should be conducted for each component and then for the overall integration.

• System-to-system tests: Once all subsystems and components have been successfully tested, the commissioning program can move into the integrated system testing phase. At this point each component system has been fully tested in a stand-alone fashion, so integrated testing does not need to cover the same ground. Rather the focus should be on the interfaces between systems; for instance, the interaction between the fire detection systems and the HVAC systems. Each scenario that requires that two or more systems operate in a unified manner should be simulated and appropriate response verified. Inviting facility staff to participate in these tests is an excellent opportunity for the CxA to provide training.

8. Ongoing operation/change control: Specify a complete, updated set of documents.

Upon successful completion of the construction and acceptance testing, the facility is ready to be turned over. As part of the turnover, the contractor should provide a complete set of updated documents and control code. A lessons-learned session that includes the construction, commissioning, and operating staff yields significant knowledge transfer to support ongoing operations.  

In addition, having the controls contractor periodically return to the site (monthly or quarterly) to work with the facility staff on small system adjustments is a good way to ensure that facility operations remain optimized. Changes in setpoints or programming should be updated in the documents. As time progresses and the various suppliers issue updates to the on-board controllers or system software, great care must be taken to determine the impact the upgrade will have on the integrated control and automation system. Advise owners to make all service technicians aware that a formal review must be undertaken before any firmware or software upgrades can be installed at the site. Manufacturers that supply heavily to the mission critical industry are familiar with this requirement and often issue full briefings on the upgrades they issue. However, owners should be made aware that equipment suppliers that deal across a broader scope of industries may not be familiar with this requirement and will require some direct interaction. The details of the upgrade should be reviewed by someone with in-depth knowledge of the control system to identify possible impacts.

Finally, periodic retesting and recommissioning of the controls and integration during scheduled maintenance windows will provide a level of assurance that the system’s performance has not degraded over time. CxAs can provide this relatively low-cost, high-value service over the lifecycle of the system.

The integration of MEP control systems in a mission critical facility is an exacting and demanding task. Careful and deliberate planning, coupled with professional installation, testing, and commissioning, is the only way to achieve a successful outcome. Assembling a team with controls integration and commissioning experience in the planning phase of a project will yield substantial benefits in avoided costs and improved schedule.


James McEnteggart is vice president of Primary Integration Solutions, a mission critical commissioning firm. He has more than 20 years of experience in MEP design and commissioning for mission critical and healthcare facilities.



ARTHUR , NC, United States, 01/16/13 08:18 AM:

Good article.
The Top Plant program honors outstanding manufacturing facilities in North America. View the 2013 Top Plant.
The Product of the Year program recognizes products newly released in the manufacturing industries.
The Engineering Leaders Under 40 program identifies and gives recognition to young engineers who...
The true cost of lubrication: Three keys to consider when evaluating oils; Plant Engineering Lubrication Guide; 11 ways to protect bearing assets; Is lubrication part of your KPIs?
Contract maintenance: 5 ways to keep things humming while keeping an eye on costs; Pneumatic systems; Energy monitoring; The sixth 'S' is safety
Transport your data: Supply chain information critical to operational excellence; High-voltage faults; Portable cooling; Safety automation isn't automatic
Case Study Database

Case Study Database

Get more exposure for your case study by uploading it to the Plant Engineering case study database, where end-users can identify relevant solutions and explore what the experts are doing to effectively implement a variety of technology and productivity related projects.

These case studies provide examples of how knowledgeable solution providers have used technology, processes and people to create effective and successful implementations in real-world situations. Case studies can be completed by filling out a simple online form where you can outline the project title, abstract, and full story in 1500 words or less; upload photos, videos and a logo.

Click here to visit the Case Study Database and upload your case study.

Maintaining low data center PUE; Using eco mode in UPS systems; Commissioning electrical and power systems; Exploring dc power distribution alternatives
Synchronizing industrial Ethernet networks; Selecting protocol conversion gateways; Integrating HMIs with PLCs and PACs
Why manufacturers need to see energy in a different light: Current approaches to energy management yield quick savings, but leave plant managers searching for ways of improving on those early gains.

Annual Salary Survey

Participate in the 2013 Salary Survey

In a year when manufacturing continued to lead the economic rebound, it makes sense that plant manager bonuses rebounded. Plant Engineering’s annual Salary Survey shows both wages and bonuses rose in 2012 after a retreat the year before.

Average salary across all job titles for plant floor management rose 3.5% to $95,446, and bonus compensation jumped to $15,162, a 4.2% increase from the 2010 level and double the 2011 total, which showed a sharp drop in bonus.

2012 Salary Survey Analysis

2012 Salary Survey Results

Maintenance and reliability tips and best practices from the maintenance and reliability coaches at Allied Reliability Group.
The One Voice for Manufacturing blog reports on federal public policy issues impacting the manufacturing sector. One Voice is a joint effort by the National Tooling and Machining...
The Society for Maintenance and Reliability Professionals an organization devoted...
Join this ongoing discussion of machine guarding topics, including solutions assessments, regulatory compliance, gap analysis...
IMS Research, recently acquired by IHS Inc., is a leading independent supplier of market research and consultancy to the global electronics industry.
Maintenance is not optional in manufacturing. It’s a profit center, driving productivity and uptime while reducing overall repair costs.
The Lachance on CMMS blog is about current maintenance topics. Blogger Paul Lachance is president and chief technology officer for Smartware Group.