Commissioning mission critical systems
The 24/7/365 nature of a mission critical facility makes its commissioning (Cx) unique. With an abundance of redundant systems and a zero tolerance for downtime, a data center's MEP infrastructure requires careful commissioning practice, which can be more expensive than commissioning less complex facilities.
Because of this complexity, increasing scrutiny of budgets, and ever-tightening schedules, gaps in commissioning can occur that lead to larger, more expensive problems down the road. However, by defining commissioning roles early, integrating schedule coordination, completing peer review, and employing simulation and demonstration methodology, commissioning can be a cost-effective ally for delivering data centers on time and within budget.
This article outlines some of the strategies that we have been successfully employing on data center and mission critical projects.
From project to project, the commissioning scope involves verification and documentation, including factory witness, field acceptance, and integrated systems testing. The biggest challenge here is defining who is obligated to perform which tests. In many cases, there is confusion regarding the roles of the design engineering firm and the commissioning agent (CxA). The lack of commissioning standardization has led to gaps in responsibility.
Clearly defining roles is crucial to attacking the commissioning scope. The CxA often is added to the team late in the design process. It is more effective if the Cx team is part of the team from initial design forward so they can review and comment on basis of design (BOD) documents, peer review the design, review construction specifications, and develop the Cx specifications. The CxA can then be on hand with the design engineer for factory witness tests, while the equipment is being programmed and fabricated, during field acceptance, and finally for the functional performance and integrated systems checks.
Additionally, in a typical building's commissioning process, usually there is only one scenario for operating the building. But, because of the variety of systems and amount of redundancy that exist within a data center, there are a number of different ways it can operate, including several scenarios and system sequence options. Failing one device during a commissioning test will often result in the system's overcompensation, shifting itself around the single point of failure to continue working. With knowledge of the design and system redundancies specific to each facility, the CxA can develop the appropriate means and methods to simulate the right failures.
Managing the schedule
In order to properly commission a data center, both the HVAC and the electrical systems infrastructure must be complete. In too many of today's fast-tracked projects, design and construction phases take longer than expected, while the project completion date and turnover to the owner do not stretch accordingly. As a result, acceptance testing of equipment and commissioning often are combined. This may result in on-site confusion (and tension) because equipment and systems have not been started up and accepted before final commissioning activities start.
Once the infrastructure work is complete, however, the CxA can begin commissioning with rented load banks and metering equipment. At this point, there might be a testing schedule set to occur over two months, but due to budget constraints, the schedule can get compressed.
Schedule changes like these hit mission critical facilities harder than other buildings because the stakes are significantly higher. If something goes wrong and changes have to be made after startup and acceptance, there is a greater risk of impacting another system or test that was already commissioned. In these cases, the remote monitoring systems (e.g., supervisory control and data acquisition, event project management system) may not have been completely installed and tested before site acceptance testing starts. This requires retesting equipment to verify conductivity and proper systems reporting and graphics. Sometimes, trying to compress the schedule can actually lengthen it.
While it costs nowhere near the amount to design and actually build the critical facility, commissioning a single data center can cost an owner as much as $200,000 for a typical 150,000-sq-ft building. In addition to the cost of the CxA, rental costs for portable load banks can be significant. While permanent on-site load banks are used whenever possible, portable load banks are required for proper testing and commissioning of a typical mission critical system. They range in size from truck-size medium-voltage, to small rack-mounted load banks. Coupled with this is the utility cost for power that can be involved in testing many of these systems at full load, which for the larger systems can add up to 20 MW or more.
Additionally, many critical facilities incorporate standby diesel engine generators, which also must be tested under full load and under varying system conditions, possibly requiring as much as 150 gal/hr of fuel at full load. The generator fuel can amount to as much as 10% of the owner's operating budget for the year. The general contractor, in conjunction with the other subcontractors, typically incorporates these fuel costs into his budget, as he has predetermined the amount of testing required. Nevertheless, there is a tremendous pressure in this economy to reduce the price of commissioning, while still maintaining its quality. The coordination of testing to minimize these costs is a challenge the CxA must overcome with detailed scheduling and the integration of startup, acceptance, and commissioning tasks in a logical order.
In addition to the general cost of commissioning, another hidden expense is its wear and tear on the mechanical and electrical systems. From the factory witness to field acceptance and finally to integrated performance testing, a data center's infrastructure has been through a lot of system operations before it is actually used to support its true load. It is not uncommon, especially if problems have occurred during factory or site acceptance testing, for some equipment (typically circuit breakers), to encounter hundreds of operations. While still technically within their operational limits, it is best practice to go back and inspect this equipment and, where required, perform necessary maintenance.
While the scope, schedule, and budget of each project present many commissioning challenges, the following solutions can provide some relief.
Early definition of the commissioning process and the roles of all parties. Define roles and responsibilities early in the design phase to scope out the commissioning process and the roles that others on the team (architect, engineer, contractors, and operators) will play in working with the commissioning team. The job of the CxA is not to be the designer or the contractor. Instead, the CxA's responsibility is to the owner, verifying that the system performs as intended per base design and the owner's predefined project requirements. When roles are defined clearly and the CxA is brought on early enough in the process, thorough understanding of the owner's intent and design requirements can be achieved.
Schedule coordination. Draw up a proper commissioning schedule and integrate it into the master construction schedule so that each team member knows when his or her part of the system needs to be complete and ready for commissioning. Defining which tests need to be done when and in what order, which rooms must be off-limits, and which systems need to be running is critical. For example, a data center's electrical system typically requires the longest testing time period, sometimes taking as much as a month to complete. During that time, the commissioning team is turning the power on and off, making it difficult to test other systems—such as the mechanical systems—simultaneously. Safety also plays a role in scheduling. Cleaning crews shouldn't be allowed into the space to touch up paint nicks, hang up room signs, etc., until commissioning is completed.
Design peer review. Often times an owner will hire the CxA to review the project drawings and specifications from an engineering and architectural perspective as well. This can help eliminate costly change orders that often occur late in the game. By performing an early design peer review, a savvy CxA will pick up on what testing will need to be done down the road and suggest to the designer what to include in the design to help accommodate future testing. For example, the CxA could suggest equipment options and modifications to make testing easier, formulate ways to hook up the load banks, and schedule how/when the tests will be performed.
Simulation and demonstration methodology. A question owners commonly pose is: What can be done for a facility designed in 2009 design to help it operate in 2015? The answer: Create very detailed documentation of commissioning testing. Providing test and script information and test results as deliverables at the end of the commissioning process will give project team members peace of mind. The engineering team will see that the CxA has verified their design, the contractor will realize what it is expected to do to support testing, and the owner will have documentation of the building's tests and test results at turnover for future recommissioning efforts.
Tie load banks in at the right spot. To place load banks in the most appropriate locations, a lot of parameters concerning air flow dynamics must be taken into account, including height and shape of the room, rack locations, hot/cold aisle layout, and varying load densities within the room. Based upon these criteria, the CxA will be able to properly position the load banks to best simulate the real-life conditions of the data center. It is the CxA's responsibility to determine the connection points and dictate the size of the load banks required. While many suitcase heaters typically are used to simulate the load of hundreds of servers, using additional, smaller-sized load banks will allow for more realistic testing and verification of airflow and air distribution within the room. There is still a lack of industry consensus on whether load banks should be placed in the hot or cold aisles. If too large a load bank is put in the cold aisle, the temperature will rise too much. On the other hand, if put in the hot aisles, the desired temperature will not spread out throughout the room.
Commissioning is especially crucial for facilities that are continuously operating and which have highly expensive or life-threatening consequences if they go down. The expense and complexity for such data centers and other types of mission critical facilities are appropriately higher than for more conventional buildings. The pressure of compacted schedules and budgets can tempt owners or construction teams to cut corners on Cx services. With proper planning, team management, and execution, Cx providers can respond to the realities of the field and the board room with confidence and cost-effectiveness.
Vallort is a vice president for Environmental Systems Design, and director of the company's Building Sciences Group. Weber is a senior vice president of Environmental Systems Design and director of electrical engineering, Mission Critical Facilities Group.
Setting commissioning principles
Commissioning often is applied in a modular manner. Not every system, component, or device is tested. This is both the blessing and the curse of commissioning.
As a true quality-control process, commissioning will only be as successful as the level of re-ward and risk associated with it, which directly corresponds to how much confidence an owner wants in his or her facility. For example, when performing commissioning on a condominium building, testing 10% of the room's fan cool units is representative enough to find a potential problem. However, for a mission critical facility—which demands 100% reliability and uptime—critical systems must be tested at 100% and other systems tested at a lower sampling rate.
Commissioning verifies the capability of the system on day one. Commissioning tests take place before the owner accepts the data center. Detailed documentation of commissioning testing and simulation give the owner confidence that when he accepts the facility on day one, all systems and equipment will be functioning as designed.
Commissioning does not change design, but instead serves as an agent of the owner to verify it, identifying the system's weak points to maximize reliability. The CxA is not the engineer of record, but rather an agent of the owner whose role is to both verify and enforce the design, not change or fix it. The CxA is the engineer and contractor's friend. By reporting that the data center is working as designed on day one, commissioning assures all project members that no future claims can be made on the initial project work.
Commissioning is a process that can be used to verify the future performance of a system. Data center infrastructures can be expensive. Instead, some owners are now opting to install only half of the cooling equipment, with the intention of adding more down the road as the critical facility expands and grows. In response, CxAs are being asked to verify that systems can work in the future, when the supplemental mechanical infrastructure is installed.
Annual Salary Survey
After almost a decade of uncertainty, the confidence of plant floor managers is soaring. Even with a number of challenges and while implementing new technologies, there is a renewed sense of optimism among plant managers about their business and their future.
The respondents to the 2014 Plant Engineering Salary Survey come from throughout the U.S. and serve a variety of industries, but they are uniform in their optimism about manufacturing. This year’s survey found 79% consider manufacturing a secure career. That’s up from 75% in 2013 and significantly higher than the 63% figure when Plant Engineering first started asking that question a decade ago.