Getting to the bottom—and top—of PUE

When developing data center energy-use estimations, energy engineers must account for all sources of energy use in the facility: computers, cooling plants, and other related equipment. Learn how to itemize power usage efficiency (PUE).

By William J. Kosik, PE, CEM, HP Critical Facilities Services, Chicago July 29, 2013

Learning objectives

  1. Learn which air economizer strategy is best in a given data center.
  2. Know how to itemize annual energy use of the various data center components. 

A common thread running through many articles about data centers is the idea that approaches to data center energy efficiency are still in the process of a paradigm shift. This shift is moving us away from what we know the most about: designing HVAC systems for office buildings, labs, hospitals, and schools. 

For example, just five years ago, a large portion of legacy data centers were still running supply air temperatures at 55 F—typical of a commercial building. Contrast that to new projects where  data centers will use supply air temperatures at or above 75 F. That is a 20 F increase in supply air temperature—effects cascading down into the entire cooling system. Just this one change has caused a wholesale rethinking of the design and operation of air conditioning systems that are used in data centers. 

Thankfully, there are many smart engineers designing data centers and several industry organizations (such as Uptime Institute, 7×24 Exchange, ASHRAE, and others) dedicated to the planning and design of data center power and cooling systems. Also, many of the manufacturers, arguably the most important link in the chain, now have complete equipment lines dedicated to data centers. All of these continue to provide solid standards, recommendations, and products to assist in the paradigm shift, and look to future data center transformation. 

As computer technology (hardware, networking, storage, and software) evolves at a blazing pace, planning and engineering of power and cooling systems are struggling to keep up. But this should come as no surprise. We see it in our daily life—mobile phones, PCs, notebook computers, and TVs that are rendered obsolete in 9 to 12 months from initial release. Certainly, there are different factors involved in the consumer electronics market, but the core idea is the same as with enterprise-level IT equipment—advances in new technology (manufacturing, materials, software) enable higher performance than the predecessors while using less energy. 

Understanding these constraints, it’s no wonder that power and cooling equipment manufacturers have a difficult time conducting R&D, planning, funding, and manufacturing their next generation of products that will support yet-to-be-developed computer technology. In the end, the power and cooling equipment manufacturers develop products that work well with the latest generation of computer technology, but integrating features that allow the equipment to adapt to future IT equipment design may simply be too cost prohibitive. 

Looking from a different perspective, we see that when a computer manufacturer releases a new generation of servers, the thermal engineers will have likely developed a novel cooling design to make the server run at lower temperatures and to use less fan energy. This is where the data center HVAC engineers and the server thermal engineers need to have a conversation in an attempt to optimize the energy use and efficacy of the servers and the data center cooling system, not just one or the other. An undesirable outcome is to have a high-performance, low-energy server that requires a data center cooling system that is inefficient, too complex, or too cost prohibitive to build. This is where detailed simulation and analysis of data center cooling system energy use come in.

It’s the heat and the humidity 

Supply air temperature is the most distinctive feature of a cooling system in a data center. In comfort cooling applications, the primary goal of the HVAC system is to provide enough cooling capacity to satisfy all internal and external loads, ensure that the building occupants feel comfortable (dry bulb temperature and moisture content of the air), and to maintain the appropriate filtration and ventilation rates to safeguard against higher-than-acceptable levels of gaseous and particulate contaminants. Data centers generally need to meet these goals as well, but the electrical equipment loads (when compared to a modern, high-tech commercial office building) are an order of magnitude greater. The good news is that, unlike people, computers don’t mind running very hot and are pretty tolerant to a wide range of moisture levels. With this tolerance to high heat and humidity comes tremendous opportunity for energy efficiency opportunities. 

The energy efficiency opportunities come from a combination of reduced compressor horsepower resulting from increased evaporator temperatures (supply air temperatures) and the fact that the compressors will run less often, especially in climates that enable full use of the economization strategy. This is where careful examination of the available cooling system alternatives is necessary; while a certain cooling option might offer a significant reduction in compressor energy, the other components (fans, pumps, etc.) may use more energy when compared to the other options. 

The comparison of the cooling system options must include a full hourly energy simulation of the data center as a whole (as defined by ASHRAE Standard 90.1) with the ability to analyze the cooling systems and subsystems to determine which components consume the largest amounts of energy. The results of the this analysis will provide raw data for the energy professional to make recommendations on the most energy efficient system, and also offer granular data on how each of the subsystems performs under different operational scenarios, such as different supply air temperatures and in different climates.

Economization techniques 

A critical component of any energy-efficient cooling system is the economizer. An economizer is simply a combination of operational sequences and equipment hardware that is intended to reduce energy use of an HVAC system by taking advantage of the positive psychrometric attributes of the outdoor air. Because different economizers rely on different psychrometric conditions, each one will have distinct performance characteristics. Depending on the economizer type and control strategy, the economizer will operate in three distinct modes: 100% off, partial operation, and 100% on. 

The partial operation mode will operate at a specified range of temperatures and humidities. Depending on the climate, partial economization could be in effect a large percentage of the year; it is important to account for these hours in determining the efficacy of the economizer solution. Calculating partial economization is done by adding up the hourly cooling load in tons (ton-hours)  in the period of hours being analyzed (8760 hours total). This sum becomes the numerator. The denominator is the sum of the hourly cooling load in tons-hours (simply 8760 x cooling load). The resulting percentage is essentially the amount of time the economizer can be used. 

The analysis using Chicago weather data depicts these efficiencies monthly by economizer type (Figure 1). When analyzing a climate that is south of the equator (Figure 2), the data will show the greatest savings during the “summer” months in the northern hemisphere. Another way to look at the efficacy of the economizer is the supply air temperature it can produce with no mechanical cooling. Depending on the climate type, some economizers can be used nearly 100% of the time with little or no mechanical cooling. These are examples of data visualization techniques that are useful to gain a quick understanding of the potential energy reduction.

Economization strategies 

Because the economizer will be a major driver in the energy efficiency of the overall system, it is useful to group cooling systems by economization technique and then by types of components used in the system, as shown in Figure 3. (Note: this analysis is intended to compare the energy use characteristics of the alternatives; no judgment on the operational efficacy of the systems is implied.) 

Direct air—When conditions allow, air is taken directly from outdoors and mixing data center return air with the outdoor air. Out-of-range moisture levels of the outdoor air will limit full use of the economizers. Adiabatic cooling can be added to extend the use of the economizer. At higher outdoor temperatures, outside air volume can be modulated to maintain the lowest return air possible with compressorized cooling handling the balance of the cooling requirement. 

Indirect air—Heat from data center return air is transferred to the outdoor air using a heat exchanger (heat wheel, heat pipe, etc.). When the outside air is cold enough, the return air can reject 100% of the heat to the outdoors. At higher outdoor temperatures, the system will maintain the lowest return air temperature possible with compressorized cooling handling the balance of the cooling requirement. Adiabatic cooling to reduce the temperature of the outdoor air can be used to extend the use of the economizer. The inherent efficiency losses of the air-to-air heat exchangers will reduce the usefulness of the outdoor air temperature. 

Direct/indirect water—Water cooled directly using outside air is usually accomplished by open cooling towers that dissipate heat from the water into the air. This water can then be used to cool the evaporator of a packaged water chiller, cool the compressors in a self-contained computer room unit, or to cool computers directly. The water typically is run through a water-to-water heat exchanger to avoid fouling of the secondary cooling equipment. The temperature of the water that can be produced is dependent on the moisture level of the outdoor air, and at cold outdoor air temperatures, additional equipment may be required to avoid freezing in the cooling towers. 

Indirect water—Typically an air-cooled chiller is used to generate chilled water for air handling units (AHU), water-cooled IT racks, or for water-cooled computers in the data center. Economization is achieved by using a chiller-integrated free cooling coil or by a separate closed-circuit cooling tower. Because the heat transfer between the outdoor air and the water is completely sensible, the moisture content of the outdoor air has no impact on the water temperature that is produced using the economization technique. An adiabatic process, such as water sprays added to the condenser coils, can be added to lower the outdoor air temperature; in this case the moisture level of the outdoor air becomes a factor in the temperature of the water.

The parts are greater than the whole 

The keys in achieving the greatest energy efficiency are to optimize the cooling systems and also to understand the dynamics of the data center as a whole. For example, the ASHRAE environmental classes (Figure 4) were developed to address the operation of the data center at elevated temperatures, as a means to reduce energy consumption. However it is essential to understand the impact that  higher temperatures have on the servers themselves.

Becauseenergy savings in the air conditioning systems is also a fundamental idea behind the development of the ASHRAE, one may assume that as the supply air gets warmer, less compressor power is needed and more hours of economization are available. This premise is generally true, but not universally. In hotter climates, increasing the supply air temperatures generally results in significant reductions in energy use. In colder climates the savings are less dramatic simply because there are more hours annually when the outdoor air can be used in an economization strategy. In these cases, increasing the supply air temperature will not accomplish much because the data center temperature may be greater than the highest annual temperature in that climate.

System and subsystems 

Each of the cooling systems consists of multiple energy-consuming devices: compressors, fans, pumps, and humidification equipment. Using the specifics of the actual project is vital in forming an itemization of the various components’ annual energy use. Nevertheless, assumptions based on ASHRAE minimum energy performance targets can be applied to the individual components. 

Compressorized cooling equipment—This equipment will range from unitary direct expansion equipment to water-cooled chillers. The basis to effective energy optimization for compressorized cooling equipment is the ability to unload the compressors (or decrease speed of variable speed compressors) at an even pace that is in lockstep with the actual cooling load. This avoids over- or under-provisioning of cooling capacity and the corresponding energy use. Also, the equipment must be able to take advantage of cooler outdoor temperatures and lower condenser temperatures. 

Supply fansThe power requirement of a supply fan is determined by the air volume, fan/motor efficiency, and static pressure drop of the components that make up the air handling system. The best energy efficiency will come when the difference between the supply and return air is maximized and the static pressure drop is made as small as possible. 

Scavenger fans—Used in the indirect air systems, these fans induce outdoor air across the heat exchanger. Because the indirect air systems vary depending on the manufacturer, it is essential to understand how these fans will operate, including the airflow rate, motor power, and operational profile (e.g., fan speed based on outdoor temperature). Scavenger fans can vary speed based on the amount of outdoor air needed to effectively transfer heat from the return air. 

Return/exhaust fans—Used primarily for direct air systems as a means of removing the outdoor air from the building to avoid overpressurization. Ultimately, depending on the building design, these fans will range from powerful centrifugal or vane-axial fans to low-powered propeller fan relief hoods. These fans should vary speed based on air volume, and in climates that can use outdoor air for economization a large percentage of the year, the fan system should be carefully designed because they will be running near 100% most of the year.

Pumps—Used in water-based systems only. Similar strategies to fans—keep head pressure as low as possible and vary pump motor speed based on flow requirement. 

Humidification/evaporative cooling systems—Using an adiabatic process to humidify or cool the air is necessary to achieve maximum energy savings. In some climates it is not necessary to add moisture to the air based on the ASHRAE temperature and humidity classes, so designing a humidification system may not be necessary. 

Water-cooled IT cabinetsThink of these as miniature data centers—the cooling and air movement are built-in. These cabinets rely on fans to move air across a coil mounted in the cabinet and pumps that distribute water to multiple cabinets. The energy used from the fans in the IT cabinet and pumps are not trivial and need to be included in the overall energy use calculation. 

Water-cooled computers (component level cooling)—Theoretically the lowest cooling energy consumer, the primary components are pumps and heat rejection (cooling towers, etc.). These are primarily used in high-performance computing applications where individual server cabinets are rated at 80 kW (or more). The goal is to avoid using vapor compression cooling and rely on cooling tower water only given the allowable high cooling water temperature. Parts of the computer, network, and storage systems are not able to be water cooled, so this air conditioning load must be accounted for and cooled by some other means and included in the energy analysis.

Itemization of energy consumers

Energy use simulation is a powerful tool that can be used to provide data to make decisions. Using energy simulation and analysis techniques gives the engineer insight into how the individual components behave based on IT load, supply air temperatures, and outdoor conditions. Applying data visualization techniques using line graphs allows for a detailed scrutiny of the energy usage of the components over the course of a year. This is necessary because the cooling systems perform very differently in cold weather than they do in hot weather. This approach also is used for evaluation of cooling system energy when analyzing different locations (climates). This type of analysis will expose cooling systems that might work well in certain climates and not in others, so worldwide prototypical solutions can be varied by location. 

Indirect cooling with direct expansion assistBefore a detailed evaluation of the energy simulation results is performed, it is often helpful to do a visual investigation of the annual energy use line graphs to make initial observations (Figure 5). This figure illustrates the difference between an indirect air cooling system (system 1) and an indirect evaporative cooling system (system 2), both with direct expansion (DX) cooling assist. These systems are designed to use DX cooling when the supply air temperature setpoints can no longer be maintained, augmenting the cooling capability of the system. 

In this specific example (excerpted from analyses based on actual project documentation), the two systems perform quite well when compared to other standard data center cooling solutions. The primary differences show up in the fan power and the effectiveness of the heat transfer mechanisms. In system 1, the scavenger fan energy is much less than for system 2, but the supply fan energy is higher than in system 2. Also, the heat transfer effectiveness of system 2 outperforms system 1. This is evident when inspecting the line representing the data points for the cooling energy; system 1 has higher peak power occurring more often during the colder months of the year. 

Notice that in both systems, the humidification energy is negligible. In simple terms, good energy performance (in a temperate climate) is exemplified by little or no compressor energy expended during the fall, winter, and spring months. During the summer months (June, July, and August in the northern hemisphere), compressor energy is depicted by a smooth curve that follows the curve represented by outdoor temperatures. The cooling energy expended in a data center has a strong correlation to outdoor air temperatures and should follow these conditions as closely as possible to avoid over-provisioning of cooling capability causing inefficiencies and unneeded consumption of energy. Figure 6 shows the same parameters as Figure 5, but Sao Paulo, Brazil, weather data is used. The overall energy usage is greater than Chicago, but there is a more uniform energy use across the year, where Chicago has higher spikes of power use in the summer months. Also, Sao Paulo is in the southern hemisphere—the seasons are the opposite of the northern hemisphere.  

Water-cooled chillers with water economizer—The first thing that becomes evident when reviewing the line graph of the energy use data for water-cooled chiller systems is the number of components used (Figure 7). Each of these uses energy, and the overall efficiency of the system is still good. This is why this type of system has been the gold standard for many years in large data centers and commercial buildings. Because the water economizer is based on the moisture contained in the outdoor air, in more humid climates less time is available annually to use the economizer. 

In Figure 7, systems 3 and 4 show the annual energy use of a water-cooled chiller with water economization and AHU, and the same cooling system using water-cooled computers in place of AHUs. The line graphs for system 3 have a telltale sign indicating room for energy efficiency improvement: the compressor power in the summer months shows little fluctuation; that is, there is little economization taking place during this time period.

In system 4, very little fan energy is needed except for cooling areas outside of the data center area. This becomes a primary driver of why system 4 is more efficient than system 3. Another important contributing factor is that in system 4, the water temperature is 20 F warmer than the water in system 3. This has a twofold effect: first, the compressor energy is lower because of the increased evaporator temperature, and second, there is an increase in the number of hours in which the water economizer can be used based on a higher acceptable wet-bulb temperature.  

Designing cooling systems for data centers is a process that has many variables and requires many decisions, including how the IT equipment will interface with the cooling. It is vital that the engineering team use a methodical design process that includes detailed energy simulation and analysis techniques which will help make decisions throughout the life the project. Also, taking into consideration that the IT equipment will consume more than 75% of the data center annual energy, there needs to be a careful assessment and optimization of the design and operational parameters to create long-lasting energy-savings synergies. 


William Kosik is principal data center energy technologist with HP Critical Facilities Services, Chicago. Kosik is one of the main technical contributors shaping HP Critical Facilities Services’ energy and sustainability expertise. He has worked on energy analysis and strategy projects in more than 25 countries, and consults on client assignments worldwide. A member of the Consulting-Specifying Engineer editorial advisory board, he has written more than 25 articles and spoken at more than 45 industry conferences.