Meeting electrical infrastructure demands in data centers

A data center’s electrical system requires a robust, reliable infrastructure that far surpasses that of its commercial and industrial facility peers.

05/16/2013


Learning objectives

  • Understand today’s data center demands and how to meet them.
  • Know the requirements for key equipment and its installation.
  • Understand how to properly specify wiring for various voltages.

To describe a data center using an analogy, a data center is a womb with no view—for computers. Designed to make complex equipment comfortable, the data center requires a robust and highly reliable electrical infrastructure that far surpasses that of its commercial and industrial facility peers.

These high-reliability infrastructure differences are achieved through meeting unique operational efficiencies, properly selecting and installing electrical equipment, and specifying proper wiring and design methods with appropriate voltages, all while meeting maintainability requirements for planned maintenance.

The first step in this process is to define key electrical system base requirements/data center goals. These are typical for a high-reliability installation:

Bus ducts in data centers have many joints, typically found every 10 ft in straight runs, resulting in equipment reliability and concurrent maintainability issues. Therefore, precise design is critical. Courtesy: Universal Electric Corp.1. Redundant components and systems are equivalent to a person leaving the house in the morning with pants that are too big, so he grabs a belt and a pair of suspenders. If the belt breaks, the suspenders will keep the pants up and vice versa. Either way, he’s covered.

2. Concurrent maintainability means assuring that every component and system (both power and cooling) that supplies the computers can be taken out of service for replacement, repair, or maintenance without shutting the computers down.

3. Fault tolerance, different from concurrent maintainability, means that when any component or system breaks or fails, the systems automatically reconfigure so that the computers don’t shut down. Fault tolerance is an automatic process; concurrent maintainability is a manual process. Part of fault tolerance is compartmentation so that a fire or explosion in one area does not result in total loss of power, cooling, or both to the computers.

4. Complete standby power backup is achieved with a generator plant that is set to provide power when the utility company is not available.

5. Selective overcurrent coordination of circuit breakers and/or fuses is achieved so that during a fault, only the minimum amount of the system is shut down. Ideally, the system will open only the circuit breakers that supply the single piece of failed equipment and nothing else upstream.

6. Modular, scalable construction allows a data center to expand down the road without overbuilding capacity on day one. This is crucial for two reasons: First, everyone is watching their pocketbooks, so if 10 MW of computers will eventually be needed but only 5 MW is needed on day one, the total cost of ownership (TCO) can be minimized by building a modular, scalable shell for 10 MW, but only 5 MW of interior infrastructure for day one. Second, the modular, scalable data center is easier to maintain. Data centers with an excess of unused capacity are a maintenance headache. Careful consideration of the ultimate facility configuration and expansion phases is necessary to minimize risk and eliminate the need for computer equipment shutdown during expansion.

7. Underground circuits are employed in data centers for two reasons: Contractors think they are less expensive to install, and they provide physical security and compartmentation for the data center’s wiring system. However, it is important to note that they require special calculations during the design phase. The Neher-McGrath calculations, found in the National Electrical Code (NEC) 310.15.C and ANNEX B, must be used to design all underground circuits. These calculations often result in the number and size of conductors run underground being substantially larger than would be required overhead. Thus, the anticipated economy compared to overhead circuits is often a false hope.

8. Emphasis on operating efficiency (reduced operating expense, or OPEX) and minimizing TCO can be met by reducing power usage effectiveness (PUE).

Each one of these requirements/goals is crucial because, unlike the typical commercial or industrial facility, the data center load is continuous, with increased ambient temperatures in many areas. For example, the back sections of data cabinets can be 104 to 113 F where branch circuit wiring is installed, while hot aisles can reach the same 104 to 113 F where branch circuit wiring is run upstream from the cabinets. These elevated temperatures result from higher supply air temperature to the computer equipment as a strategy to reduce PUE. Electrical equipment rooms (except those containing storage batteries) can operate at up to 104 F to reduce PUE. The extreme temperatures of a data center make its design for high operating temperatures in addition to code requirements that much more critical than a typical commercial or industrial facility design.

Operations and maintenance

An alternative to bus duct, the cable is assembled like a cable tray with large single-conductor power cables run within it and can be easily modified in the field to fit field conditions. With only two terminations (one at each end, with solid cable in bBeyond unique base design requirements, skilled data center designers also must consider equipment maintenance during design, as ease of maintenance will be crucial to meeting continuous and reliable data center operations. Because a lot of maintenance is required to sustain the critical environment, concurrent maintainability, arc flash labeling, and mean time to repair (MTTR) reduction all play a role in maintaining a data center’s electrical operations.

Designing the data center’s electrical systems to achieve concurrent maintainability means creating an arrangement where any piece of equipment or system that supplies the computers can be taken offline for maintenance purposes while the load continues to operate.

On occasion, maintenance is performed on a piece of equipment while it’s energized (hot work). While a high-reliability data center is designed for concurrent maintainability, some operators choose hot work to reduce maintenance time. While there are a lot of safety procedures in place for this type of maintenance, the best way to understand the risks associated with each piece of data center equipment is to understand its arc flash labeling. This label reflects the arc flash hazard calculated for each piece of equipment and identifies the level of personal protective equipment (PPE) and distances required for safe maintenance. It’s important to understand that some maintenance on small parts in restricted places cannot be performed with NFPA 70E levels 3 and 4 PPE.

Minimizing the time it takes to repair a piece of crucial data center electrical equipment and get it back in service to meet the load’s needs (MTTR) also is important in calculating data center maintenance in advance and when specifying equipment. Proper specification can reduce MTTR. For example, a 4000 A, low voltage, drawout-mounted circuit breaker can be withdrawn and replaced from shelf stock in 15 minutes, while a similar stationary-mounted circuit breaker might take an hour or more to replace.


<< First < Previous 1 2 3 Next > Last >>

HEINZ , CA, United States, 06/06/13 06:35 PM:

In the example on P40:
For a continuous load of 400A, the NEC requires a feeder that can carry 1.25 the continuous 400A = 500A.
So how does a 100% rated CB save on cabling costs.
Also, for a standard 80% rated CB, the terminal temp rating up to 100A is 60*, and 75* for over 100A. What is the temp rating for the terminals of a 100% rated CB?
The Top Plant program honors outstanding manufacturing facilities in North America. View the 2013 Top Plant.
The Product of the Year program recognizes products newly released in the manufacturing industries.
The Leaders Under 40 program features outstanding young people who are making a difference in manufacturing. View the 2013 Leaders here.
The new control room: It's got all the bells and whistles - and alarms, too; Remote maintenance; Specifying VFDs
2014 forecast issue: To serve and to manufacture - Veterans will bring skill and discipline to the plant floor if we can find a way to get them there.
2013 Top Plant: Lincoln Electric Company, Cleveland, Ohio
Case Study Database

Case Study Database

Get more exposure for your case study by uploading it to the Plant Engineering case study database, where end-users can identify relevant solutions and explore what the experts are doing to effectively implement a variety of technology and productivity related projects.

These case studies provide examples of how knowledgeable solution providers have used technology, processes and people to create effective and successful implementations in real-world situations. Case studies can be completed by filling out a simple online form where you can outline the project title, abstract, and full story in 1500 words or less; upload photos, videos and a logo.

Click here to visit the Case Study Database and upload your case study.

Bring focus to PLC programming: 5 things to avoid in putting your system together; Managing the DCS upgrade; PLM upgrade: a step-by-step approach
Balancing the bagging triangle; PID tuning improves process efficiency; Standardizing control room HMIs
Commissioning electrical systems in mission critical facilities; Anticipating the Smart Grid; Mitigating arc flash hazards in medium-voltage switchgear; Comparing generator sizing software

Annual Salary Survey

Participate in the 2013 Salary Survey

In a year when manufacturing continued to lead the economic rebound, it makes sense that plant manager bonuses rebounded. Plant Engineering’s annual Salary Survey shows both wages and bonuses rose in 2012 after a retreat the year before.

Average salary across all job titles for plant floor management rose 3.5% to $95,446, and bonus compensation jumped to $15,162, a 4.2% increase from the 2010 level and double the 2011 total, which showed a sharp drop in bonus.

2012 Salary Survey Analysis

2012 Salary Survey Results

Maintenance and reliability tips and best practices from the maintenance and reliability coaches at Allied Reliability Group.
The One Voice for Manufacturing blog reports on federal public policy issues impacting the manufacturing sector. One Voice is a joint effort by the National Tooling and Machining...
The Society for Maintenance and Reliability Professionals an organization devoted...
Join this ongoing discussion of machine guarding topics, including solutions assessments, regulatory compliance, gap analysis...
IMS Research, recently acquired by IHS Inc., is a leading independent supplier of market research and consultancy to the global electronics industry.
Maintenance is not optional in manufacturing. It’s a profit center, driving productivity and uptime while reducing overall repair costs.
The Lachance on CMMS blog is about current maintenance topics. Blogger Paul Lachance is president and chief technology officer for Smartware Group.