Give Me a Break Circuit breakers and branch circuit protection for data centers
Date: Friday 20 June 2008
Author: Raritan, Inc.
Introduction
Over the past few years, average power consumption per server has increased more than 20 percent. 
Also, consolidations and build outs are causing data centers and their racks to be more and more densely packed with power hungry IT equipment such as blade servers. To support new, power-hungry IT equipment, especially servers, data center managers have had to deliver more power to the IT equipment rack. Over the last decade, the typical power required at a rack has increased from 2 kilowatts to 10 kilowatts.
Electrical power is now 30 percent of data center operating costs and 20 percent of the overall total cost of ownership. The annual energy costs of an average data center are expected to double by 2010. To give a sense of the magnitude of data center power consumption, in the San Francisco Bay/Silicon Valley area today data centers alone consume 375 megawatts per annum. That is enough power to supply 75,000 households.
To deliver all this power, data center managers are deploying power distribution units (PDUs) capable of handling higher voltages, such as 208 Volts, and higher current, such as 30 Amps, to accommodate the increased power demands. In turn, new certification requirements for branch circuit protection and PDUs have been issued due to safety concerns over the substantial amount of power now delivered to IT equipment racks.
Regulations
Effective April 2003, Underwriters Laboratories (UL) issued standard UL 60950-1 which applies to information technology equipment. This standard requires the use of branch circuit protection for PDU configurations greater than 20 Amps. PDUs greater than 20 Amps, certified after April 2003, must have built-in circuit breakers.
UL 60950-1 permits products rated at 15 Amps and 20 Amps to be supplied with no breakers. It also grandfathers thermally-fused products that handle more than 20 Amps but were certified prior to April 2003 to continue to be sold. Such PDUs are still in use and are still being sold, though Underwriters Laboratory strongly recommends against using fused PDUs over 20 Amps.
Newly certified products that handle more than 20 Amps must comply with National Electrical Code ANSI/NFPA 70. In effect, this means these products are required to have branch circuit breakers listed under UL 489, "Standard for Molded-Case Circuit Breakers and Circuit Breaker Enclosures."
In addition to standard UL 489, Underwriters Laboratory also has the standard UL 1077, "Standard for Supplementary Protectors for Use in Electrical Equipment." These devices are called "supplementary protectors" and are labeled as "recognized components," not "listed components," as are UL 489 breakers. A UL Listed device meets more stringent requirements than a device that receives the UL Recognized designation.
Circuit Breakers
Circuit breakers are used in a variety of ways. They are mounted in panel boards and PDUs to protect branch circuit wiring, and they are built into equipment to protect components and systems. Interrupting a short circuit -current flow limited only by the resistance of wiring - is a severe test of a circuit breaker. If the interrupting capacity of the breaker is not adequate the device can literally explode.
All circuit breakers are tested for short-circuit activation, but whether a device continues to work after opening a short circuit depends on the severity of the event. UL 489 requires the breaker to be functional after being subjected to a short-circuit test. UL 1077 and the IEC standard EN 60934 allow for breakers to clear a short-circuit condition, but become safely destroyed in the process. UL 489 breakers can interrupt short circuits of 5,000 Amps or more. Typically, UL 1077 breakers can interrupt fault currents of 1,000 Amps.
Whether a breaker will or will not survive a short circuit depends on the magnitude of the current. Every circuit breaker has two ratings for interrupting capacity. One specifies the maximum amount of current the breaker can safely interrupt and still continue working (officially know as "fit for further use" or "recalibrated after testing"). The other interrupting rating specifies the maximum current that the breaker can interrupt safely but may be rendered inoperable in the process ("not fit for further use" or "not recalibrated after testing").
Overloads can be short term or long term. The protective device must not trip with a momentary or short-term over current event that is normal for the piece of equipment being protected. Servers, for example, may create inrush currents as their internal power supply and filter circuits start. These inrush currents typically last only a fraction of a second and seldom cause a problem. If an overload lasts longer than a few minutes the breaker should open to prevent overheating and damage. What gives a breaker the ability to discriminate between normal and damaging over currents is its delay curve.
Delay Curves
There are four kinds of delay curves in circuit breaker technology - thermal, thermal-magnetic, magnetic and hydraulic-magnetic. Each has a different trip profile in relation to time and current, and each has distinct mechanical characteristics.
Thermal circuit breakers incorporate a heat-responsive bimetal strip or disk. This type of technology has a slower characteristic curve that discriminates between safe, temporary surges and prolonged overloads. These circuit breakers provide a low-cost solution for appliances and printed circuit board protection, among other applications. Thermal-magnetic circuit breakers combine the benefits of a thermal and magnetic circuit breaker. These
devices have a delay to avoid nuisance tripping caused by normal inrush current, and a solenoid actuator for fast response at higher currents. Both standard thermal and thermal-magnetic circuit breakers are sensitive to ambient temperature so they are not good choices for warm environments like an IT equipment rack.
Magnetic circuit breakers operate via a solenoid and trip nearly instantly as soon as the threshold current has been reached. This type of delay curve is appropriate for sensitive installations but is not ideal for equipment like servers that typically have inrush currents 30 percent to 50 percent above their normal current draw.
A magnetic circuit breaker can be combined with a hydraulic delay to make it tolerant of current surges. These hydraulic-magnetic breakers are similar to the thermal-magnetic in that they have a two-step response curve - they provide a delay on normal over currents, but trip quickly on short circuits.
Hydraulic-magnetic circuit breakers are not affected by ambient temperature. Many hydraulic-magnetic breakers are available with a selection of delay curves to fit particular applications. Servers and networking equipment should be protected when there are short circuits but they should not lose power due to a normal over current condition. Therefore, the best data center PDUs use hydraulic-magnetic circuit breakers with delay curves that are appropriate
for IT infrastructure equipment.
Hydraulic-magnetic circuit breakers do tend to be sensitive to rotational position. Thus, the breakers themselves should be mounted in a vertical position to prevent gravity from influencing the movement of the solenoid. But since PDUs are designed for specific mounting, i.e., horizontally in a rack or vertically on an upright, the position of the breaker is well defined and stable.
Questions to Ask About Branch Circuit Protection for Data Centers
The following are questions for IT administrators and data center facilities managers to consider when purchasing high power PDUs for a data center:
Do your high-power PDUs comply with the latest standards?
Are there fuses or circuit breakers in your PDUs? If there are circuit breakers, are they UL 489 and UL Listed or UL 1077 and UL Recognized?
How much risk are you willing to assume by deploying products that do not comply with the latest requirements?
Who is liable if there is a problem, such as loss of power or equipment damage, which is determined to be a result of improper fuse replacement?
In the event of a power problem, how long can you wait to replace a fuse vs. resetting a circuit breaker?
Who is allowed to replace a fuse in a high-power PDU?
Are there service level agreements (SLAs) that require certain levels of uptime? How do fuses vs. various circuit breakers impact meeting these SLAs?
Summary
Effective April 2003, UL 60950-1 requires the use of branch circuit protection for PDU configurations greater than 20 Amps. In effect, this means these PDUs are required to have branch circuit breakers listed under UL 489. The best PDUs for data center use are those with hydraulic-magnetic circuit breakers designed with delay curves appropriate for IT equipment.
PDUs rated over 20 Amps that were certified before April 2003 may be sold with fuses and circuit breakers that do not meet UL 489 standards. However, these products are not recommended. Underwriter's Laboratories changed the requirements for good reasons. There are risks associated with using out-of-date products. (See the appendix "Branch Circuit Protection Alternatives to UL 489 Breakers" and the section above "Questions to Ask About Branch Circuit Protection for Data Centers.") It makes sense to purchase the latest, safest and most reliable technology available.
About Raritan
Raritan is a leading provider of management solutions that simplify IT operations. Based on KVM (Keyboard, Video, Mouse) switches, serial console servers, management software, power management and remote connectivity, Raritan's secure solutions drive data center and branch office efficiency and productivity in more than 50,000 locations around the world. Raritan also serves the OEM market by developing advanced, hardware-based, remote
management components based on KVM-over-IP and IPMI technologies. Founded in 1985, Raritan today has 38 offices worldwide, and its products are distributed in 76 countries. For more information, please visit Raritan.com
Appendix 1:
Branch Circuit Protection Alternatives to UL 489 Breakers
Branch circuit protection can be accomplished by either fuses or circuit breakers. Though disallowed for PDUs that handle greater than 20 Amps, there are PDUs, certified before April 2003, still being sold with fuses and UL 1077-compliant breakers.
Fuses:
Fuses are accepted for circuit protection for some products as noted above. However, they are not considered an industry best practice, particularly for mission-critical facilities.
Many PDUs use multiple fuses, some as many as twelve per PDU. For a data center with 100 PDUs this means as many as 2,400 fuses, each one a potential point of failure.
208 Volt deployments require two hot legs (two of the three hot legs in a three-phase power installation). It is possible for a fuse to blow on one of the legs but for the fuse on the second leg to remain intact. This means that despite fused protection the second leg remains hot and dangerous.
Replacing a fuse can be time consuming leading to longer mean time to repair (MTTR).
Replacing a fuse can be expensive since an electrician, and possibly an apprentice electrician as well, may be required. This is particularly true if a fuse needs to be replaced outside of normal business hours.
Product certifications may be invalidated and warranties may be voided if the PDU is physically opened or if the blown fuse is replaced incorrectly.
Thermal Circuit Breakers:
Thermal circuit breakers can be an improvement over fuses but, for high-current applications, may still have some limitations.
Many thermal breakers can only be reset after a fault. They can not be used to manually disconnect the power supply.
Thermal breakers by their nature are susceptible to tripping due to enclosure heat.
A thermal breaker, typically a UL 1077 device, can break fault currents up to 1kA or 2kA. This is far less than the capability of a UL 489 circuit breaker which is rated to break fault currents of 5kA or more.
UL 1077 devices are designated as supplementary protection devices and are UL Recognized.
They do not receive the more stringent UL Listed designation.
