Server immersion cooling

Example of an Immersion Cooling system

Example of server immersion cooling of one server

Server immersion cooling is a computer cooling practice by which computer components such as CPUs, GPUs, SSD and DDR Memory, and other electronics including complete servers are completely submerged in a thermally conductive dielectric liquid or coolant, which is then cooled through the use of a circulation system employing liquid pumps, plumbing, liquid-to-liquid heat exchangers and/or dry cooler (radiator) type coolers to reject the heat from the coolant. Common dielectrics coolants which are used in liquid immersion cooling of electronics are either mineral-oil based or entirely synthetic chemicals. These chemicals fall into two broad groups of application use cases: Single-phase, Liquid Immersion Coolants (SLIC) or 2-phase, Liquid Immersion Coolants (2PIC).

Server immersion cooling has the potential of becoming a popular server cooling solution, in particular for green data centers,^[1] as it allows operators to drastically reduce their energy usage through the elimination of the expensive air cooling infrastructure including on-board server fans, CRACs, A/C compressors, air-circulation fans, necessary duct work, air handlers, and other active ancillary systems such as dehumidifiers. These systems are replaced efficient low speed liquid circulation pumps and heat exchanger and/ordry cooler systems. The power reduction that can be achieved through the use of liquid immersion cooling is often measured in terms of Power Usage Effectiveness (PUE). However, It is important to note that the definitions of PUE for air-cooled devices and liquid immersion cooled devices differs in one important aspect, which makes such direct comparisons misleading in favor of air cooling.

- PUE for air-cooled data centers, as defined by the Green Grid, includes the power used by the fans and other active cooling components found on-board the servers as part of the IT Equipment Energy,

- PUE for liquid immersion cooling excludes these values from the IT Equipment Energy component because these system elements are generally removed from the equipment as they are not necessary to circulate the dielectric coolants, and often actually impede the circulation of the dielectric coolants.

This discrepancy in the definition of PUE for the different cooling methods results in PUE of air-cooled data centers generally being overstated by anywhere from 10-20% when compared against the PUE of an liquid immersion cooled facility of the same power usage.

Servers and other IT hardware cooled by liquid immersion cooling do not require fans to circulate the dielectric coolants, thus they are removed from the system prior to immersion. Hydrocarbon based thermal pastes typically used with heat spreaders on CPUs and other chips needs to be replaced with a different compound to avoid it from dissolving in the coolants. Indium foil^[2] and thermally conductive epoxies are commonly used as a replacement materials.

Network router and smart-phone immersed in synthetic single-phase, liquid immersion coolant.

The temperatures used in liquid immersion cooling are determined by the highest temperatures at which the devices immersed can reliably operate. For servers this temperature range is typically between 15 to 65 °C (59 to 149 °F), however in ASIC based crypto mining devices this range is often extended up to 75C. This increase in the high end of the temperature range allows data centers operators to use entirely passive dry coolers, or much more efficient evaporative or adiabatic cooling towers^[3] instead of chiller-based air cooling. This increase in the temperature range also allows operators using Single-phase, Liquid Immersion Coolants to more effectively use the change in outdoor temperatures to get more efficient cooling from their systems because the single-phase systems are not limited in their effectiveness by the boiling point of the 2-phase coolant.

Multiple relevant brands like Intel and Facebook have already validated the advantages of submerging servers.^[4][5]

Current commercial applications for immersion cooling range from data center oriented solutions for commodity server cooling,^[6][7] server clusters, HPCC applications^[8] and Bitcoin Mining.^[9] and mainstream cloud-based and web hosting architectures. Electric vehicle and battery manufacturers also employ liquid immersion cooling in batteries, drive-train, Kinetic energy recovery systems, electric motors, electric motor controllers, and other on-board electronic sub systems. Liquid immersion cooling is also used in the thermal management of LEDs, Lasers, X-Ray machines, and Magnetic Resonance Imaging devices.

History

The use of mineral or vegetable-based dielectrics liquid for cooling large power distribution components such as transformers and industrial machinery is a common practice and has been successfully deployed since the beginning of the 1900s^[10]

• Immersing electronics (specifically transformers) in dielectric fluids for thermal management was used before 1887.^[11]
• The first patent to explicitly mention the use of oil as a coolant and insulator is in the patent submitted for a Constant Current Transformer in 1899 by Richard Fleming of Lynn, Massachusetts, assignor to the General Electric Company of New York^[12]
• The first reference to the specific use of dielectric fluids being used to cool "computers" is in 1966 by Oktay Sevgin of IBM.^[13]
• In 1968, Richard C Chu and John H Seely, working for IBM patented an "Immersion cooling system for modularly packaged components."^[14]
• Seymour R. Cray, Jr. founder of Cray Research, LLC patented a "Immersion cooled high density electronic assembly" in 1982.^[15]
• The Cray T90 (released in 1995) used large liquid-to-chilled liquid heat exchangers and single or two-phase immersion cooling oils for heat removal^[16]

A different industry where immersion cooling is applied to electronic components is deep sea research where Remotely Operated Underwater Vehicles with electronic equipment are filled with newtonian Single-phase, Liquid Dielectrics to both protect them from corrosion in seawater and as a pressure compensating fluid to avoid being crushed by the extreme pressure exerted on the ROV while working in the deep sea. This application also includes the cooling of the electric motors used for under sea propulsion.

Until about 2014, the technology was typically only utilized in special very intensive supercomputing projects, similar to the Cray Computer Applications mentioned above. However, since the dramatic increase in global energy consumption by data centers, there is now a focus on energy efficiency which has driven the utilizing of liquid immersion cooling in both data centers and crypto mining operations to reevaluate its application. In addition the advent of new very high density CPUs and GPUs for use in real-time processing, Artificial intelligence, machine learning, and data mining operations is pressuring users and data center operators to evaluate liquid immersion cooling for ability to cool high density racks as well as reduce the total mechanical footprint of data centers.

Since 2005, several companies tried to claim patents based on the open bath approach with external pump installations. Most, if not all of these claims have been denied on the basis that IBM had already patented the technology.

Server Immersion Cooling techniques

Open or Semi-Open Bath Immersion Cooling

Open or semi-open bath immersion cooling is a data center cooling technique that implies fully submerging servers in dielectric coolant fluid. It is called open or semi-open bath because servers are placed side-by-side in large tanks that function as baths. These baths operate at atmospheric pressure and allow the coolant fluid to be moved through the hardware components or servers submerged in it.^[17]

Open Bath Immersion Cooling uses single-phase dielectric liquids. These liquids always remain in liquid state while operating. They never boil or freeze. The oil is usually pumped to an external heat exchanger where it is cooled thanks to heat exchange with a cooler water-circuit. A different approach uses natural convection as the driving force to circulate the oil^[18] through the servers. and a water cooled heat exchanger. This technique uses “open baths“ as there’s little (or no) risk of the coolant evaporating.

In semi-open bath Immersion Cooling, a two-phase chemically engineered liquid is the working fluid. It literally boils thanks to its low boiling point^[19][20] and thus exists in both a liquid and gas phase. The system takes advantage of a concept known as “latent heat” which is the heat (thermal energy) required to change the phase of a fluid (in this case two-phase dielectric mineral oil). The oil is only cooled by boiling and thus remains at the boiling point (“saturation temperature”). Energy transferred from the servers into the two-phase oil will cause a portion of it to boil off into a gas (this is the second phase of the oil). The gas rises above the liquid oil level where it contacts a condenser which is cooler than the saturation temperature. This causes the vapoured oil to condense back into a liquid form and fall (rain) back into the bath.^[21]

The term "semi-open bath" means, that when the system operates, it is sealed to avoid the evaporation of the coolant.

References