Data center liquid cooling cooling system solution

Data centers with air cooling can usually solve cabinet cooling within 12kW. As the power consumption per server unit increases, the server power that ordinary server cabinets of the original size can accommodate often exceeds 15kW, which has reached the ceiling of air convection cooling capacity compared to existing air-cooled data centers. As a technology with stronger heat dissipation capabilities, liquid cooling technology can support higher power density.

1. Advantages of liquid cooling

• It meets the heat dissipation needs of high-power density cabinets. The efficient cooling effect of liquid cooling effectively improves the efficiency and stability of server use, and at the same time, it can make the data center arrange more servers in a unit space and improve the efficiency of data center use.
• The circulation system consumes less energy and the system noise is low. The use of liquid working fluid with high specific heat reduces the energy consumption of cooling working fluid, and liquid cooling simplifies the heat exchange process, and also reduces the problem of part of the energy attenuation caused by turbulence at the end of the air cooling end in the process of transporting cold air in the room.
• Small footprint and easy site selection. The data center using liquid cooling system is simpler than the traditional air-cooled data center, removing the huge terminal air conditioning system, improving the building utilization, and arranging servers of sufficient scale in a small space.
• Lower TCO, lower operating PUE, and annual PUE can reach less than 1.2. Data center PUE with liquid cooling solution can be reduced by more than 0.15 compared to conventional chilled water systems with air cooling, allowing limited energy to be allocated more to computing power, thereby reducing operating costs and increasing computing power output.
• Waste heat recovery is easy to achieve. Compared with the traditional water temperature, the water temperature using the liquid cooling scheme is higher, the temperature difference is large, and the heat source taste and waste heat system efficiency are high.
• Strong adaptability. Cold plate type liquid cooling has strong compatibility, easy to develop, and does not need to change the original form and equipment materials; High space utilization, strong maintainability, layout conditions are similar to ordinary computer rooms, and can be directly compatible with the original refrigeration system (conventional chilled water system).

2. Primary side system based on cold plate liquid cooling scheme

For the different water temperature requirements at the end of the secondary side of liquid cooling, the liquid-cooled primary cooling source can use mechanical refrigeration system and natural cooling system. The mechanical refrigeration system includes air-cooled chilled water system and water-cooled chilled water system, which can provide medium-temperature chilled water from 12°C to 18°C; natural cooling is a cooling process that uses the cooling capacity of outdoor air without mechanical cooling when outdoor meteorological conditions permit, and the natural cooling system can be realized by open cooling towers, closed cooling towers and dry coolers, which can provide cooling water above 30°C. The form of liquid-cooled primary cooling source needs to be determined based on the water temperature requirements at the end of the secondary side and the outdoor environment of the project site.

3. Mechanical refrigeration system

• Air-cooled chilled water system

The air-cooled chilled water system is a way of chilled water preparation, mainly composed of air-cooled chillers, chilled water pumps and supporting facilities, and its liquid refrigerant is directly evaporated in its evaporator coil, so as to absorb heat from the chilled water outside the coil and cool it into a liquid state by air cooling.

The air-cooled chilled water system does not need to occupy a special computer room and does not need to install a cooling tower and pump room, the initial cost investment is low, the operation is convenient, there is no need for professional maintenance, there is no cooling water system, and it has the advantages of water saving and reduced maintenance costs. However, the air-cooled chiller is generally installed outdoors, the operation and maintenance environment are relatively harsh, the maintainability and reliability are not as good as the water-cooled chiller, and the air-cooled unit has poor cooling effect and low operating efficiency in summer.

• Water-cooled chilled water systems

The water-cooled chilled water system is a way of chilled water preparation, which is mainly composed of water-cooled chiller unit, chilled water pump, cooling water pump, cooling tower and supporting facilities, and its liquid refrigerant is directly evaporated in the evaporator coil to achieve heat absorption and cooling of the chilled water outside the coil, and cooled into liquid state by water cooling.

The water-cooled chilled water system has the advantages of low power consumption, good cooling effect throughout the year, high reliability and long service life. However, it requires special computer rooms, cooling towers, cooling water pumps, chilled water pumps and other equipment, the initial investment is large, and it requires circulating water, water consumption is large, and the unit body and cooling facilities need to be maintained, compared with the air-cooled unit, its maintenance cost is relatively high.

4. Natural cooling system

• Open cooling tower

The circulating cooling water is sprayed directly onto the cooling tower packing material, and the air flow in the cooling tower is driven by the fan, and the cooling circulating water is evaporated through the heat and mass exchange between the outdoor air and the cooling water, and the cooled circulating water is discharged at the bottom of the cooling tower (see Fig. 2). There is heat and mass exchange between the circulating cooling water and the outdoor air in the open cooling tower.

Figure 2 Schematic diagram of an open cooling tower

The initial investment and operating cost of the open cooling tower are low, the footprint is small, and the weight is light, but its operating water quality is poor, which is easy to cause scaling of the cooled heat exchanger, and is suitable for areas with good outdoor air quality. In addition, although the primary plate heat exchanger and cooling water pump can be added to avoid fouling of the core heat exchanger, the corresponding system is more complex and the initial investment is higher.

• Closed cooling tower

The closed cooling tower is a device that places the tubular heat exchanger in the tower and dissipates heat to the atmosphere through the heat exchange between the outdoor circulating air, spray water and the circulating cooling water in the tube (see Figure 3). The closed cooling tower has two systems: internal circulation and external circulation, and its internal circulation forms a closed system by docking with the cooled equipment, which brings the system heat to the cooling tower, that is, the internal circulation water transfers the heat to the atmosphere through the heat exchange coil; the external circulation is composed of circulating sprinkler pumps, water distribution systems, water collection trays and pipelines, and the external circulation water does not come into contact with the internal circulation water, but only absorbs the heat of the internal circulation water through the heat exchanger in the cooling tower, and then dissipates heat through direct contact with the air.

Figure 3 Schematic diagram of a closed cooling tower

The water quality of the closed cooling tower is better, the cooled heat exchanger is not easy to scale, and the service life is long, and the advantages of application in the outdoor environment are poor and the requirements for circulating water quality are obvious. However, the initial investment and operating cost of the closed cooling tower are high, with a large footprint and heavy weight.

• Dry cooler

The dry cooler is a dry cooler, and its working process does not consume water, and it cools the liquid in the pipe by taking the liquid inside the tube and the natural air outside the tube, reducing the temperature of the liquid in the tube, and achieving the purpose of cooling (see Figure 4). The refrigerant in the dry cooler usually uses ethylene glycol solution, and the solution concentration needs to be selected according to the extreme winter temperature of the project site.

Figure 4 Schematic diagram of the dry cooler

The dry cooler does not have a compressor, the overall power consumption is low, the service life of the unit is long, the initial investment is lower than that of the air-cooled chilled water system and the water-cooled chilled water system, but it is generally installed outdoors, the operating environment is relatively harsh, and in the summer hot heat dissipation area, it is necessary to configure a water spray cooling system or a wet curtain system to enhance heat exchange, resulting in an increase in the footprint.

5. Application scenarios of primary side systems

In a cold plate liquid cooling system, the heating device does not directly contact the liquid, but dissipates heat through direct contact with the cold plate containing the liquid, or the heat is transferred to the cold plate by the thermal conductive components, and then the heat is taken away through the liquid circulation inside the cold plate. Since the server chip and other heating devices do not need to be in direct contact with liquid, this method has less modification to the existing server chip components and auxiliary components, and is more operable, making it a highly mature and widely used liquid cooling cooling solution.

The secondary side is relatively stable and is configured by the coolant distribution unit (CDU) and the system architecture behind it. The primary side can be combined with a variety of usage conditions and scenarios. According to the refrigeration method, it is mainly divided into mechanical refrigeration and natural cooling refrigeration, and combined with the domestic situation, it is divided as follows:

Table 1 Reference values for primary and secondary supply temperatures

In areas with high heat and high humidity, the ambient temperature requirements of the computer room are high, and the direct use of closed cooling towers/dry coolers cannot directly meet the cooling requirements, and auxiliary mechanical refrigeration devices are required. The cold source usually adopts the combined cooling method of chiller + cooling tower (see Figure 5), which is highly adaptable and efficient, but consumes a lot of water and is not suitable for areas with water shortage.

• Option 1: Chiller + cooling tower (open) + plate replacement

There are various compositions of the primary side cooling source, and it is necessary to determine whether the water temperature needs to be lowered according to the local outdoor ambient temperature (including dry/wet bulb temperature) and the liquid inlet temperature of the liquid-cooled server. In addition, the water supply temperature should be about 2°C ~ 3°C higher than the indoor dew point temperature to prevent condensation (see Table 1).

Fig.5 Schematic diagram of cold tower + water cooling chiller + plate change system

The system is divided into two modes according to the outdoor temperature change:

Mode 1: The outdoor temperature is low, no need to turn on the cooler, and the cooling tower + plate replacement can meet the refrigeration requirements.

Mode 2: The water temperature of the cold tower effluent is higher than the demand of the CDU, and mechanical cooling and refrigeration are required to form a combination of cold tower + chiller.

• Option 2: Air-cooled chiller

The air-cooled chiller unit integrates the condenser, pump, compressor and other components into a whole, and is usually equipped with a dry cooler (free cold source module), which has a high degree of integration (see Figure 6). However, the latent heat of evaporation of water cannot be used, and the energy efficiency of the system is low, making it suitable for small environments and water-scarce areas.

Fig.6 Schematic diagram of air-cooled chiller

The usage mode is similar to Scenario 1, but also has two modes:

Mode 1: The outdoor temperature is low, no need to turn on the cooler, and the free cold source module can meet the refrigeration requirements.

Mode 2: If the free cooling module cannot meet the temperature requirements of the CDU and requires mechanical cooling and replenishment, the air-cooled chiller form is directly used.

• Option 3: Closed cooling tower / dry cooler

For low local temperatures throughout the year, closed towers/dry coolers can be used for direct cooling (see Figure 7), eliminating the need for mechanical cooling throughout the year.

Fig.7 Schematic diagram of a liquid cooling system of a closed cooling tower/dry cooler

The closed cooling tower system is still based on evaporative heat dissipation, which can output lower temperatures, and the water quality of the circulation system is better, which is friendly to CDU or other heat exchange equipment, but the water consumption is large. The dry cooler is large in size, and the cooling capacity of the single machine is small, but it is easy to arrange, equipped with a wet film, and can also be partially used for evaporative cooling.

The system is also divided into two modes:

Mode 1: Dry mode, no heat dissipation through water evaporation.

Mode 2: Wet mode, the system needs to remove heat through the latent tropics evaporated by the water jet, the closed cooling tower is the same as the open cooling tower. The dry cooler is initially cooled through the wet film of the air inlet, and then cooled down for the second time.

• Option 4: Open cooling tower

The cooling mode of the open cooling tower is exactly the same as that of the closed cooling tower (see Figure 8), except that the water channel of the open cooling tower is connected to the atmosphere, and the water quality is poor.

Fig. 8 Schematic diagram of cooling of an open cooling tower

The above scenarios are based on the demand for liquid cooling as the main consideration (see Table 2). In the actual operation of cold plate liquid cooling rooms, liquid cooling systems often need to be equipped with a small amount of air conditioning to meet the heat dissipation needs of non-liquid-cooled components in the server.

Table 2 Proposed scheme for primary side cooling sources

The secondary liquid circuit is a design that connects the cooling fluid manifold to the IT equipment from the cooling dispensing unit to the rack, and then back to the cooling dispensing unit through the manifold. The heat from the secondary cooling circuit is transferred to the primary cooling circuit through the plate heat exchanger of the cooling distribution unit, where it is finally discharged into the atmosphere or reused by heat recovery. 6. Overview of liquid cooling of the secondary side cold plate

As the power density of IT equipment increases, more efficient cooling technologies are needed to meet the increasing demand for computing power. Compared to traditional air cooling, liquid cooling solutions offer more efficient cooling efficiency. When to switch to liquid cooling depends on many different factors, including thermal performance requirements, power provision, PUE requirements, IT equipment density, cooling costs, and future IT equipment performance requirements and deployment strategies. In addition, whether to retrofit existing facilities or rebuild new data center rooms also needs to be considered in terms of TCO.

One of the immediate reasons for using liquid cooling is that traditional air-cooling solutions are no longer able to meet the cooling needs of IT equipment, so new solutions are needed to improve cooling capacity. There is no general guide to when or at what power level liquid cooling is needed for high-power components such as CPUs and GPUs. However, it should be noted that in addition to the cost analysis, there are some design considerations for liquid cooling solutions, such as the compatibility of all immersion materials in the cooling circuit with the cooling fluid used and long-term reliability, the cooling fluid used cannot be mixed with any other cooling fluid, etc.