Abstract

Transistor density trends till recently have been following Moore's law, doubling every generation resulting in increased power density. The computational performance gains with the breakdown of Moore's law were achieved by using multicore processors, leading to nonuniform power distribution and localized high temperatures making thermal management even more challenging. Cold plate-based liquid cooling has proven to be one of the most efficient technologies in overcoming these thermal management issues. Traditional liquid-cooled data center deployments provide a constant flow rate to servers irrespective of the workload, leading to excessive consumption of coolant pumping power. Therefore, a further enhancement in the efficiency of implementation of liquid cooling in data centers is possible. The present investigation proposes the implementation of dynamic cooling using an active flow control device to regulate the coolant flow rates at the server level. This device can aid in pumping power savings by controlling the flow rates based on server utilization. The flow control device design contains a V-cut ball valve connected to a microservo motor used for varying the device valve angle. The valve position was varied to change the flow rate through the valve by servomotor actuation based on predecided rotational angles. The device operation was characterized by quantifying the flow rates and pressure drop across the device by changing the valve position using both computational fluid dynamics and experiments. The proposed flow control device was able to vary the flow rate between 0.09 lpm and 4 lpm at different valve positions.

1 Introduction

Efficient thermal management of data centers is necessary for the continued safe and reliable operation of the Information Technology Equipment (ITE). Traditionally, air cooling has been widely utilized to dissipate heat loads from the servers using heat sinks and fans. However, low thermal mass and relatively high-power usage effectiveness (PUE) limit its implementation for cooling the latest high-performance central processing units (CPUs) and graphics processing units (GPUs) [1]. Several reports suggest that traditional air-cooled data centers expend approximately 30%–40% of energy for cooling purposes [2]. Recent trends have predicted that while the global data center energy trends are optimistic, the average PUE value still hovers around 1.58, which is mostly attributed to inefficient cooling methods employed [3].

Several methods have been explored to improve inefficiencies in data center cooling [46]. Among all these methods, direct liquid cooling (DLC) provides a highly efficient solution to meet the rising cooling demands. This is mainly due to the higher thermal conductivity and thermal mass of water or water-based coolants. A typical liquid-cooled server contains a manifold and cold plate arrangement instead of heat sinks and fans. The manifolds distribute the coolant to each of the servers, which then gets distributed to single or multiple cold plates, mounted on the processors. The earliest of the works in DLC dates to the 1960s but with growth in complementary metal oxide semiconductor (CMOS) technology in the 1990s, the heat loads in the electronic packages were significantly reduced and thus making air cooling yet again a viable technology [710]. However, with further developments in the CMOS technology, air cooling is once again reaching its limitations and the focus is again being shifted to liquid cooling [1113]. Liquid cooling can be deployed in data centers in two main ways. The first one can be a hybrid air and liquid cooling arrangement where the auxiliary components are cooled by air and the primary high-power heat-dissipating components using liquid cooling [14]. An advantage of this method is that it can be directly used in an air-cooled data center setup with minor changes around the compute systems. If the entire server is liquid-cooled, thermal pathways like cold rails for memory cooling are usually designed within the enclosure to ensure adequate cooling for all the components [15]. DLC can alleviate some of the issues related to fan acoustics and airborne contamination in air cooling [1619]. Also, single-phase DLC cold plates are easier to deploy than two-phase immersion-based and cold plate-based cooling technologies due to well-defined reliability and cooling guidelines without any concerns of using high global warming potential coolants [2025].

Typically, a constant flow rate is provisioned to the ITE in liquid-cooled data centers irrespective of the operating workloads of the servers. Significant savings in the pumping power can be attained if the flow rates are varied in proportion to the instantaneous server workload. Dynamic cooling has been explored in past for air-cooled data centers where control strategies for automatic computer room air conditioning (CRAC) units are experimentally validated. An improvement of 70% in data center energy performance was achieved by varying the parameters like CRAC fan speeds and electronically actuated vent tiles [26]. Other approaches include real-time monitoring of aisle temperatures and using machine learning algorithms to predict cooling requirements based on server workloads [2730]. On the package architecture side, thermally aware workload routes, workload migration, and varying server fan speed reduce power consumption [31].

This paper describes the design, development, and validation of a flow control device (FCD) that can be integrated directly with the ITE for dynamic variation of the coolant flow rate at the server level based on the server workload. A control strategy for such an FCD was recently published by Kasukurthy et al. [32,33] using computational fluid dynamics (CFD) where savings in pumping power of 64% were predicted. The present investigation details the concept design, development, and experimental testing of the said FCD. The proposed FCD design contains a V-cut ball valve connected to a microservo motor. The valve position is varied to change the flow rate through the valve by servomotor actuation based on predecided rotational angles. To validate the working of the FCD, variations in flow rates and the pressure drop across the device were analyzed by varying the valve position. The paper is divided into the following main sections: design and development of the FCD, CFD analysis of the final design, finite element analysis (FEA) analysis for valve, and discussion of the results.

2 Design and Development

2.1 Design Motivation.

The primary motivation behind this work was to develop an FCD that can contribute toward pumping power savings with a simple, cost-effective, and reliable design. Air-cooled data centers use actively controlled dampers to supply the target value of airflow rates to servers. Target delivery of flow rate not only optimizes the coolant transport energy cost but also reduces over-provisioning of coolant, thereby reducing energy expenditure related to heat removal. The purpose of this paper is to propose the design of a flow control device to control the flow of liquid in a cooling loop that can be directly integrated with liquid-cooled servers to save the energy required to pump the coolant. Another motivation behind the design of this FCD was the ease of manufacturing at a large scale and the ability to be manufacture using three-dimensional (3D) printing.

2.2 Concept Design.

Various valve designs and flow control strategies were studied during the concept design phase. Among the various designs studied, a butterfly valve was opted as the best option for the current flow control application, as shown in Fig. 1. After the valve design was finalized, the entire FCD concept design was first validated using CFD and it was found that variation in the flow through the device was not sufficient as shown in Fig. 2. This variation led us to design a better concept design to obtain the required variation of the flow rate that would suit data center liquid cooling applications. Subsequent design variations on valve design concluded that utilizing ball valves with V-cut would provide sufficient flow control at both high and low flow rates. Traditionally, ball valves have been used for flow on/off purposes and cannot produce precise control overflow. The more recent introduction of the V port in the ball valves has significantly improved the linear response of the flow in ball valves and allow a wider range in terms of flow rate control [34,35]. After analysis and experimentation on the butterfly valve design, it was concluded that the flow rate variation was very small; hence, a new design was chosen which had a ball valve with a V-cut. The final FCD comprises four parts: two symmetrical casing parts, one ball valve with a V cut, and a shaft sealing for a leakproof seal as shown in Fig. 3.

Fig. 1
Computer-aided design (CAD) model of the first concept design tested for the FCD where a Butterfly Valve design was used for the control valve
Fig. 1
Computer-aided design (CAD) model of the first concept design tested for the FCD where a Butterfly Valve design was used for the control valve
Close modal
Fig. 2
Comparison of flow rate variation between the final design of the FCD and initial design, with the final design showing the desired flow rate variation with changing valve angle
Fig. 2
Comparison of flow rate variation between the final design of the FCD and initial design, with the final design showing the desired flow rate variation with changing valve angle
Close modal
Fig. 3
CAD model of the final FCD design
Fig. 3
CAD model of the final FCD design
Close modal

2.3 Detailed Design.

Based on the variations tested during the concept design phase and the different valves tested using CFD, the ball valve with a V-cut was chosen as the final design. The inlet and outlet ports of the device were determined by considering standard pipe diameters, 3/8 inches, and a 10 mm ball valve. The detailed design of FCD was made using SolidWorks and the data file was exported to ansysfluent, a commercial CFD package, for further analysis including the determination of the flow characteristics of the FCD designs. After obtaining the required results, the SolidWorks design file was exported to a slicing software and was manufactured using extrusion 3D printing technology. The advantage of 3D printing is that it has a low manufacturing time compared to conventional manufacturing techniques and the production cost is very low. Also, design changes can be made without considering the tooling costs and extra lead time, and related high costs.

3 Computational Modeling

Over the years, CFD has become an important tool to ascertain the design flaws and optimize them further rather than going through intensive experimentation of manufactured models. To visualize the 3D flow inside the FCD and analyze the flow rate variation with changing the angle of the ball valve, momentum, energy, and mass conservation equations were solved using ansysfluent [36]. After preparing the CAD model on SolidWorks, the model was imported to the CFD software where it was first meshed using the ansys Mesh tool and then solved in the CFD solver. The following sections will detail the model setup and solution process in detail.

3.1 Meshing.

After importing the geometry to the CFD package, the meshing of only the flow domain was carried out. A good quality mesh is essential for the accuracy of results in CFD modeling. An efficient mesh is composed of a sufficient number of elements near the walls, especially for fluid flows, to resolve the shear effects in the near-wall region. Since the flow variation occurs only in the internal cavity of the device, it was not necessary to include all the components in the meshing and CFD solution. For this purpose, the fluid volume was extracted, and the solid bodies of the design were suppressed. This was repeated for various cases to be simulated in this study, generating four different tetrahedral meshed volumes for four angles studies for flow rate variation. To verify the quality of the mesh, the element skewness and orthogonality were monitored. For a mesh to be of good quality, the value of orthogonality should be closer to 1, and mesh skewness should be close to 0. The minimum element size was set to 0.3 mm and the maximum element size was 0.6 mm. The growth rate was set to the default of 1.2. for inflation, with the number of inflation layers as 10, and the growth rate was set to a default value of 1.2. In all the simulation cases studied, average orthogonality of 0.8 and average skewness of 0.2 were reported. The optimum mesh element count of 739,416 was used based on the mesh independence study as detailed in Table 1. The final mesh characteristics used in the study are summarized in Table 2. A summary of the boundary conditions used for setting up the CFD and FEA simulations domain is shown in Fig. 4.

Fig. 4
Summary of boundary conditions used in CFD (top) and FEA simulations (bottom)
Fig. 4
Summary of boundary conditions used in CFD (top) and FEA simulations (bottom)
Close modal
Table 1

Mesh independence results at 45 deg open-angle at 1 lpm flow rate

Number of elementsPressure drop (kPa)
315,89710.98
563,21711.63
739,41612.97
935,56913.0
Number of elementsPressure drop (kPa)
315,89710.98
563,21711.63
739,41612.97
935,56913.0
Table 2

CFD mesh characteristics for extracted fluid volume

Volume5898.8 mm3
Bodies3
Active bodies1
Nodes142942
Element739416
Volume5898.8 mm3
Bodies3
Active bodies1
Nodes142942
Element739416

3.2 Computational Fluid Dynamics and Finite Element Analysis Setup.

As the purpose of the CFD simulation was hydraulic verification of the FCD, the CFD simulation was done by deactivating the energy equation. This further deactivates all the thermal inputs possible in the simulation and saves considerable simulation time as well. The pressure drop and flow rate characterizations were carried out at a constant temperature of 22 °C due to which a constant value of the thermo-physical properties was used in the CFD solver. A pressure-based solver algorithm was used to simulate a velocity field through the FCD by a correction in the pressure equation in the continuity equation. Various turbulence models have been implemented in the literature based on specific applications [36]. Literature shows that, for cases with internal flows, kω turbulence model is preferable. Standard and shear stress transport (SST) are the two available variants of the kω model. It was believed that since flow across the valve in various positions will create large adverse pressure gradients, the SST kω model should be used because of its superiority in predicting such flows [37]. This has also been validated in literature where a comparison of various turbulence models was validated for a butterfly valve [38]. This study showed that the SST kω model predicted the values of the drag coefficient across the valve with the least error when compared to the standard experimental results. Also, the CFD results become less sensitive to grid size when using this turbulence model. The general governing equations of continuity and momentum used by the solver to are given below in Eqs. (1) and (2)

Continuity equation:
ρt+.(ρu)=Sm
(1)
Momentum:
ρdudt=p+·(μu)+f
(2)
Transport equation for the SST kω:
t(ρk)+xi(ρkui)=xj(kxj(гk))+G̃kYk+Sk
(3)
t(ρω)+xi(ρωui)=xj(ωxj(гω))+GωYω+Dω+Sω
(4)

In Eqs. (3) and (4), G̃k represents the generation of turbulence kinetic energy due to mean velocity gradients and Gω represents the generation of ω. гk and гω represent the effective diffusivity of k and ω, respectively. Yk and Yω represent the dissipation of k and ω due to turbulence and Dω represents the cross-diffusion term. Sk and Sω are user-defined source terms which were not utilized in the current CFD modeling setup [36]. After choosing the relevant turbulence models, the fluid boundary conditions were chosen based on the anticipated experimental test conditions. The coolant properties of 25% propylene glycol (PG25) are used in the simulation to determine the variation in flow rates and pressure drop with changes in the angle of the valve. A summary of the thermophysical properties of the temperature-dependent properties of the coolant is shown in Table 3. In the boundary conditions, the inlet velocity was calculated using the flow rate for the designed area. The outlet was set as a static pressure outlet with 0 Pa gauge pressure. The walls of the device were treated as adiabatic walls. A steady-state simulation with convergence criteria of 1e-3 for continuity and 1e-6 for 3D velocities, k, and ω values was used.

Table 3

Thermo-physical properties of 25% propylene glycol in temperature range of interest

Temperature (°F)Density (kg/m3)Viscosity (cP)Specific heat (J/kg °C)Thermal conductivity (W/mK)
7010232.4539140.470
8010202.0539300.475
9010181.7439430.480
10010151.4939550.486
12010091.1439850.489
14010020.9040100.501
1609950.7340390.506
Temperature (°F)Density (kg/m3)Viscosity (cP)Specific heat (J/kg °C)Thermal conductivity (W/mK)
7010232.4539140.470
8010202.0539300.475
9010181.7439430.480
10010151.4939550.486
12010091.1439850.489
14010020.9040100.501
1609950.7340390.506
A stress test was performed in ansys Static structural to find the working limits of the flow control device for anticipated flow rates. Internal pressure loads on the walls of the FCD were imported from the results of CFD simulations. A tetrahedral mesh with physical preference set to mechanical the element order was chosen and was set to program control. The minimum mesh element size was set to 0.5 mm with a target mesh quality of 0.0500. The mesh smoothing was set to medium, resulting in 912689 nodes and 534030 mesh elements. The ambient temperature of the environment temperature was set to 22 °C. A fixed support boundary was given on the sidewalls and the pressure load was imported from the CFD simulation running for the highest flow rate tested, which was 4 lpm. The material properties of ABS plastic were selected as shown in Table 4. The solver uses the relation given in Eq. (5) for solving linear static structural analysis problems. The basic assumptions in this theory include a linear elastic behavior and small deflections of the material
[K]{x}={F}
(5)
Table 4

Material properties of 3D printing material used for structural analysis of the FCD

Density1.04e-006 kg/mm3
Young's modulus239 MPa
Poisson's ratio0.399
Bulk modulus3943.9 MPa
Shear modulus854.18 MPa
Tensile yield strength41.4 MPa
Tensile ultimate strength44.3 MPa
Density1.04e-006 kg/mm3
Young's modulus239 MPa
Poisson's ratio0.399
Bulk modulus3943.9 MPa
Shear modulus854.18 MPa
Tensile yield strength41.4 MPa
Tensile ultimate strength44.3 MPa

Here, [K] represents the stiffness matrix of the entire geometry, {x} represents the unknown nodal displacements, and {F} represents the external forces or applied loads.

4 Experimental Setup

Experimentation was carried out in a controlled environment. The cooling liquid used for the loop is PG25 and care was taken to avoid contamination to get precise flow rates. A simplified schematic of the experimental setup is shown in Fig. 5. Pressure sensors were attached near the inlet and outlet of the flow control device to determine the pressure drops for various valve angles. The flow rates were obtained using ultrasonic flow sensors, which use ultrasonic frequencies to calculate the flow rates. The loop also consists of a sealed coolant reservoir to hold the cooling liquid. A description of the models and sensor accuracy is shown in Table 5. A 15 W DC pump was used to circulate the fluid through the entire test loop. Variation in flow rates across the FCD was achieved using changing the input voltage to the pump. The pressure and flow sensors were connected to the data acquisition system (DAQ). The DAQ is used for sampling electric signals such as voltage, current, and frequency and convert these electrical signals into numerical values such as temperature and pressure using the software. The flow and pressure sensors were supplied a voltage of 20 V from a different DC power source. The angle of the valve was changed with the help of a servomotor, which is controlled by the Arduino microcontroller. The Arduino was programmed using Arduino IDE in such a way that a change in the resistance value of the potentiometer will change the angles of the valve. Observations were recorded at valve angles of 0, 30, 45, and 90 deg. Data regarding pressure drop from pressure sensors and variation in flow rate with a change in valve angle were also recorded. The final fabricated FCD and its integration in a cooling loop are shown in Fig. 6.

Fig. 5
Schematic of the experimental setup used for characterizing the flow control device
Fig. 5
Schematic of the experimental setup used for characterizing the flow control device
Close modal
Fig. 6
Final assembly of the 3D printed FCD (top) and the integration of the FCD with one of the TTV and cold plate assembly in the rack (bottom)
Fig. 6
Final assembly of the 3D printed FCD (top) and the integration of the FCD with one of the TTV and cold plate assembly in the rack (bottom)
Close modal
Table 5

Details of sensor accuracy, measurement range, and operating voltages of the sensors used in the experiment

SensorOperating voltageRange of measurementAccuracy
Keyence FDX-A120–30 V0.02–20 L/min±0.003 ml/min
Honeywell pressure sensors4.75–5.25 V0–50 Psi±0.25%
K-type thermocouple0–400 °C±0.75%
SensorOperating voltageRange of measurementAccuracy
Keyence FDX-A120–30 V0.02–20 L/min±0.003 ml/min
Honeywell pressure sensors4.75–5.25 V0–50 Psi±0.25%
K-type thermocouple0–400 °C±0.75%

5 Results

5.1 Computational Fluid Dynamics Results.

After achieving convergence with an appropriate mesh as discussed earlier with the desired accuracy, the CFD results were analyzed in the form of the pressure drops across the device for different valve angles. This variation in the valve angle was repeated for the three chosen values of flow rates of 0.6, 1.0, and 1.5 lpm. The working of the FCD design was verified from the CFD results where a change in flow rate was obtained at all angles varied in the simulations. Figures 79 depict the variation of velocity at valve angles of 0, 75, and 45 deg, respectively. The results obtained from the simulations were compared with experimental data and showed a close agreement in the flow rate variation trend with changing valve angle. As opposed to the first design of the FCD, it was observed that with the closing of the valve, the flow rate decreased across the valve, and the pressure drop increased. The observations obtained for a particular condition had similar values with average error possibilities of 14%. The data in Table 6 summarize flow characteristics when the flow rate is 1 lpm when the valve is fully open at a 0 deg angle.

Fig. 7
Flow characteristics of the FCD showing velocity profile inside the device cavity when the valve is fully opened
Fig. 7
Flow characteristics of the FCD showing velocity profile inside the device cavity when the valve is fully opened
Close modal
Fig. 8
Flow characteristics of the FCD showing velocity profile inside the device cavity at a valve opening angle of 75 deg
Fig. 8
Flow characteristics of the FCD showing velocity profile inside the device cavity at a valve opening angle of 75 deg
Close modal
Fig. 9
Flow characteristics of the FCD showing velocity profile inside the device cavity at a valve opening angle of 45 deg
Fig. 9
Flow characteristics of the FCD showing velocity profile inside the device cavity at a valve opening angle of 45 deg
Close modal
Table 6

Summary of CFD results for pressure drop across the FCD at different valve angles

At 1.0 lpm @ fully open
v = 0.2122 m/s
AngleInlet pressure (kPa)Outlet pressure (kPa)Flow rate (lpm)Pressure drop (kPa)
012.14810.1250.991.53
301380.8253.34
45162.370.51212.97
90201.43015.9
At 1.0 lpm @ fully open
v = 0.2122 m/s
AngleInlet pressure (kPa)Outlet pressure (kPa)Flow rate (lpm)Pressure drop (kPa)
012.14810.1250.991.53
301380.8253.34
45162.370.51212.97
90201.43015.9

5.2 Stress Analysis.

The stress analysis was done using the Static Structural module in ansysWorkbench. The CFD data of the forces exerted by the fluid flow on the valve were exported to the structural module of the software. This was done to verify the structural integrity (based on material fill ratio in 3D printing) of the device at high lpm that can lead to leakages or the device or the device bursting under pressure. The ambient conditions for the simulation setup were kept at a constant temperature of 22 °C. A fixed support boundary condition was given on device outer walls and the internal pressure force was imported from the CFD simulation for the highest flow rate, 4 lpm, to identify the maximum stresses and deformations. The results of the FEA simulations are shown in Figs. 1012. The value for maximum deformation, observed near the inlet port, was 7.9 × 10−4 mm with maximum equivalent stress of 0.94 MPa, and the average deformation was 1.13 × 10−4 with average stress of 0.076 MPa. The average factor of safety for the device was obtained to be 15. The results from FEA analysis verify that the FCD body with current dimensions and material will not fail at pressure conditions at the maximum flow rate of 4 lpm. This, however, needs validation experimentally at different temperatures for prolonged periods and cyclic flow rates (internal pressure) to ascertain the device lifetime and failure rates.

Fig. 10
Equivalent (vonMises) stress on the FCD walls at a flow rate of 4 lpm in isometric (top) and front view (bottom)
Fig. 10
Equivalent (vonMises) stress on the FCD walls at a flow rate of 4 lpm in isometric (top) and front view (bottom)
Close modal
Fig. 11
Contours for total deformation in FCD material with a maximum deformation toward the inlet and around the valve shaft
Fig. 11
Contours for total deformation in FCD material with a maximum deformation toward the inlet and around the valve shaft
Close modal
Fig. 12
Contours for the equivalent elastic strain on the FCD from FEA analysis
Fig. 12
Contours for the equivalent elastic strain on the FCD from FEA analysis
Close modal

5.3 Experimental Results.

To ascertain the results obtained for the hydraulic performance of the proposed FCD, an experimental study was carried out under the same flow conditions. The pressure drop characteristics and flow rate variation at different valve angles are shown in Figs. 1315 and summarized in Table 7. The results showed that for a maximum flow rate of 4.0 lpm, a minimum flow rate of 0.2 lpm can be achieved when the FCD is in the fully closed position. It was observed that the maximum change in flow rate occurs between the range of 30–90 deg. The largest variation is obtained for the higher value of flow rates but reduces and becomes rather constant for flow rates less than 0.5 lpm.

Fig. 13
Variation of the flow rate through the FCD with the changing angle of the valve
Fig. 13
Variation of the flow rate through the FCD with the changing angle of the valve
Close modal
Fig. 14
Pressure drop variation across the FCD with changing valve angle
Fig. 14
Pressure drop variation across the FCD with changing valve angle
Close modal
Fig. 15
Device impedance curve with varying flow rates with the valve is in a fully open condition
Fig. 15
Device impedance curve with varying flow rates with the valve is in a fully open condition
Close modal
Table 7

Summary of experimental results for pressure drop variation with changing valve angle

At 1 lpm @ fully open
Inlet pressure (kPa)Outlet pressure (kPa)Flow rate (lpm)Experimental pressure drop (kPa)CFD pressure drop (kPa)
Angle
011.9810.4251.52962.0231.53
3012.4569.1250.86553.34
4516.9794.0050.51213.6312.97
9018.5632.663018.5715.9
At 1 lpm @ fully open
Inlet pressure (kPa)Outlet pressure (kPa)Flow rate (lpm)Experimental pressure drop (kPa)CFD pressure drop (kPa)
Angle
011.9810.4251.52962.0231.53
3012.4569.1250.86553.34
4516.9794.0050.51213.6312.97
9018.5632.663018.5715.9

A comparison of results from the experimental study and CFD analysis is shown in Figs. 1618. The comparison of results of flow rates with varying angles of the ball valve shows that a maximum discrepancy of 20% was observed in both the results at a flow rate of 1 lpm. This variation reduces with increasing the angle of the valve or when the valve is fully opened. A large variation in the results at fully closed conditions could be due to unresolved flow disturbances due to the current mesh size. A similar trend is observed in the pressure drop characteristics with varying angles of the valve. This is expected as there is a linearly inverse relationship between pressure drop and flow rate. The maximum error between the pressure drop variation was 16% for at 1 lpm flow rate when the valve is fully closed. The error between the results when the valve was fully opened was within 3% for both pressure drop and flow rate. This is because there could be little to no flow turbulence created by the valve and the CFD code was successfully able to replicate the experimental flow conditions. Based on the plotted graphical data, it was observed that with a change in angle, there is a drop in the flow rate and the pressure drop across the valve increases. The major variation in pressure drops and flow occurs between the angle of 30–90 deg angle. Similar trends were seen in experimental and CFD data when calculated for a starting flow rate of 0.6 lpm, 1 lpm, and 1.5 lpm when the valve is fully open.

Fig. 16
Comparison of the trends in CFD and experimental result for flow rate variation with changing valve angle
Fig. 16
Comparison of the trends in CFD and experimental result for flow rate variation with changing valve angle
Close modal
Fig. 17
Comparison of CFD and experimental results for variation in pressure drop with changing valve angle
Fig. 17
Comparison of CFD and experimental results for variation in pressure drop with changing valve angle
Close modal
Fig. 18
Comparison of the impedance curve obtained from CFD analysis and experimental data
Fig. 18
Comparison of the impedance curve obtained from CFD analysis and experimental data
Close modal

6 Conclusion and Future Work

Direct liquid cooling using cold plates continues to be the most popular method to overcome the challenges of thermal challenges in high-performance computing systems. While DLC by itself provides an energy-efficient way of dissipating high heat fluxes, further enhancements are possible such as reducing energy consumption related to coolant transport within the data center. Traditional DLC only provides a constant flow rate, typically using a centralized pumping system that delivers a constant flow rate to each server irrespective of the instantaneous workloads of the servers. DLC also provides notable advantages and better equipment reliability of ITE as issues related to contamination and fan acoustics [2933]. It also provides higher heat flux values than single-phase immersion cooling and overcomes the issues related to material compatibility and vapor pressure in two-phase open bath immersion cooling and two-phase cold plates [3438].

The FCD design proposed in this study will enable workload-based provisioning of the coolant to each of the servers in a rack. A recent study by the authors on an open compute server retrofitted with cold plates with distributed pumps showed considerable power savings when the flow control at the server level is dynamically varied [39]. The proposed FCD will therefore yield considerable pumping power savings by optimizing the pumping power of the rack-level pumps. Out of the two final designs developed, the FCD design with a V-cut ball valve produced the desired flow rate variation that would be sufficient to cool heat loads in the current high-powered electronic packages. The results from the CFD study and experimental testing show a good agreement in depicting the hydraulic performance data of the FCD. A minimum flow rate of 0.09 lpm and a maximum flow rate of 4 lpm were achieved by varying the angle of the valve inside the FCD cavity. Future work on this subject will investigate FCD implementation at the server and rack level for flow control at the rack level to assess the thermal and flow characteristics along with the maximum achievable power savings. The future work on the FCD will also present an overview of the possible cost savings not only due to reduced pumping power but also provide an estimate of a simple payback period after device deployment. Further studies will also focus on device reliability under operating conditions to assess the device lifetime in the field under transient operating conditions.

Funding Data

  • National Science Foundation IUCRC (No. IIP-1738811; Funder ID: 10.13039/100000001).

Nomenclature

     
  • ASHRAE =

    American Society of Heating, Refrigeration and Air-conditioning Engineers

  •  
  • CFD =

    computational fluid dynamics

  •  
  • CMOS =

    complementary metal oxide semiconductor

  •  
  • CRAC =

    computer room air-conditioning

  •  
  • DAQ =

    data acquisition unit

  •  
  • DLC =

    direct liquid cooling

  •  
  • FCD =

    flow control device

  •  
  • FEA =

    finite element analysis

  •  
  • ITE =

    Information Technology Equipment

  •  
  • PUE =

    power usage effectiveness

References

1.
Gao
,
T.
,
Tang
,
H.
,
Cui
,
Y.
, and
Luo
,
Z.
,
2018
, “
A Test Study of Technology Cooling Loop in a Liquid Cooling System
,” 17th IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (
ITherm
), San Diego, CA, May 29–June 1, pp.
740
747
.10.1109/ITHERM.2018.8419519
2.
Iyengar
,
M.
,
2010
, “
Energy Consumption of Information Technology Data Centers
,”
J. Electron. Cool.
,
16
(
4
), pp.
28
31
.https://www.electronicscooling.com/2010/12/energy-consumption-of-information-technology-data-centers/
3.
Lawrence
,
A.
,
2020
, “
Data Center PUEs Flat Since 2013
,”
Global Uptime Institute Survey
, Seattle WA, accessed Jan. 7, 2021, https://journal.uptimeinstitute.com/data-center-pues-flat-since-2013/
4.
Hoang
,
C. H.
,
Khalili
,
S.
,
Ramakrisnan
,
B.
,
Rangarajan
,
S.
,
Hadad
,
Y.
,
Radmard
,
V.
,
Sikka
,
K.
,
Schiffres
,
S.
, and
Sammakia
,
B.
,
2020
, “
An Experimental Apparatus for Two-Phase Cooling of High Heat Flux Application Using an Impinging Cold Plate and Dielectric Coolant
,” 36th Semiconductor Thermal Measurement, Modeling & Management Symposium (
SEMI-THERM
), San Jose, CA, Mar. 16–20, pp.
32
38
.10.23919/SEMI-THERM50369.2020.9142831
5.
Shahi
,
P.
,
Agarwal
,
S.
,
Saini
,
S.
,
Niazmand
,
A.
,
Bansode
,
P.
, and
Agonafer
,
D.
,
2020
, “
CFD Analysis on Liquid Cooled Cold Plate Using Copper Nanoparticles
,”
ASME
Paper No. IPACK2020-2592.10.1115/IPACK2020-2592
6.
Niazmand
,
A.
,
Chauhan
,
T.
,
Saini
,
S.
,
Shahi
,
P.
,
Bansode
,
P. V.
, and
Agonafer
,
D.
,
2020
, “
CFD Simulation of Two-Phase Immersion Cooling Using FC-72 Dielectric Fluid
,”
ASME
Paper No. IPACK2020-2595.10.1115/IPACK2020-2595
7.
Chu
,
R. C.
,
Simons
,
R. E.
,
Ellsworth
,
M. J.
,
Schmidt
,
R. R.
, and
Cozzolino
,
V.
,
2004
, “
Review of Cooling Technologies for Computer Products
,”
IEEE Trans. Device Mater. Reliab.
,
4
(
4
), pp.
568
585
.10.1109/TDMR.2004.840855
8.
Ellsworth
,
M. J.
,
Campbell
,
L. A.
,
Simons
,
R. E.
,
Iyengar
,
M. K.
,
Schmidt
,
R. R.
, and
Chu
,
R. C.
,
2008
, “
The Evolution of Water Cooling for Large IBM Large Server Systems: Back to the Future
,”
11th Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (ITherm)
, Orlando, FL, May 28–31, pp.
266
274
.
9.
Goth, G. F., Arvelo, A., Eagle, J., Ellsworth, M. J., Marston, K. C., Sinha, A. K., and Zitz, J. A., 2012, “Thermal and Mechanical Analysis and Design of the IBM Power 775 Water Cooled Supercomputing Central Electronics Complex,”
13th InterSociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems
, San Diego, CA, May 30–June 1, pp. 700–709.10.1109/IT HERM.2012.6231496
10.
Schmidt
,
R. R.
,
2005
, “
Liquid Cooling is Back
,”
Electron. Cool.
,
11
(
3
), pp.
34
38
.https://www.electronicscooling.com/2005/08/liquid-cooling-is-back/
11.
Patrizio
,
A.
,
2018
, “
Lenovo Introduces New Water-Cooled Server Technology
,” Network World, Framingham, MA, accessed Feb. 26, 2018, https://www.networkworld.com/article/3258646/data-center/lenovo-introduces-newwater-cooled-server-technology.html
12.
Koblentz
,
E.
,
2018
, “
How to Get Started With Liquid Cooling for Servers and Data Center Racks
,” Data Centers Trends Newsletter, TechRepublic, Nashville, TN, U.S. edition, accessed July 8, 2018, https://www.techrepublic.com/article/how-to-get-started-with-liquid-cooling-for-servers-and-data-center-racks/
13.
Iyengar
,
M.
,
David
,
M.
,
Parida
,
P.
,
Kamath
,
V.
,
Kochuparambil
,
B.
,
Graybill
,
D.
,
Schultz
,
M.
,
2012
, “
Server Liquid Cooling With Chiller-Less Data Center Design to Enable Significant Energy Savings
,” 28th Annual IEEE Semiconductor Thermal Measurement and Management Symposium (
SEMI-THERM
), San Jose, CA, Mar. 18–22, pp.
212
223
.10.1109/STHERM.2012.6188851
14.
Fan
,
Y.
,
Winkel
,
C.
,
Kulkarni
,
D.
, and
Tian
,
W.
,
2018
, “
Analytical Design Methodology for Liquid Based Cooling Solutions for High TDP CPUs
,” 17th IEEE Intersociety Conference on Thermal and Thermomechanical Phenomena in Electronic Systems (
ITherm
), San Diego, CA, May 29–Jun. 1, pp.
582
586
.10.1109/ITHERM.2018.8419562
15.
Gullbrand
,
J.
,
Luckeroth
,
M. J.
,
Sprenger
,
M. E.
, and
Winkel
,
C.
,
2019
, “
Liquid Cooling of Compute System
,”
ASME J. Electron. Packag.
,
141
(
1
), p.
010802
.10.1115/1.4042802
16.
Shah
,
J. M.
,
Anand
,
R.
,
Saini
,
S.
,
Cyriac
,
R.
,
Agonafer
,
D.
,
Singh
,
P.
, and
Kaler
,
M.
,
2019
, “
Development of a Technique to Measure Deliquescent Relative Humidity of Particulate Contaminants and Determination of the Operating Relative Humidity of a Data Center
,”
ASME
Paper No. IPACK2019-6601.10.1115/IPACK2019-6601
17.
Saini
,
S.
,
2018
,
Airflow Path and Flow Pattern Analysis of Sub-Micron Particulate Contaminants in a Data Center With Hot Aisle Containment System Utilizing Direct Air Cooling
,
The University of Texas at Arlington
,
Arlington, TX
.
18.
Saini
,
S.
,
Shahi
,
P.
,
Bansode
,
P.
,
Siddarth
,
A.
, and
Agonafer
,
D.
,
2020
, “
CFD Investigation of Dispersion of Airborne Particulate Contaminants in a Raised Floor Data Center
,”
36th Semiconductor Thermal Measurement, Modeling & Management Symposium
(
SEMI-THERM
), San Jose, CA, Mar. 16–20, pp.
39
47
.10.23919/SEMITHERM50369.2020.9142865
19.
Thirunavakkarasu
,
G.
,
Saini
,
S.
,
Shah
,
J. M.
, and
Agonafer
,
D.
,
2018
, “
Air Flow Pattern and Path Flow Simulation of Airborne Particulate Contaminants in a High-Density Data Center Utilizing Airside Economization
,”
ASME
Paper No. IPACK2018-8436.10.1115/IPACK2018-8436
20.
Saini
,
S.
,
Adsul
,
K. K.
,
Shahi
,
P.
,
Niazmand
,
A.
,
Bansode
,
P.
, and
Agonafer
,
D.
,
2020
, “
CFD Modeling of the Distribution of Airborne Particulate Contaminants Inside Data Center Hardware
,”
ASME
Paper No. IPACK2020-2590.10.1115/IPACK2020-2590
21.
Gandhi
,
D.
,
Chowdhury
,
U.
,
Chauhan
,
T.
,
Bansode
,
P. V.
,
Saini
,
S.
,
Shah
,
J. M.
, and
Agonafer
,
D.
,
2019
, “
Computational Analysis for Thermal Optimization of Server for Single Phase Immersion Cooling
,”
ASME
Paper No. IPACK2019-6587.10.1115/IPACK2019-6587
22.
Shinde
,
P. A.
,
Bansode
,
P. V.
,
Saini
,
S.
,
Kasukurthy
,
R.
,
Chauhan
,
T.
,
Shah
,
J. M.
, and
Agonafer
,
D.
,
2019
, “
Experimental Analysis for Optimization of Thermal Performance of a Server in Single Phase Immersion Cooling
,”
ASME
Paper No. IPACK2019-6590.10.1115/IPACK2019-6590
23.
Niazmand
,
A.
,
Murthy
,
P.
,
Saini
,
S.
,
Shahi
,
P.
,
Bansode
,
P.
, and
Agonafer
,
D.
,
2020
, “
Numerical Analysis of Oil Immersion Cooling of a Server Using Mineral Oil and Al2O3 Nanofluid
,”
ASME
Paper No. IPACK2020-2662.10.1115/IPACK2020-2662
24.
Kumar
,
A.
,
Shahi
,
P.
, and
Saha
,
S. K.
,
2018
, “
Experimental Study of Latent Heat Thermal Energy Storage System for Medium Temperature Solar Applications
,” Proceedings of the 4th World Congress on Mechanical, Chemical, and Material Engineering (
MCM'18
), Madrid, Spain, Aug. 16–18, pp.
16
18
.10.11159/htff18.152
25.
Hoang, C. H., Rangarajan, S., Khalili, S., Ramakrisnan, B., Radmard, V., Hadad, Y., Schiffres, S., and Sammakia, B., 2021, “Hybrid Microchannel/Multi-Jet Two-Phase Heat Sink: A Benchmark and Geometry Optimization Study of Commercial Product,”
Int. J. Heat Mass Transfer
, 169, p. 120920.10.1016/j.ijheatmasstransfer.2021.120920
26.
Boucher
,
T. D.
,
Auslander
,
D. M.
,
Bash
,
C. E.
,
Federspiel
,
C. C.
, and
Patel
,
C. D.
,
2006
, “
Viability of Dynamic Cooling Control in a Data Center Environment
,”
ASME J. Electron. Packag.
,
128
(
2
), pp.
137
144
.10.1115/1.2165214
27.
Gandhi
,
A.
,
2013
, “Dynamic Server Provisioning for Data Center Power Management,”
Ph.D. thesis
, Computer Science Department, Carnegie Mellon University, Pittsburgh, PA
.http://reportsarchive.adm.cs.cmu.edu/anon/2013/CMU-CS-13-110.pdf
28.
Hazelwood
,
K.
,
Bird
,
S.
,
Brooks
,
D.
,
Chintala
,
S.
,
Diril
,
U.
,
Dzhulgakov
,
D.
, Fawzy, M.,
2018
, “
Applied Machine Learning at Facebook: A Datacenter Infrastructure Perspective
,” IEEE International Symposium on High Performance Computer Architecture (
HPCA
), Vienna, Austria, Feb. 24–28, pp.
620
629
.10.1109/HP CA.2018.00059
29.
Arghode
,
V. K.
,
Sundaralingam
,
V.
, and
Joshi
,
Y.
,
2016
, “
Airflow Management in a Contained Cold Aisle Using Active Fan Tiles for Energy Efficient Data-Center Operation Airflow Management in a Contained Cold Aisle Using Active Fan Tiles for Energy Efficient Data-Center
,”
Heat Transfer Eng.
,
37
(
3–4
), pp.
246
256
.10.1080/01457632.2015.1051386
30.
KhaliliMohsenian
,
S.
,
Desu
,
G.
,
Ghose
,
A. K.
, and
Sammakia
,
B.
,
2019
, “
Airflow Management Using Active Air Dampers in Presence of a Dynamic Workload in Data Centers
,”
35th Semiconductor Thermal Measurement, Modeling and Management Symposium
(
SEMI-THERM
), San Jose, CA, Mar. 18–22, pp.
101
110
.https://ieeexplore.ieee.org/document/9165298
31.
Xu
,
H.
,
Feng
,
C.
, and
Li
,
B.
,
2015
, “
Temperature Aware Workload Management in Geo-Distributed Data Centers
,”
IEEE Trans. Parallel Distrib. Syst.
,
26
(
6
), pp.
1743
1753
.10.1109/TPDS.2014.2325836
32.
Kasukurthy
,
R.
,
2019
, “
Design and Optimization of Energy Conserving Solutions in Data Center Application
,” Ph.D. dissertation,
The University of Texas at Arlington
,
Arlington, TX
.
33.
Kasukurthy
,
R.
,
Rachakonda
,
A.
, and
Agonafer
,
D.
,
2020
, “
Design and Optimization of Control Strategy to Reduce Pumping Power in Dynamic Liquid Cooling
,”
ASME. J. Electron. Packag.
, 143(3), p. 031001.10.1115/1.4049018
34.
Tao
,
J.
,
Lin
,
Z.
,
Ma
,
C.
,
Ye
,
J.
,
Zhu
,
Z.
,
Li
,
Y.
, and
Mao
,
W.
,
2019
, “
An Experimental and Numerical Study of Regulating Performance and Flow Loss in a V-Port Ball Valve
,”
ASME. J. Fluids Eng.
,
142
(
2
), p.
021207
.10.1115/1.4044986
35.
Chern
,
M.
, and
Wang
,
C.
,
2004
, “
Control of Volumetric Flow-Rate of Ball Valve Using V-Port
,”
ASME J. Fluids Eng.
,
126
(
3
), pp.
471
481
.10.1115/1.1760536
36.
ANSYS® FLUENT
,
2019
, Release R3, ANSYS Fluent User's Guide, ANSYS, Inc, Canonsburg, PA.
37.
Blazek
,
J.
,
2015
,
Computational Fluid Dynamics: Principles and Applications
,
Butterworth-Heinemann
, Oxford, UK.
38.
Lin
,
F.
, and
Schohl
,
G. A.
,
2004
, “
CFD Prediction and Validation of Butterfly Valve Hydrodynamic Forces
,”
World Water and Environmental Resources Congress 2004
, June 27–July 1, Salt Lake City, UT, pp.
1
8
.10.1061/40737(2004)232
39.
Shahi
,
P.
,
Saini
,
S.
,
Bansode
,
P.
, and
Agonafer
,
D.
,
2021
, “
A Comparative Study of Energy Savings in a Liquid-Cooled Server by Dynamic Control of Coolant Flow Rate at Server Level
,”
IEEE Trans. Compon. Packag. Manuf. Technol.
,
11
(
4
), pp.
616
624
.10.1109/TCPMT.2021.3067045