1. Thermal Analysis of the SKA Electronics Cabinets
Using ANSYS Icepak/Fluent CFD
Richard Bennett
UKATC 1

2. My own experience of FE Analysis
1972 Writing Fortran routines for an in-house dynamic analysis program
Successfully avoided FEA
1992 Algor for structural analysis of an instrument space-frame.
1995 CosmosM for calculating the effect of wind-shake on optical performance of the the UKIRT telescope, design of active control of the primary mirror.
1995 Introduced Rasna Mechanica to the ATC. A big step forward in integration with CAD, modelling and meshing. Use of FEA as a design tool became more widespread within the mechanical group at UKATC.
Structural integrity of instruments under pressure, thermal and earthquake loading. Optical alignment and figure of optical elements for telescope mounted instruments with a moving gravity vector.
Thermal analysis of air-cooled electronics using CFD. 2

3. FE Analysis at the UKATC
VISTA primary mirror transient with impact during earthquake
ERIS for VLT
(David Pearson)
CERN ATLATAS Upgrade thermal analysis of composite structure 3

4. Thermal Analysis of the SKA Electronics Cabinets
Antennae
4 sub-racks
each containing 8
Tile Processing Modules
(TPM)
RF over fibre links
Artist’s Impression of dipole antennae – Swinburne Astronomy Productions/ICRAR/U. Cambridge/ASTRON. 4

5. TPM Boards and thermal loads
Amplifiers – 16 x 1W
ADCs – 16 x 1.4W
FPGAs - 2x 37.5W
Receivers - 8 each side of the pcb – 3W total 5

6. Modelling with ANSYS Icepak
board assembly imported from Inventor CAD model
Heat-sinks are created from a dialog box
Fans are 3D blocks which impart momentum and pressure rise, specified with a characteristic curve
Electronic devices are hollow blocks with an internal thermal network
FPGA network model downloaded from Xilinx (manufacturer) website
Hanging node elements
Non-conformal meshing 6

7. Validation of the modelling technique
CFD simulation
Thermal test of TPM board at INAF
Air deflector plate
AAVS1 TPM
(Front panel removed for photos)
Air inlet temperature 25°C
Fan tray 7

8. Heat sink, EMC cover and fan tray
Fan tray and heat-sink too close for air from rear fans to be directed to heat-sinks
front fans
rear fans
Fans give uneven coverage of heat-sinks 8

9. Modifications for AAVS1
(AAVS1 is a verification system with a small number of antennae for testing electronics and software)
blanking plates with apertures to concentrate airflow through the heat-sinks 9

10. AAVS1 design initial concept final design 10 air outlet air outlet
2x6 fans
6x fans
air inlet
air inlet
initial concept
final design 10

11. AAVS1 simulation and test results
Caveats:
1. “Due to the fact the FW wasn’t optimized and some error of the set parameters for the power simulation of the board, the real current drawn and so the final power dissipated is higher than the estimation.”
2. Location of actual monitoring point on the board not known. 11

12. LFAA cooling options Heat-pipes and water cooled cold plates
Cabinets with column HX, front-to-back airflow in cabinet
Vertical airflow through TPM, horizontal air inlet and outlet (Scheme A)
Horizontal airflow through TPM (Scheme B)
Blade server type arrangement and in-line HX (Scheme C) 12

13. Limitations of the software and computing facilities
Size – even simplified models typically have up to million elements and sometimes, inexplicably, 10x more requiring Gb memory
Speed – hours to mesh, from many hours to many days to solve on a virtual machine with 408Gb memory and 48 processors
Parallel processing – there is a software bug, some models have given different results to serial processing. Acknowledged by ANSYS, awaiting a fix. 13

14. Workflow Detailed models of the electronic modules
Simplified models to give the same flow/pressure characteristics.
Cabinet level models with the simplified modules to calculate flowrates.
Simplified model provides look-up table for device temperatures 14

15. Detailed and Simplified Models of TPMs15

16. Cabinet level model Cooling Scheme A 16 Cooling module:
air-water HX + fans
4x TPM sub-racks per cabinet
Sub-rack blowers with air outlet (to rear)
Sub-rack air inlet (from front)
PSU: air inlet at sides, air outlet at rear
MCCS: air inlet at front, air outlet at rear
4x clock distribution
Switches
PDU
Detailed models of 4 TPMs in inner and outer slots in top and bottom sub-racks of the right-hand cabinet. All others are simplified.
Right-hand cabinet
Left-hand cabinet 16

17. Cabinet level models Cooling Schemes B & C HX HX Fans Fans Fans
Column HX between cabinets

18. Cabinet level model
Cooling Scheme A 18

19. Cabinet level models Representation of Heat Exchangers
Manufacturer’s data sheet
Text book solution for cross-flow heat exchanger
ANSYS Icepak
U can be defined as a function of flow but not temperature difference
Insufficient information 19

20. Cabinet level model Solutions for normal operation and with failed fans
Failed Blower 20

21. LFAA Air cooled results
Comparison of cooling options
FPGA Junction Temperature (°C)
Min
Max
A Column HX between cabinets, vertical airflow in sub-rack
56.6
63.3
B Column HX between cabinets, horizontal airflow in sub-rack
66.7
72.6
C In-line HX, vertical airflow in subrack
52.2
62.5
Comparison of operational modes
A Column HX between cabinets, vertical airflow in sub-rack
FPGA Junction Temperature (°C)
Min
Max
Normal operation
66.7
72.6
Failed blower in sub-rack
66.8
89.1
Failed fan in HX unit
67.6
75.6 21

22. Q. Can I avoid using CFD?
Simple model, analysed using Icepak and directly with Fluent
Calculation using standard correlations for friction and pressure loss at change of section and direction (Mathcad)
A. Yes, if you know where the air is going (but its good to have a check using another method). 22

23. Lessons learned
I expected my lack of knowledge of turbulence models and the numerical methods used in CFD to be the main challenge but actually quirky, unpredictable meshing with Icepak, different to any FE meshing I have used. It is extremely slow with large models extending the learning curve.
Niche market software has many more bugs than widely used products.
With CFD, unlike mechanical FE, it's best to start with low expectations. 23