1. Thermal-aware Task Placement in Data Centers (part 4)
SANDEEP GUPTA
Department of Computer Science and Engineering
School of Computing and Informatics
Ira A. Fulton School of Engineering
Arizona State University
Tempe, Arizona, USA

2. Thermal-aware Task Placement Problem
Given an incoming task, find a task partitioning and placement of subtasks to minimize the (increase of) peak inlet temperature
P = a U + b
Tin
Tsup D U
XInt Algorithm
Approximation solution
(genetic algorithm)
Take a feasible solution and perform mutations until certain number of iterations
bb b b
b b (a
+ ) = + ×
heat distribution
inlet temperatures
supplied air temperatures
utilization vector

3. Recirculation coefficients: a fast thermal model
Reduce/Simplify the “thermal map” concept to points of interest: equipment air inlets
Can be computed from CFD models/simulations A
Matrix A aij: portion of heat exhausted from node i that directly goes to node j
recirculation coefficients

4. Scheduling Impacts Cooling Setting
Inlet temperature
distribution
without Cooling
Inlet temperature
distribution
with Cooling
Different demands for cooling capacity
Scheduling 1
25C
Scheduling 2
25C

5. Contrasted scheduling approaches
Uniform Outlet Profile (UOP)
Assigning tasks in a way that tries to achieve uniform outlet temperature distribution
Assigning more task to nodes with low inlet temperature (water filling process)
Minimum computing energy
Assigning tasks in a way that keeps the number of active (power-on) chassis as few as possible
Server with coolest inlet temperature first
Uniform Task (UT)
Assigning all chassis the same amount of tasks (power consumptions)
All nodes experience the same power consumption and temperature rise
Outlet Temperature
Inlet Temperature

6. Simulated Environment
Used Flometrics Flovent
Simulated a small scale data center
physical dimensions 9.6m  8.4m  3.6m
two rows of industry standard 42U racks arranged
CRAC supply at 8 m3/s
There are 10 racks
each rack is equipped with 5 chassis
1000 processors in data center.
232KWatts at full utilization

7. Results(1) Recirculation Coefficients
Consistent with datacenter observations
Large values are observed along diagonal
Strong recirculation among neighboring servers, or between bottom servers and top servers 1 2 3 4 5 10
diagonal 9 8 7 6

8. Performance Results Xint outperforms other algorithms
Data Centers almost never run at 100%
Plenty of room for benefits!
diagonal

9. Power Vector Distribution
Xint contradicts “rule of thumb” placement at bottom
key

10. Supply Heat Index (SHI)
Metric developed by HP Labs
quantifies the overall heat recirculation of data center
Xint consistently has the lowest SHI

11. Conclusions
Thermal-aware task placement can significantly reduce heat recirculation
XInt performance thrives at around 50% CPU utilization
Not much can be done at 100% utilization
Cooling savings can exceed 30% (in comparison to other schemes)
Cost of operation reduces by 15% (if initially 1:1 ratio of computing-2-cooling)

12. Related Work in Progress
Waiving simplifying assumptions
Equipment heterogeneity [INFOCOM 2008]
Stochastic task arrival
Thermal maps thru machine learning
Automated, non-invasive, cost-effective [GreenCom 2007]
Implementations
Thermal-aware Moab scheduler
Thermal-aware SLURM
SiCortex product thermal management

13. Algorithm Assumptions
HPC model in mind
Long-running jobs (finish time is the same — infinity)
One-time arrival (starting time is the same)
Utilization homogeneity (same utilization throughout task’s length)
Non preemptive/movable tasks
Data Center equipment homogeneity
power consumption
computational capability
Cooling is self-controlled

14. Thank You
Questions?
Comments?
Suggestions?

15. References
1) AMD – Power and Cooling in the Data Center A_PC_WP_en.pdf
2) HP Labs - Going beyond CPUs: The Potential of Temperature-Aware Solutions for the Data Center.
3) HP Labs - Making Scheduling Cool: Temperature-Aware Workload Placement in Data Centers.

16. Additional Slides

17. Contributions Developed Thermal models of Data Centers
Developed analytical thermal models using theoretical thermodynamic formulations
Developed online thermal models using machine learning techniques
Designed thermal aware task placement algorithms
Designed genetic algorithm based task placement algorithm that minimizes the heat recirculation among the servers and the peak inlet temperature.
Created a software architecture for dynamic thermal management of Data Centers
Developed CFD Models for Real World Data Centers for testing and validation of thermal models and task placement algorithms

18. Data Center Thermal Management
Increasing need for thermal awareness
Power density increases
Circuit density increases by a factor of 3 every 2 years
Energy efficiency increases by a factor of 2 every 2 years
Effective power density increases by a factor of 1.5 every 2 years
[Keneth Brill: The Invisible Crisis in the Data Center]
Maintenance/TCO rising
Data Center TCO doubles every three years
By 2009, the three-year cost of electricity will exceed the purchase cost of the server
Virtualization/Consolidation is a 1-time/short term solution
Thermal management corresponds to an increasing portion of expenses
Thermal-aware solutions becoming prominent
Thermal-aware solutions at various levels IC
Case/chassis
room
firmware
O/S
Application
(middleware)
Dynamic voltage scaling
Dynamic frequency scaling
Circuitry redundancy
Fan speed scaling
CPU Load balancing
Thermal-aware VM
Data center job scheduling
software dimension
physical dimension
A dynamic thermal-aware control platform is necessary for online thermal evaluation
Thermal issues
Heat recirculation
Increases as equipment density exceeds cooling capacity as planned
Hot spots
Effect of Heat Recirculation
Impact: Cooling has to be set low enough to have all inlet temperatures in safe operating range
Opportunities & Challenges
Data centers don’t run at fulll unitilization
Can choose among multiple CPUs to allocate a job
Different thermal impact per CPU
Need for fast thermal evaluation
Temporal and spatial Heterogeneity of Data Centers
In equipment
In workload
downplaying the advantages of dense server deployment
without thermal-aware management
With thermal-aware management
$100M
cooling
$10M
$1M
computation
year