Thermal-aware Task Placement in Data Centers (part 4) SANDEEP GUPTA Department of Computer Science and Engineering School of Computing and Informatics Ira A. Fulton School of Engineering Arizona State University Tempe, Arizona, USA sandeep.gupta@asu.edu

Thermal-aware Task Placement Problem Given an incoming task, find a task partitioning and placement of subtasks to minimize the (increase of) peak inlet temperature P = a U + b Tin Tsup D U XInt Algorithm Approximation solution (genetic algorithm) Take a feasible solution and perform mutations until certain number of iterations bb b b b b (a + ) = + × heat distribution inlet temperatures supplied air temperatures utilization vector

Recirculation coefficients: a fast thermal model Reduce/Simplify the “thermal map” concept to points of interest: equipment air inlets Can be computed from CFD models/simulations A Matrix A aij: portion of heat exhausted from node i that directly goes to node j recirculation coefficients

Scheduling Impacts Cooling Setting Inlet temperature distribution without Cooling Inlet temperature distribution with Cooling Different demands for cooling capacity Scheduling 1 25C Scheduling 2 25C

Contrasted scheduling approaches Uniform Outlet Profile (UOP) Assigning tasks in a way that tries to achieve uniform outlet temperature distribution Assigning more task to nodes with low inlet temperature (water filling process) Minimum computing energy Assigning tasks in a way that keeps the number of active (power-on) chassis as few as possible Server with coolest inlet temperature first Uniform Task (UT) Assigning all chassis the same amount of tasks (power consumptions) All nodes experience the same power consumption and temperature rise Outlet Temperature Inlet Temperature

Simulated Environment Used Flometrics Flovent Simulated a small scale data center physical dimensions 9.6m  8.4m  3.6m two rows of industry standard 42U racks arranged CRAC supply at 8 m3/s There are 10 racks each rack is equipped with 5 chassis 1000 processors in data center. 232KWatts at full utilization

Results(1) Recirculation Coefficients Consistent with datacenter observations Large values are observed along diagonal Strong recirculation among neighboring servers, or between bottom servers and top servers 1 2 3 4 5 10 diagonal 9 8 7 6

Performance Results Xint outperforms other algorithms Data Centers almost never run at 100% Plenty of room for benefits! diagonal

Power Vector Distribution Xint contradicts “rule of thumb” placement at bottom key

Supply Heat Index (SHI) Metric developed by HP Labs quantifies the overall heat recirculation of data center Xint consistently has the lowest SHI

Conclusions Thermal-aware task placement can significantly reduce heat recirculation XInt performance thrives at around 50% CPU utilization Not much can be done at 100% utilization Cooling savings can exceed 30% (in comparison to other schemes) Cost of operation reduces by 15% (if initially 1:1 ratio of computing-2-cooling)

Related Work in Progress Waiving simplifying assumptions Equipment heterogeneity [INFOCOM 2008] Stochastic task arrival Thermal maps thru machine learning Automated, non-invasive, cost-effective [GreenCom 2007] Implementations Thermal-aware Moab scheduler Thermal-aware SLURM SiCortex product thermal management

Algorithm Assumptions HPC model in mind Long-running jobs (finish time is the same — infinity) One-time arrival (starting time is the same) Utilization homogeneity (same utilization throughout task’s length) Non preemptive/movable tasks Data Center equipment homogeneity power consumption computational capability Cooling is self-controlled

Thank You Questions? Comments? Suggestions? http://impact.asu.edu/

References 1) AMD – Power and Cooling in the Data Center. 34146A_PC_WP_en.pdf 2) HP Labs - Going beyond CPUs: The Potential of Temperature-Aware Solutions for the Data Center. 3) HP Labs - Making Scheduling Cool: Temperature-Aware Workload Placement in Data Centers.

Additional Slides

Contributions Developed Thermal models of Data Centers Developed analytical thermal models using theoretical thermodynamic formulations Developed online thermal models using machine learning techniques Designed thermal aware task placement algorithms Designed genetic algorithm based task placement algorithm that minimizes the heat recirculation among the servers and the peak inlet temperature. Created a software architecture for dynamic thermal management of Data Centers Developed CFD Models for Real World Data Centers for testing and validation of thermal models and task placement algorithms

Data Center Thermal Management Increasing need for thermal awareness Power density increases Circuit density increases by a factor of 3 every 2 years Energy efficiency increases by a factor of 2 every 2 years Effective power density increases by a factor of 1.5 every 2 years [Keneth Brill: The Invisible Crisis in the Data Center] Maintenance/TCO rising Data Center TCO doubles every three years By 2009, the three-year cost of electricity will exceed the purchase cost of the server Virtualization/Consolidation is a 1-time/short term solution Thermal management corresponds to an increasing portion of expenses Thermal-aware solutions becoming prominent Thermal-aware solutions at various levels IC Case/chassis room firmware O/S Application (middleware) Dynamic voltage scaling Dynamic frequency scaling Circuitry redundancy Fan speed scaling CPU Load balancing Thermal-aware VM Data center job scheduling software dimension physical dimension A dynamic thermal-aware control platform is necessary for online thermal evaluation Thermal issues Heat recirculation Increases as equipment density exceeds cooling capacity as planned Hot spots Effect of Heat Recirculation Impact: Cooling has to be set low enough to have all inlet temperatures in safe operating range Opportunities & Challenges Data centers don’t run at fulll unitilization Can choose among multiple CPUs to allocate a job Different thermal impact per CPU Need for fast thermal evaluation Temporal and spatial Heterogeneity of Data Centers In equipment In workload downplaying the advantages of dense server deployment without thermal-aware management With thermal-aware management $100M cooling $10M $1M computation year