1. University of Karlsruhe, System Architecture Group Balancing Power Consumption in Multiprocessor Systems Andreas Merkel Frank Bellosa System Architecture Group University of Karlsruhe EuroSys '06 April 18 - 21, 2006 Leuven, Belgium

2. 2 University of Karlsruhe, System Architecture Group State of the Art in Thermal Design Increasing power dissipation in computer systems  Ever increasing cooling costs

3. 3 University of Karlsruhe, System Architecture Group State of the Art in Thermal Design Increasing power dissipation in computer systems  Ever increasing cooling costs Alternatives for designing cooling systems: Worst case thermal design Overprovisioning  Unnecessarily high cooling cost Lower thermal design power Throttling to handle “hot” tasks  Performance penalties

4. 4 University of Karlsruhe, System Architecture Group Alternatives in SMP Systems Multiple processors, running different tasks Tasks differ in power consumption  “hot” and “cool” tasks  Hot and cold processors  Unnecessary throttling T t temperature limit

5. 5 University of Karlsruhe, System Architecture Group Alternatives in SMP Systems Multiple processors, running different tasks Tasks differ in power consumption  “hot” and “cool” tasks  Hot and cold processors  Unnecessary throttling T t temperature limit throttling

6. 6 University of Karlsruhe, System Architecture Group Our Approach: Migration instead of Throttling Leverage cool processors Migrate hot tasks away to a cool processor Combine hot tasks with cool tasks on a processor  Balance processor temperatures T t temperature limit migrate

7. 7 University of Karlsruhe, System Architecture Group Our Approach: Migration instead of Throttling Leverage cool processors Migrate hot tasks away to a cool processor Combine hot tasks with cool tasks on a processor  Balance processor temperatures T t temperature limit

8. 8 University of Karlsruhe, System Architecture Group Our Approach: Migration instead of Throttling Leverage cool processors Migrate hot tasks away to a cool processor Combine hot tasks with cool tasks on a processor  Balance processor temperatures Contributions: Characterization of tasks  Task Energy Profiles Policy for assigning tasks to CPUs  Energy-Aware Scheduling

9. 9 University of Karlsruhe, System Architecture Group Outline Task Energy Profiles Thermal Model Energy-Aware Scheduling Evaluation Conclusion

10. 10 University of Karlsruhe, System Architecture Group Characterizing Tasks Temperature High thermal capacitance of chip and heat sink  Temperature changes slowly Short scheduling intervals  CPU temperature:  Mix of multiple tasks' characteristics  No knowledge about individual task's characteristics Power consumption 37W to 61W on Pentium 4 Xeon (2.2 GHz) for different phases of compute-intensive tasks  Characterize tasks by their individual power consumption idle

11. 11 University of Karlsruhe, System Architecture Group Task Energy Profiles Definition: Energy consumed during one timeslice Behavior of tasks depends on input data  Online energy estimation required Tasks show periods of constant power consumption  Use knowledge from the past: Energy consumed during last timeslice Requirement: Determine the amount of energy the CPU consumes during one timeslice

12. 12 University of Karlsruhe, System Architecture Group Event-Driven Energy Estimation Use performance monitoring counters for estimating energy: 1) Choose set of suitable events 2) Assign amount of energy to each event 3) Count events 4) Calculate linear combination of the counter values Error < 10% for real-world integer applications Neglecting floating point applications for lack of suitable counters

13. 13 University of Karlsruhe, System Architecture Group Thermal Model Model of processor and heat sink Relates power consumption to temperature Models temperature with exponential function

14. 14 University of Karlsruhe, System Architecture Group Thermal Model Temperature Power Model of processor and heat sink Relates power consumption to temperature Models temperature with exponential function

15. 15 University of Karlsruhe, System Architecture Group Energy Aware Scheduling Objectives: Minimize the need for throttling processors Avoid unnecessary migrations (cache affinity) Policy depends on number of tasks per runqueue More than one task Combine tasks with different characteristics  Energy balancing One task Migrate hot task before CPU overheats  Hot task migration

16. 16 University of Karlsruhe, System Architecture Group Linear Energy Balancing Goal: Balance CPU temperatures Approach: Balance CPU power consumption Metric: runqueue power Average of the energy profiles of all tasks in a runqueue Predicts future energy consumption  Balance power consumption by balancing runqueue power

17. 17 University of Karlsruhe, System Architecture Group Problem: Runqueue power changes frequently Whenever tasks start or terminate Whenever tasks block/unblock  Ping-pong effects Linear Energy Balancing runqueue power CPU A runqueue power CPU B P t cool task blocks migrate hot task cool task unblocks migrate hot task

18. 18 University of Karlsruhe, System Architecture Group Runqueue power does not consider tasks that are blocked or have terminated Heat produced by those tasks is still stored in the chip and the heat sink Runqueue power cannot distinguish between hot and cool CPUs Linear Energy Balancing

19. 19 University of Karlsruhe, System Architecture Group Fit averaging function to thermal model Use exponential average of CPU's power consumption as metric  Thermal power Reflects past energy consumption  temperature Calibrate parameters to thermal model Exponential Energy Balancing temperature t power thermal power

20. 20 University of Karlsruhe, System Architecture Group Exponential Energy Balancing Problem: Task migrations have no immediate effect on thermal power  “Over-balancing” thermal power CPU A thermal power CPU B P t migrate hot tasks

21. 21 University of Karlsruhe, System Architecture Group P t Energy Balancing Use both metrics for energy balancing Migrate a hot task from CPU A to CPU B if A outranges B in both metrics  Avoids ping-pong effects  Avoids over-balancing Introduce threshold to defer migrations  Hysteresis migrate

22. 22 University of Karlsruhe, System Architecture Group Energy Balancing P t Use both metrics for energy balancing Migrate a hot task from CPU A to CPU B if A outranges B in both metrics  Avoids ping-pong effects  Avoids over-balancing Introduce threshold to defer migrations  Hysteresis

23. 23 University of Karlsruhe, System Architecture Group Hot Task Migration Only one task in a runqueue  Balancing not possible Migrate task to cooler CPU if CPU temperature comes close to maximum Search for cool target CPU Idle CPU  Migrate task CPU executing cool task  Swap tasks

24. 24 University of Karlsruhe, System Architecture Group Evaluation Implementation of energy aware scheduling for the Linux kernel Test system: 8-way Pentium 4 Xeon, 2.2 GHz Workload scenarios Power consumption of tasks ranging from 37W to 61W Measure throughput by counting number of tasks finished per time unit Expect increase in throughput if energy-aware scheduling is enabled

25. 25 University of Karlsruhe, System Architecture Group Evaluation: Energy Balancing Disabled Enabled Scenario: 8 CPUs executing different tasks Energy balancing increases throughput by 4.7% 60W 40W 20W 60W 40W 20W 200s 400s600s 0s200s 400s600s 0s800s

26. 26 University of Karlsruhe, System Architecture Group Evaluation: Hot Task Migration Disabled Enabled One hot task Hot task migration avoids throttling 60W 40W 20W 60W 40W 20W 50s 100s150s 0s 200s 50s 100s150s 0s200s

27. 27 University of Karlsruhe, System Architecture Group Homogeneous vs. Heterogeneous Workloads Workloads consisting of different combinations of memrw (38W), pushpop (47W), and bitcnts (61W)

28. 28 University of Karlsruhe, System Architecture Group Conclusion Need to throttle processors exceeding thermal design power Approach: leverage cool processors instead of throttling hot processors Characterize tasks by power consumption  Energy profiles Use energy profiles for energy-aware scheduling Energy balancing Hot task migration Reduce thermal imbalances Minimize throttling  increase duty cycles