1. Carnegie Mellon School of Computer Science Forecasting with Cyber-physical Interactions in Data Centers Lei Li leili@cs.cmu.edu PDL Seminar 9/28/2011

2. Outline Overview of time series mining –Time series examples –What problems do we solve Motivation Experimental setup ThermoCast: the forecasting model Results Other time series models and algorithms 2(c) Lei Li 2012

3. What is co-evolving time series? 3 Correlated multidimensional time sequences with joint temporal dynamics (c) Lei Li 2012

4. Goal: generate natural human motion –Game ($57B) –Movie industry Challenge: –Missing values –“naturalness” 4 Motion Capture Right hand Left hand walking motion [Li et al 2008a] (c) Lei Li 2012

5. Environmental Monitoring Problem: early detection of leakage & pollution Challenge: noise & large data 5 Chlorine level in drinking water systems [Li et al 2009] (c) Lei Li 2012

6. Network Security Challenge: Anomaly detection in computer network & online activity 6 BGP # updates on backbone from http://datapository.net/ Webclick for news from NTT Webclick for TV (c) Lei Li 2012

7. Time Series Mining Problems Forecasting Imputation (missing values) Compression Segmentation, change/anomaly detection Clustering Similarity queries Scalable/Parallel/Distributed algorithms 7 See my thesis for algorithms covering these problems (c) Lei Li 2012

8. Outline Overview of time series mining –Time series examples –What problems do we solve Motivation Experimental setup ThermoCast: the forecasting model Results Other time series models and algorithms 8(c) Lei Li 2012

9. Datacenter Monitoring & Management Temperature in datacenter Goal: save energy in data centers –US alone, $7.4B power consumption (2011) Challenge: –Huge data (1TB per day) –Complex cyber physical systems 9(c) Lei Li 2012

10. Typical Data Center Energy Consumption LBL data center Google data center [Barroso 09] [LBNL/PUB-945] 10(c) Lei Li 2012

11. Towards Thermal Aware DC Management Data centers are often over provisioned, with ≈40% of energy spent for cooling (total=$7.4B) How can we improve energy efficiency in modern multi-MegaWatt data centers? 11 JHU data center with Genomote (c) Lei Li 2012

12. Air cycle in DC 12(c) Lei Li 2012

13. Possible Ways for Saving Cooling and Computing Cost Challenges: –airflow interaction, spatial placement, SLA, … Possible direction: –Shutdown unused machine according to workload Example MSN workload 13(c) Lei Li 2012

14. Towards Data Driven AC control and server management Reactive energy saving: –slow down cooling fan in CRAC –raise AC temperature set points Proactive data center management: –predicting temperature distribution and thermal aware placement of workload 14 supply air temperature < threshold max(active inlet air temperature)< threshold (c) Lei Li 2012

15. Big Picture: Predictive AC Control and Server Management Temperature prediction Sensor measuring Server/workload management Cooling energy model Computing energy model CRAC control 15(c) Lei Li 2012

16. Outline Overview of time series mining –Time series examples –What problems do we solve Motivation Experimental setup ThermoCast: the forecasting model Results Other time series models and algorithms 16(c) Lei Li 2012

17. Experimental setup Tested in JHU data center with 171 1U servers, instrumented with a network of 80 sensors 17(c) Lei Li 2012

18. Sample measurements 18(c) Lei Li 2012

19. Observations Temperature difference cycle (max/min temp. on the same rack) is in anti-phase with air velocity cycle. Middle and bottom sections are coldest; Top is hottest Shutting down under- utilized servers could reduce energy consumption. 19(c) Lei Li 2012

20. What happens when shutting down servers? 20 Shut down (c) Lei Li 2012

21. Outline Overview of time series mining –Time series examples –What problems do we solve Motivation Experimental setup ThermoCast: the forecasting model Results Other time series models and algorithms 21(c) Lei Li 2012

22. ThermoCast [Li et al, KDD 2011] Given: intake temperatures, outtake temperatures, workload for each server, and floor air speed Goal: forecasting temperature distribution and thermal aware placement of workload Approach: a zonal forecasting model –divide the machine room into zones, and each rack into sections. 22(c) Lei Li 2012

23. Assumptions A0: incompressible air A1: environmental temperature is constant A2: supply air temperature is constant within a period A3: constant server fan speed A4: vertical air flow at the outtake is negligible A5: vertical air flow at the intake is linear to height 23(c) Lei Li 2012

24. Sensor measurements & Air interactions 24(c) Lei Li 2012

25. ThermoCast 25(c) Lei Li 2012

26. ThermoCast Model 26 floor air speed Inlet temp outlet temp Derived from fluid dynamics and thermodynamics together with assumptions [Li et al, KDD 2011] (c) Lei Li 2012

27. Parameter Learning 27 s.t. (c) Lei Li 2012

28. Outline Overview of time series mining –Time series examples –What problems do we solve Motivation Experimental setup ThermoCast: the forecasting model Results Other time series models and algorithms 28(c) Lei Li 2012

29. ThermoCast Results 29 AR ThermoCast 75%  100%  shutdown Q1: How accurately can a server learn its local thermal dynamics for prediction? 2x better using 90 minutes as training, predicting 5 minutes away (c) Lei Li 2012

30. ThermoCast Results Q2: How long ahead can ThermoCast forecast thermal alarms? 2x faster 30 BaselineThermoCast Recall62.8%71.4% FAR45%43.1% MAT2.3min4.2 min FAR=false alarm rate MAT=mean look-ahead time (c) Lei Li 2012

31. Implication on Capacity Gain Preliminary results comparing workload placement strategies: –5 minutes forecast length –With the same cooling: Inlet temp with ThermoCast: 13.75  C Inlet temp with Static profiling: 16.5  C Assume the servers consume 200W on average (Dell PowerEdge 1950), we gain extra 26% computing power with the same cooling 31(c) Lei Li 2012

32. Contributions and Impact Predictability: a hybrid approach to integrate the thermodynamics and sensor data Scalable learning/training thanks to the zonal thermal model Real data and instrument in a data center with practical workload Projected impact: can handle extra 26% workload (e.g. PUE 1.5  PUE 1.4) 32(c) Lei Li 2012

33. Outline Overview of time series mining –Time series examples –What problems do we solve Motivation Experimental setup ThermoCast: the forecasting model Results Other time series models and algorithms 33(c) Lei Li 2012

34. DynaMMo: imputation/forecasting 34 Time sensor 1 sensor 2 … sensor m blackout Goal: recover the missing values Details in [Li et al, KDD 2009] (c) Lei Li 2012

35. DynaMMo result 35 Reconstructionerror Average missing length Ideal Our DynaMMo MSVD [Srebro’03] Linear Interpolation Spline Dataset: CMU Mocap #16 mocap.cs.cmu.edu more results in [Li et al, KDD 2009] better harder (c) Lei Li 2012

36. PLiF and CLDS for clustering 36 BGP data: hierarchical clustering + PLiF features Details in [Li et al, VLDB 2010] and [Li & Prakash, ICML 2011] (c) Lei Li 2012

37. CLDS Clustering Mocap Data 37 Accuracy = 93.9% Accuracy = 51.0% PCA top 2 components CLDS two features  walking motion running motion (c) Lei Li 2012

38. WindMine Goal: find patterns and anomalies from user- click streams 38(c) Lei Li 2012

39. Discoveries by WindMine 39 Job website Job website weather kids health (c) Lei Li 2012

40. Conclusion time series mining with many applications Numbers for energy consumption in DC, and cooling costs much Sensor networks find use in data center monitoring ThermoCast: the forecasting model Other time series models and algorithms –DynaMMo for imputation –PLiF & CLDS for clustering –WindMine for web clicks 40

41. References Lei Li, et al. ThermoCast: A Cyber-Physical Forecasting Model for Data Centers KDD 2011 Lei Li, et al. Time Series Clustering: Complex is Simpler. ICML 2011 Yasushi Sakurai, Lei Li, et al, WindMine: Fast and Effective Mining of Web-click Sequences, SDM, 2011. Lei Li, et al. Parsimonious Linear Fingerprinting for Time Series. VLDB 2010. Lei Li, et al. DynaMMo: Mining and Summarization of Coevolving Sequences with Missing Values. ACM KDD 2009. 41(c) Lei Li 2012

42. Thanks! contact: Lei Li (leili@cs.cmu.edu)leili@cs.cmu.edu papers, software, datasets on http://www.cs.cmu.edu/~leili http://www.cs.cmu.edu/~leili 42