1. 1 PRM 8: B ETTER D ATA C ENTER AND O PERATIONS M ANAGEMENT T HROUGH T HE U SE OF M ETRICS Tad Davies, Senior Vice President, Consulting Services, Bick tdavies@bickgroup.comtdavies@bickgroup.com Hector Diaz, Sr. Partner, iDiaz Advisors, LLC hdiaz@hdiaz.org hdiaz@hdiaz.org John Parker, Disaster Recovery and Data Center Operations Management jparker@esri.comjparker@esri.com

2. 2 Session Description Better Data Center Management Through the Use of Metrics Understanding how your data center is performing relative to metrics facilitates informed decision-making and improved prioritization of corrective actions. This session will review metrics from three perspectives – Heuristics (DC Math), Financial, and Operational – and provide industry best practices. You will also learn “what and how” to measure what you are managing to enable you to calculate your own metrics. Join us and see how your data center stacks up!

3. 3 Introductions Tad Davies, Senior Vice President, Consulting Services, Bick Tad leads the Consulting Services practice at Bick and has been in the data center industry for 29 years. Bick Consulting provides Business Centric and Facility Centric guidance for clients across the U.S. Tad is a board member of AFCOM’s Data Center Institute. Hector Diaz, Sr. Partner, iDiaz Advisors, LLC Hector is a Senior Partner at iDiaz Advisors as well as a Board Member of AFCOM’s Data Center Institute and President of the Denver chapter. He has extensive experience managing international data center properties. At Oracle he was the Director for Data Center Portfolio Management, a portfolio of over a million square feet of "white space”. At Agilent Technologies he was the Operations Manager for all their data centers in the Americas. At HP he managed data center moves and consolidations related to mergers, acquisitions, and divestitures. John Parker, President, Southern California AFCOM Chapter John is a Senior IT professional with over 20 years in 4 industries (Healthcare, Pharmaceuticals, Banking, and Software Development). His global management experience includes Facilities, Disaster Recovery, Data Center Operations and 24/7 support teams. He currently sits on the board of directors for AFCOM’s Data Center Institute and is also President of the Southern California chapter of AFCOM.

4. 4 Agenda DC Math and Heuristics A discussion of units of measurement and their relationship that are foundational to your understanding of your environment. Quick calcs you can do to improve your understanding of current or future environment. DC Costs and Financials A discussion of financial components that comprise the total investment in a data center facility. A look at operational costs as well as costs of colocation. Recommendations regarding gaining actions to gain control of costs. Break DC Operational Metrics A discussion of operational guidance to which you can use to generate operational reports on a weekly and monthly basis.

5. 5 3 Key Things You Will Learn During this Session 1.Help you think about how you are managing your data center 2.How to create better Metrics and KPI’s 3.Share your thoughts about what has been effective for you

6. 6 DC Math and Heuristics Space PowerCooling Common formulas you need | A perspective on leveraging math

7. 7 Definition Heuristic (/hj ʉˈ r ɪ st ɨ k/; Greek: "Ε ὑ ρίσκω", "find" or "discover") refers to experience-based techniques for problem solving, learning, and discovery that give a solution which is not guaranteed to be optimal. Where the exhaustive search is impractical, heuristic methods are used to speed up the process of finding a satisfactory solution via mental shortcuts to ease the cognitive load of making a decision. Tad’s term: napkin math

8. 8 DC Math Foundational first steps: 1.What is the unit of measurement? 2.Are we talking about the same subject? The numbers can be very misleading if you aren’t talking about the same thing

9. 9 DC Math and Heuristics Space

10. 10 Space How big is your data center? Answer: 25,000 s.f. a)Compute space only? b)Adjoining operational space? c)Including electrical – mechanical space? d)The entire building (dedicated DC)? 10,000 s.f. 3,000 s.f. 7,000 s.f.

11. 11 Space

12. 12 Space – Unit of Measurement? REU - Rack Equivalent Unit. REU creates a common metric amongst equipment of differing sizes. Metric is based upon a rack size of 24"w x 48" deep. A large piece of equipment may be multiple REUs. example: Mainframes may count as 3 - 4 REUs What is an appropriate unit of measurement? Square Feet. How to make the unit more useful? Leverage a unit you know

13. 13 Objective: Determine space utilization efficiency of an existing DC SE - Space efficiency DC s.f. – data center square feet. Compute space only SE = DC s.f. / REUs Space – calculating efficiency

14. 14 Space Clarifications: 1.Rack size of 24"w x 42" deep 2.Cold aisle is 48” 3.Hot aisle is 36” 4.CRACs, PDUs, in-row PDUs in data center 5.UPS not in data center 6.Room is rectangular and doesn’t have a higher number of or physically larger than typical column REU – SPACE RATIO Best 25 Better 30 Good 35

15. 15 Space Rack Equivalent Unit (REU) ratio 58:1 Example- Legacy data center floor plan

16. 16 Space Rack Equivalent Unit (REU) ratio 32:1 Example- Same data center floor plan

17. 17 Space Objective: Determine space needed for new DC 1.Using 30 or even 35 versus 25 for contingency purposes a)Physical features of DC are unknown b)You don’t know yet how much support equipment might have to be in there S = REUs x 35

18. 18 DC Math and Heuristics Power

19. 19 Power How big is your data center? 2 mW a)Utility Service? b)Generator capacity in place? Future capacity? c)UPS capacity in place? Future capacity? d)IT load e)Does this number include redundancy

20. 20 Power - DC Math ConvertFormulaExample Watts to kilowatts Watts/1000 = kilowatts2500watts/1000 = 2.5 kilowatts To find Watts (1P)Volts x Amps x PF= watts120v x 2.5 = 300watts To find kW (3P)Volts x Amps x PF x 1.73/1000= kW480 x 800 x.9 x 1.73/1000 = 598 To find Amps (3P)Watts/ Volts x 1.73 x PF = Amps300/120 = 2.5 To find kVA (single phase) Volts x Amps/1000= kVA208v x 4.7 /1000 =.3kVA To find kVA (three phase) Volts x Amps x 1.73/1000 = kVA208 x 4.7 x 1.73 / 1000 = 1.69kVA Convert kVA to kWkVA x (power factor)= kW 300KVA x.80 (power factor) = 240kW Convert kW to kVAkW/PF = kVA240kW/.80 = 300kVA PF – power factor

21. 21 Power - DC Math Source for conversion calculator: http://www.rapidtables.com/calc/electric /Watt_to_Amp_Calculator.htm

22. 22 ItemUOMConversion TransformerkVAkVA x Power Factor (often.9) Service entranceAmpsAmps x volts x PF x 1.73 / 1000 GeneratorkWNo conversions required UPSkVA Capacity in kVA X PF (.80 or.90 or 1.0 manufacturer dependent) CoolingtonsCapacity in kW/.28 Power - Common unit of measurement: kW

23. 23 Power - Common unit of measurement: kW This is why $/kW is used – covers electrical and mechanical -highest % of costs Although kW reflects the most telling UOM of a data center, you should still know the $/s.f. number. Why? It is a metric that is better understood by senior executives Hospital execs comparing DC build costs to their hospital build costs Manufacturing comparing DC build costs to their plant build costs Most real estate execs think in $/s.f.

24. 24 Power – kW vs kVA Power Factor The power factor of an AC electrical power system is defined as the ratio of the real power flowing to the load, to the apparent power in the circuit. Real power is the capacity of the circuit for performing work in a particular time. Apparent power is the product of the current and voltage of the circuit. When this might be important: Power available out of a legacy UPS:.8 PF 1.0 PF Unity 500kVA UPS400kW 500kW kW = kVA x PF

25. 25 8760

26. 26 Power – Why PUE is important 1 rack= 5kw 5kw rack x 8,760 hrs= 43,800 kwh/ year 43,800 kwh x $0.07 /= $3,066 per rack annual 5,000 sq ft data center with 140 racks 140 racks x $3,066= $429,240 annual IT energy costs At PUE of 2= $858,000 Total DC energy At PUE of 1.5= $643,500 Total DC energy Difference:$214,500

27. 27 PUE - Power Usage Effectiveness PUE = Total Facility Power/IT Equipment Power Measures how much energy it takes to operate your equipment Relate IT infrastructure consumption to total power consumption [Generator may be supporting non-DC loads. PDUs might be too] PUE Napkin Calc = Load at ATS/ Sum of load at PDUs

28. 28 Power – seemingly small changes can be big 91 watt vs 70 watt processors= 21 watts / processor 21 watts x 2 processors= 42 watts savings 42 watts x 8760hrs / 1000=368 kwh 368kwh x $0.07/kwh=$25.75 per server 140 racks / 5000sqft DC13 servers / rack 140 racks x 13 servers / rack= 1680 servers 1680 x $25.75 x 2= $86,520

29. 29 DC Math and Heuristics Cooling

30. 30 Cooling How much cooling do you have? a)How much of your cooling is for redundancy? b)Do you need more than one extra unit to achieve N+1 due to your data center’s shape? c)Name plate rating?

31. 31 DC Math - Cooling ConvertFormulaExample Tons of air to BTU12,000 BTUs = 1 ton of air To find BTUsWatts x 3.41= BTU2500w x 3.41 = 8525 (BTU) 1 kW =.2843 tons 1 ton= 3.517kW Tons to kWTons x 3.52 = kW20 tons x 3.52 = 70kW kW to tonskW x.28 = tons70kW x.28 = 19.6 tons Unit of measurement was always in tons but now you are seeing it in kW.

32. 32 Cooling – effectiveness Objective: How does my cooling capacity stack up against my IT load? Is the net capacity (see below) significantly greater than the result? Clarifications: 1.Net out tons that existing for redundancy 2.If you’re room is inefficient – low floor/ceiling, weird shape – take 20% of the ton quantity. Load on the UPS in kW x.28 = tons of cooling. Compare this against tons of cooling in your data center

33. 33 Cooling – effectiveness Real Life Example: DC looking to add another CRAC unit. After some napkin math, pursued a containment solution CRAC Containment CapEx$80k $45k OpEx (5 yrs.)$50k $ 0k Total$130k $45k

34. 34 Airflow – bypass air does matter 1 cfm of bypass air @ 20ºF ∆ T = 21.6 BTU’s/hr 21.6 BTUH x 8760 hrs = 189,216 BTUH 189,216 BTUH ÷ 3412kw/BTU = 55.5kwh 55.5kwh x $0.07/kwh = $3.88 for each 1 CFM or bypass air Typical 8 x 6 cutout (50% full) = 92 CFM 92 CFM x $3.88 = $356.95 Wasted Energy

35. 35 Discussion ….. What napkin math has helped you? Tad Davies, Senior Vice President, Consulting Services Bick Group tdavies@bickgroup.com 314-265-2735 cell

36. 36 TUT 1: Better Data Center Management Through the Use of Metrics Part 2: Data Center Costs and Financials Hector Diaz, Sr. Partner, iDiaz Advisors, LLC hector@idiazadvisors.com

37. 37 Objective Use financial modeling of your data center costs to optimize their utilization.

38. 38 Market demand for data centers According to Microsoft, global construction will increase from about $50 billion today to about $78 billion by 2020

39. 39 Data centers are expensive to build A 1MW 7,500 ft 2 “white space” commercial data center will cost about $16.7M to build. The building to house that data center will be about 23K ft 2. Total cost > $725/ft 2. Compare that to $100 - $150 / ft 2 for a custom home in Denver.

40. 40 Data centers are expensive to operate The monthly electric power bill for a 1MW data center is about $82.6k ($991K/yr.) (at 85% occupancy, with a 1.5 PUE) Network charges are very high as well. Data center infrastructure is the fastest growing cost of deploying IT.

41. 41 Data center CapEx The best method for predicting the cost of data center build is to separate the cost of the building (or “shell”) from the cost of the mechanical, electrical, and plumbing (MEP) components.

42. 42 New data center build cost estimation 1.Start with desired IT load at full capacity in kW. 2.State desired power density for your data center expressed in kW/rack for an average rack. 3.State the desired level of redundancy.

43. 43 New data center build cost estimation 4.Calculate the size of the white space required to support your desired IT load and power density. (30 ft 2 /rack) 5.Estimate the total required size for your building shell. Tier IITier IIITier IV “shell” : “white space” ratio1.52.55.5

44. 44 New data center build cost estimation 6.Estimate the cost of building your shell based on an average construction cost of $200 - $250/ ft 2. 7.Estimate the cost of the MEP fit-out using the following heuristics: Tier IITier IIITier IV MEP fit-out cost in $/kW of IT load$10,000$12,000$16,000

45. 45 Data center OpEx The best financial modeling strategy for data center OpEx is to boil down all operational costs into a monthly recurring cost (MRC) expressed in $/kW for every data center or colocation facility you operate in. This is your “cost of compute”.

46. 46 Components of data center OpEx Depreciation Maintenance of infrastructure components Utilities (power, water, …) Fuel (e.g. diesel for generator tests) Salaries (for data center infrastructure personnel, dedicated security personnel)

47. 47 Know your “cost of compute” Question you are answering: How much is it costing me to deploy a rack full of IT gear at a given location? Boil everything down to MRC in $/kW @ a given redundancy level (for valid comparisons)

48. 48 Cost of Compute Know this number for every data center you own and for every colocation vendor you use. Easier to compute for dedicated data centers. More difficult but possible to compute for shared use buildings.

49. 49 OpEx financial accounting issues MRC in $/kW is not constant over time. There are seasonal variations for cooling loads. Design targeted PUEs are attained at full loads For a given location, MRC in $/kW will decrease as IT HW occupancy increases.

50. 50 Typical cost of compute For a large Enterprise (economies of scale apply) in a concurrently maintainable data center $235 - $300 / kW MRC For small / medium Enterprise (no economies of scale) in a non-concurrently maintainable DC $300 - $400 /kW MRC; much higher for a concurrently maintainable DC (which a small enterprise is unlikely to have)

51. 51 Financial control over space & power Establish a capacity planning process. Detailed quarterly forecast (new and decommissioned HW) tied to budgets Project 2-3 years Publish a dashboard Add feedback loop (actual vs. forecast) Enforce decommissioning! (avoid orphan HW)

52. 52 Capacity planning example

53. 53 Capacity planning benefits Improved quality of forecast over time Reduced CapEx if you can delay build out of new data center Better decision support

54. 54 Financial control over space & power Measure and publish utilization over time. Know your metrics: (avg. kW/rack, avg. ft2/rack) Set-up a space & power governance program Must pre-authorize HW intake to data center Must follow-through with decommissioning equipment.

55. 55 Financial control over energy management You are probably spending too much! You probably don’t know your PUE (Power Utilization Efficiency) You are probably running too cold!

56. 56 Energy management strategies Measure, track, and publish your PUE Use ASHRAE’s new recommended operating temperatures. See Green Greed paper on the ROI of cooling system improvements

57. 57 Energy management ROI Generally speaking, for every 1.8°F that you raise the temperature in your data center, you save 2-4% of your total energy bill.

58. 58 Green Grid recommendations Install OEM variable speed drives (VSDs) in all computer room air handlers (CRAHs). Upgrade older CRAH units with newer more efficient models. Improve rack airflow management by adding baffles and blanking panels, which improve isolation of hot and cold air aisles. Reposition the CRAH temperature/humidity sensors from the inlets of the CRAHs to the front of the IT equipment racks. Adjust the temperature set-points of the CRAH sensors and the chiller units

59. 59 Know the cost of outsourcing You are probably not benefiting from selectively outsourcing some of your infrastructure!

60. 60 There may be financial advantages to outsourcing Consider colocation You can probably not build and operate anything smaller than a 2MW data center for less than what you would pay for colocation. (For the same level of redundancy & service levels) Consider other outsourcing options

61. 61 The outsourcing spectrum Application Middleware Operating System IT Physical Infrastructure Virtual Infrastructure Physical Hardware Data Center Environment BYODC ColocationManaged HostingSaaSPaaS IaaS

62. 62 Cost of colocation Location$/kW MRC Boston$300 - 400 Chicago$225 - 425 Dallas$200 - 500 NY/NJ$275 - 510 No VA$250 - 400 No CA$275 - 450 So CA$300 - 650

63. 63 Questions? Hector R. Diaz, Sr. Partner iDiaz Advisors, LLC hector@idiazadvisors.com M: 720.346.4446

64. 64 DC Operational Metrics John Parker| SoCal AFCOM Chapter President|Esri Data Center Management| jparker@esri.comjparker@esri.com

65. 65 Segment details This session will provide details on how to create metrics and convert to reports for yourself, customers and senior management. It will also include how metrics can grow your career and your teams career too. At the end of the meeting full reports we be explained and reviewed. This educational session works for any area of Operations and Facilities and include explanations of metrics slides from each.

66. 66 The Presentation  Part one – Key factors in creating metrics  “How can you manage what you can’t measure”  Part two – The metrics formula  Workload + Performance = Goals/Results  Part three- Summary and report examples

67. 67

68. 68 Part one –Metrics building blocks Tangibles Day to day activities(workloads) Tickets Incidents, requests, changes Facilities testing and PM’s Monitoring and Alerts Intangibles Cost savings Time savings Proactive initiatives How do I do this Stuff !

69. 69 Part one –Metrics building blocks – How well you do it Support Weekly or monthly “workload” numbers combined Applications & Hardware Users & customers Data centers Projects completed Changes Performance Availability Service Level Agreements (SLA’s) Target and Stretch

70. 70 Part one - Why Metrics are so important Did you meet your goals? Creates efficiencies Cost savings Removes subjectivity Resource and workload management

71. 71 Part One- How to create Metrics 1.Understand what is important to measure 2.Know your audience 3.Automate whenever possible

72. 72 Understand what is important to measure For your teams For your customers For senior management

73. 73 Know your audience Customize metrics for yourself, your colleagues and your customers based on: Your requirements (what is important to you?) Their requirements (what is important to them)

74. 74 Automate whenever possible Spreadsheets Tools Staff - delegation

75. 75 Part two - The Metrics formula Workload + Performance = (SLA’s or Goals) Workload Metrics (what you do) Performance Metrics (how well you do it) Goal or Results Metrics Are you meeting or exceeding expectations And how do you know without asking or surveying

76. 76 Workload + Performance = Goals/ Results Customize metrics for all: Your requirements (what is important to you?) Their requirements (what is important to them) Toot your Whistle… It’s time to prove with numbers how well you provide services

77. 77 Operations and Metrics examples

78. 78

79. 79

80. 80

81. 81

82. 82

83. 83

84. 84 Power By Data Center-KW

85. 85 Support Metrics Totals Applications Internal 96 Applications External85 Services 1200 Servers3000 Virtual Machines (VMs) 5000 Cloud Instances (AMIs)350 Tickets Created 615 Tickets Closed 579 (94%)

86. 86 Monthly metrics (Support Items)

87. 87 Availability Service Level Agreements (SLAs) May YTD Comments Data Center One Infrastructure (99.9%) 100%99.9534 Data Center Two Infrastructure (99.9%) 100% Application SLAs (99.9%) 99.299.7 Acme-1 90 minute outage on 5/2

88. 88 Monthly Metrics (Workload Items)

89. 89 Part three - Summarizing Weekly Metrics meetings Centralized Metrics repository Metrics forecasting Delegate tasks Automation Accuracy

90. 90 Summarizing the two metric types Operations (Metrics you need) Understand what is important to measure Break into smaller components Continually revise Management/Customers Know your Audience Use graphs when possible Toot your whistle Easy to read Consistent reports

91. 91 Accuracy and consistency Creates visibility for yourself and your team(s) Toot your whistle Job performance Add/promote Staff Grow teams How to use Metrics to advance your career

92. 92 Annual Metrics Reports Easy to read – an overview Year to year comparisons The formula in a simplistic example Workload (what you do) 100 tickets created + Performance (how well, how much) 97 completed successfully = Goals/Results ( Objective vs. Subjective) 97%. Did this meet/exceed… goals, SLA’s, etc.

93. 93 Full Reports Review (Time Permitting) SLA Tracker Availability Formula I use (TM-TMO/TM) x100 = SLA TM - total minutes TNO- outage minutes Annual Report To request copies of these reports, email John at: jparker@esri.comjparker@esri.com

94. 94 SLA Tracker (my best, most useful Metric tool ever) Too large to put in a slide but will demonstrate in session and can email to attendees.

95. 95 Annual Metrics Slide for Application Severity Ones

96. 96 Annualized Achievements Example Exceeded Service Level Availability for Infrastructure and Applications Increased Workload by 30% with No Increase in Staffing Levels Achieved SSAE16 Type I Audit Accreditation Infrastructure Cost Reductions and Transitions

97. 97 Unlike Maxine… I will try and answer your questions?

98. 98 Thank you Tad Davies, Senior Vice President, Consulting Services, Bick tdavies@bickgroup.comtdavies@bickgroup.com Hector Diaz, Sr. Partner, iDiaz Advisors, LLC hdiaz@hdiaz.org hdiaz@hdiaz.org John Parker, Disaster Recovery and Data Center Operations Management jparker@esri.comjparker@esri.com