Technology & Operations - Enterprise Infrastructure Enterprise Platform Services Customer Experiences with HiperDispatch & Soft Capping in IBM Mainframe Systems Prepared for presentation at CMG Canada on April 18, 2012 Jonathan Gladstone, P.Eng. Senior Tech. Specialist, Mainframe & Mid-range Systems Capacity Planning Technology & Operations BMO Financial Group

Technology - Enterprise Infrastructure Enterprise Platform Services Technology & Operations - Enterprise Infrastructure Enterprise Platform Services 2Customer Experiences with HiperDispatch & Soft Capping in IBM Mainframe Systems - J.Gladstone, BMOFGApril 18, 2012 Agenda  HiperDispatch A high-level discussion of BMO’s implementation of this feature  Soft Capping Detailed presentation with circles and arrows and a paragraph… and apologies to Arlo Guthrie  BONUS TOPIC! Transition to z196 – Performance Implications Just a preview; detailed analysis not yet complete

Technology - Enterprise Infrastructure Enterprise Platform Services Technology & Operations - Enterprise Infrastructure Enterprise Platform Services 3Customer Experiences with HiperDispatch & Soft Capping in IBM Mainframe Systems - J.Gladstone, BMOFGApril 18, 2012 HiperDispatch: how is it supposed to work?  HiperDispatch aligns workloads “vertically” on physical CPs Builds a strong affinity between logical and physical processors - details available in zJournal article viewable at  Applies to all processors by type, when logically shared: zAAPs, zIIPs, GCPs VH (100%), VM (50-99%), VL (<50%; discretionary) by weight, but avoiding VL where possible  Purports to improve performance by reducing latency times, e.g. for CP state re-loads Keep data and instructions in lowest-level (fastest) cache  Performance improvement claims vary depending on configuration Largest (8-10%) for large, multi-book CECs with many large systems sharing logical resources extensively; least (0-2%) for single-book CECs with few systems with limited sharing.  Performance improvements baked in to LSPR ratings for z/OS 1.11 and up Turning off HiperDispatch now yields less than optimal performance.

Technology - Enterprise Infrastructure Enterprise Platform Services Technology & Operations - Enterprise Infrastructure Enterprise Platform Services 4Customer Experiences with HiperDispatch & Soft Capping in IBM Mainframe Systems - J.Gladstone, BMOFGApril 18, 2012 HiperDispatch: how is it applied and working at BMO?  Turned on when we went to z/OS 1.11 Default in z/OS 1.11 and up is “ON” for HiperDispatch; we left it that way  Nasty surprises! Specialty workload flowing back to GCPs VL engines left parked while workloads not meeting WLM target performance  Causes? Investigated multiple changes: new z/OS, zAAP-on-zIIP, HiperDispatch  Fixes? Set zIIP (and zAAP) weights properly – never mattered before Changed GCP weights to minimize impacts Reviewing WLM profiles  Results? Things working much better now (see third discussion regarding z196 performance)  Conclusions HiperDispatch appears to yield performance benefits claimed by IBM, but…  Weights are now important all the time (not just when box is maxed out)  WLM profiles are more important than ever Some situations still difficult (e.g. K-LPARs for GDPS)

Technology - Enterprise Infrastructure Enterprise Platform Services Technology & Operations - Enterprise Infrastructure Enterprise Platform Services 5Customer Experiences with HiperDispatch & Soft Capping in IBM Mainframe Systems - J.Gladstone, BMOFGApril 18, 2012 Soft Capping: how is it supposed to work?  Soft Capping is available for single LPARs or “Capacity Groups” (CGs) of LPARs Available since 2005 or earlier Applies only to GCPs Uses same MSU ratings as SCRT reports for VWLC  Limits CPU utilization of LPAR or Group based on four-hour rolling average (4HRA) Checked by PR/SM every 5 minutes  Utilization can go as high as enabled capacity until 4HRA hits cap; then PR/SM will limit utilization until 4HRA drops below cap again A little more complicated for Capacity Groups: cap is applied to individual software products rather than for LPARs, and LPAR weights are used as needed  4HRA can exceed cap After Cap is reached, utilization at cap will often increase 4HRA for a few intervals until it settles back Reports suggest 4HRA will exceed cap by about 3% in these circumstances  IBM VWLC charge is based on cap rather than on actual utilization True for whole CPC if CG includes all LPARs

Technology - Enterprise Infrastructure Enterprise Platform Services Technology & Operations - Enterprise Infrastructure Enterprise Platform Services 6Customer Experiences with HiperDispatch & Soft Capping in IBM Mainframe Systems - J.Gladstone, BMOFGApril 18, 2012 Soft Capping: how is it applied and working at BMO?  In place at BMO in four instances (three current) One z10 Production CEC from Jul/09 through Jan/10 One z196 Dev/Test/QA CEC from Sep/11 through present Two z196 Production CECs from Jan-Feb/12 through present  SCRT reports show one instance of capping at BMO Nov. 19, 2011 in z196D1: SCRT report shows MSU utilization hit 321 MSU on cap of 312 MSU in capacity of 408 MSU  Analysis based on data from TDS/z and from SCRT reports as submitted to IBM All IBM tools  Interesting results, with differences to customer benefit Data clearly show Soft Capping working, as documented No charge for over-utilization, as documented Unexplained difference between computed value and SCRT report is to customer advantage

Technology - Enterprise Infrastructure Enterprise Platform Services Technology & Operations - Enterprise Infrastructure Enterprise Platform Services 7Customer Experiences with HiperDispatch & Soft Capping in IBM Mainframe Systems - J.Gladstone, BMOFGApril 18, 2012 Soft Capping: November, 2011 – Cap takes effect  Capacity 408 MSU  CG cap 312 MSU  Utilization above cap for several intervals beforehand  4HRA computed from TDS/z crosses cap around 01:20 Rises to MSU  Peak hourly 4HRA 325 MSU according to TDS/z, only 321 MSU according to SCRT 1.3% difference due to truncation instead of averaging? PR/SM not counted  Working exactly as expected!

Technology - Enterprise Infrastructure Enterprise Platform Services Technology & Operations - Enterprise Infrastructure Enterprise Platform Services 8Customer Experiences with HiperDispatch & Soft Capping in IBM Mainframe Systems - J.Gladstone, BMOFGApril 18, 2012 Soft Capping: December, 2011 – Cap doesn’t kick in  Capacity 408 MSU  CG cap 312 MSU  Utilization above cap for several intervals in morning hours  4HRA computed from TDS/z never crosses cap Rises to MSU  Peak hourly 4HRA 301 MSU according to TDS/z, only 297 MSU according to SCRT 1.2% difference due to truncation instead of averaging? PR/SM not counted  Still working exactly as expected!

Technology - Enterprise Infrastructure Enterprise Platform Services Technology & Operations - Enterprise Infrastructure Enterprise Platform Services 9Customer Experiences with HiperDispatch & Soft Capping in IBM Mainframe Systems - J.Gladstone, BMOFGApril 18, 2012 Soft Capping: January, 2012 – Effects of POR  Capacity 408 MSU  CG cap 312 MSU  POR around 00:50 drops 4HRA to 1MSU as documented SMF70LAC catches up with results computed from TDS/z around 04:40  Utilization above cap for several intervals  4HRA computed from TDS/z never crosses cap Rises to MSU  Peak hourly 4HRA 291 MSU according to TDS/z, only 285 MSU according to SCRT 2.1% difference due to truncation instead of averaging? PR/SM not counted  Still working exactly as expected! 4HRA computed from TDS/z never crosses cap Rises to MSU

Technology - Enterprise Infrastructure Enterprise Platform Services Technology & Operations - Enterprise Infrastructure Enterprise Platform Services 10Customer Experiences with HiperDispatch & Soft Capping in IBM Mainframe Systems - J.Gladstone, BMOFGApril 18, 2012 Soft Capping: February, 2012 – Cap takes effect, unreported  Capacity 408 MSU  CG cap 312 MSU  Utilization above cap for several intervals beforehand  4HRA computed from TDS/z crosses cap around 17:10 Rises to MSU  Peak hourly 4HRA 316 MSU according to TDS/z, only 303 MSU according to SCRT 4.3% difference due to truncation instead of averaging? PR/SM not counted  Capping never shows in SCRT, but capping effect still clear

Technology - Enterprise Infrastructure Enterprise Platform Services Technology & Operations - Enterprise Infrastructure Enterprise Platform Services 11Customer Experiences with HiperDispatch & Soft Capping in IBM Mainframe Systems - J.Gladstone, BMOFGApril 18, 2012 Transition to z196 – Production Site Experience  Before upgrades: z10ECs, one book with GCPs, zAAPs and zIIPs  After upgrade: z196s, one book with GCPs and zAAP-on-zIIP  Drops in CPU demand evident for GCP and Total utilization MIPS normalized using LSPR (1.9 for z10, 1.11 for z196)  CPU demand normally rises to a peak at end of February, RRSP season Driven by transactions  Analysis will look at MIPS per transaction for several workload classes CICS DB2 WebSphere Batch

Technology - Enterprise Infrastructure Enterprise Platform Services Technology & Operations - Enterprise Infrastructure Enterprise Platform Services 12Customer Experiences with HiperDispatch & Soft Capping in IBM Mainframe Systems - J.Gladstone, BMOFGApril 18, 2012 Transition to z196 – Dev/Test/QA Site Experience  Before upgrade: z10EC, two books with GCPs, zAAPs and zIIPs  After upgrade: z196, one book with GCPs and zAAP-on-zIIP  CPU demand rises for GCP and Total utilization MIPS normalized using LSPR (1.9 for z10, 1.11 for z196)  Not explained Harder to analyse D/T/Q environment  Analysis will look at MIPS per transaction for several workload classes CICS DB2 WebSphere Batch

Technology - Enterprise Infrastructure Enterprise Platform Services Technology & Operations - Enterprise Infrastructure Enterprise Platform Services 13Customer Experiences with HiperDispatch & Soft Capping in IBM Mainframe Systems - J.Gladstone, BMOFGApril 18, 2012 Summary  HiperDispatch Drives performance benefit, but… Requires extra vigilance in setting LPAR weights (GCP, zAAP, zIIP) Requires careful review of WLM profiles Has difficulty where normally low-utilization systems (e.g. GDPS K-systems) need high weights  Soft Capping Performs as expected – yay! Minor added benefits (SCRT calculation, PR/SM left out)  Transition to z196 Performance expectations based on LSPR for z/OS 1.11  Includes HiperDispatch Performing better than expected Detailed analysis pending Questions?

Technology - Enterprise Infrastructure Enterprise Platform Services Technology & Operations - Enterprise Infrastructure Enterprise Platform Services 14Customer Experiences with HiperDispatch & Soft Capping in IBM Mainframe Systems - J.Gladstone, BMOFGApril 18, 2012 About the Author  Jonathan Gladstone is an IT Capacity Management professional with well over 20 years experience in computer systems management and planning. He has been at BMO Financial Group for almost 15 years, and working in capacity planning for over a decade. He is BMO’s representative on Georgian College’s Computer Studies Advisory Committee, is certified in ITIL v2 & v3 fundamentals and holds a B.A.Sc. degree in Electrical Engineering from the University of Toronto and P.Eng. certification from the Province of Ontario.  Jonathan wishes to thank many colleagues who helped with this presentation, in particular Steve Pritchard (BMO Financial Group), Horace Dyke (independent consultant) and Don Mackay (IBM Canada).  Jonathan can be found on LinkedIn, Twitter and on his own (largely I/T) blog,