Presentation is loading. Please wait.

Presentation is loading. Please wait.

Shelter Insurance Companies © MMIX Improving z/OS Capacity and Performance with Parallel Sysplex Michael J. Giglio Shelter Mutual Insurance Companies CMG.

Similar presentations


Presentation on theme: "Shelter Insurance Companies © MMIX Improving z/OS Capacity and Performance with Parallel Sysplex Michael J. Giglio Shelter Mutual Insurance Companies CMG."— Presentation transcript:

1 Shelter Insurance Companies © MMIX Improving z/OS Capacity and Performance with Parallel Sysplex Michael J. Giglio Shelter Mutual Insurance Companies CMG #9029

2 2 Shelter Insurance Companies © MMIX Knee Ankle Bad Day Hardware

3 3 Shelter Insurance Companies © MMIX z/OS Parallel Sysplex-in-a-Box Sysplex Flavors, Features and Functions On-the-job-training for Sysplex One-piece-at-a-time Things that worked Things that needed help How much What is next

4 4 Shelter Insurance Companies © MMIX Parallel Sys-what? Five Levels of Syspelx 1.Base Sysplex features No Coupling Facility (CF) 2.Parallel Sysplex Resource Sharing CF and shared structures 3.Parallel Sysplex Data Sharing 4.Continuous operations 24 x 7 x Disaster recovery 24 x 7 x 365 Survive facility failure Ten-plus-year-old Technology Coupling Facility CF LPAR CF Hardware Internal CF External CF CF Links (IODF) Couple datasets Structures Policies PARMLIB COUPLExx members

5 5 Shelter Insurance Companies © MMIX Sysplex in a Box Conventional Parallel Sysplex Multiple System z Processors Numerous z/OS Images Multiple System z Coupling Faculty (CF) images Internal CF External CF CF Images Redundant connections among hardware Sysplex in a Box One System z processor Multiple z/OS Images One or more Internal CF (ICF) engines One or more CF Images No physical connections among Images

6 6 Shelter Insurance Companies © MMIX MVS Production MVS Development MVS Test Parallel Sys-what? Base Sysplex ESCON CTC

7 7 Shelter Insurance Companies © MMIX MVS Production MVS Development MVS Test Coupling Facility Coupling Facility Parallel Sys-what? Parallel Sysplex About 40 times faster ESCON CTC

8 8 Shelter Insurance Companies © MMIX MVS Production MVS Development MVS Test Coupling Facility Coupling Facility Parallel Sys-what? Parallel Sysplex About 40 times faster

9 9 Shelter Insurance Companies © MMIX Backup Coupling Facility Backup Coupling Facility Coupling Facilities and Structures Coupling Facility Coupling Facility List Structures Lock Structures Cache Structures MVS …

10 10 Shelter Insurance Companies © MMIX Resource Sharing Structures Structure NameNicknameType Cross System FacilityXCFList Global Resource SharingGRSLock Enhanced Catalog SharingECSCache RACF Database SharingRACFCache Intelligent Resource DirectorIRDCache MVS System LoggerLOGRList

11 11 Shelter Insurance Companies © MMIX Why Resource Sharing… Cross System Facility (XCF) Global Resource Sharing (GRS) Enhanced Catalog Sharing (ECS) RACF Database Sharing System Logger LOGREC OPERLOG Intelligent Resource Director (IRD) LIST Signaling Structures MVS talking to MVS Authorized Program communication VTAM Cross-domain traffic Simplified configuration Better performance Easier to recover

12 12 Shelter Insurance Companies © MMIX Why Resource Sharing… Cross System Facility (XCF) Global Resource Sharing (GRS) Enhanced Catalog Sharing (ECS) RACF Database Sharing System Logger LOGREC OPERLOG Intelligent Resource Director (IRD) LOCK “GRS Star” Maintain status about shared resources Uses less real storage More capacity Better response time More available Easier to recover

13 13 Shelter Insurance Companies © MMIX Why Resource Sharing… Cross System Facility (XCF) Global Resource Sharing (GRS) Enhanced Catalog Sharing (ECS) RACF Database Sharing System Logger LOGREC OPERLOG Intelligent Resource Director (IRD) CACHE Catalog information Maintains VVDS changes in CF Less I/O to locate datasets Less CPU time Less elapsed time No GRS serialization of volumes

14 14 Shelter Insurance Companies © MMIX Why Resource Sharing - ECS Shared DASD Shared ICF Catalogs Catalog components Basic Catalog Structure (BCS) VSAM Volume Data Set (VVDS) Non-VSAM Volume Record (NVR) VSAM Volume Record (VVR) MVS1 Catalog Address Space MVS1 Catalog Address Space MVS2 Catalog Address Space MVS2 Catalog Address Space CF V V R GRS serializes access to volume for VVR access

15 15 Shelter Insurance Companies © MMIX Why Resource Sharing… Cross System Facility (XCF) Global Resource Sharing (GRS) Enhanced Catalog Sharing (ECS) RACF Database Sharing System Logger LOGREC OPERLOG Intelligent Resource Director (IRD) CACHE Part of RACF DB in CF Avoid Cache Invalidation in MVS Less Memory Less I/O Easier to administer

16 16 Shelter Insurance Companies © MMIX Why Resource Sharing - RACF DB MVS1 RACF Address Space MVS1 RACF Address Space MVS2 RACF Address Space MVS2 RACF Address Space CF RACF DB Invalidating Cache Buffers on each MVS system Changes go to both CF copy and DB copy

17 17 Shelter Insurance Companies © MMIX Why Resource Sharing… Cross System Facility (XCF) Global Resource Sharing (GRS) Enhanced Catalog Sharing (ECS) RACF Database Sharing System Logger LOGREC OPERLOG Intelligent Resource Director (IRD) LIST Write log information to CF Can also write to DASD Manage combined LOGREC data Single view of Sysplex logs

18 18 Shelter Insurance Companies © MMIX Why Resource Sharing… Cross System Facility (XCF) Global Resource Sharing (GRS) Enhanced Catalog Sharing (ECS) RACF Database Sharing System Logger LOGREC OPERLOG Intelligent Resource Director (IRD) CACHE Coordinates WLM across MVS images Dynamically manage shared CPs and Channel Paths

19 19 Shelter Insurance Companies © MMIX …Why Resource Sharing Simplify configuration Systems connect to CFs instead to each other Fewer connections for new MVS images Eliminate ESCON Elegance Performance, capacity, and throughout I/O avoidance Shared cache among z/OS images No increase in MIPS or MSUs Prepare for Data Sharing Hardware Experience Education

20 20 Shelter Insurance Companies © MMIX The Parallel Sysplex Training Environment (PSTE)

21 21 Shelter Insurance Companies © MMIX Defining the PSTE Pre-generated Parallel Sysplex Three z/OS images Two CF images Use z/VM or five LPARs Tool for training System programmers Operators Hands-on exercises Alternative to going to class Big Honkin’ Book Over 1,800 pages Five inches thick Twenty-three “parts” Predefined Sysplex exercises

22 22 Shelter Insurance Companies © MMIX Why use the PSTE Tool for training Implement the structure in a non- disruptive environment Learn as you go Cheaper than sending people to class Pay by the year, not by the student Renew for as many years as needed Use to develop processes Document as you learn Not enough load to measure performance chagnes

23 23 Shelter Insurance Companies © MMIX One-piece-at-a-time Implementation Inspiration 1976 Johnny Cash hit Worked at a automobile manufacturing plant Stole a car “one piece at a time” Assembled car parts Different models Multiple years Pieces Hardware installation PSTE Implementation First structure Second structure … Last structure

24 24 Shelter Insurance Companies © MMIX Hardware Installation Existing z9 processor New z hardware for Sysplex Internal Coupling Facility (ICF) Memory IODF LPARs MVS: Three for PSTE CF: PSTE1 and 2 CF: Primary and Alternate Internal Coupling Links Between MVS MVS LPARs and CF LPARs Shared Resources System z GP … ICF z/OS LPARS: Production MVS images CF LPARs: PSTE1 PSTE2 Primary Production CF Memory Three PSTE MVS images Secondary Production CF

25 25 Shelter Insurance Companies © MMIX Failure Isolation for Production System z GP … ICF z/OS LPARS: Production MVS images CF LPARs: PSTE1 PSTE2 Primary Production CF Memory Three CF LPARs on ICF engine Dynamic Dispatch “OFF” Highest level of performance 90% Share of LPAR One CF LPAR on GP engines Dynamic Dispatch “ON” Backup structures Consumes MSUs only on failure of Primary CF or components Three PSTE MVS images Secondary Production CF

26 26 Shelter Insurance Companies © MMIX Iterations for Implementing Resource Sharing Learn Document Do Measure Adjust Repeat Read the manuals for the structure Use the PSTE for hands on learning - do more than once if you need to Notes for implementing structure Procedures for Operations Share experiences with the team Implement the structure in production CF datasets, policies, structures, parmlib, commands, etc. Observe system behavior Reporting (RMF, Omegamon) Fix anything that needs it Buffers, parameters, etc.

27 27 Shelter Insurance Companies © MMIX Iterations for Implementing EACH Structure Cross System Facility (XCF) Signaling Structures Global Resource Sharing (GRS) Star Enhanced Catalog Sharing (ECS) RACF Database Sharing System Logger LOGREC OPERLOG Intelligent Resource Director (IRD) Learn Document Do Measure Adjust Repeat ~ Three weeks per iteration

28 28 Shelter Insurance Companies © MMIX Things that worked

29 29 Shelter Insurance Companies © MMIX GRS and XCF Parallel Sysplex Implementation Parallel Sysplex benefits for GRS and XCF (communications among MVS systems) Base Sysplex ESCON device speed Channel-to-channel 17MB Parallel Sysplex Coupling Facility Speed 40 times faster than ESCON Sort of violated “one-piece-at-time” Implemented GRS and XCF together Made it simpler Same configuration changes apply to both

30 30 Shelter Insurance Companies © MMIX MVS Production MVS Development MVS Test Coupling Facility Coupling Facility GRS and XCF Parallel Sysplex About 40 times faster ESCON CTC

31 31 Shelter Insurance Companies © MMIX GRS RingGRS Star Batch Minutes Nightly Batch Throughput Hypothesis GRS and XCF Star will reduce amount of time to run MVS batch JOBs The more DD statements a JOB has The more Star will reduce the run time Reality We were correct JOBs with lots of DD statements finish faster with Star 11% Savings on Batch 260 Days a year

32 32 Shelter Insurance Companies © MMIX TSO Logon TSO Logon JCL in PROCLIB Almost 100 DD statements Each time you log onto TSO, went through GRS Ring for each dataset Logon typically takes over 30 seconds for each TSO user About 200 TSO users Lots of logging on and off several times every day After GRS and XCF Star Less than five seconds per logon Hundreds of times a TSO ;-)

33 33 Shelter Insurance Companies © MMIX Enhanced Catalog Sharing ECS Measurements Batch utility – synchronize RMM and ICF catalogs Must run with no tape activity Before ECS – ran over one hour After ECS – ran less than one minute Fast Dump Restore (FDR) reports Before ECS – about five minutes After ECS – under ten seconds Storage Admin “Since these JOBs run faster, other JOBs that accesses the cataloged VSAM datasets also run faster…” GRS / XCFRingGRS / XCF Star Run times (in minutes) for RMM / VSAM Catalog Sync Job

34 34 Shelter Insurance Companies © MMIX Solving the “Blockade” problem… Blockade product “Enterprise Security Server” Authorization for Windows applications using RACF Uses RACF security to validate access to PC applications Timing is critical Bad things can happen One or more MVS systems at high CPU utilization GRS ring delay Blockade attempted to authenticate user’s request Answer did not come back soon enough May I… Yes or no When things get ugly: RACROUTE call Enqueue RACF database GRS request to Ring MVS Dev system running near 99% GRS Ring token does not come back soon enough User request times out

35 35 Shelter Insurance Companies © MMIX …Solving the “Blockade” problem XCF and GRS Star Response time for enque is 40 times faster RACF enqeue asks the Coupling Facility Answer always comes back in time Don’t care how busy DEV system is No ring delay Now unable to recreate the problem

36 36 Shelter Insurance Companies © MMIX Intelligent Resource Director (IRD)… Workload management (WLM) LPAR CPU Management Resources are dynamically moved across LPARs where they are needed Virtual CPUs automatically configured ON and OFF IRD Decisions Uses LPAR CPU Weight settings as defined on HMC Honors WLM values across LPARs Especially useful is using Defined Capacity WLM becomes Sysplex Aware IRD WLM

37 37 Shelter Insurance Companies © MMIX …Intelligent Resource Director (IRD) PROCESSOR STATUS ID CPU SERIAL EAE W 02 + W 014EAE W 014EAE2096 PROCESSOR STATUS ID CPU SERIAL EAE W 024EAE W 024EAE EAE2096 “W” Indicates WLM managed CPU One CPU Offline (set by IRD) All CPUs Online (set by IRD) MVS Command: D M=CPU

38 38 Shelter Insurance Companies © MMIX Minor Pains and Adjustments

39 39 Shelter Insurance Companies © MMIX “Pain” of Bits Backwards… Two Bits for RACF database Resource Sharing One bit for “enable Sysplex communications” OK to communicate Need to issue RVARY command to turn ON communication One bit for “data sharing at IPL” Permanently turns on Data Sharing Turns of ON this will override communications bit Needs to match for all systems in the Sysplex Supposed to work like this Turn on “communications bit” IPL z/OS Use RVARY command Turn on data sharing Do on all MVS images in Sysplex To make permanent Turn on “sharing at IPL” bit IPL z/OS You need to know which bit is which

40 40 Shelter Insurance Companies © MMIX …Pain of Bits Backwards… PSTE book says: 00 Sysplex communication is not enabled 0101 Sysplex communication is enabled 1010 RACF requests data sharing mode at next IPL (Treated the same as the 11 setting because it requires that Sysplex communication be enabled.) 11 Sysplex communication is enabled and RACF requests the data sharing mode at the next IPL. 00 RACF is not enabled for Sysplex communication RACF is enabled for Sysplex communication, and requests non–data sharing mode at IPL RACF requests data sharing mode at IPL. Because this requires Sysplex communication, RACF is also enabled for Sysplex communication. (This bit setting is treated the same as the 11 setting.) 11 RACF is enabled for Sysplex communication, and requests data sharing mode at IPL. RACF book says: We coded this Expecting this We got this!!

41 41 Shelter Insurance Companies © MMIX …Pain of Bits Backward Did not have this problem on the PSTE PSTE ships two different PDSs One with bits off One with Communication ON Data sharing OFF Swap libraries in link list IPL PSTE MVS system Use RVARY command to trun on data sharing Put into production by Turn on bit for “communication” Assembling module IPLing z/OS (MVS TEST) MVS IPLed with sharing ON on only one image of our Sysplex Fortunately we were able to put it back before we caused any damage What did we learn? PSTE is a training tool PSTE is NOT a reference guide

42 42 Shelter Insurance Companies © MMIX Measure and Adjust - XCF Buffers… RMF XCF Activity Report Since we changed XCF, I decided to run the report and take a look What I saw was already a problem with or “Base Sysplex” configuration Nobody bothered to run the report Problem is Buffer Overrun Data moving across XCF is bigger than the biggest XCF buffer XCF goes thought overhead to Dynamically create a larger buffer Move the data using the new buffer Delete the larger buffer Non-productive overhead for XCF To fix Create right-sized (bigger) buffers

43 43 Shelter Insurance Companies © MMIX …Measure and Adjust - XCF Buffers BUFFER ALL TO TRANSPORT BUFFER REQ % % % % PATHS REQ SYSTEM CLASS LENGTH OUT SML FIT BIG OVR UNAVAIL REJECT DEVL DEFAULT EIGHTK 8,124 59, FOURK 4,028 4, , BUFFER ALL TO TRANSPORT BUFFER REQ % % % % PATHS REQ SYSTEM CLASS LENGTH OUT SML FIT BIG OVR UNAVAIL REJECT DEVL DEFAULT EIGHTK 8, , FOURK 4,028 2, SIXTEENK 16,316 20, THIRTY2K 32,700 6,

44 44 Shelter Insurance Companies © MMIX Coupling Facility Activity - RMF XCF 10% GRS 63% RACF 1% ECS 26% IRD 0% Number of CF requests XCF 49% GRS 31% RACF 1% ECS 18% IRD 1% Percent of CF Utilization

45 45 Shelter Insurance Companies © MMIX How much WhatHow Much z9 Internal Coupling Facility $140,000 z9 Memory – 8 GB$60,000 PSTE$40,000 All Together$240,000 Also available for Data Sharing Used about 3GB for this project One year of PSTE Plus our time doing the work…

46 46 Shelter Insurance Companies © MMIX What is next to Sysplex Resource Sharing to implement SMF data Research Data Sharing Determine availability requirements Use the PSTE to learn about the data sharing structures DB2 IMS MQ CICS VTAM Deliver an outline for Data Sharing implementation Work Time Money Benefit

47 47 Shelter Insurance Companies © MMIX SOME places to find out more BookWhat it helped with IBM zSeries Value of Resource SharingGetting Started IBM - S/390 Parallel Sysplex: Resource SharingGeneral Stuff Parallel Sysplex Training Environnent Installation Guide Exercise Guide Installing and using the PSTE z/OS MVS Setting Up a SysplexPretty much everything z/OS Security Server RACF System Programmer’s Guide RACF structures z/OS DFSMS Access Method Services for CatalogsECS structures z/OS Resource Measurement Facility (RMF) Report Analysis User’s Guide Monitoring and tuning of CF and Structures

48 48 Shelter Insurance Companies © MMIX z/OS Parallel Sysplex-in-a-Box Sysplex Flavors, Features and Functions On-the-job-training for Sysplex One-piece-at-a-time Things that worked Things that needed help How much What is next

49 49 Shelter Insurance Companies © MMIX Final Thoughts “This instrument can teach, it can illuminate, and yes it can even inspire. But it can only do so to the extent that humans are determined to achieve those ends. Otherwise it is nothing but wires and lights in a box.” Edward R. Murrow teach inspire illuminate

50 50 Shelter Insurance Companies © MMIX Good Day and Good Luck

51 51 Shelter Insurance Companies © MMIX


Download ppt "Shelter Insurance Companies © MMIX Improving z/OS Capacity and Performance with Parallel Sysplex Michael J. Giglio Shelter Mutual Insurance Companies CMG."

Similar presentations


Ads by Google