Building an MXG PDB from NTSMF Data Chuck Hopf chuck@chopf.com www.chopf.com/geekland.shtml

Building an MXG PDB from NTSMF Data ‘Classic’ MXG PDB Structure Why it doesn’t work What needed to change How it is implemented for NTSMF Using BLDNTPDB

Classic MXG PDB Structure Read ALL the data and place it in a PDB with possibly CICSTRAN on the side Run daily - daily contains detail of ALL data read WEEKBLD consolidates dailies into weeklies MONTHBLD consolidates weeklies into monthlies

Classic MXG PDB Structure DASD storage required: 8 daily PDB 5 weekly PDB enough WORK space for the largest dataset (STEPS or TYPE74)

Why It Doesn’t Work 40 GB of SMF data daily Needs 160GB of WORK space (SORTWK and WORK) 320GB of space for daily datasets 1+ Terror Bytes for weekly datasets Time - Daily jobs that run for more than a day are not practical

What Needed to Change Multiple Parallel Threads Run week-to-date Run month-to-date Eliminate weekly/monthly jobs Exploit summarization macros where needed to reduce volume

How Is It Implemented for NTSMF? BLDNTPDB - a very large and complex collection of SAS MACROs that allows the user to choose how the PDB should be structured. Can use the classic MXG structure or... Can use virtually any structure the user chooses

Using BLDNTPDB Parameters and Meanings Examples of use Future Directions ?????????????????????

Using BLDNTPDB - Parameters and Meanings NTSMF=C:\MXG\NTSMF\NTSMF.SMF - where is your RAW NTSMF data? WEEKSTRT=MON - on what day of the week does your week start? The normal MXG default is Monday. This controls when weekly processing takes place. RUNNTINT=YES - should we run the NTINTRV summarization?

Parameters and Meanings ASUMDUR=HOUR - what level of summarization should we apply? Valid values are QTRHOUR, HALFHOUR, HOUR, SHIFT, DATE, WEEK, MONTH KEEPERS=null - A list of datasets to be ‘kept’ (usually not used) DROPPERS=null - A list of datasets to be ‘dropped’ (built but not kept in the output PDB.)

Parameters and Meanings ZEROOBS=null - Similar to DROPPERS but datasets are created with 0 observations MAXDRVS=26 - Maximum number of logical drives/physical drive (limits and array size.) RERUN=null - If YES is specified forces the rewriting of an old PDB USE WITH CAUTION

Parameters and Meanings MON=C:\MXG\MON TUE=C:\MXG\TUE WED=C:\MXG\WED THU=C:\MXG\THU FRI=C:\MXG\FRI SAT=C:\MXG\SAT SUN=C:\MXG\SUN Dataset placement for DAILY PDBs. If a null string is used, the dataset WILL NOT BE BUILT.

Parameters and Meanings WEEK=C:\MXG\WEEK WEEK2=C:\MXG\WEEK2 WEEK3=C:\MXG\WEEK3 WEEK4=C:\MXG\WEEK4 WEEK5=C:\MXG\WEEK5 Placement for weekly datasets. If a null string is used, the dataset WILL NOT BE BUILT.

Parameters and Meanings TREND=C:\MXG\TREND MONTH=C:\MXG\MONTH MONTH1=null Placement for monthly and TREND datasets. A null string results in the dataset NOT BEING BUILT.

Parameters and Meanings RUNDAY=YES - Run the BUILD process daily and build the specified daily databases RUNWEEK=NO - YES means run the weekly BUILD process if it is the day specified by WEEKDATE and WTD means build WTD datasets daily RUNMONTH=NO - YES means run the monthly BUILD process on the 2nd day of the month and MTD means build MTD datasets daily RUNTREND=WEEKLY - Build the TREND datasets when the WEEKLY BUILD is run. If DAILY is specified, build the TREND datasets when the DAILY datasets are built.

Parameters and Meanings RPTRAW=NO - Build a table of the distribution of the RAW NTSMF record types RPTDAY=PDB - Create reports for the most recent PDB at the hourly RPTWEEK=NO - Create reports for the WEEKLY PDB if it has just been RPTMONTH=NO - Create reports for the MONTHLY database if it has just been created RPTTREND=NO - Create reports for the TREND database if it has just been updated

Parameters and Meanings GRAFDAY=PDB - Create reports for the most recent PDB at the hourly GRAFWEEK=NO - Create reports for the WEEKLY PDB if it has just been GRAFMNTH=NO - Create reports for the MONTHLY database if it has just been created GRAFTRND=NO - Create reports for the TREND database if it has just been updated

Parameters and Meanings Any combination of RUNxxxx, RPTxxxx, and GRAFxxxx specifications may be used. You can do DAILY, WEEKLY, and MONTHLY runs in a single execution as well as creating reports and graphs at each level.

Parameters and Meanings Null strings passed as some parameters means ‘dont do this!’

Examples of Use Classic MXG Structure PDB for each day of week, 5 weeks, monthly with week run on Monday %BLDNTPDB(RUNDAY=YES, RUNWEEK=YES, RUNMNTH=YES, RUNTRND=WEEKLY);

Examples of Use Single daily PDB no WEEKLY/MONTHLY/TREND %BLDNTPDB(RUNDAY=YES,MON=, TUE=, WED=, THU=, FRI=, SAT=, SUN=);

Examples of Use Single daily PDB with week-to-date, month-to-date, and trending done daily %BLDNTPDB(RUNDAY=YES, RUNWEEK=WTD,RUNMNTH=MTD, RUNTRND=DAILY,MON=,TUE=,WED=, THU=, FRI=, SAT=, SUN=);

A Few Words About MXG 16.04 and Later Each dataset now has an up to 6 character mnemonic identifier. In the case of NTSMF data it starts with NT and then has 4 unique characters. This is used to identify what datasets to KEEP DROP or set to ZEROOBS. YOU MUST REVIEW THE 16.04 CHANGES and DOCUMENTATION!

Examples of Use Build and keep only the datasets needed for NTINTRV %BLDNTPDB(KEEPERS=NTBROW NTCACH NTFTP NTMEM NTNBTC NTNBUI NTNETS NTNWLI NTNWLN NTNWLS NTOBJ NTPAGE NTPDSK NTPROC NTPROR NTRDIR NTSERV NTSYST NTTCP);

Examples of Use Build only the datasets needed for NTINTRV and DROP after NTINTRV is created: %BLDNTPDB(DROPPERS=NTBROW NTCACH NTFTP NTMEM NTNBTC NTNBUI NTNETS NTNWLI NTNWLN NTNWLS NTOBJ NTPAGE NTPDSK NTPROC NTPROR NTRDIR NTSERV NTSYST NTTCP);

Examples of Use Build the datasets needed for NTINTRV but build the PROCESS dataset with zero observations: %BLDNTPDB(KEEPERS=NTBROW NTCACH NTFTP NTMEM NTNBTC NTNBUI NTNETS NTNWLI NTNWLN NTNWLS NTOBJ NTPAGE NTPDSK NTPROR NTRDIR NTSERV NTSYST NTTCP, ZEROOBS=NTPROC);

Examples of Use Reporting can be added to any execution: %BLDNTPDB(xxxxxxxxxxxxxxx, RPTDAY=YES);

Examples of Use Reporting can be done against any PDB: %BLDNTPDB(xxxxxxxxxxxxxxx, RPTDAY=MON);

Examples of Use GRAPHS can be created for daily, weekly and trend datasets (CPU usage only at this time - more to come.) %BLDNTPDB(xxxxxxxxxxxxx, GRAFDAY=YES, GRAFWEEK=YES);

Examples of Use - A Case Study Suppose you want the following: Run the PDB DAILY keeping all datasets Maximum of 5 logical drives/physical drive Start the week on Sunday Keep a WTD/MTD/and TREND datasets Run reports of Daily usage Create Graphs of Daily usage

Examples of Use - A Case Study %BLDNTPDB(MON=, TUE=,WED=,THU=,FRI=, SAT=,SUN=,WEEK2=,WEEK3=, WEEK4=,WEEK5=, RUNDAY=YES,WEEKSTRT=SUN, RUNWEEK=WTD,RUNMNTH=MTD, RUNTRND=DAILY,MAXDRVS=5, RPTDAY=PDB,GRAFDAY=PDB);

Future Directions Largely up to YOU! My knowledge of the data is limited My experience with the data is limited I need to know what YOU want/need

Future Directions What reports are needed Data sources Data elements Levels of summarization

Future Directions What graphs are needed What data sources What kinds of graphs What levels of summarization

Future Directions Currently on the development list: Selection of datasets to place in the WEEKLY/MONTHLY databases - a WEEKKEEP= operand or something along those lines More automated summarization (this is a tough one because it supposes knowledge of the datasets and variables that is difficult to have programmatically.)

Future Directions Currently on the Development List Ability to limit variables in WTD/MTD datasets using DROP= Ability to output reports as HTML

???????????