1. Evolving DB2 CPU and Response Metrics Ned Diehl The Information Systems Manager, Inc Philadelphia CMG 17 November 2007
This presentation will discus sources and interpretations of DB2 CPU and response metrics. The intended audience is traditional large system capacity planners and performance analysts.

2. Trademarks (Omissions are unintentional)
ISM
PerfMan
IBM
Parallel Sysplex
RMF
z/OS
zIIP
MVS
DB2
CICS
IMS
VTAM
SAP

3. Objectives
Discuss sources of DB2 CPU and response metrics with focus on DB2 SMF records
Traditional MVS performance analyst and capacity planner is target audience
Present graphical examples
While the target audience is traditional performance and capacity analysts, some specific DB2 terminology must be used. While attempts will be made to provide high level definitions where appropriate, it may sometimes be desirable to reference more detailed explanations. There is a terminology section near the end. Other good definition sources are the glossary in the DB2 Installation Guide starting with DB2 V8 and DB2 Administration Guide for earlier releases, and your DB2 specialists.
The last several pages also include a list of references and where to find DB2 record layouts.

4. Disclaimers
While all examples shown use real data, they are sometimes selected to make a point. Thus they do not necessarily represent optimally running systems.
“Good” performance can be very subjective and depends on what the objectives are.
Specific field names are listed, but it is best to validate spelling and definition prior to use. Field names can be release dependent.

5. Contents zIIP Overview DB2 Structure Data Sources
Key Performance Metrics
Recommendations
Summary
Record Layouts
Glossary
References

6. zIIP Overview
zIIP is the most recent addition to the family of specialty processors
Others include zAAP (Java), IFL (Linux), ICF (Coupling Facility), and SAP (System Assist Processor)
Initially available on System z9
Cost effective offloading of eligible workloads from standard central processors (GP CP-s)
Each zIIP engine costs significantly less than a GP CP
No IBM software costs
There can be as many zIIP-s as GP CP-s
There can also be other specialty processors
Enclave SRB-s are potential units of work
These are preemptible SRB-s (pre-SRB)
Exploitation started in DB2 V8 with PTF-s
Other vendors and IBM functions have added support
Managed by WLM

7. zIIP Overview DB2 Support
DB2 V8 enabled portions of the following for zIIP:
DDF SQL that uses DRDA and TCP/IP
Approximately 50% is eligible
LOAD, REORG, and REBUILD INDEX utility work
Star schema parallel query processing
DB2 V9 added portions of the following:
Children of long running parallel queries
Over 100ms of CP time
Native SQL stored procedures
Stay in pre-SRB mode rather than switching to TCB
Most other stored procedure time is not zIIP eligible
SAP exploited DDF starting with DB2 V8
Previously it used RRSAF

8. DB2 Structure
MSTR, DBM1, and IRLM will always be present. They are the primary DB2 system address spaces.
DIST will be present if there is server work (from another DB2 or remote system). Requests are not necessarily from another MVS and could be received through TCP/IP.
SPAS(s) will probably be present if there is significant server work. They essentially contain preloaded stored procedures to provide good performance for high transaction rate environments.
RRSAF is the DB2 Recoverable Resource Manager Services attachment facility which uses MVS Transaction Management and Recoverable Resource Manager Service. It is used by SAP.
In many environments most DB2 work is received from batch, TSO, CICS, and IMS address spaces. The connection to DB2 is referred to as a thread. In some cases there might be multiple threads (and thus TCBs) for performance (e.g. a CICS region).

9. DB2 Structure System Address Spaces
MSTR or SSAS - Overall Control
DBM1 or DBAS - Database Services
Buffer Pools
EDM Pool
IRLM - Internal Resource Lock Manager
DIST or DDF - Distributed Data Facility
SPAS - Stored Procedures (Programs)
Primarily Client/Server

10. DB2 Structure General Evolving architecture and data sources
While many key metrics are not old, new functions often accompanied with related measurements
Exploits multiple engines & SYSPLEX
Requestors can have multiple connections (e.g. CICS)
Minimal performance value to multiple production DB2s on an image
Uses Sub System Interface (SSI)
Cross memory services for local
Coupling facility for data sharing
TCP/IP and VTAM for distributed
Preemptible (Enclave & Client) SRBs for DDF and parallel processing
Enclaves and preemptible SRBs are examples of the joint evolution of DB2 and MVS to provide performance, especially for large server workloads.
Enclaves are effectively virtual address spaces. They are much faster and more efficient to establish when needed, but still provide a place for a transaction to execute and be managed by WLM.
A preemptible SRB has many of the desirable characteristics of a TCB (a WLM manageable and interruptible place to execute work) but is much easier and faster to establish.
For most purposes there are still logically two kinds of CPU time – preemptible and non-preemptible. They are effectively what were previously referred to as TCB and SRB. Preemptible includes “traditional” TCB and newer manageable types (preemptible SRB). Non-preemptible is what was always referred to as SRB and is still reported separately.
For detailed explanations some of the listed references are excellent sources, especially the Enrico and Arwe articles.

11. Data Sources Overview RMF - Good starting point
SMF 30 - Job accounting
SMF Dataset performance by job
SMF DB2 System Functions
SMF DB2 Accounting
SMF DB2 Statistics (“GTF” for DB2)
Requestor Measurements - CICS 110 & IMS log
The primary focus will be on the DB2 SMF 100 and 101 records, though there will be some specific references to other sources.

12. Data Sources RMF Most application CPU time reported with requestor
DIST AS is requestor for server work
Separate 72 records for each DDF enclave service class include most server application CPU time
CPU usage reported in service units
R723CCPU: TCB plus SUP & SUC
R723CSRB: SRB (on GP CP)
R723CSUP: Pre-SRB on zIIP
R723CSUC: zIIP eligible on GP CP
R723NFFS: zIIP normalization factor
Workload resource granularity a problem
Especially CICS & IMS
Since server work is received directly from a remote system there is not a normal local requestor (e.g. CICS). The DIST AS is effectively the requestor. There is separate 72 for DDF each enclave service class, which is where most server CPU time is reported.
While DB2 system ASs (other than possibly DIST) do not normally represent a significant amount of CPU time, there are specific DB2 functions associated with reported TCB and SRB times. An easy way to track then is with RMF. If you observe any significant change, consultation with a DB2 specialist is appropriate. He would know what functions are involved and if the change is reasonable.
While RMF reporting will include DB2 application CPU time with other workloads, there is no easy way to break it out from other work.

13. Data Sources SMF 30 Good for batch and TSO if totals desired
Poor granularity for CICS and IMS
Separate records for each DB2 system AS
Totals for system functions
Records for DIST AS include system and DDF application CPU, but net system values can be calculated
The following CPU (GP CP) time fields predate zIIP :
SMF30CPT: traditional TCB (includes ASR, ENC, & DET)
SMF30SRB: traditional SRB
SMF30ASR: preemptible SRB (client, not enclave)
SMF30ENC: independent enclaves
SMF30DET: dependent enclaves
Usefulness of SMF 30 varies significantly with desire and environment. For batch and TSO you might be close to the DB2 view of a “transaction”. The desire might also be how much work someone was doing rather than how much was DB2.
Obviously not very good for IMS and CICS work if the desire is a transaction view or to separate DB2 function.
System functions are easy to measure given the combination of interval records and separate fields for TCB and SRB time.
Note that DIST includes system support for DDF and server application time. Typically most DIST reported time will be application. All of the enclave time (ENC & DET) is application, though some SRB also is.
Notice that ASR, ENC, and DET are included with CPT (TCB) because they are effectively like TCB time.

14. Data Sources SMF 30 zIIP support added the following CPU fields:
SMF30_TIME_ON_zIIP: Pre-SRB on zIIP
SMF30_ENCLAVE_TIME_ON_zIIP: Independent enclave on zIIP
SMF30_DEPENC_TIME_ON_zIIP: Dependent enclave on zIIP
SMF30_TIME_zIIP_ON_CP: zIIP eligible on CP
SMF30_ENCLAVE_TIME_zIIP_ON_CP: Independent enclave zIIP eligible on CP
SMF30_DEPENC_TIME_zIIP_ON_CP: Dependent enclave zIIP eligible on CP
SMF30_ENCLAVE_TIME_zIIP_QUAL: Normalized independent enclave on zIIP
SMF30_DEPENC_TIME_zIIP_QUAL: Normalized dependent enclave on zIIP
SMF30SNF: zIIP time normalization factor
Note that several fields are included in others

15. Data Sources SMF 100 – DB2 SYSTEM
System level interval reporting
System components
Buffer pools
DDF locations
Most metrics (including CPU times) not deltas but are accumulations from start of monitoring
Difficult to synchronize prior to DB2 V7
Consider setting STATISTICS SYNC to 59 with the DB2 DSNTIPN installation panel.
Most application CPU time not included
Some server CPU included as DIST SRB!
SPAS not reported
There are multiple logical subtypes
IFCID 0001 QWSA segments contain CPU statistics by AS
SMF 100 provides a variety of useful system level data, most of which is beyond the scope of this discussion.
All the activity metrics are measurements since DB2 monitoring started, so it is necessary to calculate deltas. Values can potentially wrap (overflow and start over). Some fields are the level of the measured thing at the time of reporting (e.g, number of active buffers).
Prior to DB2 V7 there was no reasonably easy way to synchronize SMF 100 with other SMF records. The interval length also tended to vary but the average length was close to the nominal DB2 specification. Starting with V7, a logical approach is to set STATISTICS SYNC to 59 with the DB2 DSNTIPN installation panel.
Server TCB time other than stored procedure is included with DIST SRB (probably because it really is SRB time) and SMF 101 (discussed later).
DB2 system CPU times are recorded in QWSA segments. There is a segment for each system AS with a logical identifier (QWSAPROC is typically MSTR, DBM1, IRLM, or DIST).

16. Data Sources SMF 100 – DB2 SYSTEM
QWSA fields of interest:
QWSAPROC: Identifier of MSTR, DBM1, IRLM, or DIST
QWSAEJST: TCB time
QWSASRBT: SRB time (includes QWSAPSRB)
QWSAPSRB: Pre-SRB time on GP CP
QWSAPSRB_zIIP: Pre-SRB time on zIIP
QWSA notes:
QWSAPSRB & QWSAPSRB_zIIP have only been observed non-zero for DIST
QWSAPSRB is application CPU
System DIST SRB = QWSASRBT – QWSAPSRB
Normally very small

17. Data Sources SMF 101 – DB2 Accounting
Generally one record per ended transaction
IFCID 0003 contains standard accounting data
Multiples for parallel
Optional rollup record for parallel children
Optional aggregate rollup (V8) reporting for DDF & RRSAF
Some interactive work might not appear to be
TSO
SAP without commit boundary accounting (V8)
No interval reporting
Best source of workload reporting
Summarization tools probably required
QWAC segment contains application (Class 1) and “In-DB2” (Class 2) measurements
Numerous identification fields
Correlation header (QWHC)
Accounting fields (QMDA)
While most interactive work results in an SMF 101 record for what one would logically consider a transaction, there are exceptions. In some cases records with common identifiers are optionally rolled up to reduce volume (parallel children, DDF, and RRSAF).
A DB2 TSO transaction can include many end user interactions because it measures from start to end of connection, which could be a relatively long time.
SAP, which uses RRSAF (SAP R/3 uses DDF starting in DB2 V8), looks like a very long running transaction unless commit boundary reporting is used.
Given the wealth of identifiers and metrics, this is the best source for detail accounting and analysis.
There are two basic types of metrics. Application level are essentially from start of connection until termination. It might include significant elapsed time outside of DB2. In-DB2 measurements are essentially time within DB2. For most analysis, In-DB2 times are the best ones to use.
Identifiers available include who, what, where, and type of transaction. Meaningfulness of specific identification fields varies with environment.

18. Data Sources SMF 101 – DB2 Accounting
Packages/DBRM (Class 7) in QPAC segment (QPAC)
One segment for each package used
Additional identifier of package name (QPACPKNM)
Separate subtype (IFCID 0239) for overflow, V8, or rollup
Must associate with IFCID 0003 to know if rollup
Correlation headers are identical
Subset of thread level metrics
Will not necessarily cross-foot
Often a better source of what is being executed
Stored procedures must be packages
If roll-up no reliable transaction count
Cannot assume all rolled up transactions used all packages
The basic thing identified as the execution unit in an accounting record is the plan (QWHCPLAN), which may or may not be very meaningful. Some installations use blanket plan names as the initial thing invoked. Packages are often invoked by the selected plan. A package is essentially a compiled DB2 program. One or more could be used by a specific plan execution. Assuming package level accounting is active, there will be a QPAC segment for each unique package executed. They might be included in the standard IFCID 0003 SMF 101 record (max of ten segments, not rollup, and prior to V8) or in one or more separate IFCID 0239 SMF 101 records.
If in a separate record you will not know if rollup might have been involved unless you associate it with the IFCID SMF time stamps will probably be equal (certainly very close) and correlation headers (QWHC) will be identical. In the case of rollup, you cannot be sure that all transactions necessarily used all packages.
Metrics are a subset of thread level ones but there is a good set of In-DB2 elapsed, CPU, and delay times. Obviously since not all transactions use packages and a given transaction can use multiple packages, metrics are not likely to easily cross foot with anything.
Stored procedures must be packages but not all packages are necessarily loaded as stored procedures. There is a flag (QPACAAFG) indication if a package was loaded as a stored procedure.

19. Data Sources Requestor Measurements
CICS 110 & other CICS monitors
Improving DB2 resource consumption data
CICS 2.2 added DB2 CPU time to transaction detail
USRCPUT (CMF 008) & USRDISPT (CMF 007) include DB2
L8CPUT (CMF 259) is typically DB2 CPU
Each connection (thread) is separate TCB
With recent levels, probably best source for CICS DB2 workload counts, response times, and CPU
IMS Log
Log 07 (Accounting) record includes DB2 CPU in DLRTIME
Parallel child CPU not included
Difficult to use for accounting or performance analysis
Sometimes requestor measurements are the best sources. With recent code levels, CICS 110 (and other monitors) do a good job of reporting DB2 CPU and response times. There is a further benefit of being closer to real transaction performance if a CICS transaction invokes multiple DB2 requests.
L8CPUT is not necessarily just DB2 CPU time. Thread creation and termination time, which is not reported by DB2, is included. If the application is thread safe, related CPU time will be included. In general if it is a CICS DB2 environment, you can treat L8CPUT as DB2 CPU time for the transaction.

20. Data Sources Comments
Most DB2 CPU time is application TCB & preemptible SRB
Included with requestor totals
DIST is requestor for server work
SMF 30 & RMF 72 include all 101 transaction CPU time
SMF 100 SRB includes some server application CPU
Granular DB2 workload reporting requires SMF 101
Relative use of stored procedures varies significantly among installations
For locally (e.g. batch, CICS, IMS) generated requests, application CPU time is charged to the requestor.
For workload level reporting, RMF is not very useful. SMF 30 is probably only effective for batch and TSO.
SMF 101 is the best source for workload reporting because of the combination of granularity and available metrics.

21. Key Performance Metrics CPU Usage - System
Easily reported
Normally small in prime time
Many functions use SRBs (non-preemptible)
DBM1 SRB typically largest
Primarily asynchronous database I/O
DIST levels could be high – most will normally be application
SMF 100 SRB includes server application home CPU (not stored procedure)
SMF 30 includes all server application CPU in one record, though enclave totals are also in separate fields
RMF produces separate record for each service class
Most application CPU should be in enclave records
If significant change, work with DB2 specialist
Direct relationship between TCB or SRB in an AS, and a set of DB2 functions
RMF report classes can be easiest source

22. DB2 CPU Usage System Address Spaces
This is an example of system address space measurements for a basic environment without any DDF. Note that most CPU time is DBM1 SRB, which is primarily from asynchronous database I/O.
These values are from SMF 100, but could easily have been RMF or SMF 30.

23. Key Performance Metrics CPU Usage -Application
SMF 101 CPU time notes:
Within a class all fields are separate. There are no totals.
Class 2 values are subsets of class 1 or common fields.
Total (Class 1)
QWACEJST – QWACBJST (End - Start): Home GP CP
If rollup record do not subtract QWACBJST
QWACSPCP: Stored Procedure GP CP
QWACTRTT: Trigger not enclave GP CP
QWACTRTE: Trigger enclave GP CP
QWACUDCP: User defined function GP CP
QWACCLS1_zIIP: Home zIIP
QWACTRTT_zIIP: Trigger zIIP
QWACSPNF_CP: Native Stored Proc GP CP
QWACSPNF_zIIP: Native Stored Proc zIIP
To get total Class 1, all values must be added when present.

24. Key Performance Metrics CPU Usage -Application
In-DB2 (Class 2)
QWACAJST: Home GP CP
QWACSPTT: Stored procedure SQL GP CP
QWACTRTT: Trigger not enclave GP CP
QWACTRTE: Trigger enclave GP CP
QWACUDTT: User defined function GP CP
QWACCLS2_zIIP: Home zIIP
QWACTRTT_zIIP: Trigger zIIP
QWACSPNF_CP: Native Stored Proc GP CP
QWACSPNF_zIIP: Native Stored Proc zIIP
Package (Class 7)
QPACTJST: GP CP
QPACCLS7_zIIP: zIIP
zIIP Eligible on GP CP
QWACZIIP_ELIGIBLE: No indication of related fields
To get total Class 2, all values must be added when present.
In-DB2 times are subsets of corresponding totals (Class 1).
Note that for triggers (QWACTRTT and QWACTRTE) there is no separate In-DB2 value. Values are reported when Class 1 recording is active.
Package times are a subset of In-DB2 values.
Note: Recording setup is optional so In-DB2 and package data might not be available even though application metrics are present.

25. DB2 CPU Usage System Plus Application
This example adds total application time to the earlier chart. Application data source is SMF Other data sources would only be useful in special cases (e.g. a pure server environment). Notice that most of the CPU time is application.

26. DB2 CPU Usage Application by Type
This example breaks down application CPU time, from the previous chart, by type of request. Type is one of the correlation header identifiers.

27. DB2 System CPU Usage Server Environment
Negligible zIIP CPU
DIST TCB largest value
MSTR also significant

28. DB2 Application CPU Server Environment
Server (DDF) is dominate application
Over 50% of server could be zIIP
zIIP on GP CP also shown on zIIP to show total potential

29. DB2 Application CPU Server Environment
Most CPU time is In-DB2
Minimal stored procedure use
zIIP on GP CP not shown separately due to duplication

30. DB2 Application CPU Server Environment
The complex contains 1 zIIP and 8 GP CP
Two significant images share the box
SYSA WLM weights of 100% zIIP and 45% GP CP

31. DB2 CPU Usage Server Environment
Estimate of uncaptured CPU included
PDB2 includes all DB2 system functions and monitors
Only DDF applications in separate service class

32. DB2 CPU Usage Server Environment
Significant queuing at only 50% utilization
There was also significant GP CP delay (remember 45% weight)

33. Key Performance Metrics Transaction Rates
Many different choices
SMF 101
RMF
Requestor data
SMF 101 not necessarily one-to-one with user request
Multiple SMF 101 for parallel
Optional rollup records for children
RRSAF (e.g. WebSphere) might use commit level reporting
DDF & RRSAF might be in rollup records
QWACPCNT is aggregate count for rollup records
QWACRINV = 1, 2, or 3 for DDF & RRSAF
Commit count is good for most interactive work
QWACCOMM
Choose based on requirement
Coordinate with other reporting
There are numerous potential sources for transaction counts. Choices will vary with analysis objective and workload. SMF 101 is typically the most flexible because of all the identifiers, though RMF service and report classes and requestor data might be more attractive. If RMF provides desired granularity, it is an attractive easy approach. Requestor measurements have the potential attraction of being closer to logical application counts.
For parallel there is the interesting philosophical choice of what to count. For transaction count (and response time) the root is probably all that matters.
For rollup records, QWACPCNT is the aggregate transaction count.
With most interactive work, commit count tends to be close to transaction count.

34. Key Performance Metrics Transaction Rates
Record Type
QWACPARR
QWACPCNT
QWACPACE
Normal
OFF
ZERO
Parallel Parent
CHILD COUNT
Parallel Child
PARENT QWHSACE
Parallel Rollup
SET
Simple Rollup
ROLL-UP COUNT
N/A

35. Key Performance Metrics Response Times
Several choices
SMF 101
RMF
Requestor data
Application - might include wait for work (e.g. CICS & RRSAF)
In-DB2 - typically best for DB2 analysis
For rollup records, must divide by commit (QWACCOMM) or aggregate (QWACPCNT) count
Match choice with requirement
Should have response objectives
As with transaction counts, there are several source choices for response times, with similar considerations.
When working with DB2 101 records there is the further consideration of choosing either application of In-DB2 measurements. Application times might be attractive for some workloads if more complete measurements are desired.

36. Key Performance Metrics Response Times
Application (Class 1)
QWACESC – QWACBSC
If rollup record do not subtract QWACBSC
In-DB2 (Class 2)
QWACASC: Home AS
QWACSPEB: Stored procedure SQL
QWACTRET: Trigger not enclave
QWACTREE: Trigger enclave
QWACUDEB: User defined function
QWACSPNF_ELAP: Native stored procedure
Package (Class 7)
QPACSCT
Application time is a superset of In-DB2. The two values are simply start and end time stamps unless rollup is involved.
Total In-DB2 is the sum of all 6 values.
Package is a subset of In-DB2.

37. Key Performance Metrics Response Components
SMF 101 provides much detail
CPU Time
DASD Wait
DDF Wait
Exception Delays
Waits & exception delays optionally reported
Application delay rather than event time
In-DB2 (Class 3) in QWAC & QWAX segments
Package / DBRM (Class 8) in QPAC segment
Use to direct efforts when response problems occur
While many of the details are beyond the scope of this discussion, it is worth knowing that numerous time metrics are available that allow for detail response time analysis. We have previously discussed the various CPU components. Given that response targets are not being met, an analysis of component times can direct tuning and upgrade efforts.
Note that delay fields appropriate accounting classes to be active.
Package (QPAC) data in general is not so rich as thread level, but there is a good set of response components available.
Some activities are potentially asynchronous, examples include prefetch (attempt to load records into a buffer pool prior to being needed) and parallel processing. In general the exception and I/O values reported in SMF 101 are delay times rather than total elapsed time of the function.

38. Key Performance Metrics Response Components
Package and application metrics have good resolution
Essentially all server work used packages
OTHER probably CPU delay (zIIP & GP CP)
CPU is sum of all In-DB2 times.
Different rates because some transactions used multiple packages.
OTHER WAIT is sum of service and exception delays

39. Key Performance Metrics Response Components
Package SERVICE is higher because it includes items that are separate application metrics, but not shown.

40. Recommendations Teamwork with subsystem specialists
Track key performance metrics
System
Workload
Use experts if unexpected values or significant deviations
Vary reporting periods and intervals with needs
Keep history
Incorporate with management reporting
For CPU analysis, beware of double counting and missed data
Teamwork is essential. While DB2 (and other subsystem) specialists might have some different objectives and requirements from the typical MVS analyst, there is valuable knowledge and information to share.
Track key system and workload metrics.
When something looks strange or changes significantly, confer with the appropriate subsystem specialist.
For long term analysis (e.g. month) average hours for period profiles or planning intervals for history are probably appropriate. Shorter intervals (e.g. 30 min) are better for daily glitch analysis. Note that the shortest interval is limited by the SMF 100 reporting intervals where DB2 might be different from SMF defaults.
Keep history and incorporate it with management reporting.
Be careful of potential double counting or missed data. Server workloads are especially problematic. Mixing data sources can also confuse the issue.

41. Recommendations Metric Sources
RMF for system components
System CPU usage
Memory
Paging
SMF 101 for workloads
CPU
Transaction rates
Response times
Response components
Depending on objectives, requestor measurements might be better for workloads.
RMF is probably the easiest source for system components, though SMF 100 (DB2 System) and SMF 30 are also usable. Remember that except for server work, application CPU is not included. Also server recording in different across the three sources. (DASD paging is typically not an issue, but RMF is the beat way to watch it. If an issue, check buffer pools for potential over allocation.)
If flexible workload granularity is desired, SMF 101 is good choice for almost all workloads. It is the primary source for response components.
For some workloads (e.g. CICS) requestor measurements might be a desirable choice.

42. Summary Traditional data is available
Many basic metrics are old, but function exploitation often requires corresponding changes
Detail DB2 skills not needed
Much published information
Numerous tools available
Performance analysis, capacity planning, and accounting should be changed for specialty processors
“Excess” zIIP capacity can be cost effective
Typically the primarily measures desired are CPU, response, and transaction counts. They have been available for many years.
Since DB2 and MVS are both evolving, new measurements are periodically added. This tends to have the biggest impact on CPU and response profiles for exploiting workloads. Some environments (e.g. CICS) might have minimal value from use of CPU fields added since DB2 V2.
In general the analysis discussed does not require detail DB2 skills. When something strange happens or a significant change is observer, consult a DB2 specialist.
There are many good references, some of which are listed at the end.
There are numerous tools available that can aid with analysis and reporting

43. DB2 Record Layouts
SMF record layouts are produced by assembling macros in DB2xxx.SDSNMACS, where xxx is release level.
SMF 100 IFCID 0001 : DSNDQWST SUBTYPE=0
System services
SMF 100 IFCID 0002 : DSNDQWST SUBTYPE=1
Database services & buffer pools
SMF 101 IFCID 0003 & 0239 : DSNDQWAS SUBTYPE=ALL
Accounting
Additional field description information in:
DSNxxx.SDSNIVPD(DSNWMSGS) starting with V8
DSNxxx.SDSNSAMP(DSNWMSGS) for older releases
All referenced time fields are in 8-byte TOD-clock format.
Recording options controlled with DB2 DSNTIPN installation panel.
Note that SMF records are not documented in the SMF manual. The combination of the layouts and DSNWMSGS provide good detail. Another useful field description source is IBM Tivoli OMEGAMON XE for DB2 Performance Expert on z/OS Report Reference. GOGGLE can also be a good source of descriptive information.

44. DB2 Terminology
Allied Address Space - AS connected to DB2 that can make DB2 requests.
Application Plan - Effectively a transaction definition. The control structure that is produced during the bind process.
DRDA – Distributed Relational Database Architecture
EDM Pool - Environmental descriptor management pool. A pool in memory used for database descriptors, application plans, and packages.
Enclave - One or more units of work running in the same or multiple address spaces. Effectively a virtual AS.
In-DB2 - Application processing within DB2.
Package - A set of statically bound SQL statements that is available for processing.

45. DB2 Terminology
Preemptible SRB - TCB like but less expensive to create.
Requestor - Source of “transaction” for accounting purposes.
RRSAF - Recoverable Resource Manager Services attachment facility.
Server - Target of requests from a remote system.
Stored procedure - User-written application program that can be invoked through use of the SQL CALL statement.
Thread - Connection between DB2 and requesting AS.
Trigger - A set of SQL statements that is stored in a DB2 database and executed when a certain event occurs in a DB2 table.
User-defined function (UDF) - A function that is defined to DB2 and can be referenced in SQL statements.

46. References
Ned Diehl, "DB2 CPU and Response Metrics", Proceedings, CMG05, p668 and
Ned Diehl, “Measurement and Modeling of DB2 zIIP Workloads”, Proceedings, CMG 2006
Joel Goldstein, “DB2 Version 3 & 4 Performance Metrics”, CMG96 Proceedings, p668 and
Peter Enrico, “Focus Enclaves”, Cheryl Watson’s Tuning Letter 1999, N0.2
John Arwe, “Preemptible SRBs”, CMG95 Proceedings, p646
Chuck Hoover, “Tuning DB2 From The Ground Up”, Share Feb 98 Proceedings, Session 1340
Namik Hrle & Johannes Schuetzner, “Finding Out Who Did It”, IDUG Solutions Journal Volume 11, Number 2 (2004)
Barry Merrill “Captured DB2 CPU Time”, Cheryl Watson’s Tuning Letter 2005, N0.1 & MXG Listserv December 2004
William Favero, “zIPPing Along”, ZJOURNAL August/September 2006
Even though several sources are not especially new, they can still be very useful.

47. References System Management Facilities, IBM
Resource Measurement Facility Report Analysis, IBM
IBM Tivoli OMEGAMON XE for DB2 Performance Expert on z/OS Report Reference, IBM
DB2 Administration Guide, IBM
DB2 Installation Guide, IBM
PerfMan for DB2, ISM
PerfMan for z/OS, ISM
CMG Proceedings
Share Proceedings
IDUG Proceedings & Solutions Journal
Google