1. Bill Arledge DB2 Data Management Strategist Mainframe DB Recovery
DB2 Recovery 101 Overview
Bill Arledge
DB2 Data Management Strategist
Mainframe DB Recovery
9/21/2018

2. Availability is critical …
Lost revenue…
Up to $6 million/hour for e-businesses
2 - 3% of annual revenue for every 10 hours of database outage
Lost customers…
Nearly 14 times your initial investment to win customers back
Lost market share…
Approximately 1/2 point of market share for every 8 hours of outage (It will take an estimated 3 years to win customers back)
9/21/2018
©2007 BMC Software

3. Recovery Elements CREATE FAILURE RECOVERY EXECUTE JCL RECOVERY
ANALYSIS
RECOVERY MANAGEMENT
FAST UTILITIES
APPLICATION OUTAGE
9/21/2018
©2007 BMC Software

4. When Availability is Critical, Recovery is Crucial!
Unplanned downtime is an unfortunate fact of life….
Up to 80% of all unplanned downtime is caused by software or human error*
Up to 70% of recovery is “think time”!
How many of you have mission-critical systems? How many of you have e-business applications? Did you know that industry analysts estimate that each minute of database downtime could cost your business up to $96K? That’s $6M/hour.
Unfortunately, unplanned database downtime is a fact of life. And 80% of unplanned downtime is caused by software or human error. McGladrey and Pullen estimates that 50% of all businesses who experience a critical system outage of 10 days or more will NEVER recover. In those 10 days, the average business loses 2 - 3% of its annual revenue. For every 8 hours of outage, the average business loses 1/2 point of market share, which will take a full 3 years to recover. Database downtime can also result in lost customers. If you lose a customer, it costs 14 times your initial investment to get them back. If you CAN get them back.
And industry research shows that the average DBA has less than a year of experience.
So Are you ready to recover from unplanned downtime? Are you confident you can recover from any type of outage? Do you know how to minimize your time to recovery?
*Source: Gartner, “Aftermath: Disaster Recovery”, Vic Wheatman, September 21, 2001
9/21/2018
©2007 BMC Software

5. DB2 Recovery Resource Review
Large lines = User data flow
Updates to Tablespace are Logged, Copied
Small lines = DB2 info flow
Copies and Logs are registered
Log range for Tablespace recovery tracked
MVS ICF Catalog
Watches over all
ICF Catalog
Full Copy or
Incremental Copies
SYSLGRNX
TABLESPACE
When a table space or index space is opened for update, an entry is made in SYSLGRNX to record the RBA range of the recovery log during the time when this object was being updated.
Image copies are registered in the DB2* catalog table called SYSCOPY.
Incremental image copies can be merged together with full image copies to form a new full image copy before a recovery is required.
During recovery, SYSCOPY is scanned to find the most current full image copy for the object being recovered. SYSLGRNX is then scanned to determine the RBA range on the DB2 recovery log that will be used to apply all updates forward.
The Boot Strap Data Set will be used to match the RBA ranges with the appropriate Active and Archive log data sets for the recovery.
Archive Logs
Active
Log or
SYSCOPY
BSDS
9/21/2018
©2007 BMC Software

6. ICF Catalog All DB2 pagesets must be cataloged
Cataloging updates three MVS system files
VTOC - Volume Table of Contents
VVDS - VSAM Volume Data Sets
ICF - Integrated Catalog Facility
ICF Catalog
CREATE TABLESPACE
TS IN DP000001
USING VCAT DB2PVCAT
...
All DB2 objects (tablespaces and indexes) must be cataloged. If you are using user defined objects (also known as VCAT defined), you must allocated the datasets manually (via the MVS utility IDCAMS). If you use DB2 STOGROUPs, the allocation is performed for you by DB2.
Each DASD volume with DB2 objects has a Volume Table of Contents (a VTOC), and a VSAM Volume Data Set (VVDS). There is a shared Integrated Catalog Facility dataset (sometimes called a ‘User Catalog’) which points to the appropriate volume(s) for a cataloged dataset. If the dataset spans volumes, the ICF entry contains a list of all volumes.
DB2PVCAT.DSNDBC.DP TS I0001.A001
DB2PVCAT.DSNDBD.DP TS I0001.A001
VTOC, VVDS
9/21/2018
©2007 BMC Software

7. Boot Strap Data Set (BSDS)
Active and Archive Log Inventories
Active Log is reused, Archive Log is serial up to (V8 and later)
Current Active Log is DS01
Next Archive Log will be A004
Active Logs
DS01
DS02
DS03
ACTIVE RBA
LOG START END STATUS
DS FFFF Not Reusable
DS FFFF Reusable
DS FFFF Reusable
ARCHIVE RBA
LOG START END VOLSER
A FFFF VOL=T00001
A FFFF VOL=T00002
A FFFF VOL=T00003
BSDS
Archive Logs
A001
A002
A003
The Boot Strap Data Set (BSDS) is fundamental to DB2 restart and recovery. There are usually two copies of the BSDS. Among other things, it contains the Active and Archive Log inventory. This inventory keeps track of which Active and Archive Log datasets contain particular log ranges. Once the recovery utility has determined what log data needs to be applied (from SYSLGRNX), the BSDS is used to determine the appropriate Log dataset.
There are usually at least three Active Logs defined per DB2 subsystem (there can be up to 32). The Active Logs are usually dual copied. The size of the Active log varies on shop processing characteristics and recovery requirements. A common allocation is about 300 Cylinders (3390 device). As Active Logs fill, DB2 switches to the ‘next’ Active log, and writes the full Active Log to an Archive Log dataset. Active Log datasets are reused, so that when the ‘last’ Active Log is filled, DB2 switches back to the ‘first’ Active Log. Be sure the ‘first’ Archive Log has completed writing to Archive! Otherwise, DB2 will stop until the Active Log is available.
Archive Logs are serial. Once an Archive Log has been written (typically to cart or tape), it is not reused. The Archive Log inventory can track up to the last 1000 Archive Logs (again, usually dual copied).
Naming conventions usually looks like this (n = 1 or 2):
BSDS - Vcat.BSDS0n
Active - Vcat.LOGCOPYn.DSxx (from 02 to 32)
Archive - Vcat.ARCHLOGn.Axxxxxxx (from to )
9/21/2018
©2007 BMC Software

8. BSDS and Relationship to the LOG
Incomplete UR Summary
Record
Open Pageset Summary
Database and Pageset
Exceptions Summary Record
LOG
DB2
Checkpoint
BSDS
Active & Archive Log Dataset Information
DDF Communication
Record
Checkpoint Queue
9/21/2018
©2007 BMC Software

9. Print Log Map (DSNJU004) DSNJU004 (Print Log Map) provides
Active log data set information
Archive log data set information
System checkpoints
Driven by
LOGLOAD ZPARM
Active log switch
Local Time
GMT
Note: The number of archive logs that can be recorded in the BSDS is subject to a limitation set in the installation but, in any case, no greater than 1000.
Some of the output from a DSNJU004
LTIME INDICATES LOCAL TIME, ALL OTHER TIMES ARE GMT.
DATA SHARING MODE IS ON
HIGHEST RBA WRITTEN B660F8E :51:57.8
HIGHEST RBA OFFLOADED B150FFF
ACTIVE LOG COPY 1 DATA SETS START RBA/LRSN/TIME END RBA/LRSN/TIME DATE LTIME DATA SET INFORMATION
A B150FFF :36
DSN=DSNDBE.DBE2.LOGCOPY1.DS01
B71A19155E B71C PASSWORD=(NULL) STATUS=REUSABLE
:38: :33:46.8
00097B C230FFF :14
DSN=DSNDBE.DBE2.LOGCOPY1.DS0
B71C PASSWORD=(NULL) STATUS=NOTREUSABLE
:33:
9/21/2018
©2007 BMC Software

10. DB2 Logs – A Shared Resource
Primary usage to provide for restart and recovery of DB2 subsystems and objects
Other uses including audit and data migration
Log records
Record updates for all spaces with before and/or after images
Data and index pages
Record DDL operations (updates to catalog pages)
Capture checkpoint and restart information including
transaction info
Object exception information
Are identified by a log point
RBAs (Relative Byte Addresses) for non-data sharing
LRSNs (Log Record Sequence Numbers) for data sharing
Include the page (and row if applicable) being impacted
When spaces share a subsystem (or data sharing member) for update, they share log data sets. There is one set of log data sets per member. A commit for one transaction potentially externalizes log records for other transactions. When using RECOVER for one space, log records may be read and bypassed that are system records or log records for other spaces. Since each data sharing member has its own logs, a strategy of updating applications on different members would isolate those applications for log use.
Twelve Byte Hexadecimal Numbers
9/21/2018
©2007 BMC Software

11. DB2 RECOVERY LOG MANAGER
DB2 Log Data Flow
RBA 1000
RBA 2000
DB2 RECOVERY LOG MANAGER
LOG BUFFERS
INSERT
ONE ROW
Archive
Log
LOGSWITCH
Active
BEGIN_UR
RECORD
UNDO / REDO
DATA
INDEX
COMMIT
RECORDS
END_UR
OPEN
PAGESETS
Checkpoint
Log Buffers
RBA (Relative Byte Address)
Written to Active Log
Full Active written to Archive
A quick DB2 Log refresher course.
As updates are executed on behalf of the user community, their activity is logged in the DB2 recovery log. The DB2 recovery log stores before and after images of data that is being updated, before images of data being deleted, and after images of data being inserted. These images are stored in the form of undo/redo records.
The process starts by assigning a unit of recovery ID (URID). Following this, any pagesets that need to be opened are opened; this activity is also logged to the recovery log.
Not only are data updates logged, but so are index modifications. Index modifications occur when new key values are inserted, old values are deleted, or RIDs are added to an existing key value list.
If any index maintenance, such as page splits are required, all the operations to accomplish the split are logged as well. The bottom line is that not just one record is written to the recovery log when an update is performed to a data row. In fact, over one hundred records may be written to accomplish a single update.
9/21/2018
©2007 BMC Software

12. DB2 Log Data Breakdown Other Data 15% 20% Commit 5% Checkpoint 10% 50%
Index
50%
Checkpoint
10%
Commit 5%
Other
15%
UNDO
REDO
9/21/2018
©2007 BMC Software

13. DB2 Log Archival Process
Active log offload is triggered by several events, including:
active log data set is full
Starting DB2 and an active log data set is full
ARCHIVE LOG command
Two uncommon events also trigger the offload:
An error occurring while writing to an active log data set
Filling of the last un-archived active log data set
Write to Active Log
Triggering Event
Update the BSDS
Write the Archive
Offload Process
ARCHIVING THE ACTIVE LOG
9/21/2018
©2007 BMC Software

14. DSN1LOGP - Looking at Log Records
DSN1LOGP is a standalone utility available with DB2
IBM does not set out to document everything you see in a detail report
You can still get lots of information but not easily
Most recovery experts would find a more sophisticated log tool handy to
Re-create SQL from the log
report and filter on transaction and column data more effectively and
handle compression and other issues
DSN1LOGP is free when you buy DB2. It does have quite a few limitations but can be useful. Some parts of the log records are formatted and interpreted but some are not. Some items in the unformatted portion of the output can be interpreted by reviewing the DSNDQJ00 macro provided with DB2,.
If you have a more sophisticated log processing tool available you should learn how to use it and do non-destructive testing with it.
9/21/2018
©2007 BMC Software

15. DSN1LOGP – Usage Examples
Who updated that table that's not supposed to be updated?
Who DROPPED that DATABASE?
Did BADUSER update anything?
Are GRANTs and REVOKEs being done outside of our control?
Who FREEd that PLAN or PACKAGE?
Are SAVEPOINTs being executed on this subsystem?
Can I find a common quiet point for two tablespaces for Recovery?
Sample summary log record for a Unit of Recovery (UOR)
DSN1151I DSN1LPRT UR CONNID=TSO CORRID=BADUSER AUTHID=BADUSER PLAN=DSNESPCS
START DATE= TIME=14:29:20 DISP=COMMITTED INFO=COMPLETE
STARTRBA=024AC666C4C7 ENDRBA=024AC666C76B STARTLRSN=BCA426C98AED ENDLRSN=BCA426C98C3E
NID=* LUWID=USBMCN01.DEBALU.BCA4267B984C.0001
COORDINATOR=* PARTICIPANTS=*
DATA MODIFIED:
DATABASE=0600=KMMSEGDB PAGE SET=0002=KMMSEGS
9/21/2018
©2007 BMC Software

16. Identifying Relevant Log
The SYSLGRNX directory table records log ranges containing updates to a space (or partition)
There are entries for each data sharing member updating and
these entries give the location range on the logs (relative byte address--RBA) and
the relative time range (log record sequence number--LRSN) to coordinate with copies and other logs
SYSLGRNGX output provided by REPORT RECOVERY utility
Identifies assets required for recovery
The log ranges in the SYSLGRNX table help control the log read for any given space, but ranges can be open for long periods. Ranges are kept for index spaces only if they are currently defined COPY YES.
The active log data set parameters ROSWITCH CHKPTS and ROSWITCH TIME (DSNZPARM values PCLOSEN and PCLOSET) are responsible for controlling when log ranges are terminated if a space or partition becomes inactive. The end value will reflect the last time the space was updated so infrequently updated spaces will have fairly discrete log ranges. However, frequently updated spaces may have quite long ranges extending over many logs.
LRSN (log record sequence number) values in the SYSLGRNX rows coordinate the logs from different members of a data sharing group. They are relative byte addresses if a system is non-data sharing.
9/21/2018
©2007 BMC Software

17. DB2 Directory Table SYSLGRNX Open update log ranges on the DB2 Log
Provides for faster recovery
SYSLGRNX
COPY
Current
LOG A B C D E F
Open log ranges
QUIESCE
DBID PSID Start Range End Range
F A B
F C D
F E F
Quiet Point
The Directory table SYSLGRNG/SYSLGRNX keeps track of when an object is open for update. When a tablespace is first opened for Read/Write, a row is inserted to SYSLGRNG/SYSLGRNX which contains the current Log RBA (Relative Byte Address). When the tablespace is closed for Read/Write, the SYSLGRNG/SYSLGRNX row is updated, and the End Range is populated with the current RBA. This range of RBAs (Start RBA to End RBA) represent a part of the DB2 log where the tablespace MIGHT have been updated.
DB2 automatically closes SYSLGRNG/SYSLGRNX entries for quiet or ‘dormant’ tablespaces. DB2 does this by internally switching the tablespace to Read Only mode. By default, this occurs every 5 checkpoints or 10 minutes. This has the effect of minimizing log processing during recovery. (There’s no reason to mount and read hours of log data if the tablespace was in read only mode). In the event of recovery, the SYSLGRNG/SYSLGRNX information is used to help determine which Active/Archive Logs the recovery will need to process. The BSDS has Begin and End RBA ranges for each log dataset.
9/21/2018
©2007 BMC Software

18. Spaces Absent in SYSLGRNX
Some catalog and directory spaces don’t have SYSLGRNX entries
DSNDB01.SYSUTILX
DSNDB01.DBD01
DSNDB01.SYSLGRNX
DSNDB06.SYSCOPY
DSNDB06.SYSGROUP
DSNDB01.SCT02
DSNDB01.SPT01
SYSUTILX has to be recovered first and independently so that other RECOVERs can have information about utilities executing on spaces and so they can be terminated if necessary. The copy of SYSUTILX is recorded in the log and it has no log ranges. SYSUTILX must be recovered independently and it must examine every log record after the copy looking for updates.
9/21/2018
©2007 BMC Software

19. Log Compatibility
Since log records reference pages and rows in spaces and spaces are identified by internal IDs:
certain activities make one series of log records incompatible with others
require a new copy or starting point
LOAD REPLACE completely resets the data and REORG and REBUILD change row and key entry locations
Certain DROPs can be disastrous
Log records in the subsystem log are not logical in the sense that they rely on the internal location of the information (internal space IDs, page numbers and an arbitrary ID within a page). Even log records for index spaces are by page. This causes certain activities to make different ranges of log records incompatible. This means that RECOVER needs a new starting point. LOAD REPLACE and REORG can be run with LOG YES. These activities create a new starting point without a copy. (LOAD REPLACE with a DUMMY SYSREC and LOG YES is often used to conveniently reset a space.) A REBUILD of an index in COPY YES status makes RECOVER impossible until a new copy is made. (REBUILD sets an index space with COPY YES set into the advisory status ICOPY.)
If a data base, table space or table is accidentally dropped, not only can you not necessarily use RECOVER (because SYSCOPY entries and SYSLGRNX entries are lost when a table space is dropped) but you can not readily apply log after restoring a space from an image copy you might locate. There are tools directed at this problem. Some may require preparation in advance.
9/21/2018
©2007 BMC Software

20. Log Roadblocks update row, page E2, row 1 insert row, page 1F2, row 7
log apply
DB2 LOG
log apply
REORG is executed
Row is now on page E2, row 1
COPY
COPY
RECOVER cannot process with inconsistent log records. The log apply shown on the left can be done with its corresponding copy and the log apply shown on the right can be done with its corresponding copy, but they cannot be combined. All important events for the recovery of the spaces are in the SYSCOPY table. The events are identified by the ICTYPE column. That’s how RECOVER identifies this road block.
SYSCOPY Columns
Column Description
DBNAME Name of the database.
TSNAME Name of the target table space or index space.
DSNUM Data set number or partition number. A zero indicates all.
continued
9/21/2018
©2007 BMC Software

21. Problem Categories
An expert categorizes failures and plans and performs accordingly. There are three common possibilities.
A media failure destroys data or compromises it (Disk failure or controller or cache failure occurs)
Data becomes logically compromised by an incorrect job or transaction
The data center is unusable (aka disaster)
When recovery must be performed, there are three general possibilities. One is a hardware failure that has affected the DB2 data. As RAID features protect more and more data from the head crashes of the past (by mirroring or parity recovery schemes), disk failure is less likely to cause a problem in DB2. However, the disk enclosure and connections may fail in such a way that any data in the enclosure is potentially compromised. There is more hardware and software in the enclosure that might fail than ever before.
A second possibility is that the programs and transactions acting on the data have done something wrong. This is called a logical or application failure. The DB2 subsystem generally appears to be operating normally.
The third possibility for a recovery is that a disaster has effectively destroyed the primary data center.
9/21/2018
©2007 BMC Software

22. Media Failure A media failure destroys data or compromises it
Identify volume contents and RECOVER or REBUILD for traditional DASD
Identify objects affected by storage component and RECOVER or REBUILD bearing mind that some may not be affected because they weren’t recently updated
With a storage failure, there are three steps:
Find out what DB2 objects are affected. (Volume or storage unit.)
Identify unaffected data sets if it is a RAID unit failure.
Prepare a RECOVER or REBUILD for every object affected to get them back to the exact desired state (often called ‘current’).
Note that application spaces are not single points of failure as long as multiple copies are made and dual logs are maintained. The log supports bringing copies to current. DB2 subsystems containing important data should use dual logs, actives and archives. The two copies of active logs and the two copies of the BSDS (boot strap data set) should be on separate packs or RAID units. If this rule is not followed and there is a problem with the storage containing logs or BSDS, data loss may occur and a conditional restart of the system followed by recovery of all spaces to a prior time may be required.
9/21/2018
©2007 BMC Software

23. Pop Quiz - Index Recovery
Does a table space recovery
mean indexes must be
rebuilt (or recovered)?
Maybe Not
Recovery to current (for a media failure) would not
require it . Some objects are being recovered to overcome
media failure. Related objects should still be consistent if they were unaffected by the media failure.
It is only necessary to use RECOVER or REBUILD or some other technique for the objects that are damaged or missing as long as TOLOGPOINT (or TOCOPY) is not used. Normally you wouldn’t use these options when a media failure had occurred.
9/21/2018
©2007 BMC Software

24. Logical Data Corruption
Data has become logically compromised by an incorrect job or transaction
An expert finds the cause of the problem.
An expert knows the possible tools to use
whether it is a set of RECOVER and REBUILD statements or
a special program or
a special log tool
or some combination of the above
If DB2 thinks everything is fine, but the users don’t think so...then you have a logical problem! On rare occasions the DB2 subsystem might compromise the data. For example, there have been failures to externalize changed pages, failures to properly issue locks, etc. There is also a nasty sub-class of user error when a DBA or other authorized person drops an object such as a table, table space or data base. Usually, however you are looking for a bad or duplicate batch job, a bad transaction or a user who has done something wrong.
9/21/2018
©2007 BMC Software

25. Finding Corruption Point
Look for a place where everyone agrees data
wasn’t corrupted. Get as close to now as
possible!
POINT A
What happened in between
POINT B
DB2 LOG
With data corruption, you need to find a prior time when the data was valid. This may be quite subtle. Just because no users noticed a problem between opening of business and 10AM, doesn’t mean everything was fine at the beginning of the day. If the source of the corruption is not known, the symptoms may be revealing. For example, if users are complaining that a group of customers seem to have duplicate order items and orders are grouped and processed at night, it is reasonable to assume that a duplicate batch job was run last night. If users are complaining about missing data, transactions or jobs that do deletes might be examined.
DSN1LOGP and more sophisticated log tools can be invaluable in researching such problems. Searching for log data on recent logs for the table spaces in question can reveal the problem.
Application Data
Is fine
Application Data
Is Corrupted
9/21/2018
©2007 BMC Software

26. Consistency Point Batch Job Batch Job Rerun
If RECOVER must be used, a point of consistency across affected
table spaces must be located and any good updates after that point
will be lost.
Online Trans
Online Trans
DB2 LOG
Batch Job
Batch Job Rerun
In the case shown here we have discovered a batch job that was rerun creating the problem. The corruption begins a bit earlier than when it was discovered, at the start of that job. Other work was going on for the same application tables. It might have been independent of the batch job but some or all of it might have depended on the values and rows corrupted by the batch job. This is an application question.
One possibility is to go back to the time after the first batch job using a point-in-time recovery (RECOVER with TOLOGPOINT on all the table spaces and COPY YES indexes and a rebuild for other indexes). It looks like from this diagram that nothing was happening for the spaces involved in our application prior to the batch job rerun. In a real situation, however, we have to discover this point. If we had a QUIESCE after every batch job, we can find the QUIESCE in the SYSCOPY table for our spaces.
Application Data
Now Corrupted
Application Data
Is fine
9/21/2018
©2007 BMC Software

27. Looking for QUIESCE Points
Official points are recorded in SYSCOPY generally as a result of execution of the QUIESCE utility
Useful queries for evaluating available quiesce (quiet) points on the DB2 log
Identifies all Quiesce
Points after the
specified ICDATE
SELECT DBNAME, TSNAME, ICDATE, ICTIME, HEX(START_RBA)
FROM SYSIBM.SYSCOPY WHERE DBNAME IN ('LSBX', 'LSBQ')
AND ICTYPE = 'Q' AND ICDATE > ’020124' ORDER BY ICDATE, ICTIME;
SELECT HEX(MAX(START_RBA))
AND ICTYPE = 'Q';
SELECT DBNAME, NAME FROM SYSIBM.SYSTABLESPACE WHERE
DBNAME IN ('LSBX', 'LSBQ')
AND (DBNAME, NAME) NOT IN (
SELECT DBNAME, TSNAME
FROM SYSIBM.SYSCOPY WHERE
START_RBA = (
SELECT MAX(START_RBA) FROM SYSIBM.SYSCOPY WHERE DBNAME IN ('LSBX', 'LSBQ') ) );
Identifies the latest
Quiesce point for a set
Of objects
The first SQL shown on this slide shows all QUIESCE entries after a certain date for some databases. The second gives the maximum QUIESCE point. The third determines if one of the spaces does not have an entry at the QUIESCE point. These statements could be adapted to various situations. Caution should be taken if partitioned table spaces are quiesced by PART. This is not recommended.
Identifies related objects
With no entry at the latest
quiesce point
9/21/2018
©2007 BMC Software

28. Recovering from Logical Errors Using SQL Processes
If the result of the batch job was undone with SQL then the online transactions might be preserved and it would not be necessary to find a point of consistency for RECOVER.
Online Trans
Online Trans
Online Trans
DB2 LOG
Batch Job Rerun
Batch Job Reversed
The picture has been altered here to push the bad batch job to the left and show a (possibly more realistic) situation where online transactions are constantly running and overlapping.
SQL that reverses everything done by the batch job provides several advantages.
The SQL corrections could also run with other transactions.
All the online transactions shown would still be reflected in the data.
It would not be necessary to find a point of consistency.
SQL to reverse the changes can be obtained by using a tool capable of creating UNDO SQL. (At least three vendors have one.) You could also write your own SQL. You could do this manually or using an application log and a planned procedure for processing it. It’s quite difficult to use the log as a source of manual data. For example, the log may contain incomplete updates; the log is in internal DB2 format; the log doesn’t include the key qualifiers nor column and table names; and the log data may be compressed. SQL reversing the changes must update the records with old data for updates, insert deleted rows; and delete inserted rows. To execute correctly when rows were affected multiple times, these inserts, updates and deletes should be in the reverse order from the original SQL. Cascading deletes or updates need to be accounted for as well.
Application Data
Now Corrupted
SQL INSERT
SQL DELETE
9/21/2018
©2007 BMC Software

29. Caveats for a SQL approach
Using SQL to correct logical errors has some possible pitfalls
The transactions being preserved may have depended on the incorrect data
DB2 LOG
BAD!
transaction changes
addresses
transaction changes
salesmen based on
addresses
Good!
Using SQL to undo corruption can have some striking advantages. However, there are also some difficulties. In the illustration, transactions change customer addresses using an old tape, regressing many customers’ addresses. Then good transactions switch the salesmen assigned in customer records based on zip code. Some of these transactions are incorrect because of the bad addresses. If we reverse the address changes, the salesmen assigned might still be wrong. Some tools can locate rows that had the address changes and also had salesmen changes.
Here is another example of depending on incorrect data in later transactions.
A corrupt transaction inserts rows for new customers using an incorrect credit evaluation. Some of the customers place orders and are given a confirmation on a credit plan based on the credit evaluation in the customer row. The order application requires the customer row and it’s already there so one is not inserted.
If new customer rows are deleted, the order rows are orphaned. If the new customer rows are not deleted, they all have incorrect credit evaluations.
If many rows were inserted and few orders subsequently placed by those customers, one approach is to reverse the inserts with the incorrect credit calculation except for those where the customers made orders. Some tools can create a report to help you locate those rows.
9/21/2018
©2007 BMC Software

30. Caveats for a SQL approach
Using SQL to correct logical errors has some possible pitfalls
The transactions that ran during or after the corruption may have also updated the same column in some of the rows corrupted
DB2 LOG
many employees 401K deductions set to zero
Employee requests to set new percentages for 401K processed
Good!
Columns that are updated by corrupt transactions and subsequently updated by good transactions with proper values may be incorrect if they are reversed based on the log records for the corrupt transaction.
Many employee rows are updated by a corrupt transaction to zero a column that represents the 401K deduction percentage.
Good transactions change some of these rows to have a new and correct value for employees that had submitted changes. For example, Jane Doe had a percentage of 10% that was then erroneously zeroed out. Subsequently, a request was processed from Jane to set it to 5%. Other employees had similarly submitted new amounts.
If the log records for all the updates setting the values to zero are examined in isolation, Jane might have her percentage set to 10% again. So the good transaction for her request is effectively reversed.
Some products can produce reports to point out these anomalies or have techniques to help avoid them.
In this case, using point-in-time recovery on the application spaces to a point before the bad transaction and then ‘replaying’ the good transactions would work. Some products can create REDO SQL for this purpose.
Columns that are accumulated can cause complex problems as in this example.
A corrupt transaction adds one to a column that counts the number of requests that a customer will be charged for in accessing a legal database. The column initially held a value of 110 so after the corrupt transaction it is 111.
Subsequently a good transaction adds two to the column making it 113.
The column value should be the original number in the column (before the corrupt transaction) plus two or 112.
If the update by the corrupt transaction is reversed in isolation and the original value returned, the value will be For this reason, doing a traditional point-in-time recovery in cases like this and then rerunning the application to process the increments may be better practice.
Some tools may be able to locate rows that have this type of anomaly, allowing a manual update for them and reversal for others.
BAD!
9/21/2018
©2007 BMC Software

31. Caveats for a SQL approach
Allowing access to application spaces while they are corrupted may cause problems as in these examples
If a group of customers were accidentally deleted from your data base, then allowing salesmen to continue placing orders might cause them to recreate customer rows because they don’t see the rows. When a insert is attempted for the customer to correct the delete, it will likely receive -803 or cause an improper duplicate customer record
Customer shipping addresses corrupted might cause a label to be printed (read only) and packages to be misdirected
If users may be reinserting accidentally deleted data as in the first bullet point, a log tool that bypasses -803 errors is useful.
Tools can help but you must know your data. If you are not the application specialist you must engage the specialist to help avoid compounding logical errors.
9/21/2018
©2007 BMC Software

32. Image Copy An image copy is a sequential dataset Image Copies
Contains page images from the tablespace or indexspace
Represents at least one data set of a space and at most a complete space (all data sets or partitions)
Image Copies
Can be made while changes are taking place (SHRLEVEL CHANGE) or
Can be made allowing only reads so they are consistent (SHRLEVEL REFERENCE)
Registered in SYSCOPY and accessible via SQL SELECT
REPORT RECOVERY identifies copy required for recovery
No guarantee that a copy in SYSCOPY is not deleted or not cataloged.
Can be used to UNLOAD data
Deleted by
DROP DDL against the space
Potentially by the MODIFY utility
Image copies are made with utilities, primarily COPY but also with other utilities (for example, LOAD and REORG). They are recorded in SYSCOPY, a DB2 catalog table. They contain logical pages from the space. There are also utilities to make new copies from multiple incrementals or a full and subsequent incrementals and functions to combine copies and logs to make a more up-to-date copy. For example, IBM’s MERGECOPY combines copies to create a new copy and DB2 Change Accumulation Tool uses copies and logs to create a new copy. One image copy (a separate sequential file) is for a data set, partition or the entire space. The most you can put in one image copy is one space (all partitions or data sets). You cannot have one sequential data set for all the spaces in a data base, for example. Nor can you copy a single table from a table space with multiple tables.
SYSCOPY information is deleted when a space is DROPPED and some rows are cleaned up when you run the MODIFY utility.
There are a few examples of copies that are not simple sequential data sets. Some utilities support hardware-assisted backups and record them for a special restore at RECOVER time. An example of this with the IBM COPY offering is the CONCURRENT COPY using the features of the some 3990 controllers.
9/21/2018
©2007 BMC Software

33. Image Copy Types space copy
Multiple, identical image copies (four) may be made. They are identified as:
Primary or Backup; and
Local or Recovery site.
Image copies may be made with only changed pages. These are incremental image copies.
IBM RECOVER uses the backup copy if the primary copy is unavailable. LOCAL or RECOVERY copy is selected based on the SITETYP ZPARM or can be overridden with a LOCALSITE or RECOVERYSITE option on the RECOVER. Other vendors may allow different types or syntax for selection from these identical copies. Multiple identical copies are usually made simultaneously during the COPY process. However, there are utilities that make and register additional copies later by copying the copy. IBM has a separate utility for this, COPYTOCOPY starting in V7.
The syntax for making a full or incremental copy is the FULL YES/NO option. FULL NO (incremental) may take less time to produce and may take up less space in the sequential data set.
Incrementals must be used with a FULL copy. If they can’t be used, the log after the last usable copy can replace them. Therefore you may not wish to make backups of incrementals. Incremental copies are more efficient if the table space has the TRACKMOD YES option set. This option causes an area in the space maps to be maintained that shows the pages updated since the last full or incremental copy. This option will cause updates to the space maps and logging of these updates. If no incremental copies are planned, it is more efficient during updating to have TRACKMOD NO.
You cannot make incremental copies of index spaces with IBM COPY. Incremental copies are a way to limit the log processed by RECOVER without having to make a full copy but they require a more complex RECOVER and, if indexes are being copied and updated (e.g. INSERTs are occurring) then the incrementals may not substantially reduce RECOVER time.
x nK pages
space
copy
9/21/2018
©2007 BMC Software

34. SHRLEVEL CHANGE Copiespg pg 1
page 2
copied pg 2
... pg
FFF0
copy
ends
page
FFF2
copied pg
FFF1
contents
of copy
page FFF2
update
page 2
update 1 2 3 4 5
DB2 LOG
Every expert should be confident that SHRLEVEL CHANGE copies work and why.
A SHRLEVEL CHANGE copy is registered (in SYSCOPY) at a point before any updates that might be missing from it. There is no registration of the log point of the end of the copy. The COPY utilities guarantee that every log record before the registration point is reflected in the pages on the copy. Updates made during the copy may or may not be on the copy. In this case, Page 2 is copied without the update that occurred during the copy process. Page FFF2 is copied after the update that occurred during the copy process and contains it.
During a RECOVER utility execution using the copy in this illustration, the pages from the copy and the log records representing the two updates are matched. Page 2 will not have the update shown on the copy. (It was copied before the update.) There is a log point that is on every logical page in the table space and that remains on the copied pages. This value is called PGLOGRBA in the Diagnosis Guide. It is a log record sequence number. The log point of every update is compared to the log point in the page before the update is applied. The log point of page 2 on the copy will be less than the log point of the update shown. The update will be applied. The log point in page FFF2 is equal to the log point of the update shown for it. A RECOVER utility does not attempt to apply that update.
copy
begins
9/21/2018
©2007 BMC Software

35. External Copies Unknown to DB2
Data set dumps made by DFDSS or DSN1COPY or other mechanisms
Aren’t registered but may be used by
Restoring known copies that are consistent because the space was stopped or DB2 was cleanly stopped
Restoring a complete set of system data
‘flash copied’ or ‘snapped’ between the SET LOG SUSPEND and SET LOG RESUME commands or
made while DB2 is down after it was taken done cleanly
and then restarting DB2.
Relying completely on data sets dumps is not a wise idea. Important data should be image copied on some schedule and image copies should be made after LOG NO activities, particularly REORG. (If LOAD REPLACE is executed, the LOAD file may be preserved to reload but saving and using this data requires a special process for that space and you will not be able to apply log to the reloaded space with RECOVER.)
A complete set of system data (logs, bootstrap data sets and all spaces including the catalog and directory) can be copied after a subsystem is taken down cleanly or while a SET LOG SUSPEND is active, using special fast hardware copies. These data sets can be restored and used restart the entire subsystem at that point. Using SET LOG SUSPEND is the subject of past conference presentations. All the data is made consistent at RESTART in much the same fashion as after a power failure. Procedures have also been developed to attempt to use a restart with a more up-to-date logs and BSDS and to bring the system to a point further than the state of committed transactions at the time of the dumps. This involves using DEFER on all the spaces during restart and using LOGONLY recovery. There are a number of issues to manage.
If application spaces are copied outside of DB2 while DB2 is down and it was shut down cleanly, these copies can be restored and LOGONLY RECOVER used on them. If a single data set is copied outside DB2 while the space is active, you might be able to do the same but this is not recommended by IBM.
9/21/2018
©2007 BMC Software

36. SYSCOPY contains a wealth of data!
SYSCOPY MINING
SYSCOPY contains a wealth of data!
WHO?
WHAT?
WHEN?
GROUP_MEMBER
DBNAME
TSNAME
DSNUM
DSNAME
JOBNAME
AUTHID
ICTYPE
STYPE
SHRLEVEL
ICBACKUP
PIT_RBA
OTYPE
TIMESTAMP
START_RBA
ICDATE
ICTIME
SYSCOPY Columns, cont.
Column Description
ICTYPE Type of Operation:
A - ALTER
B - REBUILD INDEX (recorded for COPY YES index space)
D - CHECKDATA LOG(NO)
F - COPY FULL YES
I - COPY FULL NO
P - RECOVER TOCOPY or RECOVER TORBA or TOLOGPOINT; PIT_RBA
reflects the log point of the copy or log point specified.
Q - QUIESCE
R - LOAD REPLACE LOG(YES)
S - LOAD REPLACE LOG(NO)
T - TERM UTILITY COMMAND (Terminated Utility)
W - REORG LOG(NO)
X - REORG LOG(YES)
Y - LOAD RESUME LOG(NO)
Z - LOAD RESUME LOG(YES)
ICDATE YYMMDD
START_RBA This is really an LRSN and orders the events in time. …
DSNAME Copy data set name if ICTYPE is ‘F’ or ‘I’. If ICTYPE is ‘P’ and
the RECOVER was TOCOPY, contains the copy data set name used for
TOCOPY. Otherwise, dbname.spacename if created after V4.
9/21/2018
©2007 BMC Software

37. SYSCOPY Example DB2 Catalog Table SYSIBM.SYSCOPY
Backup and recovery point information
The report was obtained by running the following SQL:
SELECT
DBNAME,TSNAME,ICDATE,ICTIME,ICTYPE,HEX(START_RBA)
FROM SYSIBM.SYSCOPY
ORDER BY DBNAME,TSNAME
ICTYPE has many different possible values, those are listed below:
F-full image copy
I-incremental image copy
P-partial recovery (point in time)
Q-QUIESCE point
R-LOAD REPLACE LOG(YES)
S-LOAD REPLACE LOG(NO)
W-REORG LOG(NO)
X-REORG LOG(YES)
Y-LOAD RESUME LOG(NO)
Z-LOAD RESUME LOG(YES)
T-terminated
IC Type Description
F Full
I Incremental
Q Quiesce
X REORG LOG(YES)
SHRLEVEL Description
R Reference
C Change
9/21/2018
©2007 BMC Software

38. Spaces not recorded in SYSCOPY
Three catalog and directory spaces do not have entries in the SYSCOPY table
DSNDB01.DBD01
DSNDB01.SYSUTILX
DSNDB06.SYSCOPY
Information on copies for these spaces resides in the DB2 log
The SYSCOPY entries for these spaces can still be found with REPORT RECOVERY and can be printed from the log with DSN1LOGP and the SYSCOPY option.
9/21/2018
©2007 BMC Software

39. Can you keep all logs and never make copies?
Pop Quiz - Avoid Copies?
Can you keep all logs and never make copies?
PROBOBABLY NOT
Not if REORG or LOAD are used with LOG NO.
SYSCOPY TABLE continued
Column Description
ICTIME HHMMSS
SHRLEVEL C entry is a copy and was SHRLEVEL CHANGE
R entry is a copy and was SHRLEVEL REFERENCE
blank - not a copy …
TIMESTAMP Same value as represented in ICDATE, ICTIME, IBM recommends using
this one.
ICBACKUP When multiple copies are made, represents which one the entry is for.
blank - LOCALSITE primary copy
LB - LOCALSITE backup copy
RP - RECOVERYSITE primary copy
RB - RECOVERYSITE backup copy
STYPE Additional information about the event. For example, on a QUIESCE
row (ICTYPE = ‘Q’) a ‘W’ in this column indicates QUIESCE
WRITE(YES). For partial recovery (ICTYPE = ‘P’) an ‘L’ indicates
LOGONLY.
PIT_RBA An LRSN that is the log point of a TOLOGPOINT or TORBA recovery or
the log point of a copy used in TOCOPY.
9/21/2018
©2007 BMC Software

40. DB2 Recovery Processing
LOGAPPLY
RESTORE
MESSAGES
Image
Copy
SYSCOPY
SYSLGRNG
BSDS
ACTIVE LOG
ARCHIVE LOG
TABLE SPACE
9/21/2018
©2007 BMC Software

41. RECOVER can be instructed to stop at a particular log point
RECOVER Flavors
RECOVER can use all the log records to the end of the subsystem log(s) or
RECOVER can be instructed to stop at a particular log point
RECOVER usually starts by restoring image copies except in the rare cases where everything is on the log and
RECOVER has a LOGONLY feature that assumes the space is restored outside its control
The RECOVER utility can seem to have a daunting set of options but at its simplest it restores copies and applies logs from the point of the copy if required. The copies are registered with log points or LRSNs. If TOCOPY is not specified, logs are applied from the log point of the last copy. (If incrementals are involved, the full and incremental copies are merged.) We’ve already seen why SHRLEVEL CHANGE copies work with log records.
When logs are applied, all log records in the subsystem log(s) after the copy that are indicated by directory table SYSLGRNX ranges are applied with two exceptions.
You can request that RECOVER stop at a particular log record. This is usually done to get back to a point in time when the space wasn’t logically corrupted. This is done with the TOLOGPOINT option.
When you do a TOCOPY or TOLOGPOINT RECOVER and later, before you have done another COPY, you do another RECOVER (either to current or to another point after you executed the first RECOVER), RECOVER skips the group of log records eliminated by the prior recovery. (This is shown in the following illustrations.)
It is theoretically possible to execute a RECOVER … TOLOGPOINT operation using the (undamaged) space and processing the log backward to the TOLOGPOINT, avoiding image copies. IBM’s RECOVER does not have this function.
9/21/2018
©2007 BMC Software

42. Disaster Recovery Options from weekly dumps to offsite logging
Dumps - Simple, cheap, maximum data loss
Weekly dumps means several days data loss
Offsite Logs - Complex, expensive, no data loss
Applying log data to shadow increases expense
Compromise - Periodic vaulting of Copies & Logs
Daily or hourly log shipment will minimize data loss
Good topic for a future presentation
Disaster Recovery support is always a balancing act between cost and complexity vs. data loss and recovery time. Some shops opt for the very simple (and efficient) approach of dumping all DASD volumes periodically (typically weekly). At the remote site, a full set of volume restores gets the entire data center ‘back’ to the same point in time. Unfortunately, executing the dumps requires an outage, and restoring at the remote site means losing data. (what if the disaster occurs on Friday, and your most recent dump was taken last Sunday?)
Some companies can’t accept the outage for dumps, or the loss of data. There are techniques which allow for capturing DB2 log data, and transmitting it to a remote site. The log data can even be applied at the remote site, so the business is essentially shadowed. In the event of a disaster, this method ensures very little data loss (just the inflight transactions), and minimal recovery time. However, it is incredibly complicated and expensive.
The compromise in most shops is to send a copy of all application and DB2 system object copies, along with copies of the DB2 log, to a remote vault. In the event of a disaster, recovery is to the end of the ‘last’ offsite Archive Log, which typically is from ‘last night’. This is a technique documented in the DB2 System Administration guide. It requires a complex set of procedures to support it.
Cost
Complexity
Data Loss
Outage Time
9/21/2018
©2007 BMC Software

43. Questions? Expert Summary
Know the basics and don’t be caught by the myths
Know the assets you are trying to protect
Know what you have to protect them with
Plan for each type of failure and practice if you can
Questions?
9/21/2018
©2007 BMC Software