Presentation is loading. Please wait.

Presentation is loading. Please wait.

Experiences with D/R Procedures Of ADABAS Data on Mainframes Natural Conference Boston Dieter W. Storr May 2004

Similar presentations


Presentation on theme: "Experiences with D/R Procedures Of ADABAS Data on Mainframes Natural Conference Boston Dieter W. Storr May 2004"— Presentation transcript:

1 Experiences with D/R Procedures Of ADABAS Data on Mainframes Natural Conference Boston Dieter W. Storr May 2004

2 May 2004Dieter W. Storr -- 2

3 May 2004Dieter W. Storr -- 3 Different Disaster Different Action Unplanned downtime l Machine outages l Network outages l Software failures Disaster l Site / data center loss l Catastrophic failure

4 May 2004Dieter W. Storr -- 4 Leading Causes of Downtime Source: DRJ Summer 2002, Volume 15, Number 3 Power Storm Flood Terrorism Outage Damage Sabotage

5 May 2004Dieter W. Storr -- 5 Other Causes of Downtime Fire Earthquake Computer Crime

6 May 2004Dieter W. Storr -- 6 LA Times Downtime Flood Damage 21 April 2002: l Water was flooding through the Orange County facility, 14-inch pipe that supplies the fire-sprinkler system burst, half the facility standing in more than a foot of muddy water l Affected areas: editorial, ad ops, IT,HR, ADABAS was not affected

7 May 2004Dieter W. Storr -- 7 LA Times Downtime Bomb Alarm 14 June 2002: l A bomb was believed to have been left in the Bank of America branch that’s set into the Times Building l Security swept the building, l DBA’s observed the system from home

8 May 2004Dieter W. Storr -- 8 LA Times Downtime Bomb Alarm 29 July 2002: l An intruder claimed to have a bomb, darted into the garage l Security swept the building, l OP stopped CA7 - so PLOGCOPY couldn’t start automatically, two PLOG’s got full, ADABAS was locked, DBA’s later started the PLCOPY jobs manually

9 May 2004Dieter W. Storr -- 9 LA Times Downtime Power Outage - 29 August 2002 (3:43 P.M.) City (DWP) had a power grid, flood leaked into a DWP transformer There were actually 2 spikes/outages, the first started the UPS switchover, which was interrupted by the second, which took the UPS down.

10 May 2004Dieter W. Storr LA Times Downtime Power Outage - cont’ The network was back in service after a short delay. Our Unix-based servers were restarted, and checked. There was no evidence of damage to the Sybase Adaptive Server Enterprise (ASE, formerly: Sybase SQL Server) servers.

11 May 2004Dieter W. Storr LA Times Downtime Power Outage - cont’ Mainframe recovery was delayed due to corruption to the Hardware Management Console (HMC) OP did a power-on reset, which restored the HMC Operations IPLed, and Technical Support proceeded with system checkout procedures. Although Enterprise Storage Server (ESS) had an error indicator, it was still up and did not add to any outages IBM reset error indicator without impact.

12 May 2004Dieter W. Storr LA Times Downtime Power Outages - cont’ Started ADABAS servers manually: Parm Error 23, DIB block remained after an abnormal termination Started all servers with IGNDIB=YES 18:25 ADABAS IS ACTIVE NO ADAN58 Message

13 May 2004Dieter W. Storr LA Times Downtime ADAN58 Message (ADA71: ADAN5A) ADAN58 BUFFER-FLUSH START RECORD DETECTED DURING AUTORESTART. THE NUCLEUS WILL T E R M I N A T E AFTER AUTORESTART. IN CASE OF POWER FAILURE, THE DATABASE MIGHT BE INCONSISTENT BECAUSE OF PARTIALLY WRITTEN BLOCKS. O N L Y IN THIS CASE, REPAIR THE DATABASE BY RESTORE AND REGENERATE; OTHERWISE RESTART THE NUCLEUS. ADAN5A: FILES MODIFIED DURING AUTORESTART: files

14 May 2004Dieter W. Storr Power Failure During Buffer Flush ABCD EFCH EFCD old block updated block partially updated block on disk

15 May 2004Dieter W. Storr Nucleus Restart After Power failure - IGNDIB=YES ADA User exit 2 active. ADA PLOG2 closed. ADAP3X2P submitted. ADAN PROTECTION-LOG PLOGR1 STARTED ADAN NUCLEUS-RUN WITH PROTECTION-LOG ADAL :25:18 CLOGRS IS ACTIVE ADAN ADABAS COMING UP ADAN5A FILES MODIFIED DURING AUTORESTART: ADAN5A ADAN5A ADAN5A ADAN RUNNING WITH ASYNCHRONOUS BUFFERFLUSH ADAN8Y FILE-LEVEL CACHING INITIALIZED ADAN ADABAS DYNAMIC CACHING ENVIRONMENT ESTABLISHED. ADAN A D A B A S V6.2.2 IS ACTIVE ADAN MODE = MULTI I S O L A T E D ADAN RUNNING WITHOUT RECOVERY-LOG ADA User exit 8 active.

16 May 2004Dieter W. Storr LA Times Downtime Power Outage - cont’ Switched all PLOGs Checked batch and online There was no evidence of damage to any of the ADABAS components.

17 May 2004Dieter W. Storr Other LA Times Disasters 1965: Watts riots 1971: Sylmar quake : Whittier punch : LA riots 1994: Northridge quake Feb 1998: El Niňo, flooding in B-1 computer room 15 April 1999: Power failure ‘news editing’

18 May 2004Dieter W. Storr ADABAS Recovery Command Log (CLOG) Failure - I/O Error Restore or reallocate/format the CLOG ADABAS will come up through Autorestart normally No data loss if CLOG is not used CLOG

19 May 2004Dieter W. Storr ADABAS Recovery Protection Log (PLOG) Failure - I/O Error Restore or reallocate/format the PLOG Take a full back-up of the database ADABAS will come up through Autorestart normally Restart batch jobs l Restartable batch jobs = OK l Non-restartable batch jobs = check PLOG

20 May 2004Dieter W. Storr ADABAS Recovery TEMP and SORT Failure - I/O Error Restore or reallocate/format the TEMP/SORT dataset Different actions for the utilities l See the ADABAS Utilities manuals TEMP SORT

21 May 2004Dieter W. Storr ADABAS Recovery DSIM Failure - I/O Error Restore or reallocate/format a DSIM dataset Different actions for the utilities l See the ADABAS Utilities manuals DSIM

22 May 2004Dieter W. Storr ADABAS Recovery Recovery Aid Dataset Failure - I/O Error Restore or reallocate/format a RLOG dataset Prepare the RLOG dataset l ADARAI PREPARE RLOGSIZE / RLOGDEV…. Different actions for the utilities l See the ADABAS Utilities manuals Take a full back-up of the database l This will start the first generation of the RLOG dataset RLOGR RLOGM

23 May 2004Dieter W. Storr ADABAS Recovery ASSO/DATA Failure - I/O Error Copy PLOG twice - ADARES PLCOPY Restore or reallocate/format DATA dataset(s) Instead of reallocate/format and restore all DATA volumes, System specialists can l Reallocate and format the new volume l Restore the VTOC chain l Restore and Regenerate only files that were located on the failed volume Otherwise,... DATA ASSO

24 May 2004Dieter W. Storr ADABAS Recovery ASSO/DATA Failure - I/O Error Restore entire database ADASAV RESTORE [OVERWRITE = for GCB] ADASAV RESTONL [OVERWRITE] include PLOG Start nucleus with UTIONLY=YES Regenerate updates from end of last save (SYN2) ADARES REGENERATE PLOGNUM=xxx ADARES FROMCP=SYN2,FROMBLK=xxx DATA ASSO

25 May 2004Dieter W. Storr ADABAS Recovery ASSO/DATA Failure - I/O Error Possible utilities need to be rerun (see ADARES): l ADALOD LOAD FILE=xxx l ADALOD UPDATE FILE=xxx l ADALOD UPDATE FILE=xxx,DDISN l ADAINV INVERT FILE=xxx,FIELD=xx Lock files to rerun utilities l ADADBS OPERCOM LOCKU=xx Unlock utility-only status l ADADBS OPERCOM UTIONLY=NO DATA ASSO

26 May 2004Dieter W. Storr ADABAS Recovery ASSO/DATA Failure - I/O Error Rerun the regenerate function for the relevant files Unlock the regenerated files l ADADBS OPERCOM UNLOCKU=xx Don’t repeat these steps if ADARES points out: l ADALOD LOAD FILE=nn l ADARES REGENERATE FILE=nn l ADADBS REFRESH FILE=nn Nucleus is ready DATA ASSO

27 May 2004Dieter W. Storr ADABAS Recovery WORK 1 Failure - I/O Error Restore or reallocate/format the WORK dataset Restore and regenerate the entire database to avoid inconsistencies: open transactions See ASSO/DATA failure WORK2 WORK1 WORK3

28 May 2004Dieter W. Storr ADABAS Recovery WORK 2/3 Failure - I/O Error End the database normally (ADAEND) to avoid open transactions in part 1 of WORK Restore or reallocate/format the WORK dataset Restart the database normally If database abends then restore and regenerate the entire database - see ASSO/DATA failure WORK2 WORK1 WORK3

29 May 2004Dieter W. Storr ADABAS Recovery Failure in Data Storage Blocks //DDSIIN DD DSN=SAVE.SIBA…. // DD DSN=PLCOPY.LOG1… // DD DSN=PLOCPY.LOG2… //DDCARD DD * ADARES REPAIR DSRABN=xxx-yyy ADARES FILE=n1,n2,n3 Failure in DSST ADADCK DSCHECK FILE=xxx ADADCK REPAIR DS DATA CALL SAG ! !

30 May 2004Dieter W. Storr ADABAS Recovery Nucleus Ends With RC 77 Not restartable No more space for Checkpoint File (CP) Rename old WORK Allocate/format new WORK with old space Change high-used RABN and high-used ISN Restart nucleus with new WORK and UTIONLY=YES Nucleus is in “crippled mode” - no user has access Expand the database Stop the nucleus normally Rename old WORK and restart the nucleus with old WORK (autorestart) CP ASSO DATA

31 May 2004Dieter W. Storr ADABAS Recovery Nucleus Ends With RC 77 Not restartable No more space for user files Rename old WORK Allocate/format new WORK with old space Restart nucleus with new WORK and UTIONLY=YES Nucleus is in “crippled mode” - no user access Expand database Stop nucleus normally Rename old WORK and restart nucleus with old WORK (autorestart) User ASSO DATA User

32 May 2004Dieter W. Storr ADABAS Recovery Nucleus Abends - Missed DE Values Descriptor is marked in FDT as DE, value doesn’t exist in ASSO, but in DATA. Check: ADAICK ICHECK FILE=xxx[,NOOPEN] ADAVAL VALIDATE FILE=xxx,DESCRIPTOR=yy Solution 1: ADAULD UNLOAD FILE=xxx,UTYPE=EXF ADALOD LOAD FILE=xxx,LWP=yyyyK Solution 2: ADADBS RELEASE FILE=xxx,DESCRIPTOR=yy ADAINV INVERT FILE=xxx,FIELD=yy,LWP=... ASSODATA CALL SAG ! !

33 May 2004Dieter W. Storr Back-up Possibilities ADASAV to tape / disk Including Fast Dump Restore, DFDSS Delta Save Facility (DSF) Delta Save QDUMP (Legent) Disk mirroring (hardware level) l FlashCopy of Enterprise Storage Server (ESS) l Peer-to-Peer Remote Copy Extended Distance (PPRC-XD) l OC-3 links two EMC disc arrays Replication l Stand-by systems l Restore and Regenerate l Entire Transaction Server ASSO DATA

34 May 2004Dieter W. Storr ADABAS Disaster Recovery How to back-up Collect recovery data Restore w/o nucleus Start nucleus w/ UTILONLY=YES Regenerate w/ nucleus Switch UTIONLY=NO

35 May 2004Dieter W. Storr :0001:0002:0003:008: :0012:00 ADAP1BKF Online SAVE ADAP1BKF Online SAVE ADAP1PLC (FEOFPL) ADAP1PLC (FEOFPL) ADAP1PLC PLOG Switch ADAP1PLC PLOG Switch BRM/ABARS Several Jobs BRM/ABARS Several Jobs ADAP1BKO Copy Tapes ADAP1BKO Copy Tapes ASSO / DATA / WORK / etc. Pick-up by Recall PDS, GDGs etc. DFDSS Full-Volume Back-up DFDSS Full-Volume Back-up Weekly ADABAS Back-up at LA Times

36 May 2004Dieter W. Storr Production Database Back-ups ADASAV SAVE BUFNO=2,TTSYN=60 Record format... : VB Record length... : Block size.... : BUFNO=30

37 May 2004Dieter W. Storr Back-up to SMS Disk Pool Run times are consistently at least 80% lower when writing to disk instead of cartridge Run times are consistently around 60% lower when copying from disk to cartridge (compared with cart to cart) DFSMShsm, automate your storage management tasks, SMS Production Storage Pool DFSMShsm

38 May 2004Dieter W. Storr Back-up to Disk Pool No cartridge errors No cartridge drive errors No cartridges get accidentally ejected from the silo Smaller back-up window Smaller maintenance windows Less impact to application processes Greater confidence that the data you need will be there when you need it

39 May 2004Dieter W. Storr IBM Magstar 3494/Virtual Tape Server Linear design l frames Conf. Flexibility l SCSI, FC, ESCON, FICON l 3590, 3490E, VTS High availability l Dual robotics l Dual library manager >42 old 3490 carts will fit on 1 new 3494 cart 5 x 3390 volumes fit on one 3494 cart One 3494 cart can be read in 45 seconds into the VTS disk cache (raid-5)

40 May 2004Dieter W. Storr Virtual Tape Concept Virtual tape drives l Appear as multiple 3490E tape drives l 3490E Media 1 and 2 support l Shared / partitioned like real tape drives Tape Volume Caching l All data access is to cache l Improves ‘mount’ performance l LRU Cache management Volume Stacking l Fully utilizes physical cart capacity l Reduces physical cart requirement l Reduces footprint requirement Virtual Volume 2 Magstar /60 GB capacity* Logical Volume 1... Virtual Drive 1 Virtual Drive n F Virtual Volume 1 Virtual Volume n Tape Volume Cache Virtual Drive 2 Logical Volume n * assumes 3:1 compression

41 May 2004Dieter W. Storr Performance Tests

42 May 2004Dieter W. Storr Collecting Data For Recovery Block Ranges SYN1 - SYN2 For ADASAV RESTORE From ADASAV SAVE PROTECTION LOG PLOGNUM=64, SYN1=4695, SYN2=4698 From ADAREP SYN1 06 UTI :00: DUAL ADAP1BKF SYNP 06 UTI :00: DUAL ADAP1BKF SYN2 06 UTI :01: DUAL ADAP1BKF SYNV 0A UTI :01: DUAL ADAP1BKF SYNV 0A UTI :01: DUAL ADAP1BKF SYNV 28 UTI :02: DUAL ADAP1PLC SYNP 28 UTI :02: DUAL ADAP1PLC EOD 00 ET :30: DUAL ADAPRREP SYNS 53 ET :30: DUAL ADAP1REP SYNV 28 UTI :30: DUAL ADAP1PLC SYNP 28 UTI :30: DUAL ADAP1PLC

43 May 2004Dieter W. Storr Collecting Data For Recovery Block Ranges SYN2 - End For ADARES REGENERATE From ADAREP SYN1 06 UTI :00: DUAL ADAP1BKF SYNP 06 UTI :00: DUAL ADAP1BKF SYN2 06 UTI :01: DUAL ADAP1BKF SYNV 0A UTI :01: DUAL ADAP1BKF SYNV 0A UTI :01: DUAL ADAP1BKF SYNV 28 UTI :02: DUAL ADAP1PLC SYNP 28 UTI :02: DUAL ADAP1PLC EOD 00 ET :30: DUAL ADAPRREP SYNS 53 ET :30: DUAL ADAP1REP SYNV 28 UTI :30: DUAL ADAP1PLC SYNP 28 UTI :30: DUAL ADAP1PLC

44 May 2004Dieter W. Storr Collecting Data For Recovery Dataset Name From Back-up Job (GDG) For ADASAV RESTORE ADABAS.PRODOFFD.DB1.BACKUP.FULL.G0842V00 CATALOGED

45 May 2004Dieter W. Storr Collecting Data For Recovery Dataset Names From PLOG Copy Jobs (GDG) Matching block numbers End For ADASAV RESTORE and ADARES REGENERATE DDSIAUS1 OUTPUT VOLUME=WRK015, SESSION NR=64 FROMBLK= 1214, FROMTIME= :30:24 TOBLK= 4701, TOTIME= :01:42 ADABAS.PROD.DB1.PLOG.COPY.G7170V00 DDSIAUS1 OUTPUT VOLUME=WRK015, SESSION NR=64 FROMBLK= 4702, FROMTIME= :02:08 TOBLK= 4748, TOTIME= :30:03 ADABAS.PROD.DB1.PLOG.COPY.G7171V00 DDSIAUS1 OUTPUT VOLUME=WRK004, SESSION NR=64 FROMBLK= 4749, FROMTIME= :30:25 TOBLK= 4791, TOTIME= :30:33 ADABAS.PROD.DB1.PLOG.COPY.G7172V00

46 May 2004Dieter W. Storr Recovery - Part 1 - W/O Nucleus ADASAV RESTONL //RESTONL EXEC ADASAVRD //DDREST1 DD DISP=SHR,BUFNO=30, // DSN=ADABAS.PRODOFFD.DB1.BACKUP.FULL.G0842V00 //DDPLOG DD DISP=SHR,BUFNO=30, // DSN=ADABAS.PROD.DB1.PLOG.COPY.G7170V00 //DDKARTE DD * ADASAV RESTONL BUFNO=2,OVERWRITE //REPORT EXEC ADAREP //DDKARTE DD * ADAREP NOFILE //

47 May 2004Dieter W. Storr Recovery - Part 2 Start the ADABAS nucleus with normal JCL (UTIONLY=YES) ADAN PROTECTION-LOG PLOGR1 STARTED ADAN NUCLEUS-RUN WITH PROTECTION-LOG ADAL :20:29 CLOGRS IS ACTIVE ADAN ADABAS COMING UP ADAN RUNNING WITH ASYNCHRONOUS BUFFERFLUSH ADAN8Y FILE-LEVEL CACHING INITIALIZED ADAN ADABAS DYNAMIC CACHING ENVIRONMENT ESTABLISHED. ADAN A D A B A S V6.2.2 IS ACTIVE ADAN MODE = MULTI I S O L A T E D ADAN RUNNING WITHOUT RECOVERY-LOG ADA User exit 8 active. ADA ADAP1PLC submitted.

48 May 2004Dieter W. Storr Recovery - Part 2 - With Nucleus

49 May 2004Dieter W. Storr Recovery - Part 3 - With Nucleus Lock files to re-run utilities See regenerate report l ADADBS OPERCOM LOCKU=fnr or SYSAOS: A / I / L / F or modify command /F jobname,LOCKU=fnr Unlock utility-only status for users l ADADBS OPERCOM UTIONLY=NO or SYSAOS: A / I / L / U or modify command /F jobname,UTIONLY=NO

50 May 2004Dieter W. Storr Recovery - Part 3 - With Nucleus Re-run the utilities - if necessary l ADALOD LOAD / UPDATE / DDISN l ADAINV INVERT FILE=xxx,FIELD=xx Unlock files l ADADBS OPERCOM UNLOCKF=fnr or SYSAOS: A / I / L / F / N or modify command /F jobname,UNLOCKF=fnr

51 May 2004Dieter W. Storr Delta Save Facility (DSF)

52 May 2004Dieter W. Storr Delta Save Facility

53 May 2004Dieter W. Storr Delta Save QDUMP (CCA - now: TSI)

54 May 2004Dieter W. Storr Disk Mirroring Benefits Asynchronous disk mirroring can provide better physical protection by supporting extended physical distances. No loss of committed transactions in synchronous storage (mirroring/RAID) on a CPU failure ASSO DATA ASSO DATA

55 May 2004Dieter W. Storr Disk Mirroring Limitations No protection from data corruption introduced by the hardware / software Secondary site is not guaranteed to be transitionally consistent, because data is moved at the disk/track/sector or bit level (in the case of asynchronous mirroring). Client application must be re-started after failure and need to be aware of failure ASSO DATA ASSO DATA

56 May 2004Dieter W. Storr Disk Mirroring Limitations Synchronous mirroring and RAID devices can add overhead to application performance. Redundant/specialized high availability hardware/software can be expensive and restricted to use for backup purposes only. Secondary copy of data is not available for use – low hardware utilization. Need to replicate everything on disk, no selectivity of data replication ASSO DATA ASSO DATA

57 May 2004Dieter W. Storr Example For Disk Mirroring S/390UNIX S/390UNIX miles OC-3 link EMC 5700 SRDF remote mirrored synchronized Back Up / Hot Site SRDF remote mirrored synchronized Main Platform

58 May 2004Dieter W. Storr Dedicated line broadband speeds and prices T megabits per second (24 DS0 lines) Ave. cost $400.-$650./mo. T megabits per second (28 T1s) Ave. cost $6,000.-$16,000./mo. OC megabits per second (100 T1s) Ave. cost $20,000.-$45,000./mo. OC megabits per second (4 OC3s) no price OC gigabits per seconds (4 OC12s) no price OC gigabits per second (4 OC48s) no price Source: prices updated: 16 March 2004

59 May 2004Dieter W. Storr Peer-to-Peer Remote Copy Extended Distance (PPRC-XD) PPRC = 60 miles - PPRC-XD = continent ESS Shark - IBM ESS DASD - HDS also support PPRC ESS Shark FlashCopy Also see TimeFinder from EMC

60 May 2004Dieter W. Storr External Back-up Systems Fast Copy of Data Snapshot l No data movement l A virtual copy by copying pointers Copy Process l Physical copy asynchr. from the log. Copy l No impact on applic. on the original data Specific Hardware Required l Software works only with the hardware Work on Volume Level l Some snapshot only tools work also on dataset level

61 May 2004Dieter W. Storr Snapshot & Physical Copy IBM Hardware: Enterprise Storage Server Software: Flashcopy EMC 2 Hardware: Symmetrix Remote Data Facility Software: EMC TimeFinder l

62 May 2004Dieter W. Storr How It Works Read / update Physical Backup Physical Backup Snapshot Read / update Read only snap Pre-defined time window SuspendResume Source Data Source Data Read only: update requests are queued Source: SAG

63 May 2004Dieter W. Storr Replication Benefits Warm standby systems can be configured over a Wide Area Network, providing protection from site failures. Ability to more quickly swap to the standby system in the event of failure, as backup database is already on-line. Data corruption is typically not replicated as transactions are logically reproduced rather than I/O blocks mirrored.

64 May 2004Dieter W. Storr Replication Benefits Warm standby systems can be configured over a Wide Area Network, providing protection from site failures. Ability to more quickly swap to the standby system in the event of failure, as backup database is already on-line. Data corruption is typically not replicated as transactions are logically reproduced rather than I/O blocks mirrored. ASSO DATA WORK ASSO DATA WORK

65 May 2004Dieter W. Storr Replication Benefits Automatic switch over for clients using a switching mechanism, no client restart needed. Originating applications are minimally impacted as replication takes place asynchronously after commit of the originating transaction. The warm standby database is available for read-only operations, allowing better utilization of backup systems. ASSO DATA WORK ASSO DATA WORK

66 May 2004Dieter W. Storr Replication Benefits Ability to resynchronize and easily switch back to primary system when it becomes available without loss of data. ASSO DATA WORK ASSO DATA WORK

67 May 2004Dieter W. Storr Replication Limitations Warm standby system will be out-of-date by transactions committed at the active database that have not been applied to the standby. Protection is limited to components supporting Warm Standby (e.g. DBMS data sources may be protected but file systems may not be supported). ASSO DATA WORK ASSO DATA WORK

68 May 2004Dieter W. Storr Entire Transaction Propagator The Entire Transaction Propagator allows for asynchronous data replication. Replicated data can be updated and synchronized with master data at user specified intervals.

69 May 2004Dieter W. Storr OS/390 Recovery Procedures Prepared by the Mainframe Recovery Team Recovering l The OS/390 platform l The ABARS aggregates l The ADABAS databases

70 May 2004Dieter W. Storr -- 70

71 May 2004Dieter W. Storr OS/390 D/R Times (SUNGARD) About 2400 tapes l Shipping time from storage to the mainframe ? l 4 hours ahead for tape staging OS/390 and ABARS aggregates l 5 hours planned, 7+ hours with problems ADABAS databases l Approx. 2-3 hours for tape restore and regenerate l Next test Nov 1: approx. 45 minutes from disk pool

72 May 2004Dieter W. Storr Experiences From D/R Tests Problems to IPL on a strange CPU (6 hours duration) l Initial setup (restore SYS.. Libraries) l Pre-IPL procedures (restore Adabas, work, spool volumes, etc) l Post-IPL procedures (DFHSM in disaster mode, etc.) l Application restores Tape drive offline problems, Import MVSCAT typo errors, etc. Recovered wrong volumes, generation errors Initialize work volumes - conversion to SMS (DFSMShsm) TMC recovery problems caused BRM recovery problems, too

73 May 2004Dieter W. Storr Experiences From D/R Tests Sent wrong cartridges with system dates to storage Less channels for tapes on our offsite (2 instead of 4) = double restore time

74 May 2004Dieter W. Storr Experiences From D/R Tests RESTONL abended with SB00, no PLOG restored, Recovery Aid flag was on at the saved database. REGENERATE deleted file and pointed out to repeat the ADALOD job but the input dataset was not saved We did a full volume restore (DFDSS), restored the database and forgot to format the dual protection logs. Missed protection logs BRM restored wrong aggregates  Missing full-volume restores - (Database 2)  Missing volumes in Work Storage Pool - (Database 3)

75 May 2004Dieter W. Storr Experiences From D/R Tests  BRM: Back-up and Recovery Manager ABARS: Aggregate Back-up and Recovery Support (ABARS = not: Air conditioning and refrigeration industry services ) Recovered (-1) Aggregates instead of (0) – (all Databases) Recovered only SOME files on Aggregate (0) - (Database 1) BRM/ABARS was not properly recovered (wrong version of BRM database) Once those problems were resolved (several hours later), the ADABAS recovery ran smoothly.  5 Databases (61.4GB) restored and regenerated in 3.5 hours (tape/cart)

76 May 2004Dieter W. Storr How Far is ‘Far Enough?’ (http://www.drj.com/articles/spr03/ html) Alternate Facility Offsite Storage Facility Answer = 105 miles …so the survey

77 May 2004Dieter W. Storr Lessons Learned (http://www.drj.com/articles/spr02/ html) Distance is key Streets, bridges, tunnels, airports are closed Tape recovery is not effective All applications are critical Inconsistent back-up is no back-up at all People-dependent processes do not suffice Two sites are not enough People are irreplaceable; so is information

78 May 2004Dieter W. Storr Lessons Learned (http://www.drj.com/articles/spr02/ html) Companies that relied on tape or on third-party provider found in many cases they had difficulty meeting their recovery time objectives All disasters are possible

79 May 2004Dieter W. Storr Helpful Links Software AG - ADABAS Recovery ADABAS Restart and Recovery (Operations Manual) University of Arkansas - D/R Plan Disaster Recovery Journal

80 May 2004Dieter W. Storr Helpful Links FlashCopy Shark (ESS) State of the Art Storage EMC TimeFinder Entire Transaction Propagator (SAG) p151/pdf/man.pdf

81 May 2004Dieter W. Storr Thank you! Questions?


Download ppt "Experiences with D/R Procedures Of ADABAS Data on Mainframes Natural Conference Boston Dieter W. Storr May 2004"

Similar presentations


Ads by Google