Presentation is loading. Please wait.

Presentation is loading. Please wait.

IMS Tools ………Reorganization

Similar presentations


Presentation on theme: "IMS Tools ………Reorganization"— Presentation transcript:

1 IMS Tools ………Reorganization
Janet LeBlanc Silicon Valley Lab

2 IMS Tools – Overview / Mission
DM Tools is unique business model within IBM Operates as entrepreneurial business High focus on return on investment / profitability Customer requirements driven Mission Deliver IMS based solutions that: Are profitable to IBM Meet customer expectations and needs

3 IMS Tools – New Products, Versions, Releases

4 Application Management
Fast Path IMS High Performance Fast Path Utilities Full Function IMS High Performance Load, IMS High Performance Pointer Checker IMS High Performance Prefix Resolution IMS High Performance Unload IMS Index Builder IMS Parallel Reorganization IMS Online Reorganization Facility Administration IMS Database Control Suite IMS Command Control Facility IMS ETO Support IMS HP Sysgen Tools IMS Queue Control Facility IMS Workload Router IBM Data Encryption for IMS and DB2 Databases IMS Database Repair Facility IMS HALDB Conversion and Maintenance Aid IMS HD Compression- Extended IMS Library Integrity Utilities IMS Parameter Manager IMS Sequential Randomizer Generator IMS Buffer Pool Analyzer IMS Network Compression Facility IMS Performance Analyzer IMS Problem Investigator IMS Sysplex Manager IBM Tivoli OMEGAMON XE for IMS IBM Application Recovery Tool for IMS and DB2 Databases IMS Database Recovery Facility IMS DataPropagator MS DEDB Fast Recovery IMS High Perf Image Copy IMS High Perf Change Accumulation MS Batch Terminal Simulator IMS Batch Backout Manager IMS Connect Extensions IMS MFS Reversal Utilities IMS Program Restart Facility Data Base Administration Utility Management Recovery Management End to End Management Performance Management IMS DATA BASE TOOLS Application Management TM Management

5 IMS Tools – Utilities Management
Data Base Administration Utility Management Recovery Management End to End Management Performance Management IMS DATA BASE TOOLS Application Management TM Management IMS Tools – Utilities Management Fast Path IMS High Performance Fast Path Utilities Full Function IMS High Performance Load IMS High Performance Unload IMS Index Builder IMS High Performance Prefix Resolution IMS Parallel Reorganization IMS Online Reorganization Facility IMS High Performance Pointer Checker Administration IMS Database Control Suite

6 IMS High Performance Fast Path Utilities
High Performance FP Advanced Reorganization Tool DEDB Unload/Reload DEDB Analyze DEDB Change Fast Path Basic Tools DEDB Pointer Checker DEDB Tuning Aid Fast Path Online Tools Online Pointer Checker Online Data Extract Online Area Extender

7 IMS High Performance Fast Path Utilities
Fast Path Utility Region OS Batch IMS HPFP Utilities FP Online Tools Online Pointer Checker Online Data Extract Online Area Extender FP Basic Tools DEDB Unload/Reload DEDB Pointer Checker DEDB Tuning Aid FP Advanced Tool DEDB Unload/Reload DEDB Analyze DEDB Change IMS Control Region IMS Dependent Region IMS Online z/OS

8 Highlights of FP Advanced Tool
JCL ease of use Increase the productivity of database support personnel Minimize steps for completing database Analyze, Change, Reload and Unload Minimize DD statements Single driver program with command language Integrity with IMS Increase the application availability Multiple area data sets (ADSs) support High performance Reduce the consuming time for the maintenance and database conversion

9 JCL ease of use : Minimize JCL DD statements
FP Advanced Tool dynamically allocates : DEDB area data sets for Input of the analyze, change, and unload process Output of the reload and change process (with space allocation) ACB libraries DBRC RECON data sets Data sets of unloaded segment records for Output of the unload process Input of the reload process HFPPRINT data set HFPRPTS data set SORT work data sets Above DD statements are no longer required in JCL! Operators need not care about these data set allocations !

10 JCL ease of use : Command language
Single driver program with unified command language for all functions Simplified JCL structure //HFP EXEC PGM=HFPMAIN0 //HFPSYSIN DD * GLOBAL DBRC=YES UNLOAD DBD=DEDBJN22 /* //HFP EXEC PGM=HFPMAIN0 //HFPSYSIN DD * GLOBAL DBRC=YES CHANGE DBD=DEDBJN22 /* //HFP EXEC PGM=HFPMAIN0 //HFPSYSIN DD * GLOBAL DBRC=YES RELOAD DBD=DEDBJN22 /* //HFP EXEC PGM=HFPMAIN0 //HFPSYSIN DD * GLOBAL DBRC=YES ANALYZE DBD=DEDBJN22 /*

11 JCL ease of use : Command language
Simple language structure Commands ANALYZE CHANGE GLOBAL RELOAD UNLOAD Subcommands ALLOCATE FILECTL IRENAME LOADCTL REPORT THRESHOLD Keywords many…

12 JCL ease of use : Command language
Nice features Advanced data set name specification Masks can be used for data set names e.g.) IMSVS.USERFILE.&AREA Generate data set groups (GDG) Command syntax check without run GLOBAL SCAN=YES

13 High performance : Technical topics
Reduce I/O and CPU Internal sort No SORTIN and SORTOUT I/O for sorting MMcall Asynchronous I/O Concurrent processing of multiple UOWs Data space For IOVF and SDEP Multi-tasking Parallel processing for multiple DEDB areas Multiple subtasks for analyze, change, reload, and unload The number of concurrent processes can be specified by control statement

14 Unload/Reload/Change functions
Unloads a single DEDB area or multiple DEDB areas to QSAM data sets Reload: Reloads a single DEDB area or multiple DEDB areas from QSAM data sets If necessary, the analyze function can be included in the reload process Change: Reorganizes or restructures a single DEDB or multiple DEDB areas with single job step If necessary, the analyze function can be included in the change process

15 Unload/Reload/Change: Minimize manual steps
Fast Path Basic Tools Unload/Reload EXEC steps 1 (NEW Randomizer) OLD ACB library (NEW ACB library) RECON 5 Sort Control data set IC OLD AREAS Control statements REORGANIZED AREAS 2 RELOAD UNLOAD (one job) (multi-task) RELOAD (one or multiple jobs) (single task) RELOAD SORT (optional) Control statements IDCAMs DELETE/ DEFINE DBRC utility Set RN on IOVF Work data set or data space (SDEP Work data set) 3 4

16 HPFP Unload Unload: Minimize manual steps 1 IC OLD AREAS
(NEW randomizer) OLD ACB library (NEW ACB library) RECON Sort Control data set IC OLD AREAS Control Statements REORGANIZED AREAS UNLOAD (one job) (multi-task) RELOAD (one job) (multi-task) SORT (optional) Control statements IDCAMs DELETE/ DEFINE DBRC Utility Set RN on HPFP Unload IOVF/SDEP Work data set or data space (SDEP Work data set) 1

17 HPFP Reload Reload: Minimize manual steps 2
HPFP Reorganization Tool Unload/Reload EXEC steps HPFP Reload 2 (NEW Randomizer) OLD ACB Library (NEW ACB Library) RECON Sort Control data set IC OLD AREAS Control statements REORGANIZED AREAS UNLOAD (one job) (multi-task) RELOAD (one job) (multi-task) SORT (optional) Control Statements IDCAMs DELETE/ DEFINE DBRC utility Set RN on IOVF/SDEP Work data set or data space (SDEP Work data set)

18 Change: Minimize manual steps
HPFP Reorganization Tool Change EXEC Step for reorganization ACB library RECON Reorganized AREAs Change (one job) (multi-task) Old AREAs Reload Unload Buffer HFPSYSIN IDCAMS SORT

19 Change: Minimize manual steps
HPFP Reorganization Tool Change EXEC step for restructuring Old ACB library New ACB library RECON Restructured AREAs (NEW Randomizer) Change Old AREAs Reload Unload Buffer HFPSYSIN IDCAMS SORT

20 Reload/Change: Place certain segments into overflow CIs
The LOADCTL subcommand controls the place to insert dependent segments, which is either DOVF or IOVF. //HFP EXEC PGM=HFPMAIN0 //HFPSYSIN DD * RELOAD DBD=DEDBJN22 LOADCTL SEGMENT=DD2,LOCATION=IOVF /* DEDB AREA RAP IOVF SDEP BASE ROOT DOVF DD1 DD1 DD1 DD1 DD1 DD2

21 Sample JCLs You can see: 4 sample JCLs How easy JCL are
Examples of control statements 4 sample JCLs Unload an area registered with DBRC Reload an area registered with DBRC Reorganize an area registered with DBRC Restructure an area registered with DBRC

22 Unloading an area registered with DBRC
//HFPUNL EXEC PGM=HFPMAIN0 //STEPLIB DD DISP=SHR,DSN=HFP220.SHFPMOD0 // DD DISP=SHR,DSN=IMSVS.SDFSRESL // DD DISP=SHR,DSN=IMSVS.PGMLIB //IMSACB DD DISP=SHR,DSN=IMSVS.ACBLIB //IMSDALIB DD DISP=SHR,DSN=IMSVS.MDALIB //HFPRPTS DD SYSOUT=* //HFPPRINT DD SYSOUT=* //HFPSYSIN DD * GLOBAL DBRC=YES UNLOAD DBD=DEDBJN22, IAREA=(DB22AR0), OAREA=(DB22AR0), ODSNMASK=‘IMSVS.USRFILE.&AREA‘ /*

23 Reloading an area registered in DBRC
//HFP EXEC PGM=HFPMAIN0 //STEPLIB DD DISP=SHR,DSN=HFP220.SHFPMOD0 // DD DISP=SHR,DSN=IMSVS.SDFSRESL // DD DISP=SHR,DSN=IMSVS.PGMLIB //IMSACB DD DISP=SHR,DSN=IMSVS.ACBLIB //IMSDALIB DD DISP=SHR,DSN=IMSVS.MDALIB //HFPRPTS DD SYSOUT=* //HFPPRINT DD SYSOUT=* //HFPSYSIN DD * GLOBAL DBRC=YES RELOAD DBD=DEDBJN22, IAREA=(DB22AR0), IDSNMASK=‘IMSVS.USRFILE.&AREA', OAREA=(DB22AR0) /* PTRCHKLVL=FULL REPORT /*

24 Reorganizing an area registered in DBRC
//HFP EXEC PGM=HFPMAIN0 //STEPLIB DD DISP=SHR,DSN=HFP220.SHFPMOD0 // DD DISP=SHR,DSN=IMSVS.SDFSRESL // DD DISP=SHR,DSN=IMSVS.PGMLIB //IMSACB DD DISP=SHR,DSN=IMSVS.ACBLIB //IMSDALIB DD DISP=SHR,DSN=IMSVS.MDALIB //IAREA001 DD DISP=SHR,DSN=HPFP.OLD.DB22AR0.ADS1 //HFPRPTS DD SYSOUT=* //HFPPRINT DD SYSOUT=* //HFPSYSIN DD * GLOBAL DBRC=YES CHANGE DBD=DEDBJN22, IAREA=(DB22AR0), OAREA=(DB22AR0) /* PTRCHKLVL=FULL REPORT /*

25 Restructuring an area registered in DBRC
//HFP EXEC PGM=HFPMAIN0 //STEPLIB DD DISP=SHR,DSN=HFP220.SHFPMOD0 // DD DISP=SHR,DSN=IMSVS.SDFSRESL // DD DISP=SHR,DSN=IMSVS.PGMLIB //OLDACB DD DISP=SHR,DSN=IMSVS.OLDACB //IMSACB DD DISP=SHR,DSN=IMSVS.ACBLIB //IMSDALIB DD DISP=SHR,DSN=IMSVS.MDALIB //IMSRESLB DD DISP=SHR,DSN=IMSVS.SDFSRESL //IAREA001 DD DISP=SHR,DSN=HPFP.OLD.DB22AR0.ADS1 //HFPRPTS DD SYSOUT=* //HFPPRINT DD SYSOUT=* //HFPSYSIN DD * GLOBAL DBRC=YES CHANGE DBD=DEDBJN22, IAREA=(DB22AR0), OAREA=(DB22AR0) /*

26 Reorganization Toolsets …Two Integrated IMS Tools choices
Reorg Utility Set Parallel Reorg HP Unload HP Load Index Builder HP Prefix Resolution HP Image Copy HP Pointer Checker Program Restart Facility Online Reorg Utility Set Online Reorg Facility HP Unload HP Load HP Prefix Resolution HP Image Copy HP Pointer Checker Program Restart Facility

27 IMS High Performance Unload v1.1 5655-E06
HP Unload Unloads HDAM, HIDAM, HISAM and SHISAM databases With IMS V7, support of the new HALDB database types PHDAM, PHIDAM Provides an IMS application program interface For stand-alone batch program Assembler, COBOL, or PL/1 Transparent to programmer Contains a high-performance DB retrieval engine The HSSR Engine High speed buffering technique similar to OSAM SB Different from DL/I Buffering Sequential Subset Randomizer Creates randomizing module for fast sequential processing of record subset Allows physical clustering of database records in same subset Fast Unload is key because it is the time consuming step i the DB reorg process because it is the one which creates the unload dataset by reading every disorganized part of the DB involving a lot of I/Os.

28 IMS HP Unload in the Reorganization Process
Database unload utilities Unload a database in compressed or uncompressed format Unload a broken HIDAM, HDAM, or HISAM database Skip invalid HD pointers or invalid records Create unloaded sequential data sets for use by application programs Multiple unload formats Use dynamic allocation of DB datasets Easy to read output reports and statistics DB statistics report or Randomizing statistics report Pre-tuned with defaults that should be adequate for most databases Full support for HALDB Unload one, several or all partitions Migration/fallback support New function for user exit FABHEXTR Specify that n records are to be unloaded from each HALDB Partition PARTEXTR control statement Upto three output datasets two when IPR Parallel Reorg is used same or different formats can use different user exit for each User Exit support user exits can be written in Assembler, COBOL or PL/1 enables skipping of individual segments, skipping remaining segments in DB record, and editing of segment data choice of type A or B exits type B is simpler, but limits output to a single unload file type A exits (upto three) can be used for creating upto three unload files, and have more function - such as “skip to root with specified key” Optional Statistical Reports Segment counts, DB statistics, randomising statistics, DB Call statistics, buffering and I/O statistics, CAB (Chained Anticipatory Buffering) statistics HALDB partition information

29 IMS High Performance Load v2.1 5655-M26
HP Load Utility Performance replacement of IMS HD Reorganization Load utility (DFSURGL0) Complement to IMS High Performance Unload Tool compressed/uncompressed input in various formats dynamic allocation of DB datasets Full support for HALDB Self Optimization Except for DATASPACE option where the default is N and use of Y is better SORT=YES Integrated Load SORT HDAM Physical Sequence Sort for Reload (PSSR) Utility Previously in DBTools SMU Sorts the unloaded database data set before reload Used with HALDB, when changing partition boundaries during reorg Avoids “cascading” on Reload > 71% reduction in elapse time > 92% reduction in CPU time (over base IMS load utility) New DB Load Utility V7 Support Dynamic allocation Support for HALDB DBRC support JCL compatibility with DBT:FRR Out of the box - performance tuned Initializes empty HDAM, HIDAM DBs Performs HALDB partition initialization as part of the Load User Exit Facility Uses it's own buffering & database writes - not DL/I: Chained writes to IMS databases , All buffers allocated above the 16M line Provides statistics on space usage and pointers. Physical Sequence Sort for Reload sorts an unloaded database data set from HDAM database help preventing cascading produce reports about database sizes and randomizer performance. The following new functions introduced for PSSR: – User Exit support – Segment Edit/Compression exit support – IMS/ESA Year 2000 Exit Tool (5697-E04) support

30 IMS HP Load in the Reorganization Process
Support of IMS HDAM, HIDAM, PHDAM and PHIDAM databases Supports reloading of compressed segments without calling compression routine Initializes empty HDAM and HIDAM databases DFSUINPT DD Dummy statement SEQERROR=ACCEPT option Support of HISAM and SHISAM thru maintenance DB Reload and DB Initialisation Support of logical relationship or secondary indexes Creates a DFSURWF1 data set that can be used by: Index Builder (IB) product IMS High Performance Prefix Resolution IMS Prefix Resolution utility (DFSURG10) Support of HALDB Automatically initializes HALDB partition data set before reload Provides a performance replacement for IMS Partition Initialization Utility Creates ILDS When changing partition boundaries during reorg of HALDB PSSR may be used prior load to sort segments by RAP within partitions for PHDAM. SEQERROR=ACCEPT was developed to be BMC compliant. It allows to load errors in a database! Support of HISAM and SHISAM thru PTFs PQ53316/UQ59432 PQ54113 for the DOC APAR

31 IMS HP Load in the Reorganization Process ...
User exit facility for additional processing of each segment Assembler, COBOL or PL/1 Edit segment Delete segment (or this and subsequent segments of DB record) Force segment into overflow (HDAM or PHDAM) Force segment to start new block (HIDAM or PHIDAM) Load API Used in application for intial load processing Bitmap Resetter Adjust the bitmap of HDAM/HIDAM/PHDAM/PHIDAM database to accommodate denser packing of the database blocks. Statistics reports to aid in tuning the database Reporting on space use and segment pointer statistics New New

32 IMS HP Prefix Resolution v3.1 5655-M27
Enables you to resolve and update prefixes of IMS databases involved in logical relationships as a single job step Functions: Eliminates the WF2 and WF3 data sets Executes prefix resolution and prefix update functions as replacements for the existing IMS Prefix Resolution and IMS Prefix Update utilities Supports IMS Parallel Reorganization, V3 single job step execution of database reorganization, prefix resolution, and prefix update tasks Your Value: Better performance and faster running prefix resolution and prefix updates Significant performance improvements when used in place of existing IMS utilities Use to maintain all logically related IMS databases and is required after the databases are loaded or reorganized High Performance Prefix Resolution is a replacement product for the Database Tools FPR Feature Prefix resolution is used in maintaining all logically related IMS databases and is required after the databases are loaded or reorganized. IBM IMS High Performance Prefix Resolution, Version 3.1, enables you to resolve and update prefixes of IMS databases involved in logical relationships as a single job step. A data transfer service called HPPRPIPE (eliminating Work File 2 (WF2) and Work File 3 (WF3) data sets) avoids much of the I/O, tape handling, and DASD requirements associated with prefix resolution and prefix update. This version of IMS HP Prefix Resolution supports IMS Parallel Reorganization for z/OS, Version 3.1 single job step execution of database reorganization, prefix resolution, and prefix update tasks. Also included with this latest version of IMS High Performance Prefix Resolution, Version 3.1, is the capability to perform the Prefix Resolution function and the Prefix Update function as replacements for the IMS Prefix Resolution Utility or the IMS Prefix Update Utility. Using these functions over the existing base IMS Utilities provides even more performance improvements when performing this type of database maintenance.

33 IMS Index Builder - Major Functions
Support for Full-Function non-partitioned DB and HALDB Easy-to-use one step procedure Creating New Secondary Indexes Rebuilding Secondary Indexes Using as input output from initial load or reload after a reorg (DFSURWF1) Using as input a DL/I scan of the IMS database Using as input the output from prefix resolution (DFSURIDX) Creating Input for IMS HP Prefix Resolution Split Function Initializing Empty Secondary Indexes Rebuilding a HIDAM Primary Index Using IMS IB for Recovery of Secondary Indices Change in the GENJCL.RECOV skeleton To rebuild indices after reorganization process To recover damaged secondary indices and avoid full reorganization process To avoid taking IC of secondary indices To minimize elapsed time .... IB Version 2 doesn't generate the DBRC model commands like with IB V1. IMS IB V2 notifies DBRC when tasks complete successfully. IMS IB V2.3 will be GA on 20/09/2002

34 IMS Parallel Reorganization
New Version IPR is an umbrella product or higher level framework for execution of the following tools IMS High Performance Unload IMS High Performance Load IMS Index Builder IMS High Performance Image Copy IMS High Performance Pointer Checker Functions are Parallel reorganization High Speed alternative to serial use of HP Unload High Speed alternative to serial use of HP Load Enhanced Data Base Scan (for logical relationships) using HP Unload IMS Parallel Reorganization provides the infrastructure to operate the reorganization utilities in parallel (or independently), allowing a significant reduction in the reorganization time.

35 IPR Driver V3 – Big Picture
Online Read Access Concurrent Database Backup New in v3 Single Step Execution Sep. 21 Announcement IPR Driver V3 IMS Tools New Versions/Release DB - IMS Parallel Reorganization V3 - IMS High Performance Unload V1 - IMS High Performance Load V2 - IMS Index Builder V2.3 - IMS High Performance Image Copy V3.2 - IMS High Performance Pointer Checker V2 - IMS High Performance Prefix Resolution V3 Database Image Copy Secondary Index Index Image Copies Prefix Update for Internal Logical Relationship New in V3 DB Admin DB (Shadow) Reports and DB Statistics Automated DBRC Processing Database “Health Check” New in V3 Sec. Index (Shadow)

36 IPR Driver V3 Functions Stopping the database or making it read-only before reorg IMS /DBR or /DBD DATABASE command is issued by IPR Driver Reorganizing database data sets into “shadow” data sets Unload, Reload, and Index-Builder tasks run concurrently Image copies can be taken during reorg, with optional HASH pointer check  Type-A Image Copy Updating segment prefixes after the database reload For the database that has internal logical relationship  Concurrent Prefix Update Taking image copies with optional HASH check after the prefix update  Type-B Image Copy Stopping the database at the completion of READ-ONLY reorg IMS /DBR DATABASE command is issued by IPR Driver Performing post-reorganization process Original and “shadow” data sets names are swapped DBRC is notified of the reorg completion and the image copy New New New

37 Sample JCL Statements for a Parallel Reorg with Secondary Indexes and Internal Logical Relationships
//IPR EXEC PGM=HPSGMAIN,PARM='DBD=SAMPLEDB,DBRC=Y' //STEPLIB DD DISP=SHR,DSN=TOOLS.LIBRARY // DD DISP=SHR,DSN=IMS.SDFSRESL //IMS DD DISP=SHR,DSN=IMS.DBDLIB //IMSDALIB DD DISP=SHR,DSN=IMS.DALIB //DFSURCDS DD DISP=SHR,DSN=DFSURCDS //DFSURWF1 DD DISP=(NEW,PASS,DELETE),DSN=&&DFSURWF1, // UNIT=SYSALLDA,SPACE=(CYL,(n,m)), // DCB=(RECFM=VB,LRECL=800,BLKSIZE=2400) //HPSIN DD * (REORG) IMSCMD=YES DBSHARE=YES IC=YES INDEXBLD=YES NAMESWAP=YES PREFIXRES=YES DELOLDDS=YES (PREFIXRES) UPDLPC=NO /* //ICEIN DD * GLOBAL HDPC=Y,ICHLQ=IMSICA . If IC=YES and PREFIXRES=YES are specified and the database has an internal logical relationship, ICTYPE=B is implicitly set.

38 IMS DB Reorg. – Online Solutions
IMS Version 9 HALDB OnLine Reorg. (OLR) Integrated OnLine Reorganization (OLR) of HALDBs HALDB OLR provides 100% availability of the largest databases in the world! OLR provides non-disruptive reorganization of HALDB PHDAM and PHIDAM partitions Concurrent IMS updates are allowed while OLR is active Scheduled data outage not required IMS Online Reorganization Facility (Tool) For non-HALDB and HALDB With restrictions For IMS V7 or above New

39 Reorganization Toolsets …Two Integrated IMS Tools choices
Online Reorg Utility Set Online Reorg Facility HP Unload HP Load HP Prefix Resolution HP Image Copy HP Pointer Checker 2 Reorg Utility Set Parallel Reorg HP Unload HP Load Index Builder HP Prefix Resolution # HP Image Copy # HP Pointer Checker # Program Restart Facility # 1

40 Features Reorg Index Only Near Online Unload
Reorganize a Database while still online Data sharing environments fully supported Allows DBD changes to be implemented and does not require manual intervention after reorganization Automatic online change implementation Complete restart-ability once in the Takeover Phase Controlling Takeover Delay, Abend, Window Interface to PRF/Region controller front-end for pausing BMPs Internal Logical Relationships HDAM and HIDAM HALDB All partitions in single job Reorg Index Only Primary or secondary for non-HALDB Secondary for HALDB Can be single partition if PSINDEX Near Online Unload Create HD Unload data set while DB still online Again, main feature is that any full function database can be reorganized while still online. Only unavailable for short duration, usually less than one minute, while data sets are renamed. Data sharing is fully supported – DB can be shared by any number of online IMS systems. ORF utility runs outside of IMS, and we capture and apply changes from all online systems. Use XCF as means of communications from IMS online systems to ORF utility. We do allow most DBD changes (ones that do not also require application, i.e. PSB, changes). When there are DBD changes, we will all of the steps necessary to make those changes effective in the online systems – without any manual intervention! Put new DBD into old DBD library, run ACBGEN, copy updated ACBs to ACBLIBA & ACBLIBB data sets, do something similar to IMS Online Change to have IMS load the new DMBs, then start the DB.

41 Verification Phase Copy Phase Reorg Phase Apply Phase Takeover Phase
IMS Control Region(s) Online Reorg Facility (ORF) Verification Phase Copy Phase Capture log updates Shadow Primary DB & Primary Index Copy Phase Original Primary Database Reorg Phase Unload Reload Rebuild Sec. Indexes PreReorg/Prefix Res/Update Image Copy/Pointer Checker Reorg & Apply Phases Capture application update calls Temporary Unload File Image Copies Reorganized Shadow DB, Primary & Secondary Indexes Apply Phase Takeover Phase /DBR original DB Load new DMB /STA DB Takeover Phase Request /DBR DB DBRC notifications Rename shadows to originals ACBGEN, Online change Request /STA DB Good diagram to keep open for reference as we go into detail of each phase. Left side shows what is happening in the online systems. Right side shows the different processes, or ‘phases’ of an ORF utility – batch job. We will talk about the detail as it relates to the ORF utility but keep in mind that there are corresponding things happening in the control region address spaces, both by ORF and also access to the databases by your transactions and BMPs Reorganized Database and Indexes Online Reorganization Completed Batch LOG Completion Phase

42 IMS Online Reorganization Facility for z/OS
DBD Changes permitted Controlling Takeover Phase Managing BMPs during Takeover Phase New Functions Product Name Allows databases to be reorganized without taking them offline. Uses shadow (snapshot copy) data sets to perform reorganization. Captures and saves copy of any update calls made in online systems and will replicate those calls against the shadow data sets after the shadow data sets have been reorganized. Once all caught up, DB is DBRd in all online systems, data sets are renamed, and DB is started on all online systems again. We go over these steps in a lot more detail in rest of presentation.

43 IMS Online Reorganization Facility for z/OS …BENEFITS
Increased Data Availability Outage reduced from hours to seconds Minimal Overhead Added to Control Regions Automation of DBA tasks Single step Reorg Automatic determination of necessary reorganization steps Same JCL for all databases Changes implemented across all online systems Better DB performance Can reorganize when needed Improved DBA Efficiency No coding/maintenance of IDCAMS statements No manual intervention required after reorganization process Again, we see the primary benefit as increased data availability. ORF also can be used to automate the many DBA tasks that need to be done for a database reorganization. A DB reorganization traditional includes: Unload, Prereorg (if logicals), Reload, Build Secondary Indexes, Prefix Resolution & Update, Image Copies, Pointer checker, ACB gens, online change. ORF can do all of this in a single step and without manual intervention. We will show an example towards the end, but all of the reorg processes are done in a single job step, and normally the only thing you’ll need to change is the DBD name.

44 Reorganization Toolsets …Two Integrated IMS Tools choices
Reorg Utility Set Parallel Reorg HP Unload HP Load Index Builder HP Prefix Resolution HP Image Copy HP Pointer Checker Program Restart Facility Online Reorg Utility Set Online Reorg Facility HP Unload HP Load HP Prefix Resolution HP Image Copy HP Pointer Checker

45 IMS Tools ………What’s new and exciting
Janet LeBlanc Silicon Valley Lab

46 Additional Slides

47 Reorg Slides

48 Rich in Function …….Unload Formats
Description HP Unload IPR Unload HD/UL HD Unload compatible Y CS F1 F2 F3 HS HSAM Unload compatible N VB Variable Block format VN VB + segment name A1 A2 A3 A4 F6 SH NO No unload dataset created User Format

49 HD/SH Format

50 F1 Format

51 F2 Format

52 F3 Format

53 CS Format

54 A1 Format

55 A2 Format

56 A3 Format

57 A4 Format

58 F6 Format

59 User Header Element Name Field Length Field Type Description DBDNAME 8
Character DBD Name FILLERnn 1-999 Integer Nn suffix determines length HDRLEN 2 Binary REQ if SEGKEY, SEGCKEY or ROOTKEY used ROOTKEY variable varies Sequence field of root segment ROOTRBA 4 RBA of root segment SEGCKEY Variable Varies Concatenated key SEGCKYSZ Length of segment concatenated key SEGCODE 1 binary Segment code of segment SEGDATSZ Length of segment data SEGDBYTE Delete byte of segment SEGKEY Sequence Field SEGKEYOF Offset of segment key SEGKEYSZ Length of segment key SEGLEV1 Hierarchical level of segment SEGLEV2 Hierarchical level of semgnet SEGNAME Name of Segment SEGRBA Database RBA of segment

60 Rich in Function ……Multiple DB Type Support
Supports reorganization of: HDAM, HIDAM, PHDAM, PHIDAM, HISAM and SHISAM databases

61 Rich in Function ……Compression Support
Unload Options to unload either compressed or uncompressed data Load Automatic recognition of status of unload data – compressed/uncompressed

62 Rich in Function …..HALDB Support
Reorganization: By Partition Range of partitions All partitions Rebuild ILDS Rebuild Secondary Indexes Conversion Support Unload with migration option Load has integrated partition initialization

63 Rich in Function …..Fault Tolerance
Unload Continue Processing after sequence errors Handle Corrupt Databases Incorrect HD pointers can be bypassed Processing of incorrect HISAM record can be skipped Root segments of HIDAM or PHIDAM can be processing using Index instead of pointers Load Reject/Save/Accept Sequence Errors

64 Rich in Function …..User Exits
Unload Record Formatting exit & Return Code Edit exit Assembler, COBOL or PL/1 Edit segment 5 std record format routines provided Run-Time Environment Exit Buffer Handler Initialization Exit Load User exit facility for additional processing of each segment Assembler, COBOL or PL/1 edit segment delete segment (or this and subsequent segments of DB record) force segment into overflow (HDAM or PHDAM) force segment to start new block (HIDAM or PHIDAM)

65 Rich in Function …..Application Program Interface
Unload For stand-alone batch program Assembler, COBOL, or PL/1 Transparent to programmer Load Assembler Language , COBOL for OS/390 and VM PL/I for z/OS and OS/390

66 Rich in Function ….Additional utilities
Sequential Subset Randomizer Creates randomizing module for fast sequential processing of record subset Allows physical clustering of database records in same subset Physical Sequence Sort for Reload (PSSR) Utility Sorts the unloaded database data set before reload Used with HALDB, when changing partition boundaries during reorg Avoids “cascading” on Reload Bitmap Resetter The bitmap resetter can be used to adjust the bitmap of HDAM/HIDAM/PHDAM/PHIDAM database to accommodate denser packing of the database blocks.

67 IMS Online Reorg Facility
IBM GLOBAL SERVICES IMS Online Reorg Facility New tool to allow IMS customers to achieve true 24x7 data availability for their full function data bases. Full function means HDAM, HIDAM, SHISAM, HISAM, PHIDAM (HAL/DB), and PHDAM (HAL/DB). These is a tool, and is not the same as OLR, which is part of IMS V9. © IBM Corporation 2005

68 Verification Phase Before we get started – verify we have what we need
DBD Structure Do we support current DBD definitions? If changes, can we implement all changes? DBRC definitions Data sets Can shadow datasets be allocated? Can image copy data sets be allocated? Can work/temp files be allocated? Software Are all necessary utilities available? Are these utilities at the proper level? Are online systems at compatible level? First phase is the Verification phase. We are running many ‘reorg’ processes in a single step, and sometimes the elapsed time of the entire reorg process can be long, we want to verify as much as possible up front, rather than waiting until say the Prefix Resolution step to verify we have enough space for the WF1 file. This slide lists some of the things we check in the verification phase. DBD structure – do we support the DBD specified. Currently a few restrictions, like external logical relationships, that we will list in detail later. As we mentioned we do allow DBD changes, we verify new DBD as well. Basically what user does is to run DBDGEN into a ‘staging’ DBD library then provide ORF the staging DBD library name. DBRC definitions – are requirements listed a few foils back met Data sets – we can, and by default, we can dynamically allocate all the datasets that are produced as part of the reorg (shadow data sets, image copy data sets, work files). We make sure we can get these allocated, or if you predefined them, that they exist and we can get to them. Software – ORF does use many of the current IMS HP tolols. We need to make sure that the tools needed to reorganize the type of DB specified are available and that they are at least at the minimum level needed by ORF. There were some changes in the tools announced in September that ORF needs.

69 Specify a library with the new DBD Limited to non-HALDB
DBD Changes Specify a library with the new DBD Limited to non-HALDB No changes in hierarchical structure No changes in DBRC Examples: Change of randomizing parameter Addition of segment types Segment length increase New DBD may be copied to primary DBD library during Takeover We have talked about this some but here is list of the type of DBD changes that are and are not supported. HAL/DB – currently ORF only runs against a single partition at a time. The things that are still in the DBD for HALDB effect all partitions so ORF can not change just a single partition. Can change randomizer for space – must keep data base records in same physical sequence (like DFSHDC40). Can add segments at bottom of hierarchy. Can’t add in middle because we need to replicate updates made in online systems with old DBD. Can increase segment size – we also will talk later about using USEREXIT to change or prime data. One more automation step – during takeover, we will copy the new DBD to the library that we found original DBD in (user can turn this off if needed).

70 Copy Phase Activate capture of log updates to primary database in online systems STOP/START primary database to create sync point Copy original database data sets to shadow data sets Apply log updates to shadow data sets Switch from log updates to application updates in online systems STOP/START primary database to create transition point Build shadow primary index Onto the next phase: Copy phase – create a snapshot copy of the original primary data base only – no indexes Connect to all IMS systems that are authorized for the database – another reason for requiring DB to be registered to DBRC. Activate capturing of log updates for this DB. Interface with TOI to STOP/START the primary DB – flushes buffers, updates catalog, creates syncpoint to create snapshot copy. Copy the original database to shadow data sets and apply captured log updates to the shadow. When caught up, transition to capturing application update calls that will be used to replicate calls to shadow data sets after the shadow data sets are reorganized. Also, our capture points are using standard IMS exits, and are only active for the specific DB ORF is running against. No overhead if ORF utility not being run. Any updates captured are sent thru XCF to ORF utility where they are stored in temp data set until needed. Reason we rebuild primary index rather than copy is for cases like HAL/DB or when primary index is non recoverable and IMS does not log updates for these data sets.

71 Reorg Phase Temporary unload data set
Unload shadow database Temporary unload data set Optionally create permanent HD unload Pre-reorganization if logical relationship DBR Reload into shadow data sets USEREXIT support (also used in Apply Phase) Build secondary indexes Sparse routines and NULLVAL supported Shared secondary indexes supported Prefix Resolution & Update if logical relationships Image copy all recoverable data sets Optional Pointer Checker (of shadow data sets) IC COMP/COMPRTN supported Dynamic allocation or predefined image copy data sets Registered to DBRC as batch IC during ‘Takeover Phase’ Now we are ready to begin the reorg: We will reorg the shadow data sets – keep in mind we are capturing and storing information about updates calls to the DB from all control regions. We mentioned earlier that we can create permanent copy of HD Unload data set if needed. Between unload/reload we delete/redefine Shadow data sets if needed (REUSE(NO)). Can specify a USEREXIT during reload – change segment data, purge data, etc… This USEREXIT will also be called when applying any captured changes in case these segments were updated in online systems. Same interface as documented in HP Load manual

72 Apply Phase Replicate captured application update calls against shadow data sets Reload userexit support When almost caught up… CHANGE.DB NOAUTH Request online IMS systems to DBR DBDs for primary DB and indexes Finish replicating captured application update calls Verify DBs are DBR’d Deactivate capturing application update calls in online IMS system(s) Replicate any additional update calls if needed Transition from Apply to Takeover Phase can be delayed until a specific time of day We now have completed the reorg of the shadow data sets. What we have left to do is to replicate any update calls that occurred & we captured in the online systems to the shadow data sets. If USEREXIT for load, we will call it, same interface, for each update call. When we have almost caught up to the updates in the online systems, we will asynchronously issue DBR requests to online systems. We also notify DBRC to prohibit any authorization requests (another reason we require DB to be registered). We go back and finish any updates we captured, verify DBR has completed, make sure no more updates occurred then we are ready to transition to TAKEOVER phase. We now have DB & indexes DBR’d, and the shadow data sets have been reorganized and have any updates that occurred in the online systems during the time of the reorg applied to them. The transition to takeover can also be delayed until a specific, possibly less intrusive time of day. If specified, ORF will idle (prior to issuing DBR) where it is basically capturing updates & replicating to the shadow data sets.

73 Takeover Phase Perform ACBGEN for any changed DBDs
Checkpoint restart data We are now restartable DBRC notifications NOTIFY.REORG NOTIFY.IC NOTIFY.PRILOG & SECLOG (in ERROR) Captured changes applied to shadows NOTIFY.ALLOC All timestamps adjusted to after DBR and before START Rename Data Sets Rename originals to .T Rename shadows to originals Delete or rename .T to .S Copy changed DBDs to original DBD library Implement Online Change Perform ACBGEN for any changed DBDs Copy new ACBs to all online systems’ ACBLIBA and ACBLIBB libraries Request DMB to be replaced on all online IMS systems Can be turned off DB will remain stopped and in NOAUTH status Return code will be set to 4 Change.DB AUTH Request online IMS systems to START DBs Additional checkpoints occur after each step is completed (for restart) First thing – save any data needed for restart. Restart data set is defined during installation (VSAM KSDS), we dynamically allocate, info deleted after takeover completed so large restart data set not needed. Next, do DBRC notifications. All timestamps will be after DBR completed. When were we updating shadow data sets with updates from online region we did this as a batch IMS application and created an output log data set (dual logging if needed). We notify DBRC of these but are marked in error until we can correct all of the timestamps in the log to reflect timestamps determined by ORF. This will occur later in takeover and after DB is put back online. After any DBRC notifications, we will then begin renaming data sets so that the shadow data sets we reorganized into become the original data base data sets. We can either delete the original data base data sets or we can leave them with a different name. We will also copy any changed DBDs into old DBD library at this time.

74 Controlling Takeover DELAY – Allows for manual steps NO – ‘Test’ mode
Job stops after restart data saved DB is left DBR’d and PROHIBIT AUTH Restart can be used to initiate takeover process NO – ‘Test’ mode Stops after Apply Phase No changes implemented in online system No DBRC notifications No affect on original databases Window – Only perform takeover during certain time Begin time - Earliest time of day that takeover will start ORF job ‘idles’ with DB still online End time - Latest time of day that takeover will start End Action - Action to take if end time has passed and job hasn’t reached takeover point yet WTOR NEXTDAY CANCEL That’s the process, now we can talk about a few options/specifics. TAKEOVER(DELAY) – will stop ORF job after restart information was saved. The original database is offline but has not been changed yet. Can be used to perform additional manual steps prior to making new DB available if needed. TAKEOVER(NO) – primarily for testing, or possibly even creating a copy of a production database. Reorg process is run but no changes are made to online system or original database. The job would not be restartable.

75 Completion Phase All of the final reports and messages are written.
Shadow data sets are deleted if you have DELETE(Y) specified. Data sets are deallocated. Miscellaneous cleanup tasks are performed. The IMS Online Reorganization Facility batch job ends.

76 Controlling BMPs Problem – BMPs must be stopped when ORF needs to STOP or DBR a DB Interface with Program Restart Facility or ORF region controller front-end New BMPs are ‘paused’ until DB is restarted Existing BMPs – next CHKP HALDB – BMP is paused until DB is restarted Non-HALDB – pseudo U3303 Job restarted from last checkpoint after DB is restarted

77 New Transactions When DB is Offline
Problem – New transaction arrives when DB is DBR’d Transaction placed on suspend queue Exit/Automation to process suspend queue and reissue transaction Takeover WINDOW to reduce impact

78 New Functionality Interface to PRF/Region controller front-end for pausing BMPs Internal Logical Relationship for HALDB All partitions in single job Reorg Index Only Primary or secondary for non-HALDB Secondary for HALDB Can be single partition if PSINDEX Near Online Unload Create HD Unload data set while DB still online

79 IMS Tools ………What’s new and exciting
Janet LeBlanc Silicon Valley Lab


Download ppt "IMS Tools ………Reorganization"

Similar presentations


Ads by Google