Presentation on theme: "IMS Tools ………Reorganization"— Presentation transcript:
1IMS Tools ………Reorganization Janet LeBlancSilicon Valley Lab
2IMS Tools – Overview / Mission DM Tools is unique business model within IBMOperates as entrepreneurial businessHigh focus on return on investment / profitabilityCustomer requirements drivenMissionDeliver IMS based solutions that:Are profitable to IBMMeet customer expectations and needs
4Application Management Fast PathIMS High Performance Fast Path UtilitiesFull FunctionIMS High Performance Load,IMS High Performance Pointer CheckerIMS High Performance Prefix ResolutionIMS High Performance UnloadIMS Index BuilderIMS Parallel ReorganizationIMS Online Reorganization FacilityAdministrationIMS Database Control SuiteIMS Command Control FacilityIMS ETO SupportIMS HP Sysgen ToolsIMS Queue Control FacilityIMS Workload RouterIBM Data Encryption forIMS and DB2 DatabasesIMS Database Repair FacilityIMS HALDB Conversion andMaintenance AidIMS HD Compression- ExtendedIMS Library Integrity UtilitiesIMS Parameter ManagerIMS Sequential Randomizer GeneratorIMS Buffer Pool AnalyzerIMS Network Compression FacilityIMS Performance AnalyzerIMS Problem InvestigatorIMS Sysplex ManagerIBM Tivoli OMEGAMON XE for IMSIBM Application Recovery Tool forIMS and DB2 DatabasesIMS Database Recovery FacilityIMS DataPropagatorMS DEDB Fast RecoveryIMS High Perf Image CopyIMS High Perf Change AccumulationMS Batch Terminal SimulatorIMS Batch Backout ManagerIMS Connect ExtensionsIMS MFS Reversal UtilitiesIMS Program Restart FacilityData Base AdministrationUtility ManagementRecovery ManagementEnd to End ManagementPerformance ManagementIMS DATA BASE TOOLSApplication ManagementTM Management
5IMS Tools – Utilities Management Data Base AdministrationUtility ManagementRecovery ManagementEnd to End ManagementPerformance ManagementIMS DATA BASE TOOLSApplication ManagementTM ManagementIMS Tools – Utilities ManagementFast PathIMS High Performance Fast Path UtilitiesFull FunctionIMS High Performance LoadIMS High Performance UnloadIMS Index BuilderIMS High Performance Prefix ResolutionIMS Parallel ReorganizationIMS Online Reorganization FacilityIMS High Performance Pointer CheckerAdministrationIMS Database Control Suite
6IMS High Performance Fast Path Utilities High Performance FP Advanced Reorganization ToolDEDB Unload/ReloadDEDB AnalyzeDEDB ChangeFast Path Basic ToolsDEDB Pointer CheckerDEDB Tuning AidFast Path Online ToolsOnline Pointer CheckerOnline Data ExtractOnline Area Extender
7IMS High Performance Fast Path Utilities Fast Path Utility RegionOS BatchIMS HPFP UtilitiesFP Online ToolsOnline Pointer CheckerOnline Data ExtractOnline Area ExtenderFP Basic ToolsDEDB Unload/ReloadDEDB Pointer CheckerDEDB Tuning AidFP Advanced ToolDEDB Unload/ReloadDEDB AnalyzeDEDB ChangeIMS Control RegionIMS Dependent RegionIMS Onlinez/OS
8Highlights of FP Advanced Tool JCL ease of useIncrease the productivity of database support personnelMinimize steps for completing database Analyze, Change, Reload and UnloadMinimize DD statementsSingle driver program with command languageIntegrity with IMSIncrease the application availabilityMultiple area data sets (ADSs) supportHigh performanceReduce the consuming time for the maintenance and database conversion
9JCL ease of use : Minimize JCL DD statements FP Advanced Tool dynamically allocates :DEDB area data sets forInput of the analyze, change, and unload processOutput of the reload and change process (with space allocation)ACB librariesDBRC RECON data setsData sets of unloaded segment records forOutput of the unload processInput of the reload processHFPPRINT data setHFPRPTS data setSORT work data setsAbove DD statements are no longer required in JCL!Operators need not care about these data set allocations !
10JCL ease of use : Command language Single driver program with unified command language for all functionsSimplified JCL structure//HFP EXEC PGM=HFPMAIN0//HFPSYSIN DD *GLOBALDBRC=YESUNLOADDBD=DEDBJN22/*//HFP EXEC PGM=HFPMAIN0//HFPSYSIN DD *GLOBALDBRC=YESCHANGEDBD=DEDBJN22/*//HFP EXEC PGM=HFPMAIN0//HFPSYSIN DD *GLOBALDBRC=YESRELOADDBD=DEDBJN22/*//HFP EXEC PGM=HFPMAIN0//HFPSYSIN DD *GLOBALDBRC=YESANALYZEDBD=DEDBJN22/*
11JCL ease of use : Command language Simple language structureCommandsANALYZECHANGEGLOBALRELOADUNLOADSubcommandsALLOCATEFILECTLIRENAMELOADCTLREPORTTHRESHOLDKeywordsmany…
12JCL ease of use : Command language Nice featuresAdvanced data set name specificationMasks can be used for data set namese.g.) IMSVS.USERFILE.&AREAGenerate data set groups (GDG)Command syntax check without runGLOBAL SCAN=YES
13High performance : Technical topics Reduce I/O and CPUInternal sortNo SORTIN and SORTOUT I/O for sortingMMcallAsynchronous I/OConcurrent processing of multiple UOWsData spaceFor IOVF and SDEPMulti-taskingParallel processing for multiple DEDB areasMultiple subtasks for analyze, change, reload, and unloadThe number of concurrent processes can be specified by control statement
14Unload/Reload/Change functions Unloads a single DEDB area or multiple DEDB areas to QSAM data setsReload:Reloads a single DEDB area or multiple DEDB areas from QSAM data setsIf necessary, the analyze function can be included in the reload processChange:Reorganizes or restructures a single DEDB or multiple DEDB areas with single job stepIf necessary, the analyze function can be included in the change process
15Unload/Reload/Change: Minimize manual steps Fast Path Basic Tools Unload/Reload EXEC steps1(NEWRandomizer)OLD ACB library(NEWACBlibrary)RECON5SortControl data setICOLD AREASControl statementsREORGANIZED AREAS2RELOADUNLOAD(one job)(multi-task)RELOAD(one or multiple jobs)(single task)RELOADSORT(optional)Control statementsIDCAMs DELETE/ DEFINEDBRC utilitySet RN onIOVF Work data setor data space(SDEPWorkdata set)34
16HPFP Unload Unload: Minimize manual steps 1 IC OLD AREAS (NEWrandomizer)OLD ACB library(NEWACBlibrary)RECONSortControl data setICOLD AREASControl StatementsREORGANIZED AREASUNLOAD(one job)(multi-task)RELOAD(one job)(multi-task)SORT(optional)Control statementsIDCAMs DELETE/ DEFINEDBRC UtilitySet RN onHPFP UnloadIOVF/SDEPWork data setor data space(SDEPWorkdata set)1
20Reload/Change: Place certain segments into overflow CIs The LOADCTL subcommand controls the place to insert dependent segments, which is either DOVF or IOVF.//HFP EXEC PGM=HFPMAIN0//HFPSYSIN DD *RELOADDBD=DEDBJN22LOADCTL SEGMENT=DD2,LOCATION=IOVF/*DEDB AREARAPIOVFSDEPBASEROOTDOVFDD1DD1DD1DD1DD1DD2
21Sample JCLs You can see: 4 sample JCLs How easy JCL are Examples of control statements4 sample JCLsUnload an area registered with DBRCReload an area registered with DBRCReorganize an area registered with DBRCRestructure an area registered with DBRC
22Unloading an area registered with DBRC //HFPUNL EXEC PGM=HFPMAIN0//STEPLIB DD DISP=SHR,DSN=HFP220.SHFPMOD0// DD DISP=SHR,DSN=IMSVS.SDFSRESL// DD DISP=SHR,DSN=IMSVS.PGMLIB//IMSACB DD DISP=SHR,DSN=IMSVS.ACBLIB//IMSDALIB DD DISP=SHR,DSN=IMSVS.MDALIB//HFPRPTS DD SYSOUT=*//HFPPRINT DD SYSOUT=*//HFPSYSIN DD *GLOBALDBRC=YESUNLOADDBD=DEDBJN22,IAREA=(DB22AR0),OAREA=(DB22AR0),ODSNMASK=‘IMSVS.USRFILE.&AREA‘/*
23Reloading an area registered in DBRC //HFP EXEC PGM=HFPMAIN0//STEPLIB DD DISP=SHR,DSN=HFP220.SHFPMOD0// DD DISP=SHR,DSN=IMSVS.SDFSRESL// DD DISP=SHR,DSN=IMSVS.PGMLIB//IMSACB DD DISP=SHR,DSN=IMSVS.ACBLIB//IMSDALIB DD DISP=SHR,DSN=IMSVS.MDALIB//HFPRPTS DD SYSOUT=*//HFPPRINT DD SYSOUT=*//HFPSYSIN DD *GLOBALDBRC=YESRELOADDBD=DEDBJN22,IAREA=(DB22AR0),IDSNMASK=‘IMSVS.USRFILE.&AREA',OAREA=(DB22AR0)/*PTRCHKLVL=FULLREPORT/*
24Reorganizing an area registered in DBRC //HFP EXEC PGM=HFPMAIN0//STEPLIB DD DISP=SHR,DSN=HFP220.SHFPMOD0// DD DISP=SHR,DSN=IMSVS.SDFSRESL// DD DISP=SHR,DSN=IMSVS.PGMLIB//IMSACB DD DISP=SHR,DSN=IMSVS.ACBLIB//IMSDALIB DD DISP=SHR,DSN=IMSVS.MDALIB//IAREA001 DD DISP=SHR,DSN=HPFP.OLD.DB22AR0.ADS1//HFPRPTS DD SYSOUT=*//HFPPRINT DD SYSOUT=*//HFPSYSIN DD *GLOBALDBRC=YESCHANGEDBD=DEDBJN22,IAREA=(DB22AR0),OAREA=(DB22AR0)/*PTRCHKLVL=FULLREPORT/*
27IMS High Performance Unload v1.1 5655-E06 HP UnloadUnloads HDAM, HIDAM, HISAM and SHISAM databasesWith IMS V7, support of the new HALDB database types PHDAM, PHIDAMProvides an IMS application program interfaceFor stand-alone batch programAssembler, COBOL, or PL/1Transparent to programmerContains a high-performance DB retrieval engineThe HSSR EngineHigh speed buffering technique similar to OSAM SBDifferent from DL/I BufferingSequential Subset RandomizerCreates randomizing module for fast sequential processing of record subsetAllows physical clustering of database records in same subsetFast Unload is key because it is the time consuming step i the DB reorg process because it is the one which creates the unload dataset by reading every disorganized part of the DB involving a lot of I/Os.
28IMS HP Unload in the Reorganization Process Database unload utilitiesUnload a database in compressed or uncompressed formatUnload a broken HIDAM, HDAM, or HISAM databaseSkip invalid HD pointers or invalid recordsCreate unloaded sequential data sets for use by application programsMultiple unload formatsUse dynamic allocation of DB datasetsEasy to read output reports and statisticsDB statistics report or Randomizing statistics reportPre-tuned with defaults that should be adequate for most databasesFull support for HALDBUnload one, several or all partitionsMigration/fallback supportNew function for user exit FABHEXTRSpecify that n records are to be unloaded from eachHALDB PartitionPARTEXTR control statementUpto three output datasetstwo when IPR Parallel Reorg is usedsame or different formatscan use different user exit for eachUser Exit supportuser exits can be written in Assembler, COBOL or PL/1enables skipping of individual segments, skipping remaining segments in DB record, and editing of segment datachoice of type A or B exitstype B is simpler, but limits output to a single unload filetype A exits (upto three) can be used for creating upto three unload files, and have more function - such as skip to root with specified keyOptional Statistical ReportsSegment counts, DB statistics, randomising statistics, DB Call statistics, buffering and I/O statistics, CAB (Chained Anticipatory Buffering) statisticsHALDB partition information
29IMS High Performance Load v2.1 5655-M26 HP Load UtilityPerformance replacement of IMS HD Reorganization Load utility (DFSURGL0)Complement to IMS High Performance Unload Toolcompressed/uncompressed input in various formatsdynamic allocation of DB datasetsFull support for HALDBSelf OptimizationExcept for DATASPACE option where the default is N and use of Y is betterSORT=YESIntegrated Load SORTHDAM Physical Sequence Sort for Reload (PSSR) UtilityPreviously in DBTools SMUSorts the unloaded database data set before reloadUsed with HALDB, when changing partition boundaries during reorgAvoids “cascading” on Reload> 71% reduction in elapse time> 92% reduction in CPU time(over base IMS load utility)NewDB Load UtilityV7 SupportDynamic allocationSupport for HALDBDBRC supportJCL compatibility with DBT:FRROut of the box - performance tunedInitializes empty HDAM, HIDAM DBsPerforms HALDB partition initialization as part of the LoadUser Exit FacilityUses it's own buffering & database writes - not DL/I: Chained writes to IMS databases , All buffers allocated above the 16M lineProvides statistics on space usage and pointers.Physical Sequence Sort for Reloadsorts an unloaded database data set from HDAM databasehelp preventing cascadingproduce reports about database sizes and randomizer performance.The following new functions introduced for PSSR: User Exit support Segment Edit/Compression exit support IMS/ESA Year 2000 Exit Tool (5697-E04) support
30IMS HP Load in the Reorganization Process Support of IMS HDAM, HIDAM, PHDAM and PHIDAM databasesSupports reloading of compressed segments without calling compression routineInitializes empty HDAM and HIDAM databasesDFSUINPT DD Dummy statementSEQERROR=ACCEPT optionSupport of HISAM and SHISAM thru maintenanceDB Reload and DB InitialisationSupport of logical relationship or secondary indexesCreates a DFSURWF1 data set that can be used by:Index Builder (IB) productIMS High Performance Prefix ResolutionIMS Prefix Resolution utility (DFSURG10)Support of HALDBAutomatically initializes HALDB partition data set before reloadProvides a performance replacement for IMS Partition Initialization UtilityCreates ILDSWhen changing partition boundaries during reorg of HALDBPSSR may be used prior load to sort segments by RAP within partitions for PHDAM.SEQERROR=ACCEPT was developed to be BMC compliant. It allows to load errors in a database!Support of HISAM and SHISAM thru PTFsPQ53316/UQ59432PQ54113 for the DOC APAR
31IMS HP Load in the Reorganization Process ... User exit facility for additional processing of each segmentAssembler, COBOL or PL/1Edit segmentDelete segment (or this and subsequent segments of DB record)Force segment into overflow (HDAM or PHDAM)Force segment to start new block (HIDAM or PHIDAM)Load APIUsed in application for intial load processingBitmap ResetterAdjust the bitmap of HDAM/HIDAM/PHDAM/PHIDAM database to accommodate denser packing of the database blocks.Statistics reports to aid in tuning the databaseReporting on space use and segment pointer statisticsNewNew
32IMS HP Prefix Resolution v3.1 5655-M27 Enables you to resolve and update prefixes of IMS databases involved in logical relationships as a single job stepFunctions:Eliminates the WF2 and WF3 data setsExecutes prefix resolution and prefix update functions as replacements for the existing IMS Prefix Resolution and IMS Prefix Update utilitiesSupports IMS Parallel Reorganization, V3 single job step execution of database reorganization, prefix resolution, and prefix update tasksYour Value:Better performance and faster running prefix resolution and prefix updatesSignificant performance improvements when used in place of existing IMS utilitiesUse to maintain all logically related IMS databases and is required after the databases are loaded or reorganizedHigh Performance Prefix Resolution is a replacement product for the Database Tools FPR FeaturePrefix resolution is used in maintaining all logically related IMS databases and is required after the databases are loaded or reorganized. IBM IMS High Performance Prefix Resolution, Version 3.1, enables you to resolve and update prefixes of IMS databases involved in logical relationships as a single job step. A data transfer service called HPPRPIPE (eliminating Work File 2 (WF2) and Work File 3 (WF3) data sets) avoids much of the I/O, tape handling, and DASD requirements associated with prefix resolution and prefix update.This version of IMS HP Prefix Resolution supports IMS Parallel Reorganization for z/OS, Version 3.1 single job step execution of database reorganization, prefix resolution, and prefix update tasks. Also included with this latest version of IMS High Performance Prefix Resolution, Version 3.1, is the capability to perform the Prefix Resolution function and the Prefix Update function as replacements for the IMS Prefix Resolution Utility or the IMS Prefix Update Utility.Using these functions over the existing base IMS Utilities provides even more performance improvements when performing this type of database maintenance.
33IMS Index Builder - Major Functions Support for Full-Function non-partitioned DB and HALDBEasy-to-use one step procedureCreating New Secondary IndexesRebuilding Secondary IndexesUsing as input output from initial load or reload after a reorg (DFSURWF1)Using as input a DL/I scan of the IMS databaseUsing as input the output from prefix resolution (DFSURIDX)Creating Input for IMS HP Prefix ResolutionSplit FunctionInitializing Empty Secondary IndexesRebuilding a HIDAM Primary IndexUsing IMS IB for Recovery of Secondary IndicesChange in the GENJCL.RECOV skeletonTo rebuild indices after reorganization processTo recover damaged secondary indices and avoid full reorganization processTo avoid taking IC of secondary indicesTo minimize elapsed time....IB Version 2 doesn't generate the DBRC model commands like with IB V1.IMS IB V2 notifies DBRC when tasks complete successfully.IMS IB V2.3 will be GA on 20/09/2002
34IMS Parallel Reorganization New VersionIPR is an umbrella product or higher level framework for execution of the following toolsIMS High Performance UnloadIMS High Performance LoadIMS Index BuilderIMS High Performance Image CopyIMS High Performance Pointer CheckerFunctions areParallel reorganizationHigh Speed alternative to serial use of HP UnloadHigh Speed alternative to serial use of HP LoadEnhanced Data Base Scan (for logical relationships) using HP UnloadIMS Parallel Reorganization provides the infrastructure to operate the reorganization utilities in parallel (or independently), allowing a significant reduction in the reorganization time.
35IPR Driver V3 – Big Picture OnlineRead AccessConcurrentDatabaseBackupNew in v3Single Step ExecutionSep. 21 AnnouncementIPR Driver V3IMS Tools New Versions/ReleaseDB- IMS Parallel Reorganization V3- IMS High Performance Unload V1- IMS High Performance Load V2- IMS Index Builder V2.3- IMS High Performance Image Copy V3.2- IMS High Performance Pointer Checker V2- IMS High Performance Prefix Resolution V3DatabaseImage CopySecondaryIndexIndexImage CopiesPrefix Update forInternal LogicalRelationshipNew in V3DBAdminDB(Shadow)Reports andDB StatisticsAutomatedDBRCProcessingDatabase“Health Check”New in V3Sec. Index(Shadow)
36IPR Driver V3 FunctionsStopping the database or making it read-only before reorgIMS /DBR or /DBD DATABASE command is issued by IPR DriverReorganizing database data sets into “shadow” data setsUnload, Reload, and Index-Builder tasks run concurrentlyImage copies can be taken during reorg, with optional HASH pointer check Type-A Image CopyUpdating segment prefixes after the database reloadFor the database that has internal logical relationship Concurrent Prefix UpdateTaking image copies with optional HASH check after the prefix update Type-B Image CopyStopping the database at the completion of READ-ONLY reorgIMS /DBR DATABASE command is issued by IPR DriverPerforming post-reorganization processOriginal and “shadow” data sets names are swappedDBRC is notified of the reorg completion and the image copyNewNewNew
37Sample JCL Statements for a Parallel Reorg with Secondary Indexes and Internal Logical Relationships //IPR EXEC PGM=HPSGMAIN,PARM='DBD=SAMPLEDB,DBRC=Y'//STEPLIB DD DISP=SHR,DSN=TOOLS.LIBRARY// DD DISP=SHR,DSN=IMS.SDFSRESL//IMS DD DISP=SHR,DSN=IMS.DBDLIB//IMSDALIB DD DISP=SHR,DSN=IMS.DALIB//DFSURCDS DD DISP=SHR,DSN=DFSURCDS//DFSURWF1 DD DISP=(NEW,PASS,DELETE),DSN=&&DFSURWF1,// UNIT=SYSALLDA,SPACE=(CYL,(n,m)),// DCB=(RECFM=VB,LRECL=800,BLKSIZE=2400)//HPSIN DD *(REORG)IMSCMD=YESDBSHARE=YESIC=YESINDEXBLD=YESNAMESWAP=YESPREFIXRES=YESDELOLDDS=YES(PREFIXRES)UPDLPC=NO/*//ICEIN DD *GLOBAL HDPC=Y,ICHLQ=IMSICA.If IC=YES and PREFIXRES=YES are specified and the database has an internal logical relationship, ICTYPE=B is implicitly set.
38IMS DB Reorg. – Online Solutions IMS Version 9 HALDB OnLine Reorg. (OLR)Integrated OnLine Reorganization (OLR) of HALDBsHALDB OLR provides 100% availability of the largest databases in the world!OLR provides non-disruptive reorganization of HALDB PHDAM and PHIDAM partitionsConcurrent IMS updates are allowed while OLR is activeScheduled data outage not requiredIMS Online Reorganization Facility (Tool)For non-HALDB and HALDBWith restrictionsFor IMS V7 or aboveNew
40Features Reorg Index Only Near Online Unload Reorganize a Database while still onlineData sharing environments fully supportedAllows DBD changes to be implemented and does not require manual intervention after reorganizationAutomatic online change implementationComplete restart-ability once in the Takeover PhaseControlling TakeoverDelay, Abend, WindowInterface to PRF/Region controller front-end for pausing BMPsInternal Logical RelationshipsHDAM and HIDAMHALDBAll partitions in single jobReorg Index OnlyPrimary or secondary for non-HALDBSecondary for HALDBCan be single partition if PSINDEXNear Online UnloadCreate HD Unload data set while DB still onlineAgain, main feature is that any full function database can be reorganized while still online. Only unavailable for short duration, usually less than one minute, while data sets are renamed.Data sharing is fully supported – DB can be shared by any number of online IMS systems. ORF utility runs outside of IMS, and we capture and apply changes from all online systems. Use XCF as means of communications from IMS online systems to ORF utility.We do allow most DBD changes (ones that do not also require application, i.e. PSB, changes).When there are DBD changes, we will all of the steps necessary to make those changes effective in the online systems – without any manual intervention!Put new DBD into old DBD library, run ACBGEN, copy updated ACBs to ACBLIBA & ACBLIBB data sets, do something similar to IMS Online Change to have IMS load the new DMBs, then start the DB.
41Verification Phase Copy Phase Reorg Phase Apply Phase Takeover Phase IMS Control Region(s)Online Reorg Facility (ORF)Verification PhaseCopy PhaseCapture log updatesShadow Primary DB & Primary IndexCopy PhaseOriginalPrimaryDatabaseReorg PhaseUnloadReloadRebuild Sec. IndexesPreReorg/Prefix Res/UpdateImage Copy/Pointer CheckerReorg & Apply PhasesCapture application update callsTemporaryUnload FileImage CopiesReorganized Shadow DB, Primary & Secondary IndexesApply PhaseTakeover Phase/DBR original DBLoad new DMB/STA DBTakeover PhaseRequest /DBR DBDBRC notificationsRename shadows to originalsACBGEN, Online changeRequest /STA DBGood diagram to keep open for reference as we go into detail of each phase.Left side shows what is happening in the online systems. Right side shows the different processes, or ‘phases’ of an ORF utility – batch job. We will talk about the detail as it relates to the ORF utility but keep in mind that there are corresponding things happening in the control region address spaces, both by ORF and also access to the databases by your transactions and BMPsReorganizedDatabaseand IndexesOnline Reorganization CompletedBatchLOGCompletion Phase
42IMS Online Reorganization Facility for z/OS DBD Changes permittedControlling Takeover PhaseManaging BMPs during Takeover PhaseNew FunctionsProduct NameAllows databases to be reorganized without taking them offline. Uses shadow (snapshot copy) data sets to perform reorganization. Captures and saves copy of any update calls made in online systems and will replicate those calls against the shadow data sets after the shadow data sets have been reorganized. Once all caught up, DB is DBRd in all online systems, data sets are renamed, and DB is started on all online systems again.We go over these steps in a lot more detail in rest of presentation.
43IMS Online Reorganization Facility for z/OS …BENEFITS Increased Data AvailabilityOutage reduced from hours to secondsMinimal Overhead Added to Control RegionsAutomation of DBA tasksSingle step ReorgAutomatic determination of necessary reorganization stepsSame JCL for all databasesChanges implemented across all online systemsBetter DB performanceCan reorganize when neededImproved DBA EfficiencyNo coding/maintenance of IDCAMS statementsNo manual intervention required after reorganization processAgain, we see the primary benefit as increased data availability.ORF also can be used to automate the many DBA tasks that need to be done for a database reorganization. A DB reorganization traditional includes: Unload, Prereorg (if logicals), Reload, Build Secondary Indexes, Prefix Resolution & Update, Image Copies, Pointer checker, ACB gens, online change. ORF can do all of this in a single step and without manual intervention.We will show an example towards the end, but all of the reorg processes are done in a single job step, and normally the only thing you’ll need to change is the DBD name.
59User Header Element Name Field Length Field Type Description DBDNAME 8 CharacterDBD NameFILLERnn1-999IntegerNn suffix determines lengthHDRLEN2BinaryREQ if SEGKEY, SEGCKEY or ROOTKEY usedROOTKEYvariablevariesSequence field of root segmentROOTRBA4RBA of root segmentSEGCKEYVariableVariesConcatenated keySEGCKYSZLength of segment concatenated keySEGCODE1binarySegment code of segmentSEGDATSZLength of segment dataSEGDBYTEDelete byte of segmentSEGKEYSequence FieldSEGKEYOFOffset of segment keySEGKEYSZLength of segment keySEGLEV1Hierarchical level of segmentSEGLEV2Hierarchical level of semgnetSEGNAMEName of SegmentSEGRBADatabase RBA of segment
60Rich in Function ……Multiple DB Type Support Supports reorganization of:HDAM,HIDAM,PHDAM,PHIDAM,HISAM andSHISAM databases
61Rich in Function ……Compression Support UnloadOptions to unload either compressed or uncompressed dataLoadAutomatic recognition of status of unload data – compressed/uncompressed
62Rich in Function …..HALDB Support Reorganization:By PartitionRange of partitionsAll partitionsRebuild ILDSRebuild Secondary IndexesConversion SupportUnload with migration optionLoad has integrated partition initialization
63Rich in Function …..Fault Tolerance UnloadContinue Processing after sequence errorsHandle Corrupt DatabasesIncorrect HD pointers can be bypassedProcessing of incorrect HISAM record can be skippedRoot segments of HIDAM or PHIDAM can be processing using Index instead of pointersLoadReject/Save/Accept Sequence Errors
64Rich in Function …..User Exits UnloadRecord Formatting exit & Return Code Edit exitAssembler, COBOL or PL/1Edit segment5 std record format routines providedRun-Time Environment ExitBuffer Handler Initialization ExitLoadUser exit facility for additional processing of each segmentAssembler, COBOL or PL/1edit segmentdelete segment (or this and subsequent segments of DB record)force segment into overflow (HDAM or PHDAM)force segment to start new block (HIDAM or PHIDAM)
65Rich in Function …..Application Program Interface UnloadFor stand-alone batch programAssembler, COBOL, or PL/1Transparent to programmerLoadAssembler Language , COBOL for OS/390 and VM PL/I for z/OS and OS/390
66Rich in Function ….Additional utilities Sequential Subset RandomizerCreates randomizing module for fast sequential processing of record subsetAllows physical clustering of database records in same subsetPhysical Sequence Sort for Reload (PSSR) UtilitySorts the unloaded database data set before reloadUsed with HALDB, when changing partition boundaries during reorgAvoids “cascading” on ReloadBitmap ResetterThe bitmap resetter can be used to adjust the bitmap of HDAM/HIDAM/PHDAM/PHIDAM database to accommodate denser packing of the database blocks.
68Verification Phase Before we get started – verify we have what we need DBD StructureDo we support current DBD definitions?If changes, can we implement all changes?DBRC definitionsData setsCan shadow datasets be allocated?Can image copy data sets be allocated?Can work/temp files be allocated?SoftwareAre all necessary utilities available?Are these utilities at the proper level?Are online systems at compatible level?First phase is the Verification phase. We are running many ‘reorg’ processes in a single step, and sometimes the elapsed time of the entire reorg process can be long, we want to verify as much as possible up front, rather than waiting until say the Prefix Resolution step to verify we have enough space for the WF1 file.This slide lists some of the things we check in the verification phase.DBD structure – do we support the DBD specified. Currently a few restrictions, like external logical relationships, that we will list in detail later. As we mentioned we do allow DBD changes, we verify new DBD as well. Basically what user does is to run DBDGEN into a ‘staging’ DBD library then provide ORF the staging DBD library name.DBRC definitions – are requirements listed a few foils back metData sets – we can, and by default, we can dynamically allocate all the datasets that are produced as part of the reorg (shadow data sets, image copy data sets, work files). We make sure we can get these allocated, or if you predefined them, that they exist and we can get to them.Software – ORF does use many of the current IMS HP tolols. We need to make sure that the tools needed to reorganize the type of DB specified are available and that they are at least at the minimum level needed by ORF. There were some changes in the tools announced in September that ORF needs.
69Specify a library with the new DBD Limited to non-HALDB DBD ChangesSpecify a library with the new DBDLimited to non-HALDBNo changes in hierarchical structureNo changes in DBRCExamples:Change of randomizing parameterAddition of segment typesSegment length increaseNew DBD may be copied to primary DBD library during TakeoverWe have talked about this some but here is list of the type of DBD changes that are and are not supported.HAL/DB – currently ORF only runs against a single partition at a time. The things that are still in the DBD for HALDB effect all partitions so ORF can not change just a single partition.Can change randomizer for space – must keep data base records in same physical sequence (like DFSHDC40).Can add segments at bottom of hierarchy. Can’t add in middle because we need to replicate updates made in online systems with old DBD.Can increase segment size – we also will talk later about using USEREXIT to change or prime data.One more automation step – during takeover, we will copy the new DBD to the library that we found original DBD in (user can turn this off if needed).
70Copy PhaseActivate capture of log updates to primary database in online systemsSTOP/START primary database to create sync pointCopy original database data sets to shadow data setsApply log updates to shadow data setsSwitch from log updates to application updates in online systemsSTOP/START primary database to create transition pointBuild shadow primary indexOnto the next phase:Copy phase – create a snapshot copy of the original primary data base only – no indexesConnect to all IMS systems that are authorized for the database – another reason for requiring DB to be registered to DBRC. Activate capturing of log updates for this DB.Interface with TOI to STOP/START the primary DB – flushes buffers, updates catalog, creates syncpoint to create snapshot copy.Copy the original database to shadow data sets and apply captured log updates to the shadow.When caught up, transition to capturing application update calls that will be used to replicate calls to shadow data sets after the shadow data sets are reorganized.Also, our capture points are using standard IMS exits, and are only active for the specific DB ORF is running against. No overhead if ORF utility not being run.Any updates captured are sent thru XCF to ORF utility where they are stored in temp data set until needed.Reason we rebuild primary index rather than copy is for cases like HAL/DB or when primary index is non recoverable and IMS does not log updates for these data sets.
71Reorg Phase Temporary unload data set Unload shadow databaseTemporary unload data setOptionally create permanent HD unloadPre-reorganizationif logical relationshipDBRReload into shadow data setsUSEREXIT support (also used in Apply Phase)Build secondary indexesSparse routines and NULLVAL supportedShared secondary indexes supportedPrefix Resolution & Updateif logical relationshipsImage copy all recoverable data setsOptional Pointer Checker (of shadow data sets)IC COMP/COMPRTN supportedDynamic allocation or predefined image copy data setsRegistered to DBRC as batch IC during ‘Takeover Phase’Now we are ready to begin the reorg:We will reorg the shadow data sets – keep in mind we are capturing and storing information about updates calls to the DB from all control regions.We mentioned earlier that we can create permanent copy of HD Unload data set if needed.Between unload/reload we delete/redefine Shadow data sets if needed (REUSE(NO)).Can specify a USEREXIT during reload – change segment data, purge data, etc…This USEREXIT will also be called when applying any captured changes in case these segments were updated in online systems.Same interface as documented in HP Load manual
72Apply PhaseReplicate captured application update calls against shadow data setsReload userexit supportWhen almost caught up…CHANGE.DB NOAUTHRequest online IMS systems to DBR DBDs for primary DB and indexesFinish replicating captured application update callsVerify DBs are DBR’dDeactivate capturing application update calls in online IMS system(s)Replicate any additional update calls if neededTransition from Apply to Takeover Phase can be delayed until a specific time of dayWe now have completed the reorg of the shadow data sets. What we have left to do is to replicate any update calls that occurred & we captured in the online systems to the shadow data sets.If USEREXIT for load, we will call it, same interface, for each update call.When we have almost caught up to the updates in the online systems, we will asynchronously issue DBR requests to online systems. We also notify DBRC to prohibit any authorization requests (another reason we require DB to be registered).We go back and finish any updates we captured, verify DBR has completed, make sure no more updates occurred then we are ready to transition to TAKEOVER phase.We now have DB & indexes DBR’d, and the shadow data sets have been reorganized and have any updates that occurred in the online systems during the time of the reorg applied to them.The transition to takeover can also be delayed until a specific, possibly less intrusive time of day. If specified, ORF will idle (prior to issuing DBR) where it is basically capturing updates & replicating to the shadow data sets.
73Takeover Phase Perform ACBGEN for any changed DBDs Checkpoint restart dataWe are now restartableDBRC notificationsNOTIFY.REORGNOTIFY.ICNOTIFY.PRILOG & SECLOG (in ERROR)Captured changes applied to shadowsNOTIFY.ALLOCAll timestamps adjusted to after DBR and before STARTRename Data SetsRename originals to .TRename shadows to originalsDelete or rename .T to .SCopy changed DBDs to original DBD libraryImplement Online ChangePerform ACBGEN for any changed DBDsCopy new ACBs to all online systems’ ACBLIBA and ACBLIBB librariesRequest DMB to be replaced on all online IMS systemsCan be turned offDB will remain stopped and in NOAUTH statusReturn code will be set to 4Change.DB AUTHRequest online IMS systems to START DBsAdditional checkpoints occur after each step is completed (for restart)First thing – save any data needed for restart. Restart data set is defined during installation (VSAM KSDS), we dynamically allocate, info deleted after takeover completed so large restart data set not needed.Next, do DBRC notifications. All timestamps will be after DBR completed.When were we updating shadow data sets with updates from online region we did this as a batch IMS application and created an output log data set (dual logging if needed). We notify DBRC of these but are marked in error until we can correct all of the timestamps in the log to reflect timestamps determined by ORF. This will occur later in takeover and after DB is put back online.After any DBRC notifications, we will then begin renaming data sets so that the shadow data sets we reorganized into become the original data base data sets. We can either delete the original data base data sets or we can leave them with a different name.We will also copy any changed DBDs into old DBD library at this time.
74Controlling Takeover DELAY – Allows for manual steps NO – ‘Test’ mode Job stops after restart data savedDB is left DBR’d and PROHIBIT AUTHRestart can be used to initiate takeover processNO – ‘Test’ modeStops after Apply PhaseNo changes implemented in online systemNo DBRC notificationsNo affect on original databasesWindow – Only perform takeover during certain timeBegin time - Earliest time of day that takeover will startORF job ‘idles’ with DB still onlineEnd time - Latest time of day that takeover will startEnd Action - Action to take if end time has passed and job hasn’t reached takeover point yetWTORNEXTDAYCANCELThat’s the process, now we can talk about a few options/specifics.TAKEOVER(DELAY) – will stop ORF job after restart information was saved. The original database is offline but has not been changed yet. Can be used to perform additional manual steps prior to making new DB available if needed.TAKEOVER(NO) – primarily for testing, or possibly even creating a copy of a production database. Reorg process is run but no changes are made to online system or original database. The job would not be restartable.
75Completion Phase All of the final reports and messages are written. Shadow data sets are deleted if you have DELETE(Y) specified.Data sets are deallocated.Miscellaneous cleanup tasks are performed.The IMS Online Reorganization Facility batch job ends.
76Controlling BMPsProblem – BMPs must be stopped when ORF needs to STOP or DBR a DBInterface with Program Restart Facility or ORF region controller front-endNew BMPs are ‘paused’ until DB is restartedExisting BMPs – next CHKPHALDB – BMP is paused until DB is restartedNon-HALDB –pseudo U3303Job restarted from last checkpoint after DB is restarted
77New Transactions When DB is Offline Problem – New transaction arrives when DB is DBR’dTransaction placed on suspend queueExit/Automation to process suspend queue and reissue transactionTakeover WINDOW to reduce impact
78New FunctionalityInterface to PRF/Region controller front-end for pausing BMPsInternal Logical Relationship for HALDBAll partitions in single jobReorg Index OnlyPrimary or secondary for non-HALDBSecondary for HALDBCan be single partition if PSINDEXNear Online UnloadCreate HD Unload data set while DB still online
79IMS Tools ………What’s new and exciting Janet LeBlancSilicon Valley Lab