Presentation on theme: "ADABAS Extraction & Change Data Capture"— Presentation transcript:
1ADABAS Extraction & Change Data Capture Presented byChris S. BradleyNatWorks, Inc.
2Where Do You Need Data Today? The Question…Where Do You NeedYOURADABASData Today?
3Web The Problems… End-User Extraction Data Warehouse Extraction Web Publishing / Data ExchangeAdabasDATADATAData WarehouseExtractionWebEnd-UserExtractionWeb Publishing /Data Exchange
4A Software AG customer who has ADABAS & NATURAL The Message…A Software AG customer who hasADABAS & NATURALalready has the best tools to handleallrequirements forData Extraction & Change Data Capture
5ADABAS - Two Major Issues How to Access ADABASADABAS data structures
6The Only Way to communicate directly to ADABAS is through Accessing ADABAS DataADABASADABAS Direct CallsThe Only Way to communicate directly to ADABAS is throughDirect Calls
7Accessing ADABAS Data continued Option #5VendorUtilitiesOption #4ADABASUtilitiesADABAS Direct CallsOption #1ImbeddedDirect CallsOption #2SQL / ODBCOption #3Natural
8ADABAS Data Structures All standard data formats are supported:alphanumeric, binary, fixed point, floating point, packed decimal, unpacked decimal, ...Supports two basic field typesElementary fields“recurring fields” (MUs)Consecutive fields may be groupedA group may be repeated - Periodic Groups (PEs)Periodic Groups may contain one or more Multiple-Value Fields
9ADABAS Data Structures continued ADABAS has unique data structures:Multi-Valued Fields - “MUs” (array structure)Periodic-Groups - “PEs” (table structure)MUs in PEs (multi-dimensional structure)ADABAS has some “difficult” data typesIBM STCK-based Date and TimeStamp fieldsEBCDIC to ASCIIPacked FieldsSign Byte Handling
10ADABAS Extraction Facts NATURAL was developed by Software AG specifically to access ADABASFACT #2 NATURAL represents the most widely used AND best understood solution for accessing ADABASFACT #3 What NATURAL should not be used to do, ADABAS Utilities handle (with support from NATURAL)FACT #4 Extraction / CDC should be done in BATCH
11ADABAS Extraction - Conclusions Natural allows flexible ADABAS AccessNatural easily handles all ADABAS data structuresNatural easily handles all ADABAS data typesNatural will always work with ADABASNatural for maximum flexibilityADABAS Utilities for maximum performance
12Solving ADABAS Data Access Option #4ADABASUtilitiesADABAS Direct CallsOption #3NaturalEmbrace What Exists (ROI)Best Understood SolutionHigh Performing Solution
13NO GENERATION The Real Problem Generation is needed for: Required Natural ObjectsRequired ADABAS ParametersRequired JCL / Script Processes
16The NatQuery Extraction Solution NatQuery works by acting as an on-demand Natural Programmer.From a graphical user interface a User is enabled to:Create Query SpecificationsGenerate Natural Data Extraction programs from these SpecificationsSubmit Generated Extract programs for executionRemotely monitor execution statusDownload Extracted DataLoad extracted data into MS Access, MS Excel or convert data into XML (with optional XSL)
17The NatQuery Extraction Solution Internally, NatQuery can be thought of as having three components:Administrative ComponentEnd-User ComponentGeneration ComponentNatQueryAdministration ComponentEnd-userComponentGenerationComponent
18The NatQuery Extraction Solution The Administrative Component is used by an Administrator to capture information that is specific to the platform, environment, and ADABAS data files that NatQuery will be used against.NatQueryAdministration ComponentEnd-userComponentGenerationComponentThe Administration Component provides NatQuery with the ability to capture application-specific intelligence.
19The NatQuery Extraction Solution The End-User Componentallows for the easy entry of extract specifications.The Generation componentgenerates a Natural program from a specification.Administration ComponentEnd-userComponentGenerationComponentNaturalProgramEnd-userRequiredFiles / FieldsUser-specifiedSelect LogicOptionalVariablesDesiredTarget
20The NatQuery Extraction Solution Server EnvironmentExtraction With NaturalAdabasNaturalNaturalProgramDATADATAOther EnvironmentNatQueryNaturalProgramUserExtractSpecificationDATAAccessExcelXML...Workstation Environment
21The NatQuery Extraction Solution Server EnvironmentExtraction With ADABAS Utilities And NaturalAdabasNaturalADACMPUtilityNaturalProgramDATADATAOther EnvironmentADACMPParametersNatQueryNaturalProgramUserExtractSpecificationDATAWorkstation Environment
22The NatQuery Extraction Solution Requests Execute in “batch”Better control of requestsRequest execution can be easily scheduledImpact on online production applications are controlledMore efficient execution environment over “online”Significantly Less overheadNatQuery handles Job Control Language (JCL)Template approach provides easy set-up / maintenanceDynamic substitution makes templates executable
23NatQuery Administration Overview 1) Provide Site-Specific Job Control Language (JCL)JCLJCL Templates Provide:Integration to Natural / ADABASDynamic Process CustomizationAdministration ComponentEnd-userComponentGenerationComponentNatQuery
24NatQuery Administration Overview 2) Capture Natural Data Definition Modules (DDMs)JCLDDMsDDMs are obtained:Automatically via a User RequestManually via an Import functionAdministration ComponentEnd-userComponentGenerationComponentNatQuery
26NatQuery Generation Overview The End-User Component allows for the easy entry of extract specifications.The Generation Component converts an extract specification to an optimized Natural program.Workstation EnvironmentAdministration ComponentEnd-userComponentGenerationComponentNaturalProgramEnd-userRequiredFiles / FieldsUser-specifiedSelect LogicOptionalVariablesDesiredTarget
27Server Integration Overview File Transfer Protocol (FTP) IntegrationDirect FTP into batchIndirect FTP into batchJust FTP (Manual Execution)Manual IntegrationIND$FILE (IBM)Manual FTPOther Methods...Mainframe EnvironmentWorkstation EnvironmentAutomatedCommunicationIs Achieved UsingStandard FTPNatQuery
28Direct FTP Integration Overview User Builds Query SpecificationUser Submits RequestProgram is generated and imbedded into JCL / ScriptProgram and JCL / Script is FTP’ed to the ServerLocal Log File is writtenServer EnvironmentJES (MVS), POWER (VSE)Request ExecutesExecution updates Remote Log, creates OutputBatchUserRequestUser Retrieves OutputOutput automatically FTP’ed to workstationUserRequestNaturalProgramWorkstation EnvironmentFTPNatQueryRequestOutputRemoteLogRequestOutputLocalLog
29In-Direct FTP Integration Overview User Builds Query SpecificationServer EnvironmentUser Submits RequestProgram is generated and imbedded into JCL / ScriptProgram and JCL / Script is FTP’ed to the ServerLocal Log File is writtenBatch NaturalServer Submits RequestExecution updates remote log, creates outputNatQueryFTP ServerUser Retrieves OutputOutput automatically FTP’ed to workstationUserRequestUserRequestUserRequestNaturalProgramWorkstation EnvironmentFTPNatQueryRequestOutputUserLogRequestOutputLocalLog
30Just FTP Integration Overview User Builds Query SpecificationUser Submits RequestProgram is generated and imbedded into JCL / ScriptProgram and JCL / Script is FTP’ed to the ServerLocal Log File is writtenServer EnvironmentBatch NaturalUser Manually Submits RequestExecution updates remote log, creates outputUser Retrieves OutputOutput automatically FTP’ed to workstationUserRequestUserRequestUserRequestNaturalProgramWorkstation EnvironmentFTPNatQueryRequestOutputUserLogRequestOutputLocalLog
31NatQuery Integration to ETL Tools NatQuery Generates Descriptions of Extract LayoutDSX GenerationDataStage Exchange file (DataStage proprietary format)Allows for Full Integration of Predict MetadataCFD GenerationCOBOL File Definition (in copybook format)DATAWorkstation EnvironmentDSXFilesCFDSGTNatQueryETL toolImportGenerationComponent
32NatQuery FeaturesOptimized Access to Source File(s), Based on User-Entered Selection CriteriaAutomatic determination and generation of best access methodDescriptors, Super-Descriptors, Sub-Descriptors…Read Logical, Read Physical, Find, GetZero coding effort requiredFull Sensitivity of SuppressionAutmomatic Support for Multi-Fetch (Pre-Fetch)Automated Integration to Server (FTP)Download DDMs (direct support for SYSTRANS utility)Automatic Generation of required Descriptor StatisticsAutomated Upload, Execute and Download of ResultsAutomated Extraction of PREDICT Meta Data
33NatQuery Features - continued... Full handling of All ADABAS Field TypesDate, TimeStamp, Packed, Integer, Binary, ...Data Conversion at Extract LevelConversion of ADABAS formats to ASCII equivalentsFull Ability to handle Sign Byte for numeric fieldsFull handling of All ADABAS “recurring” dataMUs, PEs, MUs in PEsAdministratively defined defaults and maximumsUser over-ride of defaults (within allowed maximums)Ability to “Flatten” or “Concatenate” at field level
34NatQuery Features - continued... “Flattening” a recurring data structureFirst-Name Last-Name Address-Line CityOneSourceRecordSuite 100Two occurrences454 South Main StreetNatQuery built IndexTwoExtractRecordsChris Bradley South Main Street Northfield ...Chris Bradley Suite Northfield ...
35NatQuery Features - continued... “Concatenating” a recurring data structureFirst-Name Last-Name Address-Line CityOneSourceRecordSuite 100Two occurrences454 South Main StreetOneExtractRecordChris Bradley South Main Street Suite Northfield ...
36NatQuery Features - continued... Direct Integration to ETL ToolsAutomatic generation of interface filesDSX and / or CFD filesAutomatic Linking of up to 5 “Primary” filesNo User Knowledge RequiredSupport for Automatic “Look-up” filesCode-to-Text conversions, Administratively definedLook-ups are “transparent” to the user
37NatQuery Features - continued... Support “Full Extract” or “Intersection Set”Multi-File Flexible ExtractionAbilitity to Define VariablesRedefines, Constants, Expressions, Compress and Dynamic (date-based and user input)Direct Support for ADABAS utilitiesADACMP, ADASEL, ADACDCUse of “Batch” Provides Controllable Extraction
38NatQuery Features - continued... Full Manipulation of Query SpecificationsSave, Save As, DeleteQuery specifications stored with a long and short query descriptionUser Specifiable Data Extract Limits“Test” ExtractsAdministratively Contolled User Data LimitsAbility to disallow Read PhysicalAbility to set Record LimitsOne Tool - Dual UseEnd-User ExtractionData Warehouse Extraction
39NatQuery Features - continued... Minimal Mainframe FootprintJust Natural, JCL and mainframe disk spaceExisting ADABAS utilities (optional)Extraction Capability to Any Data Source Natural Can Talk toADABAS, VSAM, DB2...Integration to Desktop ToolsMS AccessMS ExcelXML (with optional XSL)Tab Delimited or User-Specified Delimiters
40NatQuery Features - continued... Integration to PREDICTAll Field-Level PREDICT Metadata is made available within NatQueryAdministrator can create their ownAbility to Trace I/O generationreview I/O generation processFull Support of Native SecurityNatural SecurityADABAS SecurityAutomatic Update AbilityAllows for centralized roll-out of new versions...
41Change Data Capture Solution NatCDCtheChange Data Capture SolutionforADABASNatWorks, Inc.
43The Source of ADABAS Changes ADABAS Protection Log (PLOG)ADABAS’ transaction recovery mechanism100% data integrity, all transactions recordedTrue “Point-in-Time” snapshot of ADABASChanged Data Available w/o ADABAS AccessMainframe EnvironmentPLOG 2AdabasPLOG 1
44ADABAS CDC (Change Data Capture) Issues in accessing PLOG:PLOG contains all transactions against all FilesPLOG is in compressed formatThe same compression used by ADABASPLOG data is stored in Variable-Length recordsDifferent from file to file and within same filePLOG data requires “conversion”EBCDIC to ASCII, date / time formats
45ADABAS CDC ADABAS utilities solve most PLOG issues: ADASEL utility (ADABAS 6):“Splits” PLOG transactions into separate filesOne File for each requested ADABAS FileHandles Expanded FilesDecompresses PLOG recordsADACDC utility (ADABAS 7):everything ADASEL doesdirect delivery of Delta changes
46ADABAS CDC NatCDC converts variable-length to fixed-length NatCDC / NatQuery solves remaining issues:NatCDC converts variable-length to fixed-lengthUser-Specified number of MU and PE occurrencesPLOG Header convertedIBM STCK time, Expanded File ISNs are normalizedNatQuery generates all required objectsAll Parameters and Programs
47NatCDC Base Components ADASEL / ADACDC utility supplied with ADABASRawPLOGDataDiskTapeNatCDCSORTADASELSystem Sort ProgramNatCDC utility (Single Optimized Natural Program)NatQueryServer EnvironmentWorkstation Environment
48NatCDC Processing Overview One JCL Stream for each file(or expanded file chain)One JCL Streamfor each 20 filesRawPLOGDataDiskTapeRawFILEDataDiskTapeFixedLengthDataDiskTapeDWHCDCDataDiskTapeGeneratedProcessingProgramNatCDCSORTADASELGeneratedParametersGeneratedParametersGeneratedParametersNatQueryDDMServer EnvironmentWorkstation Environment
49NatCDC Features 100% Data Integrity All transactions handled, even BackoutsSimple Mainframe InstallationOne Single Natural object program (NatCDC)One Natural Program for each fileOne JCL Stream per fileThe Fastest and Most Trusted DecompressionSAG knows their own compression the bestPerformance is CriticalCDC is a frequently occurring activity
50NatCDC Features Variable-Length to Fixed Length conversion Final Layout is User-DeterminedRecurring Fields Padded or TruncatedException Reports Produced AutomaticallyFull Handling of all ADABAS data structuresMUs, PEs, and MUs in PEsAutomatic format translations:EBCDIC to ASCIIDate and TimestampSign handling of all numeric-based fields
51NatCDC Features Physical to Logical ISN conversion All Parameters Full Support for Expanded FilesPhysical to Logical ISN conversionFull Generation of all Required ObjectsAll ParametersAll ProgramsAll JCLData is supplied with Standard HeaderTransaction Date, Time, ISN, Seq#, ...
52NatCDC Features Field Selection Options C* values available as data Fields may be selectively omittedIntegration to ETL ToolsCFD generation“DSX” generation (Ascential DataStage)Time Differential Handling OptionsAutomaticManual
53NatCDC Features Data Output Options Logical Last Single Record flagged as Store, Update or DeleteLogical First and LastOne or two records flagged as Before or AfterAllExtensive Reporting OptionsOccurrence Exception ProcessingStore, Update, Delete CountsTotal Before and After images...