Presentation on theme: "ADABAS Extraction & Change Data Capture NatWorks, Inc. Chris S. Bradley Presented by."— Presentation transcript:
ADABAS Extraction & Change Data Capture NatWorks, Inc. Chris S. Bradley Presented by
2 TSI 4/05 The Question… Where Do You Need YOUR ADABAS Data Today?
3 TSI 4/05 The Problems… l End-User Extraction l Data Warehouse Extraction l Web Publishing / Data Exchange Adabas DATA End-User Extraction DATA Data Warehouse Extraction Web Publishing / Data Exchange
4 TSI 4/05 The Message… A Software AG customer who has ADABASNATURAL ADABAS & NATURAL already has the best tools to handleall requirements for Data Extraction & Change Data Capture
5 TSI 4/05 ADABAS - Two Major Issues l How to Access ADABAS l ADABAS data structures
6 TSI 4/05 Accessing ADABAS Data ADABAS ADABAS Direct Calls The Only Way to communicate directly to ADABAS is through Direct Calls
7 TSI 4/05 Accessing ADABAS Data continued ADABAS ADABAS Direct Calls Option #1 Imbedded Direct Calls Option #2 SQL / ODBC Option #3 Natural Option #4 ADABAS Utilities Option #5 Vendor Utilities
8 TSI 4/05 ADABAS Data Structures l All standard data formats are supported: ä alphanumeric, binary, fixed point, floating point, packed decimal, unpacked decimal,... l Supports two basic field types ä Elementary fields ä “recurring fields” (MUs) l Consecutive fields may be grouped ä A group may be repeated - Periodic Groups (PEs) ä Periodic Groups may contain one or more Multiple- Value Fields
9 TSI 4/05 ADABAS Data Structures continued l ADABAS has unique data structures: ä Multi-Valued Fields - “MUs” (array structure) ä Periodic-Groups - “PEs” (table structure) ä MUs in PEs (multi-dimensional structure) l ADABAS has some “difficult” data types ä IBM STCK-based Date and TimeStamp fields l EBCDIC to ASCII ä Packed Fields ä Sign Byte Handling
10 TSI 4/05 ADABAS Extraction Facts l FACT #1 NATURAL was developed by Software AG specifically to access ADABAS l FACT #2 NATURAL represents the most widely used AND best understood solution for accessing ADABAS l FACT #3 What NATURAL should not be used to do, ADABAS Utilities handle (with support from NATURAL) l FACT #4 Extraction / CDC should be done in BATCH
11 TSI 4/05 ADABAS Extraction - Conclusions l Natural allows flexible ADABAS Access l Natural easily handles all ADABAS data structures l Natural easily handles all ADABAS data types l Natural will always work with ADABAS Naturalfor maximum flexibility Natural for maximum flexibility ADABAS Utilities for maximum performance
12 TSI 4/05 Solving ADABAS Data Access ADABAS ADABAS Direct Calls Option #3 Natural Option #4 ADABAS Utilities 4 Embrace What Exists (ROI) 4 Best Understood Solution 4 High Performing Solution
13 TSI 4/05 The Real Problem NO GENERATION Generation is needed for: Required Natural Objects Required ADABAS Parameters Required JCL / Script Processes
a Data Extraction Solution for ADABAS
15 TSI 4/05 Leveraging Proven Technology ADABAS™ Utilities Predict™ Natural™ EXCEL ACCESS Desktop Tools Desktop Tools XML/XSL Load Ready Data Load Ready Data DB2 ™ Load Ready Data Load Ready Data RDBMS of Choice RDBMS of Choice XML Tamino™ Target of Choice Target of Choice ETL Tool of Choice ETL Tool of Choice ADASAV Backup ADABAS Extraction
16 TSI 4/05 The NatQuery Extraction Solution NatQuery works by acting as an on-demand Natural Programmer. From a graphical user interface a User is enabled to: l Create Query Specifications l Generate Natural Data Extraction programs from these Specifications l Submit Generated Extract programs for execution l Remotely monitor execution status l Download Extracted Data l Load extracted data into MS Access, MS Excel or convert data into XML (with optional XSL)
17 TSI 4/05 The NatQuery Extraction Solution Administration Component NatQuery End-user Component Generation Component Internally, NatQuery can be thought of as having three components: l Administrative Component l End-User Component l Generation Component
18 TSI 4/05 The NatQuery Extraction Solution Administration Component NatQuery End-user Component Generation Component The Administrative Component is used by an Administrator to capture information that is specific to the platform, environment, and ADABAS data files that NatQuery will be used against. The Administration Component provides NatQuery with the ability to capture application-specific intelligence.
19 TSI 4/05 The NatQuery Extraction Solution Administration Component End-user Component Generation Component End-user Required Files / Fields Desired Target User-specified Select Logic Optional Variables Natural Program l The End-User Component ä allows for the easy entry of extract specifications. l The Generation component ä generates a Natural program from a specification.
20 TSI 4/05 The NatQuery Extraction Solution Server Environment Workstation Environment DATA Natural Program Natural Program DATA User Extract Specification AccessExcelXML NatQuery DATA Other Environment Adabas l Extraction With Natural...
21 TSI 4/05 Adabas The NatQuery Extraction Solution Server Environment Workstation Environment DATA Natural Program Natural Program DATA User Extract Specification NatQuery DATA Other Environment ADACMP Parameters ADACMP Utility l Extraction With ADABAS Utilities And Natural
22 TSI 4/05 The NatQuery Extraction Solution l Better control of requests ä Request execution can be easily scheduled ä Impact on online production applications are controlled l More efficient execution environment over “online” ä Significantly Less overhead l NatQuery handles Job Control Language (JCL) ä Template approach provides easy set-up / maintenance ä Dynamic substitution makes templates executable Requests Execute in “batch”
23 TSI 4/05 NatQuery Administration Overview Administration Component NatQuery End-user Component Generation Component JCL 1) Provide Site-Specific Job Control Language (JCL) JCL Templates Provide: Integration to Natural / ADABAS Dynamic Process Customization
24 TSI 4/05 NatQuery Administration Overview 2) Capture Natural Data Definition Modules (DDMs) JCL DDMs DDMs are obtained: Automatically via a User Request Manually via an Import function Administration Component NatQuery End-user Component Generation Component
25 TSI 4/05 NatQuery Administration Overview 3) Capture Expanded DDM Info JCL Descriptor Statistics File Relationships Occurrence Information File I/O Parameters Predict Metadata DDMs Administration Component NatQuery End-user Component Generation Component Administration Information provides NatQuery with application-specific generation intelligence
26 TSI 4/05 NatQuery Generation Overview Administration Component End-user Component Generation Component End-user Required Files / Fields Desired Target User-specified Select Logic Optional Variables Natural Program l The End-User Component allows for the easy entry of extract specifications. l The Generation Component converts an extract specification to an optimized Natural program. Workstation Environment
27 TSI 4/05 Server Integration Overview l File Transfer Protocol (FTP) Integration ä Direct FTP into batch ä Indirect FTP into batch ä Just FTP (Manual Execution) l Manual Integration ä IND$FILE (IBM) ä Manual FTP ä Other Methods... Workstation Environment NatQuery Mainframe Environment Automated Communication Is Achieved Using Standard FTP
28 TSI 4/05 FTP Direct FTP Integration Overview Server Environment l User Submits Request ä Program is generated and imbedded into JCL / Script ä Program and JCL / Script is FTP’ed to the Server ä Local Log File is written Workstation Environment NatQuery JES (MVS), POWER (VSE) Batch Request Output Remote Log l Request Executes ä Execution updates Remote Log, creates Output l User Retrieves Output ä Output automatically FTP’ed to workstation User Request User Request Natural Program Request Output Local Log l User Builds Query l User Builds Query Specification
29 TSI 4/05 FTP In-Direct FTP Integration Overview Server Environment Workstation Environment Batch Natural NatQuery FTP Server User Request Output User Log l Server Submits Request ä Execution updates remote log, creates output l User Retrieves Output ä Output automatically FTP’ed to workstation User Request User Request Natural Program Request Output NatQuery Local Log l User Builds Query Specification l User Submits Request ä Program is generated and imbedded into JCL / Script ä Program and JCL / Script is FTP’ed to the Server ä Local Log File is written
30 TSI 4/05 FTP Just FTP Integration Overview Server Environment Workstation Environment Batch Natural User Request Output User Log l User Manually Submits Request ä Execution updates remote log, creates output l User Retrieves Output ä Output automatically FTP’ed to workstation User Request User Request Natural Program Request Output NatQuery Local Log l User Builds Query Specification l User Submits Request ä Program is generated and imbedded into JCL / Script ä Program and JCL / Script is FTP’ed to the Server ä Local Log File is written
31 TSI 4/05 NatQuery Integration to ETL Tools Workstation Environment l DSX Generation ä DataStage Exchange file (DataStage proprietary format) ä Allows for Full Integration of Predict Metadata l CFD Generation ä COBOL File Definition (in copybook format) Generation Component NatQuery ETL tool NatQuery Generates Descriptions of Extract Layout Import DATA DSX Files CFD Files SGT Files
32 TSI 4/05 NatQuery Features l Optimized Access to Source File(s), Based on User- Entered Selection Criteria ä Automatic determination and generation of best access method –Descriptors, Super-Descriptors, Sub-Descriptors… –Read Logical, Read Physical, Find, Get –Zero coding effort required –Full Sensitivity of Suppression ä Autmomatic Support for Multi-Fetch (Pre-Fetch) l Automated Integration to Server (FTP) ä Download DDMs (direct support for SYSTRANS utility) ä Automatic Generation of required Descriptor Statistics ä Automated Upload, Execute and Download of Results ä Automated Extraction of PREDICT Meta Data
33 TSI 4/05 NatQuery Features - continued... l Full handling of All ADABAS Field Types ä Date, TimeStamp, Packed, Integer, Binary,... l Data Conversion at Extract Level ä Conversion of ADABAS formats to ASCII equivalents ä Full Ability to handle Sign Byte for numeric fields l Full handling of All ADABAS “recurring” data ä MUs, PEs, MUs in PEs ä Administratively defined defaults and maximums ä User over-ride of defaults (within allowed maximums) ä Ability to “Flatten” or “Concatenate” at field level
34 TSI 4/05 NatQuery Features - continued... First-Name Last-Name Address-Line City... Suite South Main Street “Flattening” a recurring data structure One Source Record Two Extract Records Chris Bradley 2 Suite 100 Northfield... Chris Bradley South Main Street Northfield... NatQuery built Index Two occurrences
35 TSI 4/05 NatQuery Features - continued... First-Name Last-Name Address-Line City... “Concatenating” a recurring data structure One Source Record One Extract Record Chris Bradley 454 South Main Street Suite 100 Northfield... Two occurrences Suite South Main Street
36 TSI 4/05 NatQuery Features - continued... l Direct Integration to ETL Tools ä Automatic generation of interface files –DSX and / or CFD files l Automatic Linking of up to 5 “Primary” files ä No User Knowledge Required l Support for Automatic “Look-up” files ä Code-to-Text conversions, Administratively defined ä Look-ups are “transparent” to the user
37 TSI 4/05 NatQuery Features - continued... l Support “Full Extract” or “Intersection Set” ä Multi-File Flexible Extraction l Abilitity to Define Variables ä Redefines, Constants, Expressions, Compress and Dynamic (date-based and user input) l Direct Support for ADABAS utilities ä ADACMP, ADASEL, ADACDC l Use of “Batch” Provides Controllable Extraction
38 TSI 4/05 NatQuery Features - continued... l Full Manipulation of Query Specifications ä Save, Save As, Delete ä Query specifications stored with a long and short query description l User Specifiable Data Extract Limits ä “Test” Extracts l Administratively Contolled User Data Limits ä Ability to disallow Read Physical ä Ability to set Record Limits l One Tool - Dual Use ä End-User Extraction ä Data Warehouse Extraction
39 TSI 4/05 NatQuery Features - continued... l Minimal Mainframe Footprint ä Just Natural, JCL and mainframe disk space ä Existing ADABAS utilities (optional) l Extraction Capability to Any Data Source Natural Can Talk to ä ADABAS, VSAM, DB2... l Integration to Desktop Tools ä MS Access ä MS Excel ä XML (with optional XSL) ä Tab Delimited or User-Specified Delimiters
40 TSI 4/05 NatQuery Features - continued... l Integration to PREDICT ä All Field-Level PREDICT Metadata is made available within NatQuery ä Administrator can create their own l Ability to Trace I/O generation ä review I/O generation process l Full Support of Native Security ä Natural Security ä ADABAS Security l Automatic Update Ability ä Allows for centralized roll-out of new versions l...
the Change Data Capture Solution forADABAS NatWorks, Inc.
42 TSI 4/05 Leveraging Proven Technology ADABAS™ Utilities Natural™ EXCEL ACCESS Desktop Tools Desktop Tools XML/XSL Load Ready Data Load Ready Data Load Ready Data Load Ready Data RDBMS of Choice RDBMS of Choice ETL Tool of Choice ETL Tool of Choice ADABAS PLOG ADABAS Change Data Capture & Transaction Auditing
43 TSI 4/05 The Source of ADABAS Changes l ADABAS’ transaction recovery mechanism ä 100% data integrity, all transactions recorded l True “Point-in-Time” snapshot of ADABAS l Changed Data Available w/o ADABAS Access PLOG 2 ADABAS Protection Log (PLOG) Adabas Mainframe Environment PLOG 1
44 TSI 4/05 ADABAS CDC (Change Data Capture) l PLOG contains all transactions against all Files l PLOG is in compressed format ä The same compression used by ADABAS l PLOG data is stored in Variable-Length records ä Different from file to file and within same file l PLOG data requires “conversion” ä EBCDIC to ASCII, date / time formats Issues in accessing PLOG:
45 TSI 4/05 l ADASEL utility (ADABAS 6): ä “Splits” PLOG transactions into separate files –One File for each requested ADABAS File –Handles Expanded Files ä Decompresses PLOG records l ADACDC utility (ADABAS 7): ä everything ADASEL does ä direct delivery of Delta changes ADABAS CDC ADABAS utilities solve most PLOG issues:
46 TSI 4/05 l NatCDC converts variable-length to fixed-length ä User-Specified number of MU and PE occurrences ä PLOG Header converted –IBM STCK time, Expanded File ISNs are normalized l NatQuery generates all required objects ä All Parameters and Programs ADABAS CDC NatCDC / NatQuery solves remaining issues:
47 TSI 4/05 NatCDC Base Components Workstation Environment Server Environment Raw PLOG Data Disk Tape ADASEL NatQuery NatCDC SORT ADASEL / ADACDC utility supplied with ADABAS NatCDC utility (Single Optimized Natural Program) System Sort Program
48 TSI 4/05 NatCDC Processing Overview Workstation Environment Server Environment Raw PLOG Data Disk Tape Raw FILE Data Disk Tape Fixed Length Data Disk Tape DWH CDC Data Disk Tape Generated Parameters ADASEL NatQuery NatCDC Generated Parameters Generated Processing Program SORT One JCL Stream for each file (or expanded file chain) One JCL Stream for each 20 files DDM Generated Parameters
49 TSI 4/05 NatCDC Features l 100% Data Integrity ä All transactions handled, even Backouts l Simple Mainframe Installation ä One Single Natural object program (NatCDC) ä One Natural Program for each file ä One JCL Stream per file l The Fastest and Most Trusted Decompression ä SAG knows their own compression the best ä Performance is Critical –CDC is a frequently occurring activity
50 TSI 4/05 NatCDC Features l Variable-Length to Fixed Length conversion ä Final Layout is User-Determined ä Recurring Fields Padded or Truncated –Exception Reports Produced Automatically l Full Handling of all ADABAS data structures ä MUs, PEs, and MUs in PEs ä Automatic format translations: –EBCDIC to ASCII –Date and Timestamp –Sign handling of all numeric-based fields
51 TSI 4/05 NatCDC Features l Full Support for Expanded Files ä Physical to Logical ISN conversion l Full Generation of all Required Objects ä All Parameters ä All Programs ä All JCL l Data is supplied with Standard Header ä Transaction Date, Time, ISN, Seq#,...
52 TSI 4/05 NatCDC Features l Field Selection Options ä C* values available as data ä Fields may be selectively omitted l Integration to ETL Tools ä CFD generation ä “DSX” generation (Ascential DataStage) l Time Differential Handling Options ä Automatic ä Manual
53 TSI 4/05 NatCDC Features l Data Output Options ä Logical Last –Single Record flagged as Store, Update or Delete ä Logical First and Last –One or two records flagged as Before or After ä All l Extensive Reporting Options ä Occurrence Exception Processing ä Store, Update, Delete Counts ä Total Before and After images ä...
54 TSI 4/05 NatCDC Benefits l Cost Effective ä Uses vendor supplied utility ä Uses Natural l One Tool - Dual Use ä Data Warehouse Change Data Capture (CDC) ä End-User Extraction l...