ADABAS Extraction & Change Data Capture Presented by Chris S. Bradley NatWorks, Inc.

Where Do You Need Data Today? The Question… Where Do You Need YOUR ADABAS Data Today?

Web The Problems… End-User Extraction Data Warehouse Extraction Web Publishing / Data Exchange Adabas DATA DATA Data Warehouse Extraction Web End-User Extraction Web Publishing / Data Exchange

A Software AG customer who has ADABAS & NATURAL The Message… A Software AG customer who has ADABAS & NATURAL already has the best tools to handle all requirements for Data Extraction & Change Data Capture

ADABAS - Two Major Issues How to Access ADABAS ADABAS data structures

The Only Way to communicate directly to ADABAS is through Accessing ADABAS Data ADABAS ADABAS Direct Calls The Only Way to communicate directly to ADABAS is through Direct Calls

Accessing ADABAS Data continued Option #5 Vendor Utilities Option #4 ADABAS Utilities ADABAS Direct Calls Option #1 Imbedded Direct Calls Option #2 SQL / ODBC Option #3 Natural

ADABAS Data Structures All standard data formats are supported: alphanumeric, binary, fixed point, floating point, packed decimal, unpacked decimal, ... Supports two basic field types Elementary fields “recurring fields” (MUs) Consecutive fields may be grouped A group may be repeated - Periodic Groups (PEs) Periodic Groups may contain one or more Multiple-Value Fields

ADABAS Data Structures continued ADABAS has unique data structures: Multi-Valued Fields - “MUs” (array structure) Periodic-Groups - “PEs” (table structure) MUs in PEs (multi-dimensional structure) ADABAS has some “difficult” data types IBM STCK-based Date and TimeStamp fields EBCDIC to ASCII Packed Fields Sign Byte Handling

ADABAS Extraction Facts NATURAL was developed by Software AG specifically to access ADABAS FACT #2 NATURAL represents the most widely used AND best understood solution for accessing ADABAS FACT #3 What NATURAL should not be used to do, ADABAS Utilities handle (with support from NATURAL) FACT #4 Extraction / CDC should be done in BATCH

ADABAS Extraction - Conclusions Natural allows flexible ADABAS Access Natural easily handles all ADABAS data structures Natural easily handles all ADABAS data types Natural will always work with ADABAS Natural for maximum flexibility ADABAS Utilities for maximum performance

Solving ADABAS Data Access Option #4 ADABAS Utilities ADABAS Direct Calls Option #3 Natural Embrace What Exists (ROI) Best Understood Solution High Performing Solution

NO GENERATION The Real Problem Generation is needed for: Required Natural Objects Required ADABAS Parameters Required JCL / Script Processes

Data Extraction Solution NatQuery a Data Extraction Solution for ADABAS

Leveraging Proven Technology ADABAS Extraction Web ADABAS™ ADASAV Backup DB2™ Predict™ ADABAS™ Utilities Tamino™ Target of Choice Natural™ XML NatQuery Load Ready Data RDBMS of Choice ETL Tool of Choice EXCEL ACCESS Desktop Tools XML/XSL Load Ready Data

The NatQuery Extraction Solution NatQuery works by acting as an on-demand Natural Programmer. From a graphical user interface a User is enabled to: Create Query Specifications Generate Natural Data Extraction programs from these Specifications Submit Generated Extract programs for execution Remotely monitor execution status Download Extracted Data Load extracted data into MS Access, MS Excel or convert data into XML (with optional XSL)

The NatQuery Extraction Solution Internally, NatQuery can be thought of as having three components: Administrative Component End-User Component Generation Component NatQuery Administration Component End-user Component Generation Component

The NatQuery Extraction Solution The Administrative Component is used by an Administrator to capture information that is specific to the platform, environment, and ADABAS data files that NatQuery will be used against. NatQuery Administration Component End-user Component Generation Component The Administration Component provides NatQuery with the ability to capture application-specific intelligence.

The NatQuery Extraction Solution The End-User Component allows for the easy entry of extract specifications. The Generation component generates a Natural program from a specification. Administration Component End-user Component Generation Component Natural Program End-user Required Files / Fields User-specified Select Logic Optional Variables Desired Target

The NatQuery Extraction Solution Server Environment Extraction With Natural Adabas Natural Natural Program DATA DATA Other Environment NatQuery Natural Program User Extract Specification DATA Access Excel XML ... Workstation Environment

The NatQuery Extraction Solution Server Environment Extraction With ADABAS Utilities And Natural Adabas Natural ADACMP Utility Natural Program DATA DATA Other Environment ADACMP Parameters NatQuery Natural Program User Extract Specification DATA Workstation Environment

The NatQuery Extraction Solution Requests Execute in “batch” Better control of requests Request execution can be easily scheduled Impact on online production applications are controlled More efficient execution environment over “online” Significantly Less overhead NatQuery handles Job Control Language (JCL) Template approach provides easy set-up / maintenance Dynamic substitution makes templates executable

NatQuery Administration Overview 1) Provide Site-Specific Job Control Language (JCL) JCL JCL Templates Provide: Integration to Natural / ADABAS Dynamic Process Customization Administration Component End-user Component Generation Component NatQuery

NatQuery Administration Overview 2) Capture Natural Data Definition Modules (DDMs) JCL DDMs DDMs are obtained: Automatically via a User Request Manually via an Import function Administration Component End-user Component Generation Component NatQuery

NatQuery Administration Overview 3) Capture Expanded DDM Info JCL DDMs Descriptor Statistics File Relationships Occurrence Information File I/O Parameters Predict Metadata Administration Component Administration Information provides NatQuery with application-specific generation intelligence End-user Component Generation Component NatQuery

NatQuery Generation Overview The End-User Component allows for the easy entry of extract specifications. The Generation Component converts an extract specification to an optimized Natural program. Workstation Environment Administration Component End-user Component Generation Component Natural Program End-user Required Files / Fields User-specified Select Logic Optional Variables Desired Target

Server Integration Overview File Transfer Protocol (FTP) Integration Direct FTP into batch Indirect FTP into batch Just FTP (Manual Execution) Manual Integration IND$FILE (IBM) Manual FTP Other Methods... Mainframe Environment Workstation Environment Automated Communication Is Achieved Using Standard FTP NatQuery

Direct FTP Integration Overview User Builds Query Specification User Submits Request Program is generated and imbedded into JCL / Script Program and JCL / Script is FTP’ed to the Server Local Log File is written Server Environment JES (MVS), POWER (VSE) Request Executes Execution updates Remote Log, creates Output Batch User Request User Retrieves Output Output automatically FTP’ed to workstation User Request Natural Program Workstation Environment FTP NatQuery Request Output Remote Log Request Output Local Log

In-Direct FTP Integration Overview User Builds Query Specification Server Environment User Submits Request Program is generated and imbedded into JCL / Script Program and JCL / Script is FTP’ed to the Server Local Log File is written Batch Natural Server Submits Request Execution updates remote log, creates output NatQuery FTP Server User Retrieves Output Output automatically FTP’ed to workstation User Request User Request User Request Natural Program Workstation Environment FTP NatQuery Request Output User Log Request Output Local Log

Just FTP Integration Overview User Builds Query Specification User Submits Request Program is generated and imbedded into JCL / Script Program and JCL / Script is FTP’ed to the Server Local Log File is written Server Environment Batch Natural User Manually Submits Request Execution updates remote log, creates output User Retrieves Output Output automatically FTP’ed to workstation User Request User Request User Request Natural Program Workstation Environment FTP NatQuery Request Output User Log Request Output Local Log

NatQuery Integration to ETL Tools NatQuery Generates Descriptions of Extract Layout DSX Generation DataStage Exchange file (DataStage proprietary format) Allows for Full Integration of Predict Metadata CFD Generation COBOL File Definition (in copybook format) DATA Workstation Environment DSX Files CFD SGT NatQuery ETL tool Import Generation Component

NatQuery Features Optimized Access to Source File(s), Based on User-Entered Selection Criteria Automatic determination and generation of best access method Descriptors, Super-Descriptors, Sub-Descriptors… Read Logical, Read Physical, Find, Get Zero coding effort required Full Sensitivity of Suppression Autmomatic Support for Multi-Fetch (Pre-Fetch) Automated Integration to Server (FTP) Download DDMs (direct support for SYSTRANS utility) Automatic Generation of required Descriptor Statistics Automated Upload, Execute and Download of Results Automated Extraction of PREDICT Meta Data

NatQuery Features - continued... Full handling of All ADABAS Field Types Date, TimeStamp, Packed, Integer, Binary, ... Data Conversion at Extract Level Conversion of ADABAS formats to ASCII equivalents Full Ability to handle Sign Byte for numeric fields Full handling of All ADABAS “recurring” data MUs, PEs, MUs in PEs Administratively defined defaults and maximums User over-ride of defaults (within allowed maximums) Ability to “Flatten” or “Concatenate” at field level

NatQuery Features - continued... “Flattening” a recurring data structure First-Name Last-Name Address-Line City ... One Source Record Suite 100 Two occurrences 454 South Main Street NatQuery built Index Two Extract Records Chris Bradley 1 454 South Main Street Northfield ... Chris Bradley 2 Suite 100 Northfield ...

NatQuery Features - continued... “Concatenating” a recurring data structure First-Name Last-Name Address-Line City ... One Source Record Suite 100 Two occurrences 454 South Main Street One Extract Record Chris Bradley 454 South Main Street Suite 100 Northfield ...

NatQuery Features - continued... Direct Integration to ETL Tools Automatic generation of interface files DSX and / or CFD files Automatic Linking of up to 5 “Primary” files No User Knowledge Required Support for Automatic “Look-up” files Code-to-Text conversions, Administratively defined Look-ups are “transparent” to the user

NatQuery Features - continued... Support “Full Extract” or “Intersection Set” Multi-File Flexible Extraction Abilitity to Define Variables Redefines, Constants, Expressions, Compress and Dynamic (date-based and user input) Direct Support for ADABAS utilities ADACMP, ADASEL, ADACDC Use of “Batch” Provides Controllable Extraction

NatQuery Features - continued... Full Manipulation of Query Specifications Save, Save As, Delete Query specifications stored with a long and short query description User Specifiable Data Extract Limits “Test” Extracts Administratively Contolled User Data Limits Ability to disallow Read Physical Ability to set Record Limits One Tool - Dual Use End-User Extraction Data Warehouse Extraction

NatQuery Features - continued... Minimal Mainframe Footprint Just Natural, JCL and mainframe disk space Existing ADABAS utilities (optional) Extraction Capability to Any Data Source Natural Can Talk to ADABAS, VSAM, DB2... Integration to Desktop Tools MS Access MS Excel XML (with optional XSL) Tab Delimited or User-Specified Delimiters

NatQuery Features - continued... Integration to PREDICT All Field-Level PREDICT Metadata is made available within NatQuery Administrator can create their own Ability to Trace I/O generation review I/O generation process Full Support of Native Security Natural Security ADABAS Security Automatic Update Ability Allows for centralized roll-out of new versions ...

Change Data Capture Solution NatCDC the Change Data Capture Solution for ADABAS NatWorks, Inc.

Leveraging Proven Technology ADABAS Change Data Capture & Transaction Auditing ADABAS™ ADABAS PLOG Natural™ ADABAS™ Utilities NatQuery NatCDC Load Ready Data RDBMS of Choice ETL Tool of Choice EXCEL ACCESS Desktop Tools XML/XSL Load Ready Data

The Source of ADABAS Changes ADABAS Protection Log (PLOG) ADABAS’ transaction recovery mechanism 100% data integrity, all transactions recorded True “Point-in-Time” snapshot of ADABAS Changed Data Available w/o ADABAS Access Mainframe Environment PLOG 2 Adabas PLOG 1

ADABAS CDC (Change Data Capture) Issues in accessing PLOG: PLOG contains all transactions against all Files PLOG is in compressed format The same compression used by ADABAS PLOG data is stored in Variable-Length records Different from file to file and within same file PLOG data requires “conversion” EBCDIC to ASCII, date / time formats

ADABAS CDC ADABAS utilities solve most PLOG issues: ADASEL utility (ADABAS 6): “Splits” PLOG transactions into separate files One File for each requested ADABAS File Handles Expanded Files Decompresses PLOG records ADACDC utility (ADABAS 7): everything ADASEL does direct delivery of Delta changes

ADABAS CDC NatCDC converts variable-length to fixed-length NatCDC / NatQuery solves remaining issues: NatCDC converts variable-length to fixed-length User-Specified number of MU and PE occurrences PLOG Header converted IBM STCK time, Expanded File ISNs are normalized NatQuery generates all required objects All Parameters and Programs

NatCDC Base Components ADASEL / ADACDC utility supplied with ADABAS Raw PLOG Data Disk Tape NatCDC SORT ADASEL System Sort Program NatCDC utility (Single Optimized Natural Program) NatQuery Server Environment Workstation Environment

NatCDC Processing Overview One JCL Stream for each file (or expanded file chain) One JCL Stream for each 20 files Raw PLOG Data Disk Tape Raw FILE Data Disk Tape Fixed Length Data Disk Tape DWH CDC Data Disk Tape Generated Processing Program NatCDC SORT ADASEL Generated Parameters Generated Parameters Generated Parameters NatQuery DDM Server Environment Workstation Environment

NatCDC Features 100% Data Integrity All transactions handled, even Backouts Simple Mainframe Installation One Single Natural object program (NatCDC) One Natural Program for each file One JCL Stream per file The Fastest and Most Trusted Decompression SAG knows their own compression the best Performance is Critical CDC is a frequently occurring activity

NatCDC Features Variable-Length to Fixed Length conversion Final Layout is User-Determined Recurring Fields Padded or Truncated Exception Reports Produced Automatically Full Handling of all ADABAS data structures MUs, PEs, and MUs in PEs Automatic format translations: EBCDIC to ASCII Date and Timestamp Sign handling of all numeric-based fields

NatCDC Features Physical to Logical ISN conversion All Parameters Full Support for Expanded Files Physical to Logical ISN conversion Full Generation of all Required Objects All Parameters All Programs All JCL Data is supplied with Standard Header Transaction Date, Time, ISN, Seq#, ...

NatCDC Features Field Selection Options C* values available as data Fields may be selectively omitted Integration to ETL Tools CFD generation “DSX” generation (Ascential DataStage) Time Differential Handling Options Automatic Manual

NatCDC Features Data Output Options Logical Last Single Record flagged as Store, Update or Delete Logical First and Last One or two records flagged as Before or After All Extensive Reporting Options Occurrence Exception Processing Store, Update, Delete Counts Total Before and After images ...

NatCDC Benefits Uses vendor supplied utility Uses Natural Cost Effective Uses vendor supplied utility Uses Natural One Tool - Dual Use Data Warehouse Change Data Capture (CDC) End-User Extraction ...

www.treehouse.com | tsi@treehouse.com NatQuery and NatCDC Simple ideas with enormous potential. www.treehouse.com | tsi@treehouse.com