Introduction to Heterogeneous Data Replication Spring COMMON 1999 Richard Sinn IBM Santa Teresa Lab

Agenda n What is and Why Replication ? n Introduction to IBM Data Propagator Relational (DPropR, DPR) –DB2 to DB2 family including DB2/400 n Introduction to DataJoiner –DB2 to non-DB2 –non-DB2 to DB2 –non-DB2 to non-DB2 n Putting Them Together

What is Replication ? n Moving Data from one place to another n Replication vs Copying n From Source to Target n Maintain changes automatically

Why Replication ? n Data is everywhere n Use Information Intelligently and improve decision making n Stay competitive with both Data access and Data movement n Speedup application deployment n Reengineer business processes n Increase online throughput n Improve system availability n Support audit requirements n Support Data Warehousing

IBM Data Propagator Solution n Multiple platforms (AS/400, NT, AIX, etc) n Supported by IBM n Simple Administration n Proven Technology n Speed Implementation with IBM services organization n Built on standard SQL

DPropR Components Capture Apply Admin Tools Source Server Target Server Control Server (Flexible location) Windows NT, Windows 95, OS/2 Platform Web Browser

Replication Admin Tools n Define replication sources –Also called Registration –This is what Capture works with logically n Defines subscriptions –This is what Apply works with logically n Create replication control tables –AS/400 DPR install program will create all the necessary replication control tables

Admin Tool: DJRA

Define Registration n Register a source table as a replication source n Process data using vertical fragmentation –Replicate only selected columns n Select before and after image of a column

Registration Capture Registered Source Tables At the Source Server

Registration Using DRJA

Define Subscription n Subscribe to registered table n Defines source and target relationship –For example, many target tables can subscribe to one source table n Horizontal fragmentation –Columns can be further sub-selected

Subscription Apply Target Tables Subscription Definitions Registered Source Tables

Subscription Using DJRA

Replication Flow n Source data changes are captured by Capture component according to registration n Change Data are then put into Staging Tables n According to subscription, Apply component apply changes to target(s) using the data in Staging Tables

The Capture Component Journals, Logs Capture Registered Source Tables Staging Tables Unit of Work Table At the Source Server Register Table

Journal/Log based Capture n Capture reads the DB2/400 Journal –Non-AS/400 DB2 will be reading Log n Inserts rows into the change data table –One of the staging tables n Time stamps or relative byte address is used as a log sequence number n Prunes the change data table n Communicates with Apply via ASN/IBMSNAP_PRUNCNTL table

The Apply Component Apply Register Table Staging Tables Unit of Work Table Target Tables Subscription Definitions

Capture and Apply

DPropR Functions n Subscription Set n Time based Replication n Event based Replication n Multiple Target Types n Blocking Factor n Update Anywhere n Programming Capability –SQL Statement / Call Procedure –ASNDONE –ASNLOAD n Support Transaction Consistency

Different Target Types n User copy tables –Represent source data at a particular point in time n Point-in-time tables –Represent source data at a particular point in time plus some overhead columns n Replica tables –Updates to a Replica table are automatically applied to the original source table of the Replica, Auto conflicts detection

Consistent Change Data n Consistent Change Data (CCD) n Maintain complete histories of data changes (Audit) n Condense hot spot updates before transmitting data n Loading dock n Used by NotesPump or other applications

Agenda n What is and Why Replication ? n Introduction to IBM Data Propagator Relational (DPropR, DPR) –DB2 to DB2 family including DB2/400

What if I have non-DB2 data? What Happens to my: n Oracle Database n Microsoft SQL Server n Sybase SQL Server n Informix Database n And all the applications that built with them ?

Introduction to DataJoiner n Transparent Access to Heterogeneous Data Sources n Transparent DDL Support n Heterogeneous Database Joins n Global Optimization n Two-Phase Commit Support n Integrated Data Replication n Spatial Data Management/Access

DataJoiner Overview Setup n NT Server (on IPCS) n Install DJ n Install other DBs protocols –SQL*Net for Oracle –Informix-net for Informix –ODBC drivers or Sybase open client for MS SQL Server or Sybase SQL Server Basic Data Access n On NT with DJ: –Create database –Connect to DB –Create Server Mapping (Create Nickname) n Setup client to go to NT with DJ n All access will map to real target DB

DataJoiner Overview NT Server IPCS DataJoiner DB Protocol Client System Non-DB2 (e.g. Oracle) Non-DB2 (e.g. Oracle)

Total Access with DataJoiner

DB2/Non-DB2 Replication AS/400 with DPR/400 (Capture/400 Running; Could be any DB2 family) AS/400 with DPR/400 (Capture/400 Running; Could be any DB2 family) Source Server DataJoiner on NT IPCS Apply Running DataJoiner on NT IPCS Apply Running Sybase, Informix, Oracle Microsoft SQLServer, DB2 Family Sybase, Informix, Oracle Microsoft SQLServer, DB2 Family Target Server Admin (DJRA)

Non-DB2/Non-DB2 Replication DataJoiner on NT IPCS Apply Running DataJoiner on NT IPCS Apply Running Sybase, Informix, Oracle Microsoft SQLServer, DB2 Family Sybase, Informix, Oracle Microsoft SQLServer, DB2 Family Target Server Capture Triggers Using CCD Sybase, Microsoft Oracle, Informix SQL Anywhere Source Server Admin (DJRA)

Trigger Based Capture I n Trigger: Wake up and do actions based on event (such as insert, delete, update) n Insert, Update, and Delete triggers created on the source table n Committed changes inserted into the CCD table n Capture both before and after image of the column

Trigger Based Capture II n Prunes CCD tables via trigger on IBMSNAP_PRUNCNTL table –Clean up CCD n Update Synchpoint via Trigger on IBMSNAP_REG_SYNCH Table –Communication with Capture

Trigger Based Capture III n Log sequence numbers generated dependent on OEMs n Pre-existing Triggers –Run-time error –Check by DJRA and inform users –Depends on OEM DB

Heterogeneous Replication

n DJRA tolerates the pre-existing target table n DataJoiner does the data type mapping n Read only copies (one direction with User Copy, PIT) n No Support for Replica yet n Target table created in data source n Nickname for target table created in DJ Database

Putting Them Together Replication n DPR on AS/400 and other platforms n Integrated support with DPNR, Data Refresher, NotesPump n Expert Info and Support from IBM n Proven Technology Heterogeneous Data Access n DataJoiner on IPCS n Server consolidation advantages with IPCS

Putting Them Together n Data from Any to Any n Replicate Changes n Time & Event based replication n Update Anywhere n Data Transformation for Datawarehouse n Build Powerful Applications

Reference Information n n n n

Trademark Information n AIX, AS/400, DataHub, DataJoiner, DataPropagator, DataRefresher, DB2, DProp, DRDA, IBM, IMS, MVS, OS/2, OS/400, RS/6000, VM/ESA and VSE/ESA are trademarks or registered trademarks of the IBM Corpopration in the United States or other countries or both. n Microsoft, Windows, and the Windows 95 and the Windows NT logos are trademarks or registered trademarks of Microsoft Corporation. n UNIX is a registered trademark in the United States and other countries licensed exclusively through X/Open Company Limited. n Lotus Notes is a trademark of the Lotus Development Corporation. n Other company, product, and service names may be trademarks or service marks of others.