1. DB2 Information Integrator 8
DB2 Information Integrator 8.2 Q Replication and Event Publishing - the elevator tale …
Presentation given by: Pav Kumar-Chatterjee,
IBM Product Introduction Center, Hursley
(IDUG – 2 December 2004)
Welcome!

2. DB2 Information Integrator 8.2: Q Replication - Overview

3. Current Architecture for SQL Replication
Admin
DB2
IMS
DB2
Log based
Staging tables
Sybase
CD1 CD
Sybase
Oracle
SQL Server
Informix
Teradata
Nicknames
Control
Oracle
Control
SQL Server
Capture
Apply
Informix
Federation
Engine
Trigger based
DB2 Data Propagator
DB2 II Replication Edition
IMS Data Propagator

4. Why Create Another Replication Architecture?
Performance: Combine high throughput with low latency.
Capability: Significantly improve multi-directional replication support.
New function: Event publishing, table difference utility.
Manageability: Reduce the number of replication objects to be defined and managed, ease the definition process with new Replication Center wizards .

5. Q Replication Components
Admin
Utilities
Control
Control
Source
Target
Capture
Apply
Federation
Engine
Log based
WebSphere MQ
Substantially modified
New
New and modified

6. DB2 Information Integrator 8.2: Event Publishing - Overview

7. What is DB2 II Event Publishing?
Capture changed-data from DB2, IMS and CICS/VSAM
Correlate by transactions within single database
Extract to consistent and documented XML format
“Publish” to WebSphere MQ queue
Received & Processed by any MQ “listener”
Pick up messages
Take action
Two Event Publisher infrastructures:
DB2
IMS and CICS/VSAM
VSAM
IMS
DB2 UDB
for z/OS
DB2 Information Integrator
Event Publishers for z/OS
Capture, Externalize (XML) and Deliver to MQ

8. Event Publishing Components
Admin
Utilities
Control
Source
XML
Capture
Federation
Engine
Log based
WebSphere MQ
Substantially modified
New
New and modified

9. Event Publishing - Publication Options
Format
Only data from committed transactions is published
Data is self describing with XML tags
Row based = one row per message
Transaction based = one transaction per message
Row Content
Subset by column
Subset by predicate
Changed column values only or all column values
New data values only or include old values

10. DB2 Information Integrator 8
DB2 Information Integrator 8.2: Replication and Event Publishing Products

11. Replication and Event Publishing Products: z/OS
DB2 DataPropagator for z/OS
DB2 UDB sources and targets (DB2 for z/OS V7 and V8)
SQL architecture
DB2 Information
Integrator Replication
for z/OS
DB2 UDB sources and targets (DB2 for z/OS V7 and V8)
SQL and Queue architectures (includes DB2 Event Publisher)
DB2 DataPropagator
DB2 Information
Integrator Event Publisher for z/OS
Event publishing to message queues
Available for DB2, IMS, or VSAM
IMS
DataPropagator
for z/OS
IMS sources and DB2 UDB for z/OS target

12. IMS & VSAM Implementation on z/OS
DataMapper
Federation Server
z/OS
OS/390
Metadata
Catalog
Changed-data
Capture Agents
IMS log data
CICS log data
MQ listener
WebSphere
MQ for z/OS
MQ Client
Leverages DB2 II Classic Federation mappings and Server infrastructure
Install changed data capture agents for IMS and/or VSAM. Identify data to be monitored in the metadata.
Install/Configure WebSphere MQ
Deliver data to MQ listener
or WMB !

13. DB2 Implementation on z/OSQ
Capture
z/OS & Distributed
Metadata
Replication
Center
DB2 log
MQ listener
WebSphere MQ
MQ Client
Each message represents a DB2 transaction
XML format consistent with IMS & VSAM
Replication Center for metadata management
“Front-end” of Q Replication solution
Install/Configure WebSphere MQ
No Q Capture on i-Series
Deliver data to MQ listener
or WMB !

14. Replication and Event Publishing Products: Distributed Platforms and iSeries
DB2 LUW and Informix IDS sources and targets.
SQL architecture.
DB2 LUW
DB2 DataPropagator
DB2 Information
Integrator Replication
Edition
Multi-vendor sources and targets (SQL architecture)
DB2 LUW sources and targets (Queue architecture) – note that Websphere MQ is bundled with this product.
DB2 DataPropagator
DB2 Information
Integrator Event Publisher Edition
DB2 LUW sources (Queue architecture) – note that Websphere MQ is bundled with this product.
DB2
DataPropagator for iSeries
DB2 iSeries sources and targets.

15. DB2 Implementation z/OS and LUW: Process Flow
SOURCE
SOURCE2
SOURCE1
User Application
METADATA
DB2 Log
Q Capture
XML
User Application
WMB
ADMINISTRATION
DB2 MQ Listener
User Stored Procedure
Replication
Center
Replication
Monitor
TARGET

16. Co-existing DB2 Event Publishing with DB2 Replications (Q and SQL)
CD2
CD1
Staging Tables
Log Reader 1
Capture Schema
„CAP1“
TARGET1
DB2 Log
SQL Apply
Q Sub
TARGET2
Q Apply
Q Capture Schema
„CAP2“
Log Reader 2
SOURCE3
SOURCE2
SOURCE1
Event Pub
User Application
SQL Replication and Q Replication can co-exist
Managed at source by using multiple capture schemas
One Q Capture can handle both Publications and Subscriptions

17. DB2 Information Integrator 8.2: Q Replication - Terms

18. Terminology SQL Replication – Replication via “staging tables”.
Q Replication – Replication via WebSphere MQ.
Q Capture, Q Apply – Q Replication engine components.
Q Subscription – The process of defining Q objects for data replication.
Q Publication – The process of defining Q objects for XML publication.
Event publishing – Publishing changed rows or data-events as XML messages via WebSphere MQ.
Do I need a Websphere Application Server to run Q Replication?
Do I need an DB2 II license to run Q Replication?
What happens to DataPropagator and will it go away?

19. Sample Q Replication Scenarios
CD1
SOURCE
REPLICA
Subsets
Transformations
Conflict Detection/Resolution
Updateable Predicates
Updateable Primary Keys
High-Availability Applications
REPLICA
REPLICA
COPY
REPLICA
Data Distribution (1:m)
Peer to Peer
COPY
CD1
PRIMARY
CD1
SECONDARY
CD1
SOURCE
CD1
SOURCE
CD1
SOURCE
Data Consolidation (n:1)
Bidirectional

20. Requires at least 4 queues (minimum):
Q Replication queues
Requires at least 4 queues (minimum):
Adminq
Adminq
Local queue
Remote queue
Q Apply
Recvq
Sendq
Q Capture
DB2
Target
DB2
Source
Remote queue
Local queue
Spillq
Restartq
Local queue
Local queue
1. Recvq for QApply to receive the transaction and informational messages from Q Capture
2. Spillq, dynamic queue for QApply to hold the transaction messages as the target table is being loaded
1. Adminq for Q Capture to receive control messages from QApply or subscribing app
2. Restartq holds the Q Capture position in the DB2 log

21. What exactly is a Replication Queue Map? (1)
Queue Manager
Queue Manager
Q Capture
Q Apply
Send Queue (remote)
Receive Queue (local)
Admin Queue (local)
Admin Queue (remote)

22. What exactly is a Replication Queue Map? (2)
Queue Manager
Queue Manager
Q Capture
Q Apply
Send Queue (remote)
Receive Queue (local)
Admin Queue (remote)
Admin Queue (local)
Q Apply
Receive Queue (local)
Admin Queue (remote)
Send Queue (remote)
Queue Manager

23. Subscriptions and Publications
Subscriptions – used for replication
Point to point, a 1-3 fan-out requires 3 subscriptions.
Unidirectional, bidirectional, or peer to peer.
Unidirectional subs offer more options than bidirectional, bidirectional offers more options than peer to peer (as Apply complexity grows, less options are offered).
Publications – used for event publishing
More options available here, because characteristics of downstream application are unknown.
There is no overlap or interaction between publications and subscriptions.
They use separate queues and are independent objects.

24. Conflict Detection and Resolution
“one size does not fit all” – choices provided, care must be taken in selecting the options most appropriate for a given application!
VERSION based:
Based upon time zone adjusted timestamps, most recent timestamp “wins”
Users must add two extra columns to support it (timestamp, tie-breaker) – GUI driven
Extra columns maintained by triggers (insert/update)
VALUE based:
Conflict level options offered:
Check all columns on update - requires transmission of all old/new values
Check only changed columns on update - allows for column merge
Check only key columns
Resolution choices offered: Force or Ignore
Force Action - requires transmission of all new values
force convergence on conflicts
log the conflict
Ignore Action
Force/Ignore used together in a pair provides "very good" convergence
- Example: HA environment (Force on the copy, and Ignore on the Server)

25. Q Subscription Process Flow
SOURCE
METADATA
SOURCE2
SOURCE1
Q Apply
Browser
METADATA
Apply Agent
DB2 Log
Q Capture
Apply Agent
Apply Agent
ADMINISTRATION
TGT3
TGT1
TGT2
Replication
Center
Replication
Monitor
TARGET

26. DB2 Information Integrator 8.2: Q Replication - Utilities

27. Replication Utilities (Q and SQL)
asntdiff
Builds a table of differences between the source and target tables
Differences can be used to effect changes to source or target
Can also be used just to verify target on a periodic basis
asntrepair
Repairs target based on differences found by tdiff
The Alert Monitor
Provides automated monitoring of your replication environment
Alerts you to error and other conditions
The Replication Analyzer
Generates report about the state of your replication
environment
Technical support tool, but used by many customers
All utilities work with both Q and SQL replication

28. Utilities: Alert Monitor
Source server
Target server
Log
DProp & Q Apply
Target
DProp & Q Capture
Status:
System health
Event:
Errors/Warnings
Thresholds exceeded:
Throughput
Latency
Queue depth
Replication
Monitor
Poll capture and apply at user defined interval
Operates on user defined alerts and thresholds
Notification via or pager

29. DB2 Information Integrator 8.2: Q Replication - Administration

30. Replication Administration
Replication Center GUI
Launchpads, Wizards, Online Help
Definitions, Operations, Monitoring
Command Line Interface
Scripts or interactive mode
Example:
Java API’s
Typically used when replication is embedded
C:\asnclp
REPL > CREATE QSUB USING REPLQMAP ...
REPL > CREATE SUBSCRIPTION SET SETNAME ...
REPL > CREATE MEMBER IN SETNAME ...

31. Replication Launchpad (Q-Replication)

32. Create Q Subscriptions

33. ASNCLP for Command Line Operation and Scripting
Command line processor to define Replication Scenarios
Calls same Java API‘s as the Replication Center
Interactive and Script Mode supported
What is is ?
Example
Interactive Mode
Script Mode
C:\asnclp REPL > CREATE QSUB USING REPLQMAP ... (Q Replication commands)
REPL > CREATE SUBSCRIPTION SET SETNAME ... (SQL Replication commands)
REPL > CREATE MEMBER IN SETNAME ...
C:\asnclp -f replscript.asn

34. Supported Platforms DB2 II Replication Edition for Multiplatform 8.2
AIX
AIX 5.1 or later
DB2 ESE for AIX Version 8.2
WebSphere MQ for Linux for Intel and Linux for zSeries Version 5.3
Linux
Linux for intel distribution that supports DB2 ESE 8.2 and WebSphere MQ 5.3
DB2 ESE for Linux Version 8.2
Windows
DB2 II Replication for z/OS 8.2
z/OS
z/OS Version 1.2 or later
XML Tool kit for z/OS and OS/390 Version 1.4
DB2 UDB V7 & V8 for z/OS and OS/390
WebSphere MQ for z/OS Version 5.3
DB2 II Q Replication does not support iSeries

35. DB2 Information Integrator: Websphere MQ - Overview

36. Topics Quick overview of Websphere MQ objects.
Websphere MQ queues required for Q Replication.

37. Quick overview of MQ objects
Queue Manager
Manages queues of messages for application programs
Provides application programming interface
Coordinates updates to databases and queues using *two-phase commit
Sends one message to more than one destination using distribution lists, etc
Queue Manger objects
Queues
store **messages sent by programs
different types of queues, each with a specific purpose
Message channels
connects two queue managers
these channels are unidirectional
*Note: Q Replication does not require two-phase commit
** Note: Q Replication uses persistent messages

38. MQ queues
Local queue
is a real queue, owned by the qmgr to which the application program is connected.
Remote queue
is a queue owned by a different qmgr. A remote queue definition is the local definition of a remote queue. It is associated with a transmission queue.
Transmission queue (xmitq)
is a local queue with a special purpose. Transmission queues used as staging area when sending messages to queues that are owned by different queue managers.
Dynamic queue
local queue created “on the fly”, when the application needs it.
Model queue
is not a real queue. It is a collection of attributes that are used when a dynamic queue is created.

39. Q Replication queues - Remote setup Q Capture /Q Apply do not share Queue Manager
Adminq
Adminq
1.Local queue
2.Remote queue
Q Apply
Recvq
Q Capture
Target
Sendq
Source
1.Remote queue
3.Local queue
Spillq
Restartq
4.Local queue
2.Local queue
3. Recvq for QApply to receive the transaction and control messages from Q Capture
1. Adminq for Q Capture to receive control messages from QApply or subscribing app
4. Spillq, dynamic queue for QApply to hold the transaction messages as the target table is being loaded
2. Restartq holds the Q Capture position in the DB2 log
1. Sendq for Q Capture to put messages..Remote def
for the local recvq on the QApply side
2. Adminq for Q Apply to put messages..Remote def
for the local adminq on the QCapture side

40. Examples of MQ commands on LUW
Create queue manager
crtmqm –u <dead.letter.queue.name> CSQ
Start queue manager
strmqm CSQ
Start the MQ interactive command line
runmqsc CSQ
define qlocal (‘IBMQREP.ASN.ADMINQ’)
define qlocal (‘IBMQREP.ASN.RESTARTQ’)
define qlocal (‘IBMQREP.ASN.SEND_RECVQ’) MAXDEPTH(640000) MAXMSGL( )
define qmodel (IBMQREP.SPILL.MODELQ) DEFSOPT(shared) MAXDEPTH(500000) MSGDLVSQ(fifo) DEFTYPE(permdyn)
For z/OS, the queues would have to be defined with SHARE REPLACE
Use the RC or asnclp to create control tables and define Replication Queue Map.
Next page for the remote setup.

41. Remote setup System 1 (QMGR= CSQ1) System 2 (QMGR = CSQ2)
DEFINE QREMOTE(‘IBMQREP.ASN.SENDQ’)+
RNAME(‘IBMQREP.ASN.RECVQ’)+
RQNAME(‘CSQ2’)+XMITQ(‘CSQ2’)
DEFINE QLOCAL(‘IBMQREP.ASN.RECVQ’)
DEFINE QLOCAL(‘CSQ2’) USAGE(xmitq)
DEFINE CHANNEL(‘CSQ1TOCSQ2’)+
CHLTYPE(SDR) TRTYPE(TCP)+
CONNAME(IP address of CSQ2 port#)+
XMITQ(‘CSQ2’)+DISCNT(0)
CHLTYPE(RCVR) TRTYPE(TCP)
DEFINE QLOCAL(‘IBMQREP.ASN.ADMINQ’)
DEFINE QREMOTE(‘IBMQREP.ASN.ADMINQ’)+
RNAME(‘IBMQREP.ASN.ADMINQ’)+
RQMNAME(‘CSQ1’)+XMITQ(‘CSQ1’)
DEFINE QLOCAL(‘CSQ1’)+USAGE(xmitq)
DEFINE CHANNEL (‘CSQ2TOCSQ1’)+
DEFINE CHANNEL(‘CSQ2TOCSQ1’)+
CONNAME(IP address of CSQ1 port#)+
XMITQ(‘CSQ1’)+DISCNT(0)
DEFINE QLOCAL(‘IBMQREP.ASN.RESTARTQ’)
DEFINE QMODEL('IBMQREP.SPILL.MODELQ') DEFSOPT(SHARED) +
MAXDEPTH(500000) MSGDLVSQ(FIFO) DEFTYPE(PERMDYN)

42. MQ / Q Replication interaction
Q Replication will not work with the distribution lists.
Q Replication can work in the MQ cluster environment.
Cluster configuration is not a Q Replication requirement
A receive cluster queue must be defined exactly once in a cluster, so not a good fit where the cluster setup is for load balancing
Make sure the DISCINT is large enough if no channel initiator is in place, and with channel initiator the user can tune the timeout attribute.
DISCINT – is the channel attribute, to control how long the channel should remain running, when there are no messages to send
Q Replication can detect missing messages.
MQSeries highly recommend to have Dead-letter-queue defined.
We do not provide DLQ handler, however QApply knows when the message is missing from the queue, as Q Replication messages are “dense”.
MQ Series Primer – a good easy reading….highly recommended.

43. DB2 Information Integrator: Q Replication The Good the bad and the ugly (configuration wise!)

44. The Good (1)
In order to pick one over the other, will depend on the workload, need for throughput, latency requirement, etc.

45. The Good (2)
Represents single Q Capture instance sending data over to multiple target databases. There is one Q Apply instance per target.

46. The Good (3)
Represents the functionality of Q Capture, where it can publish DB2 transactions in the XML format.
The messages in the XML and Compact format cannot be sent over the same data queue.

47. or potentially to provide higher Q Capture throughput.
The Good (4)
One could require multiple instances of Q Capture, depending on the latency requirements for various applications
or potentially to provide higher Q Capture throughput.

48. where three Q Apply instances are running.
The Good (5)
Three Q Capture instances running on the single source and sending data over to a single target,
where three Q Apply instances are running.
This configuration could be used to parallelize traffic in the sysplex environment.

49. The Good (6)
This figure represents a consolidation scenario, of data from multiple sources going to the same target database.
Recommendation is to use this configuration , where each browser is dealing with a set of partitioned data.
For example, all state data getting consolidated in the federal database.

50. Implications: dense numbering, which adminq to send the message?
The Bad (1)
Cannot map 2 sendq’s to same recvq. Checked during setup, but not if aliases…
Implications: dense numbering, which adminq to send the message?

51. RI’s defined between TI,T2,T3,T4. Cannot detect dependencies….
The downright ugly!
RI’s defined between TI,T2,T3,T4. Cannot detect dependencies….

52. Questions?

53. And so finally …

54. Q Replication delivers low latency, high throughput replication.
Summary
Q Replication delivers low latency, high throughput replication.
Event Publishing provides the blueprint for publishing data from many sources.