Oracle Streams--Simplifying Information Sharing in Oracle10 g Patricia McElroy Product Manager Oracle Corporation Session id: 40208

Patricia McElroy Product Manager, Distributed Systems Oracle Corporation

Oracle Streams-- Simplifying Information Sharing in Oracle10 g

Agenda  Oracle Streams Overview  Streams Architecture  Hints and Tips

Oracle Streams  Simple solution for information sharing  Provides – uniquely flexible replication – message queuing – data warehouse loading – event management and notification

Streams Basic Elements  Capture  Staging  Consumption (apply) Consumption StagingCapture

Multi-Database Streams  A stream can contain multiple elements from multiple databases  Events flow between staging areas Consumption Staging Capture Consumption Staging Capture

Capture  Streams captures events – Implicitly: log-based capture of DML and DDL – Explicitly: Direct enqueue of user messages  Captured events are published in the staging area  SQL and messaging APIs in multiple languages – JDBC, JMS, PL/SQL, C, SOAP Capture

Log-Based Change Capture  Low overhead, low latency change capture – Changes to the database are written to the online redo log – Oracle9i Streams can extract changes from the log as it is written (hot mining) – Changes are formatted as a Logical Change Record (LCR), a representation of the change Capture

Logical Change Record (LCR)  Database change = LCR –DML  Object name, owner, Type of DML, SCN  Row change = LCR  OLD, NEW values –DDL  Object name, owner, Type of DDL, SCN  DDL text –LOB  Multiple LCRs per LOB  Piecewise chunks

Staging  Streams publishes captured events into a staging area – Implemented as a queue – Supports for new type “any” datatype allows a single staging area to hold any type of data – All events, LCRs and user-messages, can be staged in the same queue – Messages remain in staging area until consumed by all subscribers Staging

Staging Area Propagation  Other staging areas can subscribe to events – in same database – in a remote database  Events can be routed through a series of staging areas Propagation Staging Staging

Transformations  Transformations can be performed – as events enter the staging area – as events leave the staging area – as events propagate between staging areas  Transformation examples – change format, data type, column name, table name Staging

Consumption  Staged events are consumed by subscribers – Implicitly: Apply Process  Default Apply  User-Defined Apply – Explicitly: Application dequeue via open interfaces  JMS, C, C++, PLSQL, SOAP (XML/HTTP) Consumption

Default Apply  The default apply engine will directly apply the DML or DDL represented in the LCR – apply to local Oracle table – apply via DB Link to non-Oracle table  Automatic conflict detection with optional resolution – unresolved conflicts placed in exception queue  Parallel apply maximizes concurrency Consumption

User-defined Apply  User-written custom apply procedures  Written in PL/SQL, Java, C, C++  Uses: – full control over apply – normalizing or denormalizing data – populating related fields or tables Consumption

Rule-based Configuration  Consumers subscribe to published events  Content-based subscription  Rule is expressed as SQL WHERE clause dbms_rule_adm.create_rule( rule_name=>‘scott.rule1', condition=>':dml.get_object_owner() = ''SCOTT'' AND :dml.get_object_name()=''EMP''');  Rule sets for simplicity  Rule sets govern capture,staging, and apply – Negative, Inclusion  Dynamic rule maintenance

Replication Rules  Rule-based – DML, DDL – content – True/False  Granularity – Table – Subset of Table – Schema – Database – Tablespace  Tailored Replication API DBMS_STREAMS_ADM – ADD_TABLE_RULES – ADD_SCHEMA_RULES – ADD_GLOBAL_RULES – ADD_SUBSET_RULES – MAINTAIN_TABLESPACES – ADD_MESSAGE_RULE

NY(master) London(subset) Milan(subset)Paris(subset) Directed Networks  Propagation independent of Apply  Rules-based subscription determine if event is locally applied – London applies UK only  WAN Friendly – Send once, fan out – NY-->London, London-->Milan, London-->Paris INSERT … VALUES (‘EUROPE’,’ ITALY’) ITALY EUROPE FRANCE

Automatic Conflict Detection  Automatic conflict detection with user- selectable conflict resolution routines – latest timestamp, earliest timestamp, maximum or minimum value, overwrite, discard – User-definable resolution routines  Conflict detection compares current row values at receiving site with “old” values of changed row from the originating site – if match, “new” values are applied to row – if not, conflict resolution method is used, if supplied – if still unresolved, place transaction in exception queue  Ability to disable conflict detection by column or table

Streams Capture and Apply EMP Update EMP set job=‘coding’ where empid=510; Redo Log Capture EMP empid|job | | sales |… 510 | coding|... Apply Queue LCRs Queue LCRs Propagation ACK

Capture Background process, CPnn Invokes Logminer Staged in buffer queue (SGA) Restartable Automatic Flow Control Redo Log Queue LCRs Capture Ack

Capture Restart Every database change is in redo Ordered by SCN Last enqueued SCN capture restart Last acknowledged SCN instance restart Redo Log Queue LCRs Capture Ack

Downstream Capture  Zero impact at source site  Changes are captured by another database – Same Platform – Archive logs  Log transport services/ RFS to move logs

IMPLICIT APPLY Propagation Job Queue processes Stream LCRs Maintain status information 9i:Re-propagate on instance crash 10g: Reuse spilled messages during recovery In memory Queue/ LCRs IMPLICIT CAPTUR E In memory Queue/ LCRs Acknowledgements LCRs

Apply Reader Coordinator Server ….. Background processes Assembles transactions Exactly once apply Dependency computation Conflict detection EMP empid|job | | sales |… 510 | coding|... Apply Queue LCRs Ack

Transformation Example Redo Log Queue LCRs EMP Update EMP set job=‘coding’ where empid=510; Capture Queue LCRs Person Apply Propagation empid|job | | sales |… 510 |coding|... Emp->Person T 1-1 Mapping LCR based ACK

Customized Apply Example Queue LCRs EMP Update EMP set state=‘CA’ where empid=100; Redo Log Capture Queue LCRs Emp Apply Propagation empid|state|.. 95 | GA |… 105 | LA |... Do not apply Changes to EMP for state=‘CA’; Maintain AUDIT table of each LCR U Full Control Transaction Based Audit 100 | CA |… Bit Bucket ACK

Data Subsetting Queue LCRs EMP Update EMP set job=‘CA’ where empid=100; Redo Log Capture Queue LCRs EMP Apply Propagation empid|state| | CA |… 205 | CA |... Queue LCRs EMP Apply empid|state| | LA |… 105 | LA |... LA CA Insert empid 100 Delete empid 100 ACK

Streams and RAC  Oracle9i – No hot mining – All Streams activities done on a single instance (owning instance) – Streams restart after failure not automatic  Oracle10i – Hot mining – Primary and secondary owning instance – Failover of Streams processes automatic

Messaging Enhancements  Streams encompasses AQ  Simplified messaging APIs  Batch enqueue/dequeue  Criteria-based purge  Gateway to Tibco

Heterogeneous Support  Oracle to non-Oracle Apply via gateway – Apply process on Oracle node applies change  Non-Oracle to Oracle change capture supported via explicit enqueue of LCRs  Message Gateways – MQ Series LCR or user message Gateway MessageGateway MQ Series Sybase

Heterogeneous Example Queue LCRs EMP Update EMP set state=‘CA’ where empid=100; Redo Log Capture Queue LCRs EMP Apply Propagation empid|state| | CA |… 105 | LA |... ACK Apply2 Sybase EMP TG4 Sybs

US Gateway Sybase DR UK JP Report Customer Example  Maintain application tables in 3 regional centers – autonomous – low overhead – low latency  Provide Special Services – Reporting Database – Disaster Recovery – Legacy Application hosted on Sybase

Streams Replication Features  Log-based Change Capture  Customizable Apply Engine  Directed Networks  Schema Evolution  Transformations  Heterogeneous Support  Explicit Enqueue/Dequeue

Streams Benefits for Replication  Flexible Configurations  Reduced Network Traffic  Low Overhead  No Downtime Requirements

Software Updates  patch, minimum  Streams Wizard patch  Watch OTN Streams website for custom software –

Configuration Tips  Use separate queues – Capture – Apply  Increase shared_pool_size and max_sga_size init.ora parameters  Relocate Streams data dictionary tables from SYSTEM – Set tablespace before configuring streams – Including apply-only sites

Rule Tips  Spelling counts!  Eliminate duplicate rules in a rule set.  Make sure that the source_database_name is correctly specified. – At source site: SELECT global_name FROM GLOBAL_NAME;  Ensure that rules do not allow objects with invalid data types – When using GLOBAL rules, modify existing rule to eliminate the Streams Administrator schema and other schemas with queues, ADTs or tables with invalid data types.

Rule Management Tips  Having no rule set defined is NOT the same as an empty rule set.  Use DBMS_STREAMS_ADM package to create replication rules  Use same package to create or remove rules – DBMS_STREAMS_ADM – DBMS_RULE_ADM

Troubleshooting Tips  Message Number columns = source DML,DDL SCN – CAPTURE_MESSAGE_NUMBER, ENQUEUE_MESSAGE_NUMBER – DEQUEUE_MESSAGE_NUMBER, APPLIED_MESSAGE_NUMBER  ORA no instantiation scn set – Set Instantiation SCN – Exp/Imp with appropriate clauses – DBMS_APPLY_ADM.SET_TABLE_INSTANTIATION_SCN  ORA insufficient privilege – Explicitly grant privileges to apply user  MISSING Streams data dictionary information ! – DBMS_CAPTURE_ADM.PREPARE_TABLE_INSTANTIATION

Troubleshooting #2  Determine the scope of the problem : – Apply not working for this site only  Check DBA_APPLY_ERROR for errors ­Procedure to display error details in doc  Check trace files  Check rules at this site  Still not working? … – Apply not working for any site  Check rules for both propagation and capture  Check propagation job is working ­Job queue processes ­DB Link  Make sure COMMIT was performed for transaction

Operational Tips  Replicating DDL? – Ensure that privileges are explicitly granted to the apply user. – Modify any manual hotbackup scripts to set an apply tag before starting the backup. – Configure Instantiation SCN at next higher level  Removing Archive log files from disk? – Ensure that scripts do not automatically remove files that may be needed for capture to restart.  Implement “heartbeat” table – Update periodically.

Next Steps….  Recommended sessions – Data Integration with Oracle Streamsata Wed 4:30 Room 130 Integration Using Oracle St  Recommended demos and/or hands-on labs – Hands-On Lab: Integrate your Information.  See Your Business in Our Software – Visit the Streams booth in the DEMOgrounds for more information.  Relevant web sites to visit for more information –

A Q & Q U E S T I O N S A N S W E R S