2. Oracle Data Integrator – Technical Deck
Mark Pare
Sr. Sales Consultant
Oracle Higher Education

3. <Insert Picture Here>
Agenda
<Insert Picture Here>
4 Key Differentiators
Architecture
6 Steps to Production
ODI or ESB?
Popular Usage Scenarios

4. Oracle Data Integrator 4 Key Differentiators

5. Why Data Integration? NEED… HAVE…
Information How and Where you Want It
Business Intelligence
Corporate Performance Management
Business Activity Monitoring
Business Process Management
Data Integration
Migration
Data Warehousing
Master Data Management
- - -
Data Synchronization
Federation
Real Time Messaging
HAVE…
Data in Disparate Sources
ERP
---
CRM
- - -
Legacy
Best-of-breed Applications

6. Challenges & Emerging Solutions In Data Integration
Increasing data volumes; decreasing batch windows
Non-integrated integration
Complexity, manual effort of conventional ETL design
Lack of knowledge capture
Shift from E-T-L to E-LT
Convergence of integration solutions
Shift from custom coding to declarative design
Shift to pattern-driven development

7. Oracle Data Integrator
Data Movement and Transformation from Multiple Sources to Heterogeneous Targets
Performance: Heterogeneous “E-LT”
Flexibility: Active Integration Platform
Productivity: Declarative Design
Hot-Pluggable: Knowledge Modules
BENEFITS
KEY DIFFERENTIATED FEATURES

8. Differentiator: E-LT Architecture High Performance1
Differentiator: E-LT Architecture High Performance
Conventional ETL Architecture
Extract
Load
Transform
Transform in Separate ETL Server
Proprietary Engine
Poor Performance
High Costs
IBM & Informatica’s approach
Transform in Existing RDBMS
Leverage Resources
Efficient
High Performance
Benefits
Optimal Performance & Scalability
Easier to Manage & Lower Cost
Next Generation Architecture
“E-LT”
Load
Extract
Transform

9. Oracle Data Integrator Data-oriented Integration2
Differentiator: Active Integration Batch, Event-based, and Service-oriented Integration
Evolve from Batch to Near Real-time Warehousing on Common Platform
Unify the Silos of Data Integration
Data Integrity on the Fly
Services Plug into Oracle SOA Suite
Benefits
Oracle Data Integrator
Data-oriented Integration
Event Conductor
Event-oriented
Integration
Service Conductor
Service-oriented
Declarative Design
Metadata
Data Conductor
Enables real-time data warehousing and operational data hubs
Services plug into Oracle SOA Suite for comprehensive integration

10. Differentiator: Declarative Design Developer Productivity3
Differentiator: Declarative Design Developer Productivity
Conventional ETL Design
Specify ETL Data Flow Graph
Developer must define every step of Complex ETL Flow Logic
Traditional approach requires specialized ETL skills
And significant development and maintenance efforts
Declarative Set-based Design
Simplifies the number of steps
Automatically generates the Data Flow whatever the sources and target DB
Example: [SALAH]
Benefits
Significantly reduce the learning curve
Shorter implementation times
Streamline access to non-IT pros
ODI Declarative Design
Define How: Built-in Templates
Define
What
You Want
Automatically Generate
Dataflow 1 2

11. 4
Differentiator: Knowledge Modules Hot-Pluggable: Modular, Flexible, Extensible
Pluggable Knowledge Modules Architecture
Reverse
Engineer Metadata
Journalize
Read from CDC Source
Load
From Sources to Staging
Check
Constraints before Load
Integrate
Transform and Move to Targets
Service
Expose Data and Transformation Services
Reverse WS WS WS
Staging Tables
Load
Integrate
Services
CDC
Journalize
Check
Target Tables
Sources
Error Tables
Sample out-of-the-box Knowledge Modules
SAP/R3
Log Miner
SQL Server Triggers
Oracle DBLink
JMS Queues
Check MS Excel
TPump/ Multiload
Oracle Merge
Oracle Web Services
Siebel
DB2 Journals
DB2 Exp/Imp
Oracle SQL*Loader
Check Sybase
Type II SCD
Siebel EIM Schema
DB2 Web Services
Benefits
Tailor to existing best practices
Ease administration work
Reduce cost of ownership

12. Oracle Data Integrator Architecture

13. Architecture: Conceptual View
Design-Time
Metadata Management
Runtime
Agent
Data Flow Conductor
Service Interfaces and Developer APIs
User
Interfaces
Thin Client
Data Flow Generator
Knowledge Module Interpreter
Knowledge Modules
Master
Repository
Work
Repositories
Runtime Session Interpreter
Data Flow
Operator
Designer
Java design-time environment
Runs on any platform
Thin client for browsing Metadata
Java runtime environment
Orchestrates the execution of data flows
Metadata repository
Pluggable on many RDBMS
Ready for deployment
Modular and extensible metadata

14. Architecture: Component View
ODI Design-Time Environment
Development Servers and Applications
Design-time Repositories
Code
Execution
Execution Log
Return Codes
Agent
Data Flow Conductor
CRM
Legacy
ERP
Data Warehouse
Files / XML
User Interfaces
Administrators
Designers
Topology/Security
Metadata/Rules
Development
ESB
Scenarios
and Projects
Releases
Production Servers and Applications
ODI Runtime Environment
Runtime Repository
Return Codes
Code
Execution Log
Execution
Metadata Navigator
Production
CRM
Legacy
ERP
Data Warehouse
Files / XML
ESB
User Interfaces
Administrators
Operators
Thin Client
Data Stewarts
Topology/Security
Metadata Lineage
Agent
Data Flow Conductor

15. Oracle Data Integrator 6 steps to Production

16. Overview: 6 steps to Production
Retrieve/Enrich metadata
Design transformations
Orchestrate data flows
Generate/Deploy data flows
Monitor executions
Analyze impact / data lineage
Development
Production
Development Servers and Applications
Production Servers and Applications
Data Warehouse
Data Warehouse
CRM
CRM
Legacy
Legacy
ERP
ERP
Files / XML
Files / XML
ESB
ESB
ODI Design-Time Environment
ODI Runtime Environment
Agent
Data Flow Conductor
User Interfaces
Administrators
Designers
Agent
Data Flow Conductor
User Interfaces
Design-time Repositories
Runtime Repository
Operator
Metadata Navigator

17. Retrieve/Enrich Metadata1
Retrieve/Enrich Metadata
Development Servers and Applications
Design-Time Environment
Reverse-engineer Metadata
Automatic
Customizable
40+ technologies supported
Enrich Metadata
Documentation
Declarative rules for Data Integrity
Cross-technologies references
ODI Designer
ERP
Data Warehouse
Design-time Repositories
CRM
Files / XML
Legacy
ESB

18. Design Transformations2
Design Transformations 1
Define What You Want 3
Automatically Generate
Data flows
Oracle Data Integrator “Interface”
Declarative Design 2
Define How to Do It: Select Template
Bulk Load • Changed Data Capture • Incremental Update • Slowly Changing Dimension

19. Orchestrate Data Flows3
Orchestrate Data Flows
Sequence Transformations
Leverage OracleDI Tools
Data Quality Processes
Files/Archives Management
Send/Receive s
Web Services Invokation
Event Detection
Create your Own Tools
Use Control Structures
Loops
Conditions
Error Handling

20. Generate and Deploy Data Flows4
Generate and Deploy Data Flows
Create Scenarios
Compile Data Flows for Run-time
Version the Data Flows
Advanced Version Management
Deploy to Production
Design-time Repositories
Scenarios
and Projects
Releases
Runtime Repository

21. Monitor Executions 5 View sessions running in real-time
Review generated code
Detailed run-time statistics
Restart failed sessions

22. Analyze impact / data lineage6
Analyze impact / data lineage
Maintain a large number of data flows in a complex environment
Web-based end-to-end data lineage
Understand your data flows
Follow the path of data
Drill-down to transformations ?

23. Oracle Data Integrator ODI or ESB?

24. What tool is best suited for task X?
Requirement
ESB
ODI
Recommended
Latency / Volume
Synchronous Integration 
Asynchronous Integration with routing and transformation
Asynchronous Integration for Active Data Warehousing (mini-batch)
Batch Integration with High Volume
Transformations
In-memory XSLT Transformations (XML to XML)
Transformations in App Server
Transformations in Database (E-LT)
Integration Topology
Data Warehouse Loading (E-LT)
JMS to JMS
JMS to DB/App with routing and transformation (real-time or synchronous)
JMS to DB/App with bulk transformation (mini-batch)
DB/App to DB/App (batch or mini-batch with CDC)
DB/App to DB/App (synchronous or real-time with CDC Adapters) …

25. ESB and ODI in real-life scenarios
Data
Latency
Oracle Data Integrator
Batch
(over 2 hours)
Oracle Enterprise Service Bus
Asynchronous
Synchronous
(immediate)
Real-life
Scenarios
Message by Message
Mini Batches
Large Volume
(over 1M)
Data Volume Processing

26. Oracle Data Integrator Extended Capabilities

27. Extended Capabilities
Master Data Management enabled
Common Format Designer
Automated generation of canonical format and transformations
Built-in Data Integrity
Real-time enabled
Changed Data Capture
Message Oriented Integration (JMS)
SOA enabled
Generation of Data Services
Generation of Transformation Services
Extensibility
Knowledge Modules Framework
Scripting Languages
Open Tools

28. Oracle Data Integrator Master Data Management Enabled

29. MDM Enabled: Canonical Format Design
Use in conjunction with packaged MDM solution
Design and Populate Canonical Format
Use existing metadata artifacts to design MDM application (entities, fields, relationships)
Generate and maintain Master Data structure
Generate and deploy transformations using metadata artifacts
Master Data
Enterprise Service Bus
CRM
SCM
Legacy
ERP

30. MDM Enabled: Built-in Data Integrity
Data Integrity Firewall
Auditing, cleansing and recycling
Declare constraints at table level
Design mappings and check flow integrity
Audit, cleanse or recycle rejected records
Message Id
Name
City
Duplicated Record
001
John Doe
New York
022
Boston
Invalid City Reference
230
Albert Fresh
Maris

31. Oracle Data Integrator Real-time Enabled

32. Real-time enabled: Changed Data Capture
Publish and Subscribe CDC Framework
Database logs
Triggers
Third-tier solutions
Ensures “read” transaction integrity across multiple tables
Design or generate Mappings
Select Journalized Data Only
Start Journals
CDC

33. Real-time enabled: Message Oriented Integration
Connect to Publish and Subscribe JMS Message Providers
Ensure messages delivery with transaction integrity
High-volume bulk transformations
Design complex bulk transformations mixing Queues, Databases and Applications
Use JMS Queues and topics as sources or targets
Subscribe
Publish
CDC
JMS Provider
(MOM, ESB)

34. Oracle Data Integrator SOA Enabled

35. SOA Enabled: Data Access Services
SOA Infrastructure
Business Processes
Generate and share data access services
Generate and deploy data services
Test data services
Leverage data services in your SOA infrastructure
Services
Data Access
Transform
ESB
Business

36. SOA Enabled: Data Flow Services
SOA Infrastructure
Expose transformations as Web Services
Orchestrate data flows
Publish data flows as web services in your SOA infrastructure
Bulk Transf.
Business Processes
Services
Data Access
Transform
ESB
Business

37. Oracle Data Integrator Extensible Framework

38. Extensibility: Knowledge Modules
KM’s Meta Code
120+ KMs out-of-the-box
Tailor to existing best practices
Ease administration work
Reduce cost of ownership
Customizable and extensible
Executed Code KM
Interpreter
Metadata
Journalize
Read from CDC Source
Load
From Sources to Staging
Check
Constraints before Load
Integrate
Transform and Move to Targets
Service
Expose Data and Transformation Services
Reverse
Engineer Metadata
Services
Pluggable Knowledge Modules Architecture
CDC
Sources
Staging Tables
Error Tables
Target Tables WS

39. Extensibility: Scripting Framework
Extend data flows with scripting procedures
Leverage all database languages
SQL, PL/SQL, Transact SQL, etc.
Use Operating Systems shell scripts
Win32 DOS, sh, ksh, csh, OS400 commands, JCL, etc.
Choose from compatible Bean Scripting Framework languages
Java, JavaScript, Jython (Java Python), Perl, etc.

40. Extensibility: Open Tools
Extend ODI tools
Add your own tools to the Design Palette
Implement OdiOpenToolAbstract Java Interface
Register Open Tool in ODI Designer
Use Open Tool in your design environment

41. Popular Usage Scenarios

42. E-LT for Data Warehouse Create Data Warehouse for Business Intelligence Populate Warehouse with High Performance ODI
Data Warehouse
Cube
----
Operational
Analytics
Metadata
Load
Transform
Capture Changes
Incremental Update
Data Integrity
Aggregate
Export
Heterogeneous sources and targets
Incremental load
Slowly changing dimensions
Data integrity and consistency
Changed data capture
Data lineage

43. SOA Initiative Establish Messaging Architecture for Integration Incorporate Efficient Bulk Data Processing with ODI
Services
Data Access
Transformation
Others
----
Operational
Metadata
Generate Data
Expose Transformation Services
Deploy and reuse
Business Processes
Invoke external services for data integration
Deploy data services
Deploy transformation services
Integrate data and transformation services in your SOA infrastructure

44. Master Data Management Create Single View of the Truth Synchronize Data with ODI
----
Metadata
Change Data Capture
Master Data Load
Canonical Format Design
Cleansing and Reconciliation
Master Data Publishing
CDC
Use in conjunction with packaged MDM solution
Use as infrastructure for designing your own hub
Create declarative data flows
Capture changes (CDC)
Reconcile and cleanse the data
Publish and share master data
Extend metadata definitions

45. Migration Upgrade Applications or Migrate to New Schema Move Bulk Data Once and Keep in Sync with ODI
Old Applications
New Application
Metadata
Initial bulk load
CDC for synchronization
Transformation to new application format
CDC for loop-back synchronization
CDC
----
Bulk-load historical data to new application
Transform source format to target
Synchronize new and old applications during overlap time
Capture changes in a bi-directional way (CDC)

46. ODI Enhances Oracle BI Populate Warehouse with High Performance ODI
Siebel CRM
Oracle EBS
PeopleSoft
SAP/R3
Other Sources
Oracle Data Integrator
E-LT Metadata
E-LT Agent
Oracle BI Enterprise Data Warehouse
Oracle BI Suite EE
Oracle BI Server
Oracle BI Presentation Server
Answers
Interactive Dashboards
Publisher
Delivers
Bulk E-LT
Oracle Business Intelligence Suite EE:
Simplified Business Model View
Advanced Calculation & Integration Engine
Intelligent Request Generation
Optimized Data Access
Oracle Data Integrator:
Populate Enterprise Data Warehouse
Optimized Performance for Load and Transform
Extensible Pre-packaged E-LT Content

47. Oracle Data Integrator
ODI Enhances Oracle SOA Suite Add Bulk Data Transformation to BPEL Process
Oracle SOA Suite
Business Activity Monitoring
Web Services Manager
Business Rules Engine
Enterprise Service Bus
BPEL Process Manager
Bulk Data Processing
Oracle Data Integrator
E-LT Metadata
E-LT Agent
Oracle SOA Suite:
BPEL Process Manager for Business Process Orchestration
Oracle Data Integrator:
Efficient Bulk Data Processing as Part of Business Process
Interact via Data Services and Transformation Services

48. Oracle Data Integrator
ODI Enhances Oracle SOA Suite Populate BAM Active Data Cache Efficiently
Data Warehouse
Oracle SOA Suite
BPEL Process Manager
Web Services Manager
Business Rules Engine
Enterprise Service Bus
Bulk and
Real-Time
Data Processing
SAP/R3
PeopleSoft
Message Queues
CDC
Business Activity Monitoring
Active Data Cache
Event Engine
Report Cache
Event Monitoring Web Applications
Oracle Data Integrator
Metadata
Agent
Oracle SOA Suite:
Business Activity Monitoring for Real-time Insight
Oracle Data Integrator:
High Performance Loading of BAM’s Active Data Cache
Pre-built and Integrated

49. Roadmap and Direction

50. Oracle Data Integrator: Roadmap
Focus Areas for Next Major Release
Deep Integration with Fusion Middleware
Runtime, Design time, Security, Administration, Events
Functional Integration with Oracle Warehouse Builder
Runtime Integration, Metadata Sharing, Knowledge Module Sharing
Deployment of ODI for Embedded Data Integration
OracleBI Enterprise Edition, Data Hubs, Application Migrations
Enhanced Usability and Debuggability
Wizards, New Views, User-definable Debugging
Improved Support for Native Oracle Database Features
Oracle OLAP

51. ODI Statement of Direction
Key Points of Direction
Commitment to heterogeneous systems support
Including: DB2, Teradata, Netezza, Hyperion, etc.
Commitment to Fusion design principles
Including: J2EE compliance, container portability
Commitment to best-of-class E-LT performance
Across platforms, batch & realtime, high complexity