Presentation is loading. Please wait.

Presentation is loading. Please wait.

Oracle Data Integrator – Technical Deck

Similar presentations

Presentation on theme: "Oracle Data Integrator – Technical Deck"— Presentation transcript:


2 Oracle Data Integrator – Technical Deck
Mark Pare Sr. Sales Consultant Oracle Higher Education

3 <Insert Picture Here>
Agenda <Insert Picture Here> 4 Key Differentiators Architecture 6 Steps to Production ODI or ESB? Popular Usage Scenarios

4 Oracle Data Integrator 4 Key Differentiators

5 Why Data Integration? NEED… HAVE…
Information How and Where you Want It Business Intelligence Corporate Performance Management Business Activity Monitoring Business Process Management Data Integration Migration Data Warehousing Master Data Management - - - Data Synchronization Federation Real Time Messaging HAVE… Data in Disparate Sources ERP --- CRM - - - Legacy Best-of-breed Applications

6 Challenges & Emerging Solutions In Data Integration
Increasing data volumes; decreasing batch windows Non-integrated integration Complexity, manual effort of conventional ETL design Lack of knowledge capture Shift from E-T-L to E-LT Convergence of integration solutions Shift from custom coding to declarative design Shift to pattern-driven development

7 Oracle Data Integrator
Data Movement and Transformation from Multiple Sources to Heterogeneous Targets Performance: Heterogeneous “E-LT” Flexibility: Active Integration Platform Productivity: Declarative Design Hot-Pluggable: Knowledge Modules BENEFITS KEY DIFFERENTIATED FEATURES

8 Differentiator: E-LT Architecture High Performance
1 Differentiator: E-LT Architecture High Performance Conventional ETL Architecture Extract Load Transform Transform in Separate ETL Server Proprietary Engine Poor Performance High Costs IBM & Informatica’s approach Transform in Existing RDBMS Leverage Resources Efficient High Performance Benefits Optimal Performance & Scalability Easier to Manage & Lower Cost Next Generation Architecture “E-LT” Load Extract Transform

9 Oracle Data Integrator Data-oriented Integration
2 Differentiator: Active Integration Batch, Event-based, and Service-oriented Integration Evolve from Batch to Near Real-time Warehousing on Common Platform Unify the Silos of Data Integration Data Integrity on the Fly Services Plug into Oracle SOA Suite Benefits Oracle Data Integrator Data-oriented Integration Event Conductor Event-oriented Integration Service Conductor Service-oriented Declarative Design Metadata Data Conductor Enables real-time data warehousing and operational data hubs Services plug into Oracle SOA Suite for comprehensive integration

10 Differentiator: Declarative Design Developer Productivity
3 Differentiator: Declarative Design Developer Productivity Conventional ETL Design Specify ETL Data Flow Graph Developer must define every step of Complex ETL Flow Logic Traditional approach requires specialized ETL skills And significant development and maintenance efforts Declarative Set-based Design Simplifies the number of steps Automatically generates the Data Flow whatever the sources and target DB Example: [SALAH] Benefits Significantly reduce the learning curve Shorter implementation times Streamline access to non-IT pros ODI Declarative Design Define How: Built-in Templates Define What You Want Automatically Generate Dataflow 1 2

11 4 Differentiator: Knowledge Modules Hot-Pluggable: Modular, Flexible, Extensible Pluggable Knowledge Modules Architecture Reverse Engineer Metadata Journalize Read from CDC Source Load From Sources to Staging Check Constraints before Load Integrate Transform and Move to Targets Service Expose Data and Transformation Services Reverse WS WS WS Staging Tables Load Integrate Services CDC Journalize Check Target Tables Sources Error Tables Sample out-of-the-box Knowledge Modules SAP/R3 Log Miner SQL Server Triggers Oracle DBLink JMS Queues Check MS Excel TPump/ Multiload Oracle Merge Oracle Web Services Siebel DB2 Journals DB2 Exp/Imp Oracle SQL*Loader Check Sybase Type II SCD Siebel EIM Schema DB2 Web Services Benefits Tailor to existing best practices Ease administration work Reduce cost of ownership

12 Oracle Data Integrator Architecture

13 Architecture: Conceptual View
Design-Time Metadata Management Runtime Agent Data Flow Conductor Service Interfaces and Developer APIs User Interfaces Thin Client Data Flow Generator Knowledge Module Interpreter Knowledge Modules Master Repository Work Repositories Runtime Session Interpreter Data Flow Operator Designer Java design-time environment Runs on any platform Thin client for browsing Metadata Java runtime environment Orchestrates the execution of data flows Metadata repository Pluggable on many RDBMS Ready for deployment Modular and extensible metadata

14 Architecture: Component View
ODI Design-Time Environment Development Servers and Applications Design-time Repositories Code Execution Execution Log Return Codes Agent Data Flow Conductor CRM Legacy ERP Data Warehouse Files / XML User Interfaces Administrators Designers Topology/Security Metadata/Rules Development ESB Scenarios and Projects Releases Production Servers and Applications ODI Runtime Environment Runtime Repository Return Codes Code Execution Log Execution Metadata Navigator Production CRM Legacy ERP Data Warehouse Files / XML ESB User Interfaces Administrators Operators Thin Client Data Stewarts Topology/Security Metadata Lineage Agent Data Flow Conductor

15 Oracle Data Integrator 6 steps to Production

16 Overview: 6 steps to Production
Retrieve/Enrich metadata Design transformations Orchestrate data flows Generate/Deploy data flows Monitor executions Analyze impact / data lineage Development Production Development Servers and Applications Production Servers and Applications Data Warehouse Data Warehouse CRM CRM Legacy Legacy ERP ERP Files / XML Files / XML ESB ESB ODI Design-Time Environment ODI Runtime Environment Agent Data Flow Conductor User Interfaces Administrators Designers Agent Data Flow Conductor User Interfaces Design-time Repositories Runtime Repository Operator Metadata Navigator

17 Retrieve/Enrich Metadata
1 Retrieve/Enrich Metadata Development Servers and Applications Design-Time Environment Reverse-engineer Metadata Automatic Customizable 40+ technologies supported Enrich Metadata Documentation Declarative rules for Data Integrity Cross-technologies references ODI Designer ERP Data Warehouse Design-time Repositories CRM Files / XML Legacy ESB

18 Design Transformations
2 Design Transformations 1 Define What You Want 3 Automatically Generate Data flows Oracle Data Integrator “Interface” Declarative Design 2 Define How to Do It: Select Template Bulk Load • Changed Data Capture • Incremental Update • Slowly Changing Dimension

19 Orchestrate Data Flows
3 Orchestrate Data Flows Sequence Transformations Leverage OracleDI Tools Data Quality Processes Files/Archives Management Send/Receive s Web Services Invokation Event Detection Create your Own Tools Use Control Structures Loops Conditions Error Handling

20 Generate and Deploy Data Flows
4 Generate and Deploy Data Flows Create Scenarios Compile Data Flows for Run-time Version the Data Flows Advanced Version Management Deploy to Production Design-time Repositories Scenarios and Projects Releases Runtime Repository

21 Monitor Executions 5 View sessions running in real-time
Review generated code Detailed run-time statistics Restart failed sessions

22 Analyze impact / data lineage
6 Analyze impact / data lineage Maintain a large number of data flows in a complex environment Web-based end-to-end data lineage Understand your data flows Follow the path of data Drill-down to transformations ?

23 Oracle Data Integrator ODI or ESB?

24 What tool is best suited for task X?
Requirement ESB ODI Recommended Latency / Volume Synchronous Integration Asynchronous Integration with routing and transformation Asynchronous Integration for Active Data Warehousing (mini-batch) Batch Integration with High Volume Transformations In-memory XSLT Transformations (XML to XML) Transformations in App Server Transformations in Database (E-LT) Integration Topology Data Warehouse Loading (E-LT) JMS to JMS JMS to DB/App with routing and transformation (real-time or synchronous) JMS to DB/App with bulk transformation (mini-batch) DB/App to DB/App (batch or mini-batch with CDC) DB/App to DB/App (synchronous or real-time with CDC Adapters)

25 ESB and ODI in real-life scenarios
Data Latency Oracle Data Integrator Batch (over 2 hours) Oracle Enterprise Service Bus Asynchronous Synchronous (immediate) Real-life Scenarios Message by Message Mini Batches Large Volume (over 1M) Data Volume Processing

26 Oracle Data Integrator Extended Capabilities

27 Extended Capabilities
Master Data Management enabled Common Format Designer Automated generation of canonical format and transformations Built-in Data Integrity Real-time enabled Changed Data Capture Message Oriented Integration (JMS) SOA enabled Generation of Data Services Generation of Transformation Services Extensibility Knowledge Modules Framework Scripting Languages Open Tools

28 Oracle Data Integrator Master Data Management Enabled

29 MDM Enabled: Canonical Format Design
Use in conjunction with packaged MDM solution Design and Populate Canonical Format Use existing metadata artifacts to design MDM application (entities, fields, relationships) Generate and maintain Master Data structure Generate and deploy transformations using metadata artifacts Master Data Enterprise Service Bus CRM SCM Legacy ERP

30 MDM Enabled: Built-in Data Integrity
Data Integrity Firewall Auditing, cleansing and recycling Declare constraints at table level Design mappings and check flow integrity Audit, cleanse or recycle rejected records Message Id Name City Duplicated Record 001 John Doe New York 022 Boston Invalid City Reference 230 Albert Fresh Maris

31 Oracle Data Integrator Real-time Enabled

32 Real-time enabled: Changed Data Capture
Publish and Subscribe CDC Framework Database logs Triggers Third-tier solutions Ensures “read” transaction integrity across multiple tables Design or generate Mappings Select Journalized Data Only Start Journals CDC

33 Real-time enabled: Message Oriented Integration
Connect to Publish and Subscribe JMS Message Providers Ensure messages delivery with transaction integrity High-volume bulk transformations Design complex bulk transformations mixing Queues, Databases and Applications Use JMS Queues and topics as sources or targets Subscribe Publish CDC JMS Provider (MOM, ESB)

34 Oracle Data Integrator SOA Enabled

35 SOA Enabled: Data Access Services
SOA Infrastructure Business Processes Generate and share data access services Generate and deploy data services Test data services Leverage data services in your SOA infrastructure Services Data Access Transform ESB Business

36 SOA Enabled: Data Flow Services
SOA Infrastructure Expose transformations as Web Services Orchestrate data flows Publish data flows as web services in your SOA infrastructure Bulk Transf. Business Processes Services Data Access Transform ESB Business

37 Oracle Data Integrator Extensible Framework

38 Extensibility: Knowledge Modules
KM’s Meta Code 120+ KMs out-of-the-box Tailor to existing best practices Ease administration work Reduce cost of ownership Customizable and extensible Executed Code KM Interpreter Metadata Journalize Read from CDC Source Load From Sources to Staging Check Constraints before Load Integrate Transform and Move to Targets Service Expose Data and Transformation Services Reverse Engineer Metadata Services Pluggable Knowledge Modules Architecture CDC Sources Staging Tables Error Tables Target Tables WS

39 Extensibility: Scripting Framework
Extend data flows with scripting procedures Leverage all database languages SQL, PL/SQL, Transact SQL, etc. Use Operating Systems shell scripts Win32 DOS, sh, ksh, csh, OS400 commands, JCL, etc. Choose from compatible Bean Scripting Framework languages Java, JavaScript, Jython (Java Python), Perl, etc.

40 Extensibility: Open Tools
Extend ODI tools Add your own tools to the Design Palette Implement OdiOpenToolAbstract Java Interface Register Open Tool in ODI Designer Use Open Tool in your design environment

41 Popular Usage Scenarios

42 E-LT for Data Warehouse Create Data Warehouse for Business Intelligence Populate Warehouse with High Performance ODI Data Warehouse Cube ---- Operational Analytics Metadata Load Transform Capture Changes Incremental Update Data Integrity Aggregate Export Heterogeneous sources and targets Incremental load Slowly changing dimensions Data integrity and consistency Changed data capture Data lineage

43 SOA Initiative Establish Messaging Architecture for Integration Incorporate Efficient Bulk Data Processing with ODI Services Data Access Transformation Others ---- Operational Metadata Generate Data Expose Transformation Services Deploy and reuse Business Processes Invoke external services for data integration Deploy data services Deploy transformation services Integrate data and transformation services in your SOA infrastructure

44 Master Data Management Create Single View of the Truth Synchronize Data with ODI
---- Metadata Change Data Capture Master Data Load Canonical Format Design Cleansing and Reconciliation Master Data Publishing CDC Use in conjunction with packaged MDM solution Use as infrastructure for designing your own hub Create declarative data flows Capture changes (CDC) Reconcile and cleanse the data Publish and share master data Extend metadata definitions

45 Migration Upgrade Applications or Migrate to New Schema Move Bulk Data Once and Keep in Sync with ODI Old Applications New Application Metadata Initial bulk load CDC for synchronization Transformation to new application format CDC for loop-back synchronization CDC ---- Bulk-load historical data to new application Transform source format to target Synchronize new and old applications during overlap time Capture changes in a bi-directional way (CDC)

46 ODI Enhances Oracle BI Populate Warehouse with High Performance ODI
Siebel CRM Oracle EBS PeopleSoft SAP/R3 Other Sources Oracle Data Integrator E-LT Metadata E-LT Agent Oracle BI Enterprise Data Warehouse Oracle BI Suite EE Oracle BI Server Oracle BI Presentation Server Answers Interactive Dashboards Publisher Delivers Bulk E-LT Oracle Business Intelligence Suite EE: Simplified Business Model View Advanced Calculation & Integration Engine Intelligent Request Generation Optimized Data Access Oracle Data Integrator: Populate Enterprise Data Warehouse Optimized Performance for Load and Transform Extensible Pre-packaged E-LT Content

47 Oracle Data Integrator
ODI Enhances Oracle SOA Suite Add Bulk Data Transformation to BPEL Process Oracle SOA Suite Business Activity Monitoring Web Services Manager Business Rules Engine Enterprise Service Bus BPEL Process Manager Bulk Data Processing Oracle Data Integrator E-LT Metadata E-LT Agent Oracle SOA Suite: BPEL Process Manager for Business Process Orchestration Oracle Data Integrator: Efficient Bulk Data Processing as Part of Business Process Interact via Data Services and Transformation Services

48 Oracle Data Integrator
ODI Enhances Oracle SOA Suite Populate BAM Active Data Cache Efficiently Data Warehouse Oracle SOA Suite BPEL Process Manager Web Services Manager Business Rules Engine Enterprise Service Bus Bulk and Real-Time Data Processing SAP/R3 PeopleSoft Message Queues CDC Business Activity Monitoring Active Data Cache Event Engine Report Cache Event Monitoring Web Applications Oracle Data Integrator Metadata Agent Oracle SOA Suite: Business Activity Monitoring for Real-time Insight Oracle Data Integrator: High Performance Loading of BAM’s Active Data Cache Pre-built and Integrated

49 Roadmap and Direction

50 Oracle Data Integrator: Roadmap
Focus Areas for Next Major Release Deep Integration with Fusion Middleware Runtime, Design time, Security, Administration, Events Functional Integration with Oracle Warehouse Builder Runtime Integration, Metadata Sharing, Knowledge Module Sharing Deployment of ODI for Embedded Data Integration OracleBI Enterprise Edition, Data Hubs, Application Migrations Enhanced Usability and Debuggability Wizards, New Views, User-definable Debugging Improved Support for Native Oracle Database Features Oracle OLAP

51 ODI Statement of Direction
Key Points of Direction Commitment to heterogeneous systems support Including: DB2, Teradata, Netezza, Hyperion, etc. Commitment to Fusion design principles Including: J2EE compliance, container portability Commitment to best-of-class E-LT performance Across platforms, batch & realtime, high complexity



Download ppt "Oracle Data Integrator – Technical Deck"

Similar presentations

Ads by Google