Presentation is loading. Please wait.

Presentation is loading. Please wait.

Oracle Data Integrator – Technical Deck Mark Pare Sr. Sales Consultant Oracle Higher Education.

Similar presentations


Presentation on theme: "Oracle Data Integrator – Technical Deck Mark Pare Sr. Sales Consultant Oracle Higher Education."— Presentation transcript:

1

2 Oracle Data Integrator – Technical Deck Mark Pare Sr. Sales Consultant Oracle Higher Education

3 3 4 Key Differentiators Architecture 6 Steps to Production ODI or ESB? Popular Usage Scenarios Agenda

4 4 Oracle Data Integrator 4 Key Differentiators

5 5 Data Integration Data Warehousing Master Data Management Real Time Messaging FederationMigration Data in Disparate Sources ERP --- CRM Legacy --- Best-of-breed Applications Information How and Where you Want It Business Intelligence Corporate Performance Management Business Activity Monitoring Business Process Management HAVE… NEED… Data Synchronization Why Data Integration?

6 6 Challenges & Emerging Solutions In Data Integration CHALLENGEEMERGING SOLUTION 1.Increasing data volumes; decreasing batch windows 2.Non-integrated integration 3.Complexity, manual effort of conventional ETL design 4.Lack of knowledge capture Shift from E-T-L to E-LT Convergence of integration solutions Shift from custom coding to declarative design Shift to pattern-driven development

7 7 Oracle Data Integrator Data Movement and Transformation from Multiple Sources to Heterogeneous Targets 1.Performance: Heterogeneous E-LT 2.Flexibility: Active Integration Platform 3.Productivity: Declarative Design 4.Hot-Pluggable:Knowledge Modules BENEFITSKEY DIFFERENTIATED FEATURES

8 8 Differentiator: E-LT Architecture High Performance Conventional ETL Architecture Extract Load Transform Next Generation Architecture E-LT LoadExtract Transform Transform in Separate ETL Server Proprietary Engine Poor Performance High Costs IBM & Informatica s approach Transform in Existing RDBMS Leverage Resources Efficient High Performance Benefits Optimal Performance & Scalability Easier to Manage & Lower Cost 1 1

9 9 Enables real-time data warehousing and operational data hubs Services plug into Oracle SOA Suite for comprehensive integration Oracle Data Integrator Data-oriented Integration Event Conductor Event-oriented Integration Service Conductor Service-oriented Integration Declarative Design Metadata Data Conductor Differentiator: Active Integration Batch, Event-based, and Service-oriented Integration Evolve from Batch to Near Real-time Warehousing on Common Platform Unify the Silos of Data Integration Data Integrity on the Fly Services Plug into Oracle SOA Suite Benefits 2 2

10 10 Differentiator: Declarative Design Developer Productivity Conventional ETL Design Specify ETL Data Flow Graph Developer must define every step of Complex ETL Flow Logic Traditional approach requires specialized ETL skills And significant development and maintenance efforts Declarative Set-based Design Simplifies the number of steps Automatically generates the Data Flow whatever the sources and target DB Example: [SALAH] Benefits Significantly reduce the learning curve Shorter implementation times Streamline access to non-IT pros ODI Declarative Design Define How : Built-in Templates Define What You Want Automatically Generate Dataflow

11 11 Journalize Read from CDC Source Load From Sources to Staging Check Constraints before Load Integrate Transform and Move to Targets Service Expose Data and Transformation Services Reverse Engineer Metadata Tailor to existing best practices Ease administration work Reduce cost of ownership Reverse Journalize Load Check Integrate Services Pluggable Knowledge Modules Architecture CDC Sources Staging Tables Error Tables Target Tables WSWS WSWS WSWS SAP/R3 Siebel Log Miner DB2 Journals SQL Server Triggers Oracle DBLink DB2 Exp/Imp JMS Queues Check MS Excel Check Sybase Oracle SQL*Loader TPump/ Multiload Type II SCD Oracle Merge Siebel EIM Schema Oracle Web Services DB2 Web Services Sample out-of-the-box Knowledge Modules Benefits Differentiator: Knowledge Modules Hot-Pluggable: Modular, Flexible, Extensible 4 4

12 12 Oracle Data Integrator Architecture

13 13 Java design-time environment Runs on any platform Thin client for browsing Metadata Java runtime environment Runs on any platform Orchestrates the execution of data flows Metadata repository Pluggable on many RDBMS Ready for deployment Modular and extensible metadata Design-Time Metadata Management Runtime Agent Data Flow Conductor Service Interfaces and Developer APIs User Interfaces Thin Client Data Flow Generator Knowledge Module Interpreter Knowledge Modules Master Repository Work Repositories Runtime Repositories Data Flow Generator Runtime Session Interpreter Data Flow Operator Designer Architecture: Conceptual View

14 14 Architecture: Component View ODI Design-Time EnvironmentDevelopment Servers and Applications Design-time Repositories Code Execution Execution Log Return Codes Agent Data Flow Conductor CRM Legacy ER P Data Warehouse Files / XML User Interfaces Administrators Designers Topology/Security Metadata/Rules Development ESB Production Servers and Applications ODI Runtime Environment Runtime Repository Return Codes Code Execution Log Execution Metadata Navigator Production CRM Legacy ER P Data Warehouse Files / XML ESB User Interfaces Administrators Operators Thin Client Data Stewarts Topology/Security Execution Log Metadata Lineage Agent Data Flow Conductor Scenarios and Projects Releases

15 15 Oracle Data Integrator 6 steps to Production

16 16 ODI Design-Time EnvironmentODI Runtime Environment User Interfaces Overview: 6 steps to Production 1.Retrieve/Enrich metadata 2.Design transformations 3.Orchestrate data flows 4.Generate/Deploy data flows 5.Monitor executions 6.Analyze impact / data lineage Development Development Servers and Applications Agent Data Flow Conductor CRM Legacy ERP Data Warehouse Files / XML User Interfaces Administrators Designers ESB Design-time Repositories Production Production Servers and Applications Agent Data Flow Conductor CRM Legacy ERP Data Warehouse Files / XML Operator Metadata Navigator ESB Runtime Repository

17 17 1.Reverse-engineer Metadata Automatic Customizable 40+ technologies supported 2.Enrich Metadata Documentation Declarative rules for Data Integrity Cross-technologies references Design-Time Environment ODI Designer Design-time Repositories Development Servers and Applications CRM Legacy ERP Data Warehouse Files / XML ESB Retrieve/Enrich Metadata 1 1

18 18 Oracle Data Integrator Interface Declarative Design 1 Define What You Want3 Automatically Generate Data flows 2 Define How to Do It: Select Template Bulk Load Changed Data Capture Incremental Update Slowly Changing Dimension Design Transformations 2 2

19 19 1.Sequence Transformations 2.Leverage OracleDI Tools Data Quality Processes Files/Archives Management Send/Receive s Web Services Invokation Event Detection Create your Own Tools 3.Use Control Structures Loops Conditions Error Handling 3 3 Orchestrate Data Flows

20 20 1.Create Scenarios Compile Data Flows for Run-time 2.Version the Data Flows Advanced Version Management 3.Deploy to Production Design-time Repositories Scenarios and Projects Releases Runtime Repository Generate and Deploy Data Flows 4 4

21 21 View sessions running in real- time Review generated code Detailed run-time statistics Restart failed sessions 5 5 Monitor Executions

22 22 Analyze impact / data lineage Maintain a large number of data flows in a complex environment Web-based end-to-end data lineage 1.Understand your data flows 2.Follow the path of data 3.Drill-down to transformations ? 6 6

23 23 Oracle Data Integrator ODI or ESB?

24 24 What tool is best suited for task X? RequirementES B ODIRecommende d Latency / Volume Synchronous Integration ESB Asynchronous Integration with routing and transformation ESB Asynchronous Integration for Active Data Warehousing (mini-batch) ODI Batch Integration with High Volume ODI Transformations In-memory XSLT Transformations (XML to XML) ESB Transformations in App Server ESB Transformations in Database (E-LT) ODI Integration Topology Data Warehouse Loading (E-LT) ODI JMS to JMS ESB JMS to DB/App with routing and transformation (real-time or synchronous) ESB JMS to DB/App with bulk transformation (mini-batch) ODI DB/App to DB/App (batch or mini-batch with CDC) ODI DB/App to DB/App (synchronous or real-time with CDC Adapters) ESB …

25 25 Oracle Data Integrator ESB and ODI in real-life scenarios Data Volume Processing Data Latency Message by Message Mini BatchesLarge Volume (over 1M) Synchronous (immediate) Asynchronous Batch (over 2 hours) Oracle Enterprise Service Bus Real-life Scenarios

26 26 Oracle Data Integrator Extended Capabilities

27 27 Extended Capabilities Master Data Management enabled Common Format Designer Automated generation of canonical format and transformations Built-in Data Integrity Real-time enabled Changed Data Capture Message Oriented Integration (JMS) SOA enabled Generation of Data Services Generation of Transformation Services Extensibility Knowledge Modules Framework Scripting Languages Open Tools

28 28 Oracle Data Integrator Master Data Management Enabled

29 29 MDM Enabled: Canonical Format Design Use in conjunction with packaged MDM solution Design and Populate Canonical Format 1.Use existing metadata artifacts to design MDM application (entities, fields, relationships) 2.Generate and maintain Master Data structure 3.Generate and deploy transformations using metadata artifacts CRM Enterprise Service Bus SCMLegacyERP Master Data

30 30 MDM Enabled: Built-in Data Integrity Data Integrity Firewall Auditing, cleansing and recycling 1.Declare constraints at table level 2.Design mappings and check flow integrity 3.Audit, cleanse or recycle rejected records MessageIdNameCity Duplicated Record001John DoeNew York Duplicated Record022John DoeBoston Invalid City Reference230Albert FreshMaris

31 31 Oracle Data Integrator Real-time Enabled

32 32 Real-time enabled: Changed Data Capture Publish and Subscribe CDC Framework Database logs Triggers Third-tier solutions Ensures read transaction integrity across multiple tables 1.Design or generate Mappings 2.Select Journalized Data Only 3.Start Journals CDC

33 33 Real-time enabled: Message Oriented Integration Connect to Publish and Subscribe JMS Message Providers Ensure messages delivery with transaction integrity High-volume bulk transformations 1.Design complex bulk transformations mixing Queues, Databases and Applications 2.Use JMS Queues and topics as sources or targets JMS Provider (MOM, ESB) CDC Subscribe Publish

34 34 Oracle Data Integrator SOA Enabled

35 35 SOA Enabled: Data Access Services Generate and share data access services 1.Generate and deploy data services 2.Test data services 3.Leverage data services in your SOA infrastructure SOA Infrastructure Services Data Access Transform Business Business Processes ESB

36 36 SOA Enabled: Data Flow Services Expose transformations as Web Services 1.Orchestrate data flows 2.Publish data flows as web services in your SOA infrastructure SOA Infrastructure Services Data Access Transform Business Business Processes ESB Bulk Transf.

37 37 Oracle Data Integrator Extensible Framework

38 38 Journalize Read from CDC Source Load From Sources to Staging Check Constraints before Load Integrate Transform and Move to Targets Service Expose Data and Transformation Services Reverse Engineer Metadata Reverse Journalize Load Check Integrate Services Pluggable Knowledge Modules Architecture CDC Sources Staging Tables Error Tables Target Tables WSWS WSWS WSWS Extensibility: Knowledge Modules 120+ KMs out-of-the-box Tailor to existing best practices Ease administration work Reduce cost of ownership Customizable and extensible KM Interpreter KMs Meta Code Metadata Executed Code

39 39 Extensibility: Scripting Framework Extend data flows with scripting procedures Leverage all database languages SQL, PL/SQL, Transact SQL, etc. Use Operating Systems shell scripts Win32 DOS, sh, ksh, csh, OS400 commands, JCL, etc. Choose from compatible Bean Scripting Framework languages Java, JavaScript, Jython (Java Python), Perl, etc.

40 40 Extensibility: Open Tools Extend ODI tools Add your own tools to the Design Palette 1.Implement OdiOpenToolAbstract Java Interface 2.Register Open Tool in ODI Designer 3.Use Open Tool in your design environment

41 41 Popular Usage Scenarios

42 42 E-LT for Data Warehouse Create Data Warehouse for Business Intelligence Populate Warehouse with High Performance ODI Heterogeneous sources and targets Incremental load Slowly changing dimensions Data integrity and consistency Changed data capture Data lineage Data Warehouse Cube ---- Operational Analytics Metadata Load Transform Capture Changes Incremental Update Data Integrity Aggregate Export

43 43 SOA Initiative Establish Messaging Architecture for Integration Incorporate Efficient Bulk Data Processing with ODI Invoke external services for data integration Deploy data services Deploy transformation services Integrate data and transformation services in your SOA infrastructure Services Data Access Transformation Others ---- Operational Metadata Generate Data Services Expose Transformation Services Deploy and reuse Services Business Processes

44 44 Master Data Management Create Single View of the Truth Synchronize Data with ODI Use in conjunction with packaged MDM solution Use as infrastructure for designing your own hub Create declarative data flows Capture changes (CDC) Reconcile and cleanse the data Publish and share master data Extend metadata definitions Master Data ---- Metadata Change Data Capture Master Data Load Canonical Format Design Cleansing and Reconciliation Master Data Publishing ---- CDC

45 45 Migration Upgrade Applications or Migrate to New Schema Move Bulk Data Once and Keep in Sync with ODI Bulk-load historical data to new application Transform source format to target Synchronize new and old applications during overlap time Capture changes in a bi- directional way (CDC) Old Applications New Application Metadata Initial bulk load CDC for synchronization Transformation to new application format CDC for loop- back synchronization CDC ----

46 46 ODI Enhances Oracle BI Populate Warehouse with High Performance ODI Oracle Business Intelligence Suite EE: Simplified Business Model View Advanced Calculation & Integration Engine Intelligent Request Generation Optimized Data Access Oracle Data Integrator: Populate Enterprise Data Warehouse Optimized Performance for Load and Transform Extensible Pre-packaged E-LT Content Siebel CRM Oracle EBS PeopleSoft SAP/R3 Other Sources Oracle Data Integrator E-LT Metadata E-LT Agent Oracle BI Enterprise Data Warehouse Oracle BI Suite EE Oracle BI Server Oracle BI Presentation Server Answers Interactive Dashboards PublisherDelivers Bulk E-LT

47 47 ODI Enhances Oracle SOA Suite Add Bulk Data Transformation to BPEL Process Oracle SOA Suite: BPEL Process Manager for Business Process Orchestration Oracle Data Integrator: Efficient Bulk Data Processing as Part of Business Process Interact via Data Services and Transformation Services Oracle SOA Suite Business Activity Monitoring Web Services Manager Business Rules Engine Enterprise Service Bus BPEL Process Manager Bulk Data Processing Oracle Data Integrator E-LT Metadata E-LT Agent

48 48 ODI Enhances Oracle SOA Suite Populate BAM Active Data Cache Efficiently Oracle SOA Suite: Business Activity Monitoring for Real-time Insight Oracle Data Integrator: High Performance Loading of BAMs Active Data Cache Pre-built and Integrated Data Warehouse Oracle SOA Suite BPEL Process Manager Web Services Manager Business Rules Engine Enterprise Service Bus Bulk and Real-Time Data Processing SAP/R3 PeopleSoft Me ss ag e Qu eu es CDC Business Activity Monitoring Active Data Cache Event EngineReport Cache Event Monitoring Web Applications Oracle Data Integrator Metadata Agent

49 49 Roadmap and Direction

50 50 Oracle Data Integrator: Roadmap Focus Areas for Next Major Release Deep Integration with Fusion Middleware Runtime, Design time, Security, Administration, Events Functional Integration with Oracle Warehouse Builder Runtime Integration, Metadata Sharing, Knowledge Module Sharing Deployment of ODI for Embedded Data Integration OracleBI Enterprise Edition, Data Hubs, Application Migrations Enhanced Usability and Debuggability Wizards, New Views, User-definable Debugging Improved Support for Native Oracle Database Features Oracle OLAP

51 51 ODI Statement of Direction Statement of Direction Key Points of Direction Commitment to heterogeneous systems support Including: DB2, Teradata, Netezza, Hyperion, etc. Commitment to Fusion design principles Including: J2EE compliance, container portability Commitment to best-of-class E-LT performance Across platforms, batch & realtime, high complexity

52

53


Download ppt "Oracle Data Integrator – Technical Deck Mark Pare Sr. Sales Consultant Oracle Higher Education."

Similar presentations


Ads by Google