Data Integration in Service Oriented Architectures Rahul Patel Sr. Director R & D, BEA Systems Liquid Data – XML-based data access and integration for enterprise data Project Alchemy – Mobile, disconnected applications for enterprise data June 8, 2004

Service Oriented Architecture EmployeesCustomersPartners Specialty Infrastructure Directory, Search, Content, etc Enterprise Applications SFA, CRM, Custom, Legacy, Mainframe Database & Tuxedo Data & Transaction Services Infrastructure Services Message Bus (Transport, Routing, Transformation) Shared Business Services 1 Composite Applications, Portals and B2B Services Management Security

Enterprise Data Services Customer (DB) Order Mgt (DB) Customer Portal GetCust_by_Custid GetCust _by_Orderid Update_Address Create_Order Customer: Jack Customer (Package App) Order Mgt (Web Service) Division 1 Division 2 Requires accessing and integrating data from multiple resources

Service Clients Data Services on Logical Model Physical Model Addresses Customer Profile Pending Payments Credited Payments Orders History Data Services Platform Repository Enterprise Data JDBC Model Driven Integration (XQuery) Customer Order Payment Programming Model (SDO) Web Srvcs Adapters Custom J2EE, Web Service, XQuery, JMS Logical models capture the data access & integration complexity ONCE SAME data model, programming model, & API for whole enterprise

Enterprise Data Services Model Based Approach Model-based unification of disparate data sources –Define virtual model(s) –Map physical data source models to virtual model –Data Mediator converts operations against virtual model into Queries and transactions against individual sources, plus Middleware coordination and computations to finish the job Technology for Enterprise Data Services –Data model mapping and integration –SOA-appropriate programming model –Translation of data models, translation of queries –Performance and Caching

Choosing Appropriate Technologies XML is the Answer Required CapabilitiesStandards XML A standard for data format and data interchange A standard for querying both relational & non-relational data XQuery A standard for interfacing into applications Web Services, JCA XML Schema A standard for describing and modeling data A standard for publishing available services Web Services

Data Services Infrastructure Must tie all the pieces together Various parts of data architecture –XMLSchema (shape) –Logical Views (nodes in the information model) –Relationships (arcs in the information model) –Data sources (enterprise data source) –XQuery functions/source (aggregate/transform logic) –Programming model for these (SDO) –Business logic (Java, Workflows, etc.) –Publishing the models for application use (Web Services, SDO, Controls..) –Policies (meta-data, security, etc.) How to connect all these in a meta-data environment –Dependencies –Integrated development / management experience –A logical of reusable unit of data representing a particular business entity – e.g. Single view of Customer

Model Driven “Read” Data Services Optimized Distributed Queries on Logical Model SQL sub-query pushdown to relational sources –Selects, projects, functions, inner/outer joins, sorts, aggregates Distributed join optimization –Batched parameter passing Join, Index Join, Merge Join Slow data source handling –Intra-query function result caching –Parallel invocation of slow functional sources Query result cache for stored queries (optional) –Consistency based on TTL plus invalidation API Check security & cache Cache results Create optimized Execution plan Merge & Transform Data Data Service Request Query Result Get data from underlying sources Sub-query Sub-queryFunction Call 3

Model Driven “Write” Services Update Logic specified on the same Model Physical Source 1 Physical Source 2 Update_Address Update_CreditCard Mapping Meta-data Decomposition Change Log Time Stamp Source Info. Update Plan User Defined Logic, Validation, Rules, etc. User Defined Logic, Validation, Rules, etc. User Exit on View

Caching Latency in SOA drives tremendous need for caching Caching in middleware is inherently “optimistic” –Optimized for reads –Optimistic for writes Caching granularity should at logical data views and navigations between them –Caching individual data source results is not sufficient