Enterprise Data Integration For Service Oriented Architectures Enterprise Architect Summit – June, 2004 Christopher Keene – CEO, Persistence Software

2 2 Presentation Overview 1. SOA requires underlying enterprise data architecture that provides consistent, timely data 2. Data virtualization ensures data consistency for SOA 3. Data services meet performance, scalability, availability needs for SOA 4. Case studies demonstrate benefits of well thought-out data architecture for SOA

3 3 Distributed Computing Cycles CORBA Java SOA Connectivity: allow any app to talk to any other app Homogeneity: single, monolithic stack (J2EE) Virtualization: data from any source to any user

4 4 The Optimists View of SOA Messaging Services - SOAP - XML - UDDI/LDAP Looser coupling of commonly performed tasks Reuse at long last through shared services Eliminates tyranny of silos Everything just works SOA

5 5 SOA Data Consistency Problem Data DB2 Check_AvailPlace_Order DB1 Item 3 = out of stockItem 3 = in stock Nightly Sync Data silos can cause inconsistent results

6 6 SOA Server Migration Problem Shared Data Multi-threaded apps share data cache Multi-host apps each have separate data cache = more database traffic Data Big Server = $$$ Grid Servers = $ Data DB Check_Avail Place_Order Show_Status Check_Avail App Place_OrderShow_Status DB

7 7 SOA Data Bottlenecks DB Potential Data Bottlenecks Application level – data wait can erase grid gains Network level – greater flexibility can add latency Database level – fewer DBs may slow response High availability – more dependencies creates more points of failure Functional Services Data Services Data Bottlenecks? App Data App Data App Data DB Mass-market J2EE and.NET servers do not solve these problems!

8 8 The Pessimist's View of SOA Looser coupling of commonly performed tasks… –But, tighter consistency for commonly used data Reuse at long last through shared services… –But, lengthier development time for shared services Eliminates tyranny of silos… –But, lose application boundaries Everything just works… –But, nothing ever works as advertised

9 9 Platform 6 Platform 7 Platform 8 Platform 9 Platform 10 Platform 1 Platform 2 Platform 3 Platform 4 Platform 5 Case Study: Sell-Side Bank Real-Time Trading Applications Application 1 Application 2 Application 3 Application 4 Application 5 Complex Legacy Data Critical Data Spread Across 10 Platforms Consequences: Data mapping accounts for up to 50% of each app Each new application adds to data complexity Developer Re-Invents Custom Data Logic for Each App

10 The Iceberg Model For SOA SOA Strengths Loose task coupling Reuse of shared tasks Eliminate silos Messaging Services Data Services Functional Services Legacy Environment SOA Data Gotchas Data consistency Data bottlenecks Data availability

11 Presentation Overview 1. SOA requires underlying enterprise data architecture that provides consistent, timely data 2. Data virtualization ensures data consistency 3. Data services meet performance, scalability, availability needs for SOA 4. Case studies demonstrate benefits of well thought-out data architecture for SOA

12 Typical Data Consistency Challenges Many replicated data copies banks have on average 43 separate counterparty databases, and 37 systems with securities data Inconsistent data causes errors banks attribute 45% of automated trade failures are due to inaccurate reference data Source: Tower Group, 2003

13 Duplicate Data Everywhere

14 Data Integration Approaches Enterprise database: use existing RDBMS to access heterogeneous sources (IBM Information Integrator) Data warehouse: real-time ETL capabilities blue lines between types of databases (Sybase IQ) Enterprise Information Integration: new class of data servers with specialized join query (Metamatrix) O-R Mapping tools: complements other approaches, can integrate a few (3-5) data sources (Persistence)

15 Data Integration Alternatives Enterprise Database EII Technology Strengths Use if… O-R Mapping Data cleansing/transformation Complex, on-the-fly analysis Automates data layer development Ensures applications access data consistently Integrates with corporate standard database Join across multiple sources Need complex transformations and queries Building custom business apps Need read/write data access Data sources are homogeneous R-T Data Warehouse Meta-data management Store any object as opaque (serialized) structure Many heterogeneous data sources Few updates

16 Platform 6 Platform 7 Platform 8 Platform 9 Platform 10 Platform 1 Platform 2 Platform 3 Platform 4 Platform 5 Case Study: Sell-Side Bank Real-Time Trading Applications Application 1 Application 2 Application 3 Application 4 Application 5 Complex Legacy Data Critical Data Spread Across 10 Platforms Consequences: Data mapping accounts for up to 50% of each app Each new application adds to data complexity Developer Re-Invents Custom Data Logic for Each App Data Services Layer Single source of truth Consistent data model across applications

17 Presentation Overview 1. SOA requires underlying enterprise data architecture that provides consistent, timely data 2. Data virtualization ensures data consistency 3. Data services meet performance, scalability, availability needs for SOA 4. Case studies demonstrate benefits of well thought-out data architecture for SOA

18 Reasons For SOA Performance Fears Met User Performance Requirements? (% of all J2EE apps) Source of Bottleneck? Inefficient Data Access Other Bottleneck Yes No 57% 43% Source: 2003 Wily J2EE Benchmark Survey (n=360) 60% 40%

19 Data Services Eliminate Bottlenecks Caching Eliminates Data Bottlenecks Application level – data cache eliminates wait Network level – caching reduces network traffic Database– caching reduces query volumes HA – replication ensures no single point of failure Distributed caching eliminates bottlenecks between applications and databases DB App Cache Data Services Distributed Caching O-R Mapping Replication Functional Services DB

20 Requirements For Data Services DB App Cache Data Services Distributed Caching O-R Mapping Replication Functional Services DB Data Caching Services: stage data with app for performance and scalability Data Replication Services: position data for distributed computing, high availability Data Mapping Services: native language bindings for optimal performance

21 When To Worry: The 50/50 Rule Object Model 50+ classes < 50 classes Request Rate (Peak transactions/sec) < 50 TPS50+ TPS Data- intensive applications Model- intensive applications Transaction- intensive applications Basic applications Requires intelligent caching Requires data services layer Requires model-driven O/R mapping

22 Data Services Stack DB 1DB 3DB 2 Performance –caching improves response time Scalability –cache replication enables scaling Availability–reliable sync improves app resilience Data Integration/EII Virtual Database C# AppJava AppC++ App Compute Grid Distributed Execution Cache Data Services Distributed Caching

23 Presentation Overview 1. SOA requires underlying enterprise data architecture that provides consistent, timely data 2. Data virtualization ensures data consistency 3. Data services meet performance, scalability, availability needs for SOA 4. Case studies demonstrate benefits of well thought-out data architecture for SOA

24 Data Services Product Example - EdgeXtend Linux NT Unix Persistence EdgeXtend Data Services O-R Mapping Replication C# AppJava AppC++ App Compute Grid Distributed Execution Cache Oracle DB/2Sybase

25 Case Study (Cont): Sell-Side Bank Business Requirements Project requirements –Front & middle office equity trading: support >40 global apps –High transaction volumes: >5,000 TPS, millions per day –High availability: maximum downtime from failure <30 seconds –High scalability: support 5x volume at minimal cost –Reference data bottlenecks: all applications share common reference & order book data = huge potential for bottleneck Deployment architecture –Service Oriented Architecture: trading tasks exposed as shared functional services –Persistence Data Services: Java binding, replication, caching –Grid Deployment: Unix servers (>100 CPUs), multi-site (US, Europe, Asia), Tibco RV, MQ, Sybase

26 NY Order Service Reporting Service NY Exchange Service Case Study (cont): Data Architecture A-LS-ZM-R Persistence Data Services Distributed Caching Counterparty Service Securities Service Counterparty Service Securities Service Counterparty Service Securities Service Order Book Service NJ Partitioned databases O/R Mapping Caching Replication App examples Trading desk Auto-exec engine VWAP Sybase Vendor Feeds Reuters Bloomberg Validation Workflow Extract Transform Data cleanse Change mgmt

27 Case Study (cont): Benefits Achieved Scalability: grid data services infrastructure scaled to $7B/day in trades (mainframe savings > $4m/yr) Availability: stateful failover between grid data services caches helped cut failover time from 5 min to 30 sec Productivity: SOA delivered 50% productivity through service reuse, required up-front resource (~30% of team) Grid Data Services: distributed caching required to grid enable stateful SOA services to run in compute grid

28 NetJets Case Study Plane Crew Pilot Limo CatererTime Customer Weather Business Goal RequirementsRequirements Results With SOA Gain Competitive Advantage Through Improved Customer Service Enterprise Data Layer Across 20 Flight Mgmt Systems –WW Flight Data Synched –Saved 30 Engineering Years vs. Piece Parts –Will Deploy in Europe and Asia Without Additional DBs Accuracy – 300 Flight Management and Customer Tables Real-Time Response – Customer Schedule Changes Including 2,000 Pilots World-Wide Scalable, flexible, reliable 24x7

29 Complex object model

30 J2EE Server IntelliJet2 Logical Architecture Persistence Data Services (Entity Beans) Functional Services (Session Beans) Visual Basic.NET SOAP Java Portal JSR 168 Oracle 103,000 function points 300 Entity Beans (implement Data Services) 100 Session Beans (implement Functional Services) 21Gb collected in 6 months 500 concurrent user 30 developers 3 years of development

31 NetJets Results Best customer service in the industry – NetJets now market leader Common Data Services layer reduced development task for each new module by 30-50% Common Functional Services have enabled more agile development cycle = competitive gap is widening Independent consulting firm estimated Netjets would need 15 engineers to maintain system – with SOA and MDA, they are doing maintenance with just 5 people!

32 ROI For SOA 2x Developer productivity: after 2 years, shared services should account for > 50% of new application functionality 3x maintenance productivity: mission-critical system deployed using SOA can be maintained with 75% fewer resources Grid deployment savings: virtualization of data and functional services enables distributed deployment to low cost grid computers, with 40% capital cost savings, 30% annual operating cost savings

33 SOA Data Architecture Roadmap 1. Start with data virtualization: create golden master data 2. Add data services: provide consistent language bindings, distributed caching 3. Migrate functionality to SOA: plan to invest 30% of dev resources into shared services Consolidate SW infrastructure: eliminate silos, DBs (2+yrs)

34 The Iceberg Model For SOA Messaging Web Services MQ, JMS, CORBA Messaging Services Data Services Functional Services Legacy Environment Functional Services Java, J2EE,.NET Grid deployment flexibility Data Services Data virtualization O-R mapping Distributed caching

35 Q&A