Presentation is loading. Please wait.

Presentation is loading. Please wait.

IBM Data Virtualization within Cloud Pak for Data July 25, Mukta Singh

Similar presentations


Presentation on theme: "IBM Data Virtualization within Cloud Pak for Data July 25, Mukta Singh"— Presentation transcript:

1 IBM Data Virtualization within Cloud Pak for Data July 25, 2019 Mukta Singh
© 2019 IBM Corporation

2 A Modern Information Architecture
Data Consumers Web Services Mobile Apps Impact of modern Information Architecture Applications Analytics Tools Portals – One data fabric, on a cloud native architecture Governed Enterprise Data Catalog (Metadata) Consumer Layer (Interface Provisioning) Reduce data copies and movement Enable self-service for data Data Virtualization Virtualization Layer Connection Layer (Adaptors) Hadoop Data Data Marts Etc Warehouses Data Sources Ability to connect to data sources globally For the Data Consumer layer data were inside a single database

3 Unified platform of foundational Data and AI cloud services
Built for Multi-cloud Avoid vendor lock-in and get started on your cloud journey today Customize and Extend with add-on microservices PICK YOUR ADD-ON Containerized Services Db2 Db2 ES Mongo Cognos DSP ISV OS Custom Unified platform of foundational Data and AI cloud services IBM Cloud Cloud Pak DATA PLATFORM #1 Ranked by Forrester KUBERNETES BASED Containerized, easy to manage ICP for Data Platform : Core Cloud-native Data platform Services with Configurable Installation Options Infrastructure options : On-premise or any public cloud of your choice Cloud Pak PICK YOUR CLOUD Private or Public On-Premise Credit goes to Clay Davis

4 … Cloud Pak for Data– What’s included ?
Open, Extensible API Platform Personalized, Collaborative Team Platform App Developers Business Partners Data Engineers Data Stewards Data Scientists Business Users Extensible “add-ons” from IBM’s Data and AI Portfolio and 3rd Party Services IBM IBM IBM IBM IBM ISV OS Custom The Ladder to AI Collect Data Organize Data Analyze Data Data virtualization Data warehousing Databases on-demand Data source ingestion Distributed processing Discovery and search Data transformation Data cataloging Business glossary Policies, rules and privacy AI Machine learning learning AI Model build and deploy AI Model management Dashboards and reporting Analytical visualization When clients start with IBM Cloud Private for Data, they get the foundational capabilities for the COLLECT, ORGANIZE and ANALYZE rungs on the AI Ladder, all integrated via the flexibility of cloud-native microservices. They can access data, create policy-based catalogs, build open source AI/ML models, visualize data with dashboards, and more.  You can think of these capabilities as cloud-native microservice versions of our core offerings (like Db2, InfoSphere, DataStage, Watson Studio, and Cognos Analytics), integrated into a platform. No assembly required. Powered by: Db2 Family Powered by: InfoSphere Powered by: Watson and Cognos Multicloud Services Logging Monitoring Metering Persistent Storage Kubernetes Security Identity Access Mgmt. Docker Registry / Helm

5 What if you could query 2 or 2,000 data systems with a single query?
What if you could tap into all of your critical data assets no matter where they physically are? What if you could query 2 or 2,000 data systems with a single query?

6 IBM Data Virtualization
Query across multiple data sources: Oracle, Db2, SQLServer, Informix, Netezza, MySQL, PostgreSQL, Big SQL, Apache Hive, HDP Hive, Cloudera Impala and more! Rich application capabilities Connect to Data Virtualization with your favorite SQL apps and tools RStudio, Jupyter Notebook, Cognos, Tableau, Microstrategy Scale! 1 or 1,000 at once More than 10x better for several important use cases Secure! Strict access controls Fully encrypted communications. Schema discovery and folding Automatically find and match tables across systems so you can query them as a single virtual table. Deeply integrated w Cloud Pak for Data Enterprise Data Catalog, governance and security. e.g. Automatic publishing of virtualized data into the data catalog. Immediate access via Cognos and Watson Studio

7 Data virtualization key Use Cases
Driven by patterns needing low data latency, high flexibility with transient schema EDW prototyping and migration (mergers/acquisitions) Virtualization with Big Data (Hadoop, NoSQL, and Data Science) EDW augmentation (offload workload) On-demand Virtual Data Marts saves on cost of standing up EDW Data discovery for ”what if” scenarios across hybrid platforms Data Caching of combined data for frequently accessed data Combine MDM with IoT for Systems of Insight (IT/OT) Data integration preparation tool to complement ETL Master data hub extension to enrich 360 View (e.g. multi-channel CRM) single View across your business real-time analytics without moving data fast TTV with high ROI

8 Governed Data Virtualization in Cloud Pak for Data
The ability to view, access, manipulate and analyze data without the need to know or understand its physical format or location, and without having to move or copy it. Portals Analytics Tools Web Services Applications Mobile Apps Consumers Governance Catalog (Metadata) Consumer Layer (Interface Provisioning) Business Terms Data Privacy & Protection Rules Lineage Virtualization Layer Virtualization Platform Caching & Optimization Connection Layer (Adaptors) Lakes NoSQL Warehouses Files Marts Web Services Data Sources Cloud Applications

9 Our Data Catalog Strategy
One Scalable Catalog Available Multi-Cloud Embeddable Catalog Micro- Services Open Metadata Persona Specific Experiences With a single roadmap IBM’s catalog capabilities will lead the market At a high level our combined catalog strategy aligns with 4 key tenets: 1 – A single catalog of ML based capabilities serving metadata management and data catalog use cases across the public and private clouds and on premise. 2 – Commoditization of base catalog capabilities that will be embedded across IBM products. Anyone buying certain IBM products will get a rich catalog for free. IBM will offer value add features on top at a premium that differentiate us in the market 3 – Open metadata will provide the foundation for integration. ODPi interfaces will ensure the IBM platform is open and can be easily integrated with. Proprietary metadata stores are a challenge for our clients 4 – Persona specific UI’s will ensure that each persona that needs to create/manage/govern/consume metadata can do so in an experience that is fit for their purpose. And IT user requires different views of metadata then a knowledge worker. We will ensure this is continued to be enforced in the experiences. © 2019 IBM Corporation

10 Accelerate with Industry Content
IBM Analytics capabilities support the shift from collection to consumption Data Architects Data Engineers Data Stewards Knowledge Worker Govern the Data Activate the Data Data Collection Discover Understand Consume Prepare Data Science Define Business Glossary, Policies & Rules Connect & Access Data Classify Data Profile Data Assess Quality Curate & Catalog Search & Find Relevant Data Prepare Data for Analysis Build and Train ML/DL Models If we look across the typical Data Management ecosystem we see how the capabilities that we all know and love fit across the personas that are typically involved with the process. The lighter boxes on the left are where our clients have typically invested to date. Extremely important business critical capabilities that are required to collect, govern and manage data. The right hand side contains newer areas that are required to put information in the hands of the knowledge worker and allow them to to do something useful with the data in an efficient manner. All capabilities operate against common metadata that flows through the organization but surfaces up different capabilities through targeted experiences to each persona. Both the left hand side and the right hand side are growing, but its fair to say that we observe a much higher growth and level of interest in the features on the right hand side as clients look to embrace AI and there is a growing need to understand how to monetize data. Data Monetization Accelerate with Industry Content

11 Data Virtualization (DV) & Data Governance – with Policy Enforcement
Watson Knowledge Catalog Data Lake Repositories Optional: Policy Enforcement via Ranger / Guardium Data Virtualization Services DV Data Access Engine (SQL, APIs, NLQ) Knowledge Catalog Data Assets Details Policies & Rules Lineage …. Compliance Reporting Discovery & Exploration Self-Services Analytics BI Reporting, Dashboard Business Applications AI, ML & Optimization Relational NoSQL Hadoop Object Store Flat Files Data Integration Deny access Mask/ De-Identify Data Movement Data Replication Mask / De-Identify When a new DV Table is created DV Table is published to catalog Lineage of DV table is generated and published to catalog (what tables make up the new DV table) Data Profile/Discovery/Classification is triggered --> Data Curator Notified. If any personal data found  Automatic assign a PII flag for DV table When a DV Table is accessed DV services call Policy Enforcement Services for any required action (Mask Data, Deny Accesss, etc.) Policy Engine Logs CRM IOT Data Front Office Transactional Policy Enforcement Policy Enforcement

12 Key Architectural Differentiation
Hub and spoke execution models: Lacks scalability Performance constrained Basis for Federation and our competitors IBM is first to market with a parallel processing model: Theoretically unlimited scalability Ease of addition/removal of sources Execution pushed down into the constellation mesh Hub and Spoke Model coordinator Query Query Result Query issued against the system A coordinator receives the request and fans the work out to edge nodes Edge nodes individually perform as much work as they can based on their own data. Individual results are sent back to the coordinator for final merging and remaining analytics. Coordinator receives intermediary results from all edge nodes, merges results, and performs remaining analytics New Computational Mesh coordinator Query Query Result Query issued against the system A coordinator receives the request and fans the work out to edge nodes Edge nodes self organize into a constellation where they can communicate with a small number of peers. Nodes collaborate to perform almost all analytics, not only analytics on their own data. Coordinator receives mostly finalized results from just a fraction of nodes. Completes the final work for the query result. © 2019 IBM Corporation

13 Smart Query Processing
IBM Data Virtualization reduces query time by using Parallel Processing, Pushdown Optimization and Connection Pooling Read total quantity sold in Hawaii sold in California sold in New York Total quantity sold across 3 states Order by quantity, return top 10 products Query Results in 1/3 the time! Timeline Data Virtualization Others Using harden technology from IBM’s Enterprise Database team, with advancements from IBM Research, our Data Virtualization (DV) engine is able to rapidly deliver query results from multiple Data Sources by leveraging advanced parallel processing and pushdown optimization Not only is Data Virtualization (DV) fast, DV removes complexity when querying information from multiple Data Sources. Rapidly query, consolidate, sort and report information from a handful to thousands of unique Data Sources (DS) using one SQL statement that uses Virtual Table(s) (VT). Each VT refers to one or more actual Data Sources and Tables. Queries can combine data from a multiplicity of sources including relational databases, spreadsheets, and flat files Read total quantity sold in Hawaii Read total quantity sold in California Read total quantity sold in New York Total quantity sold across 3 states Order by quantity, return top 10 products © 2019 IBM Corporation IBM Analytics © 2018 IBM Corporation

14 Supported Data Sources
Cloud Pak for Data In the roadmap Db2 family for HDM Db2 for iSeries, zSeries Db2 for z/OS Big SQL IIAS, PDA (Netezza) Informix Derby Oracle SQL Server MySQL PostgreSQL Apache Hive, HDP Hive Cloudera Impala Teradata* MongoDB Hive Excel, CSV, Text* Sybase Snowflake IBM DVM for Z SAP HANA MAP-R Apache Drill SAS MariaDB Amazon Redshift , Dynamo, Aurora Support for any Generic JDBC driver VSAM, IMS, CICS, Adabas CouchDB Stream / MQ Apache Spark SQL Apache Kafka Cloudant Pivotal Greenplum Cassandra Clients Ask - Employer - Mongo – Atlas service ( AWS) Postgres ( AWS) Oracle © 2019 IBM Corporation *Note - manual upload of file /driver needed Note – REST APIs planned

15 Details of our Design

16 So …what is Data Virtualization?
The ability to view, access, manipulate and analyze data without the need to know or understand its physical format or location.

17 Data Virtualization The mission …
Provide simplicity and rich analytics capabilities when virtualizing over large numbers of data sources, flexible schema & data types, automated data & results caching, and compute scaling with industry leading performance. 

18 Going beyond with Simplicity and Scalability
Adding data sources is easy. Many sources can be automatically discovered so you only need to provide credentials Discovering data in the sources is automated. No need to manually import the meta data to Data Virtualization Automatically match similar table structures across data sources. Scaling for the Enterprise Supporting large number of systems. > 20 sources with only the DV service > 1000 sources with optional data source connectors Geographically dispersed sources Multiple remote sources are able to work together for efficient analytics and processing. Data source connectors provide resiliency and performance across high latency and unstable networks.

19 Giving you Confidence in the Analytics
Governance Provides data classification and quality metrics for virtualized data from any source. Data provenance and lineage to show how each source contributes to the analytics. Security Data lives in the original source with only results ever being transmitted. Fully encrypted communications between all components of the solution. Strong authentication and permission management. Policy enforcement and masking capabilities to come.

20 Remote Source Connector
Client Applications Common SQL Engine MS SQL Server Netezza (PDA) Oracle Remote Agent Adjacent to data Sybase SparkSQL Administration UI S3… Information Governance (existing at remote Data Source) Governance included within CP4D Network OPTIONAL installation of Remote Agent Architecture

21 Data Virtualization Query processing
SQL Queries received from client applications Db2 SQL engine used for parsing and optimization DV Virtual Tables are represented as Federated Nicknames with additional attributes Federation pushdown analysis and statement generation is used to build remote statements, both for the distributed DV processing as well as optimized statements targeted to particular remote sources. Special Db2 optimizations exist for driving distributed aggregation, remote join filters. Remote query contains a lot of annotations Distributed agg parameters Join key filter build and apply information Source Parallelism Column and type massaging

22 DV Engine Remote Statement
After remote statement generation, ordinary Db2 Runtime Section is built. Runtime section represents Db2 processing with individual subsections representing DPF node processing It may contain one more invocations of different remote statements. Each remote statement will be an independent thread executing concurrently in the distributed runtime. Each Remote statement may contain references to one or more remote tables or nested query operations. Driving further parallelism as additional threads are used for each construct. At the data source level each source is accessed in parallel. Another item for August time frame is to further parallelize individual source access.

23 Deploying Data Virtualization
Fully containerized BigSQL deployment in Docker and Kubernetes. BigSQL Workers, aka Db2 DPF partitions, can be dynamically added to service without REORG / Redistribute. Even ”local” data such as cache data is stored in HDFS and not local Db2 tablespaces Operate as the external interface for Data Virtualization Broad programming/query language support Built-in Federation capabilities with integrated deployment and configuration Query optimization and plan generation Management and Configuration APIs

24 Creating the Constellation
Small packet of Data Virtualization software deployed on each data node, approximately 50MB. Automatic dynamic and resilient organization of data sources. Query compiler builds a collaborative query plan, forcing the nodes to collaborate 11/24/2020

25 Real System test Growing a Constellation
Video of constellation growing to 349 Nodes. Network stays compact. 2 and 10 links between nodes No manual configuration. Latency aware connection between nodes Which nodes connect to which others? Fastest reply strategy Diameter of the constellation (i.e. the number of hops between the two furthest nodes) grows logarithmically. Small diameter is ideal for communications. Actual system test performed by Emerging Technology Services, IBM Hursley, United Kingdom

26 Autonomic Result & Data Caching (Target 2019)
A critical capability of Data Virtualization is result and data caching. Avoid the latency of remote data source queries for commonly used subsets of the data. Caches have to be fully automatic. The administrative overhead of manually defining the caches to use is too high and breaks our goal of simplicity. Result cache design points Result data is cached from the same remote query used to execute a query as part of the DV-FMP processing. Most likely cached into Hadoop from the edge cluster nodes Information about high value queries to cache will be based on the Db2 Package cache and remote statement execution logs. May require query recompile to make use of the cached results for MQT matching

27 Bringing parallelism to each data source.
Traditional processing queries the data source (on right) then processes large results on a single thread (executor) on left. Solution: Data Virtualization queries data source (on right). Merges results in many parallel threads on left by leveraging parallel read streams. Data Source Single channel is bottleneck for any large result set. Data Virtualization resolves this through parallel read from sources. 64 core

28 Efficient Cross Source Joins
Oracle Marketing db 1MB Problem: Joining data between multiple databases is a common pattern, but brutally slow. With traditional processing, data from both tables is shipped over the network to the server, where the join is processed. Solution: Data Virtualization uses the early filtering techniques to dramatically reduce transmission costs. Data from the smaller table (inner) is used to build a small filter query that is applied to the larger (outer) table before data is transmitted. In many cases this is 85% effective at filtering data that will be removed by the join before the join is processed, leading commonly to a ~10x acceleration. Db2 Sales db 50GB 64 core © 2019 IBM Corporation

29 20 databases? Traditional processing gets worse as the number of databases increases
Other Informix Db2 Data Sources Oracle 64 core Simultaneous processing and parallel fetch continues to scale up. © 2019 IBM Corporation

30 DV Internal Processing

31 DV Engine Generated SQL for Distribute Aggregation
Original SELECT COUNT(PROMOVALUE2) FROM PROMOTION Remote SQL SELECT SUM( A0.C0) FROM ( SELECT A1.C0 C0 FROM new com.ibm.db2j.GaianQuery( 'SELECT COUNT( A2."PROMOVALUE2") C0 FROM new com.ibm.db2j.GaianTable( ''PROMOTION'', ''SOURCELIST=(MYSQL10000:"POPS_node1", MYSQL10001:"POPS_node2", MYSQL10002:"POPS_node3", MYSQL10003:"POPS_node4", MYSQL10004:"POPS_node5")'', ''"PROMOKEY" INTEGER, "PROMOTYPE" INTEGER, "PROMODESC" CHAR(30), "PROMOVALUE" DECIMAL(5, 2), "PROMOVALUE2" DECIMAL(5, 2),"PROMO_COST" DECIMAL(9, 2)'' ) A2', 'GAIAN_EXTENDER=[pAgg]','', 'C0 DECIMAL(5, 2)') A1 ) A0

32 Query Processing in the Constellation
Toronto San Hose United Kingdom Toronto Fixed execution within the constellation is impossible because of the highly dynamic nature of the network. Each node instead simultaneously sends the relevant portions of the query to both the connected data source to it’s peers in the network. Combines and process the results as they are received. Duplicate results are avoided by a given node only returning results to the first peer that requested them. Implicitly results in balanced processing of the query through the constellation.

33 Language Translation in Data Virtualization
Broad set of data sources supported by Data Virtualization each with unique syntax variations. Constellation is not limited only a single data source type. A logical schema is created across all connected sources. Multiple levels of translation as we move from the applications through the constellation down to the data source. User Application JDBC, ODBC, R, Python, etc DV Service Embedded Db2 Supports major language variations, SQL, Oracle SQL, Netezza, etc DV Remote Connector Translates from received SQL to data source dialect. Compensate for missing functionality. Remote Data source Db2, Oracle, MySQL, Excel, CSV, etc.

34 Aggregation and Joins in DV
Aggregation is one of the major data reduction operations that can be distributed. DV uses the fundamental concepts of distribution aggregation, namely the partial, intermediate and final phases. Each endpoint will push the partial phase into the remote data source or perform it locally in the case of flat files or non-relational sources. Each endpoint will further perform intermediate aggregation combining the result from the remote data sources as well as adjacent nodes in the network. Data flowing over any link is as fully aggregated as possible at the given point. Fully distributed joins in DV is still under development and will include several aspects Distributed join key filters. Utilize the structure of the constellation to allow the filter to be built and used in a single traversal through the constellation. Pushed into the remote data sources as basic SQL predicates Application of the partial-intermediate-final techniques from aggregation to a join model. Most useful when the operands of a join have too low a join key cardinality for join key filters to be effective. Partial aggregation pushdown through Joins Similar concept to PEA in Db2.

35 DV Engine – Parallel Processing DV-FMP – Constellation – Data Source
Parallel processing of query results is a key element of performance for remote system. Goals: Parallel data source queries, parallel network transmission. Query Processing Query is compiled as normal through the Db2 Query Compiler and the remote statement(s) are generated. One component of the remote query syntax is the degree of parallelism to use when accessing the Data Virtualization constellation and the remote sources. Initially expected to correspond to the number of BigSQL Edge Cluster Nodes associated with DV. Each edge cluster node receives the same remote statement to execute in the subsection and invokes a thread within each DV-FMP for processing. Propagation of the query propagates through the constellation as normal. Endpoint agents with a relevant data source connection will receive the remote query from multiple paths and will understand the appropriate parallelism parameter. Each connection will access a unique range of the data based upon local indexes and statistics. Future may use hash partitioning or other partitioning methods. This ensures each connection path from data source to controlling DV-FMP will receive an approximately equal subset of the result data

36 DV Engine Multisource Distributed Join Key Filters
Performant Multisource Joins is a component enabling comprehensive distributed processing within the constellation. Goal: Provide early filtering of result data at source. Query processing Based on Db2 Access plan, remote statements will have annotations for building and application of distributed and remote join filters. Build phase happens in parallel at all sources providing components of the first argument of the join. This does not require a separate scan of the data. Filter is distributed asynchronously and in parallel within the constellation to endpoints with an instance of the second argument of the join. Filter is incorporated into the remote query for the execution with the data source of the second argument of the join. Such remote data source query is available to begin processing as soon as the filter is received.

37 Summary

38 IBM’s Unique Approach to Data Virtualization
Parallel processing mesh providing execution performance and scalability: Quickly deliver analytics results and easily evolve with new data source demands Central Node scalability and Enterprise robustness (via an HA cluster): Reliability and ability to quickly adapt to increasing business demand Governance Integration and Security: Controlled, managed and secure access to virtual data Common SQL engine, rich set of SQL dialects, application portability Retain the use of existing applications and tools within the business Richness of automation and discovery of Data: Enable more self service and increase productivity (while retaining access control) Underlying platform flexibility with Cloud Pak for Data: Grow or move your Analytics and Virtualization platform with any environment Service Cluster Constellation Caching Policy Data Source Node Analytics Application © 2019 IBM Corporation

39 Increase Productivity
IBM Cloud Pak for Data Shift to next-gen workloads: Modernize through microservices Reduce Costs Ship Faster Increase Productivity Run your data & analytics services more efficiently – single product stack and better resource utilization Increase agility and respond to changing business requirements through DevOps and continuous delivery Enable governed self-service, collaboration, and provisioning, across teams on an integrated platform Shift from Monolithic to Microservices for agility and efficiency Another top use case is obvious given we’ve built this integrated platform on private cloud – “Modernize your Data & Analytics Workloads”. Shift from monolithic to microservices for agility and efficiency. Reduce costs by running your data & analytics services more efficiently – single product stack and better resource utilization Ship faster increase agility and respond to changing business requirements through DevOps and continuous delivery Increase workforce productivity through governed self-service, collaboration, and provisioning across teams on an integrated platform ICP for Data and IBM Z/ © 2019 IBM Corporation

40 © 2019 IBM Corporation


Download ppt "IBM Data Virtualization within Cloud Pak for Data July 25, Mukta Singh"

Similar presentations


Ads by Google