Scalable Data Management with DB2

Scalable Data Management with DB2
This presentation describes the technology capabilities of InfoSphere Warehouse software. Matthias Nicola IBM Silicon Valley Lab

Agenda Introduction DB2 Scalability for OLTP and Data Warehousing
DB2's Database Partitioning Feature (DPF) Overview Data partitioning, clustering, placement Join Methods TPoX Scalability in a DPF database Scalability vs. Performance Benchmark configuration & results pureScale Overview Summary

DB2 Data Server Editions
DB2 for z/OS DB2 DB2 Enterprise Edition / IBM InfoSphere Warehouse DB2 Workgroup Edition DB2 Express-C (free!) DB2 Everyplace

Business Value of Scalability
More historical data = more precise forecasts Data mining needs a lot of data for pattern accuracy OLAP needs a lot of data for forecast accuracy Predictable costs when growth occurs Often the budget is the controlling factor, not technology Low maintenance cost is important No forced migrations from technology limitations Enabling very large databases There are many attributes to scalability, but the most common one is data growth and the ability to support large volumes of data. This type of scalability is important for a number of reasons, some of which are noted on this slide. For example, having more historical data can help improve the accuracy of the analysis and forecasting that customers want do to. When the system scales in a predictable manner, I/T organizations can more easily plan for system upgrades and the inevitable growth that occurs when mergers and acquisitions happen. And the customer does not have to worry about hitting a ceiling and having to switch technology platforms because the current platform is unable to scale.

DB2 Scalability for OLTP and Data Warehousing
Database Partitioning Feature (DPF) DB2 pureScale Range partitioning Multi-Dimensional Clustering (MDC) Compression Self-Tuning Memory Management (STMM) Automatic Storage Workload Management High Availability, Disaster Recovery Recovery Security and Compliance Utilities: Load, Backup & Restore, Redistribute Archiving etc.

DB2's Database Partitioning Feature (DPF)
select … from table Tables FCM network Database … Partition n data+log Engine Partition 3 Partition 2 Partition 1 With DPF, the database is split into multiple partitions, each of which owns it’s own portion of the database and has dedicated resources. These partitions exist within a server or across servers, enabling a parallel database within a single computer or across a cluster of computers. To the application, however, it still appears as a single system. Note that there is no need for a distributed lock manager, since the data in each partition can only be updated by that partition. This enables linear scalability by adding partitions (along with the associated CPU, I/O and memory). Database is divided into multiple database partitions Database partitions run on same or separate servers (shared-nothing) Each partition has its own table spaces, log, configuration, etc. Data is spread over N database partitions Queries are executed in parallel on all database partitions

Flexible configuration options
Possible hardware configurations All database partitions on a single machine (logical partitions)  easy exploitation of multi-core systems All database partitions on separate machines (physical partitions) Hybrid: multiple machines with several logical partitions on each FCM (Fast Communication Manager) Storage server I/O Channels SMP server DB2 Partition Storage server I/O Channels SMP server DB2 Partition The shared nothing architecture of DB2 maps directly to the shared nothing hardware architecture of the IBM InfoSphere Balanced Warehouse, or the new IBM Smart Analytics system. These complete data warehouse offerings include the necessary IBM servers, storage, and networking to deliver a solution ready to create tables and load data. These offerings are based on one or more data servers, that contain the DB2 partitions that make up the database. When system growth is required to support more data, more servers and therefore DB2 partitions are added to scale out the system. Animation on the slide shows this. Also, when more users need to access the system then additional coordinator partitions can be added – these are called User Modules. The coordinator partition (User Module) is where the user connects to, and assigns work to the other partitions. So the system is scalable to support additional data and users. Example: 4 physical machines, 2 database partitions per machine

DB2's Database Partitioning Feature (DPF)
…. ….

The Distribution Map Distribution key can consist of one or multiple columns. Avoid low cardinality columns, such as "gender", "state", etc. Unique indexes must contain all columns of the distribution key Distribution key column name column value C1 000120 DB2 hash algorithm 5 Distribution map 1 2 3 4 5 6 7 … 32k i p(i) DB2 uses a partition map (distribution map) to automatically distribute the data, as shown in this slide. The value of the “partitioning key” ( EMPNO in this example) is passed through a hashing algorithm, resulting in a value between 0 and 32k, which then corresponds to a data partition. The row is then added to the corresponding partition. db2GetDistMap API: allows client applications to obtain the distribution map for a table Partition1 Partition2 Partition3 Partition4

Single Server The key to fast DB2 performance is in reducing the I/O in the data warehouse workload. Take for example a traditional non-DB2 SMP solution with one large server connected to a large storage array. By default if I want to find the sum of all sales for a certain product in the month of February, I would have to either rely on index keys to find those products but even in that case the amount of data can be so large that a brute force table scan may be the best access path. Oracle and others try to solve this problem by performing “big block I/O” whereby they try to read large quantities of data pages in one I/O in order to improve performance. However the problem is still the I/O throughput is not sufficient. Take for example the above example where the red triangles is the records I am looking for. Because they are scattered throughout the table I need to scan every page in a sequential fashion in order to find the rows I’m looking for. This is very inefficient. Animation on the slide shows this.

DB2 Database Partitioning Feature = Divide Work
Using DB2’s database partitioning feature (DPF) you can divide the database into a set of smaller partitions where each database partition is managing a subset of the entire database (and therefore a subset of the table). Furthermore, each partition is on a separate server so there is 3 times the processing power. By doing this, the scan in the previous example now runs 3 times as fast because each server is only scanning 1/3rd of the data. This is done using hash-partitioning to evenly distribute the data across the data partitions.

Range Partitioning Further Reduces I/O
Database Partition 1 Database Partition 2 Database Partition 3 January CREATE TABLE sales (recordID INT, salesdate DATE, ... details XML) DISTRIBUTE BY HASH (recordID) PARTITION BY RANGE (salesdate) EVERY 1 MONTHS ; February Now in addition to hash partitioning, you can also add table partitioning (by range). This allows the table to be broken down by month (for example) so that the amount of data being scanned is again reduced. Taking the previous example, we can now scan only 1/3rd of the data because we are only looking for February sales. By combining DPF and table partitioning, there is less data returned to the servers and all servers are processing that smaller amount in parallel. March

Multi-Dimensional Clustering to Further Reduce I/O
Database Partition 1 Database Partition 2 Database Partition 3 January CREATE TABLE sales (recordID INT, salesdate DATE, productID INTEGER, storeID INTEGER, ... details XML) DISTRIBUTE BY HASH (recordID) PARTITION BY RANGE (salesdate) EVERY 1 MONTHS ORGANIZE BY (productID, storeID) ; February Next we can add Multi Dimensional Clustering into the mix which allows DB2 to store data on disk in an organized fashion such that all the red triangle rows are in the same contiguous set of blocks on disk. DB2 is aware of this layout and will therefore only scan data pages that qualify to the query. In this example we have cut the scan down to only reading ½ of the February data because DB2 knows that it does not need to scan the purple triangle pages. Compared to the original SMP example, each server in this case is only scanning 1/18th of the original data and therefore the results should be returned at least 18 times faster. March

Compression Reduces I/O by a Factor of 3x to 4x
February March January Database Partition 1 Database Partition 2 Database Partition 3 The final piece of the I/O challenge is to store more data in fewer data pages. In DB2 this is accomplished with unique compression capabilities that allow up to 80% compression of the data. DB2 9.7 adds similar compression for indexes and temp tables. By storing more data on fewer pages, DB2 is able to scan those pages up to 4 times faster.

Data Partitioning and Placement Options
Can distribute a table across some or all database partitions. Can replicate a table to have an identical copy on each partition. Database Partitions Part. 1 Part. 2 Part. 3 Part. 4 Part. 5 Part. 6 Part. 7 Part. 8 Table 1: Sales Table 2: Customer Table 3: Product Table 3: Product (copy) Table 3: Product (copy) Table 3: Product (copy) Table 3: Product (copy) Table 3: Product (copy) Table 3: Product (copy) Table 3: Product (copy)

Join Processing - Example
create table tab1(pk1 int, c1 int,...) distribute by hash (pk1); create table tab2(pk2 int, c2 int,...) distribute by hash (pk2); Logical data in the tables: Physical data distribution: database partition 1 database partition 2 tab1 tab2 pk1 c1 1 3 2 3 3 4 7 7 8 12 11 10 12 15 pk2 c2 3 2 4 8 5 3 7 4 8 15 10 10 12 12 15 7 tab1 tab2 tab1 tab2 pk1 c1 1 3 3 4 7 7 11 10 pk2 c2 3 2 5 3 7 4 15 7 pk1 c1 2 3 8 12 12 15 pk2 c2 4 8 8 15 10 10 12 12 distribute by hash* *For simplicity, this example hashes odd key values to partition 1 and even key values to partition 2

Collocated Join create table tab1(pk1 int, c1 int,...) distribute by hash (pk1); create table tab2(pk2 int, c2 int,...) distribute by hash (pk2); select * from tab1, tab2 where tab1.pk1 = tab2.pk2; Both tables are partitioned by the join key Any join matches are guaranteed to be within any given partition ("co-located") No join matches across partitions Allows local joins within each partition, no data movement Best case, best performance partition 1 partition 2 tab1 tab2 tab1 tab2 pk1 1 3 7 11 pk2 3 5 7 15 pk1 2 8 12 pk2 4 8 10 12

Directed Join select * from tab1, tab2 where tab1.c1 = tab2.pk2;
permanent storage on the fly / in memory partition 1 partition 2 pk1 c1 3 4 11 10 8 12 tab1' partition 1 pk2 3 5 7 15 partition 2 4 8 10 12 tab2 1 3 2 3 7 7 12 15 DTQ tab1 tab2 tab1 tab2 pk1 c1 1 3 3 4 7 7 11 10 pk2 3 5 7 15 pk1 c1 2 3 8 12 12 15 pk2 4 8 10 12 Send rows from tab1 to those partitions where they can find join matches in tab2, i.e. redistribution of tab1, based on hashing of the join key c1.

Single Partition Directed Join
select * from tab1, tab2 where tab1.c1 = 3 and tab1.c1 = tab2.pk2; partition 1 partition 2 partition 1 partition 2 tab1 tab2 tab1 tab2 tab1' tab2' pk1 c1 1 3 3 4 7 7 11 10 pk2 3 5 7 15 pk1 c1 2 3 8 12 12 15 pk2 4 8 10 12 pk1 c1 1 3 2 3 pk2 3 DTQ Value predicates are used to optimize (reduce) the data flow and eliminate irrelevant partitions from the join processing.

Repartitioned Join select * from tab1, tab2 where tab1.c1 = tab2.c2;
pk1 c1 1 3 3 4 7 7 11 10 pk2 c2 3 2 5 3 7 4 15 7 pk1 c1 2 3 8 12 12 15 pk2 c2 4 8 8 15 10 10 12 12 pk1 c1 1 3 2 3 7 7 12 15 pk2 c2 5 3 15 7 4 11 8 15 pk1 c1 3 4 11 10 8 12 pk2 c2 3 2 7 4 10 10 12 12 DTQ DTQ Redistribute both tables by hashing on their join keys so that matching rows end up on the same partition.

Broadcast Join select * from tab1, tab2
partition 1 partition 2 partition 1 partition 2 tab1 tab2 tab1 tab2 tab1' tab2 tab1' tab2 pk1 1 3 7 11 pk2 3 5 7 15 pk1 2 8 12 pk2 4 8 10 12 pk1 1 3 7 11 2 8 12 pk2 3 5 7 15 pk1 1 3 7 11 2 8 12 pk2 4 8 10 12 BTQ Broadcast a copy of one table to all database partitions.

Data Placement Option: Replicated Table
permanent storage partition 1 partition 2 tab1 tab2 tab1(copy) tab2 pk1 1 3 7 11 2 8 12 pk2 3 5 7 15 pk1 1 3 7 11 2 8 12 pk2 4 8 10 12 Good choice for small tables with infrequent insert/update/delete activity, such as dimension tables in a star schema.

Scalability vs. Performance
Performance: Time to complete a given task with given resources Scalability: Ability to add resources to complete the same task more quickly handle a bigger task in about the same time Example: Mowing the lawn… Peter does it alone in 8 hours Peter and Bob work together and take 4 hours  Scalability is perfect, performance is poor! Jim does it alone in 1 hour Jim and John together do it in 1hrs20min  Performance is great, scalability is awful ! Mary mows the lawn in 30 minutes Mary and Susan together need 15 minutes  Performance is great, scalability is also great !

two approaches are equivalent….
Scalability Metrics Fixed Database Size Database size & # of partitions Query elapsed time Increasing Database Size Mathematically, these two approaches are equivalent…. Query elapsed time # of partitions Make queries against a DB of a fixed size faster by adding partitions (“speedup”). Amount of data per partition shrinks. Hold response time constant for a growing database by adding partitions in proportion (“scaleup”/"scale-out"). Amount of data per partition remains constant. Basic assumption: Queries executed against a bigger database examine more data

Our Test Design Increasing database size: 250GB / 500GB / 1TB
Increasing number of database partitions Fixed ratio of data volume to number of partitions Show constant query elapsed times to prove scalability Query elapsed time n partitions 250GB n*2 partitions 500GB n*4 partitions 1 TB

TPoX Benchmark TPoX = Transaction Processing over XML Data
Open Source Benchmark: Financial transaction processing scenario: “online brokerage” Realistic test for XML databases Custacc Cu s tAcc.xsd 1 n 1 1 n Cu s tomer Account Holding 1 n n 1 n 1 Order Security For more details on TPoX: This slide shows the main logical data entities: Customers have one or more accounts. For each account, one or more orders are executed. Each order buys or sells shares of exactly one security. A security is a stock, bond or mutual fund. Each account contains one or more holdings. A holding, also called position, is a certain number of shares of a particular security in an account. Each security typically has many orders and holdings across the customers’ accounts. TPoX’s data entities are represented by three XML schemas. Orders are represented using the FIXML 4.4 schema. Then there is one XML document per customer that includes personal data and information about all of his accounts and holdings, i.e. account and holding data are inlined with the customer data (“CustAcc”). This is an intentional design choice to reward technological progress for updates and concurrency control on a sub-document level. A fixed number of security documents represents the vast majority of US-traded stocks, bonds and funds with real ticker symbols, real fund families, etc. Security documents range from 2KB to 9KB because stock and fund descriptions have relatively large text values of variable size. This allows for some text operations despite the data-centric focus of TPoX. CustAcc documents are between 4KB and 20KB in size. Orders are 1KB to 2KB and characterized by many attributes and a high ratio of nodes to data. All TPoX documents contain namespaces. The three document collections are interrelated, e.g. Order documents contain security symbols and account numbers that exist in the Security and the CustAcc documents, respectively. FIXML (41 XS D files) Security.xsd 4 – 20 kb – 2 kb – 9 kb FIXML: Standardized Financial XML Schema for Securities Trading !

CustAcc Order Security Document structures and join relationships ID
Name DateOfBirth Address Phone … Account ID OrignDt TrdDt Acct Side Qty Sym … ID Symbol Name SecurityType SecurityInformation StockInformation Sector Industry Category OutstShares FundInformation FundFamily AssetGroup FixedIncome ExpenseRatio TotalAssets MinInitialInvestment MinSubsequentInvest. Price/LastTrade Ask/Bid 50DayAvg 200DayAvg … CustAcc Order Security ID Currency OpeningDate Balance … Holding Holding… Symbol Name Type Quantity Symbol Name Type Quantity ID Currency OpeningDate Balance … Holding Symbol Name Type Quantity

TPoX Data & Schema create table custacc ( cadoc XML )
FIXML: financial industry XML Schema CustAcc: modeled after a real banking system that uses XML Security: information similar to investment web sites Database schema for a non-DPF DB2 database: create table custacc ( cadoc XML ) create table security ( sdoc XML ) create table order ( odoc XML ) The full DDL script for the database, tables, and indexes in this benchmark is available as open source at: Scale Factor “M”, 1 TB raw data 500M Order documents, 50M CustAcc documents 20,833 Securities, independent of scale factor 3 Simple Tables + XML Indexes

TPoX Database Schema for DPF
- Extract certain XML element values into relational cols as distribution keys - Goal: enable partitioning of both tables by a common key ! custid integer secsym varchar odoc XML custid integer cdoc XML order table (500M rows) custacc table (50M rows)

What is TPoX-DSS*? Decision Support workload on top of the regular XML data of the TPoX benchmark A set of complex SQL/XML queries Includes massive table scans, aggregation, grouping, OLAP functions, etc. Focus on single-user query response time * we might come up with a better name in the near future

Business Questions  Complex SQL/XML Queries
Q1: Popular Securities Find securities that have more shares bought than sold across all orders. List their order quantities grouped by year. Q2: Top 10 Most Popular Trading Weeks, Ranked by Order Volume For each year, find the ten most active weeks and return the buy, sell, and total order volumes for each week. Q3: Average Account Balance of Premiun Customers Calculate the average account balance of all premium customers, grouped by their number of accounts. Q4: Average Balance per Number Of Accounts Calculate the average account balance of all customers, grouped by their number of accounts. Q5: Percentage of buy orders per sector and gender For each stock in a given sector of securities, find the percentage of buy orders placed by male vs. female clients.

Business Questions  Complex SQL/XML Queries
Q6: Max Stock Orders for an Industry List the 20% (or: x%) most expensive orders for customer in a given state and for a given industry (subset of securities). Q7: Order Amounts for Two Major Currencies Calculate the min, max and avg order amount for all orders in a given timeframe grouped by buy/sell for two major currencies. Q8: Order Amounts for All Currencies Calculate the min, max and avg order amount for all orders in a given timeframe grouped by buy/sell and the order’s currency. Q9: Balance per Currency Each account is in a specific currency. Calculate the average account balance for each currency. Q10: Sleeping Customers Find all customers having less than x orders in a given timeframe.

TPoX DSS: Query Characteristics
Tables Characteristics Q1 Popular Securities O, S 2 x XMLTABLE, Group By, Order By Q2 Top 10 Most Popular Trading Weeks O Full scan of all orders, OLAP Function rank() Q3 Average Account Balance of Premiun Customers C Indexed access to premium customers, Group By, Order By Q4 Average Balance per Number Of Accounts Full scan of all customers Q5 Percentage of buy orders per sector and gender C, O, S Aggregation, SQL OLAP Functions, 3 x XMLTABLE, 2 x XMLEXISTS Q6 Max Stock Orders for an Industry 2 x XMLTABLE, 2 x XMLEXISTS Q7 Order Amounts for Two Major Currencies Several predicates, CASE expression Q8 Order Amounts for All Currencies 4 aggregation functions, Group By two XML attributes Q9 Balance per Currency Full scan of all accounts, aggregation and grouping Q10 Sleeping Customers C, O Common table expression All queries available upon request, in SQL/XML notation.

Q5: Percentage of buy orders per sector and gender
SELECT DISTINCT secsector, gender, SUM(ordqty) OVER (PARTITION BY secsector, gender) AS orderqty, SUM(ordqty) OVER (PARTITION BY secsector, gender) * 100 / SUM(ordqty) OVER (PARTITION BY secsector) AS percentage FROM security, order, custacc, XMLTABLE(' declare namespace s=" $SDOC/s:Security' COLUMNS secsector VARCHAR(30) PATH '*:SecurityInformation//*:Sector', secname VARCHAR(50) PATH '*:Name') AS T1, XMLTABLE(' declare default element namespace " $ODOC/FIXML/Order' COLUMNS ordqty BIGINT PATH AS T2, XMLTABLE(' declare namespace c=" $CADOC/c:Customer' COLUMNS gender VARCHAR(10) PATH '*:Gender') AS T3 WHERE order.secsym = security.secsym AND order.custid = custacc.custid AND XMLEXISTS(' declare namespace s=" $SDOC/s:Security/s:SecurityInformation/*[s:Industry="OfficeSupplies" and s:MinInitialInvestment=5000]') AND XMLEXISTS(' declare default element namespace " = "2"]') ORDER BY secsector, gender;

Data Partitioning in a Cluster
Each node has 2 Intel Xeon 5169 dual-core CPUs, and 32GB RAM. 4 cores per node  we use 4 database partitions per node. 8 processing nodes Node 1 Node 2 Node 3 Node 4 Node 5 Node 6 Node 7 Node 8 8 database partitions, 250GB 16 database partitions, 500 GB 32 database partitions, 1TB

Scalability Results: Cluster
Source: IBM internally measured results, September 2009 Query response times for 500GB and 1TB are close to the 250GB results!

DB2 pureScale Goals Unlimited Capacity Application Transparency
Any transaction processing or ERP workload Start small Grow easily, with your business Application Transparency Avoid the risk and cost of tuning your applications to the database topology Continuous Availability Maintain service across planned and unplanned events Webcast: Web site:

DB2 pureScale : Technology Overview
Clients connect anywhere,… … see single database Clients connect into any member Automatic load balancing and client reroute may change underlying physical member to which client is connected Clients DB2 engine runs on several host computers Co-operate with each other to provide coherent access to the database from any member Single Database View Integrated cluster services Failure detection, recovery automation, cluster file system In partnership with STG (GPFS,RSCT) and Tivoli (SA MP) Member Low latency, high speed interconnect Special optimizations provide significant advantages on RDMA-capable interconnects (eg. Infiniband) CS Cluster Interconnect PowerHA pureScale technology Efficient global locking and buffer management Synchronous duplexing to secondary ensures availability Database Log Shared Storage Access 2nd-ary Primary Data sharing architecture Shared access to database Members write to their own logs Logs accessible from another host (used during recovery)

Scale with Ease Without changing applications
Efficient coherency protocols designed to scale without application change Applications automatically and transparently workload balanced across members Without administrative complexity No data redistribution required To 128 members in initial release Limited by testing resources Single Database View Log DB2 DB2 DB2 DB2 DB2 Log Log Log Log

What is a PowerHA pureScale ?
Software technology that assists in global buffer coherency management and global locking Derived from System z Parallel Sysplex & Coupling Facility technology Software based Services provided include Group Bufferpool (GBP) Global Lock Management (GLM) Shared Communication Area (SCA) Members duplex GBP, GLM, SCA state to both a primary and secondary Done synchronously Duplexing is optional (but recommended) Set up automatically, by default db2 agents & other threads db2 agents & other threads log buffer, dbheap, & other heaps log buffer, dbheap, & other heaps bufferpool(s) bufferpool(s) Primary Log Log GBP GLM SCA Secondary Shared database (Single database partition)

The Role of the GBP x GBP acts as fast disk cache
Client A : Select from T where C2=Y Client C : Select from T where C2=Y Client B : Update T1 set C1=X where C2=Y The Role of the GBP Commit GBP acts as fast disk cache Dirty pages stored in GBP, then later, written to disk Provides fast retrieval of such pages when needed by other members GBP includes a “Page Registry” Keeps track of what pages are buffered in each member and at what memory address Used for fast invalidation of such pages when they are written to the GBP Force-at-Commit (FAC) protocol ensures coherent access to data across members DB2 “forces” (writes) updated pages to GBP at COMMIT (or before) GBP synchronously invalidates any copies of such pages on other members New references to the page on other members will retrieve new copy from GBP In-progress references to page can continue Member 1 Member 2 bufferpool(s) bufferpool(s) x Write Page “Silent” Invalidate Read Page GBP GLM SCA Page Registry M1 M2

Stealth System Maintenance
Goal: allow DBAs to apply system maintenance without negotiating an outage window Procedure: Drain (aka Quiesce) Remove & Maintain Re-integrate Repeat until done Enables continuous availability Single Database View DB2 DB2 DB2 DB2 Log Log Log Log

Achieving Efficient Scaling : Key Design Points
Deep RDMA exploitation over low latency fabric Enables round-trip response time ~10-15 microseconds Silent Invalidation Informs members of page updates requires no CPU cycles on those members No interrupt or other message processing required Increasingly important as cluster grows Hot pages available without disk I/O from GBP memory RDMA and dedicated threads enable read page operations in ~10s of microseconds Lock Mgr Lock Mgr Lock Mgr Lock Mgr Can I have this lock ? Buffer Mgr Yup, here you are. x x x Read Page New page image GBP GLM SCA

of Transaction Throughput

Questions / Discussion
InfoSphere Warehouse provides unprecedented capabilities to meet the demands of today’s data warehouse customers, including support for mixed workloads. The scalable shared nothing architecture ensures predictable, linear scaling to support very large databases. I/O reduction techniques such as table partitioning, MDC, and compression help improve performance. Additional features such as MQTs and Cubing Services enable even faster performance through smart software. Sophisticated workload management enables SLAs to be met while supporting various workloads. Administration is easy with autonomics and GUI tools. Top notch security is provided to meet the most stringent government and corporate requirements. Integrated analytics are provided for both structured and unstructured data. In summary, InfoSphere Warehouse is best of breed software technology for all of your data warehouse needs.

Backup Slides

Features to Minimize Planned Outages
Backup: Fast, scalable, granular Online or offline Fully parallel and scalable Can be throttled Partition-level backup Table space-level backup Full, Incremental, or Delta Volume snapshot support Load: Fast, scalable and granular Partition-level Online load Online index rebuild Automatic log management Other utilities Online statistics collection Online index create and reorganization Online reorganization Online inspect Dynamic operations Configuration parameters Buffer pool operations Container operations Space management Online container management Automatic storage Online index reorganization Availability is very important to most systems. Many warehouses need to be up 24x7 especially as they become more integrated into operational systems. You need to minimize the downtime caused by your database server. There are two aspects to this: planned and unplanned outages. Planned outages are the times you need to take your database out of service in order to perform maintenance. ISW offers many features focused on reducing the need for offline maintenance and minimizing the duration of any outages that are required. ISW backups can be run either online or offline. If running offline they are designed to use the maximum amount of parallelism available on the disks, the cpu, and the backup media to minimize the time required, but of course can be throttled back if desired. Online they are designed to be throttled so that they don’t interfere with concurrent operations. They can be run on individual table spaces or partitions and at the full, incremental or delta levels to control the amount that needs to be backed up. With appropriate hardware support they can exploit disk snapshots to do “offline” backups without disrupting operations. The high speed load utility is scalable and can load individual table partitions independently. It also does not require a table to be taken offline during the load or the rebuilding of indexes. Log files are rotated and archived automatically, with integration into several archive mechanisms. Most other ISW utilities also operate online to avoid outages, for example rebalance, runstats, index creation and reorganizations, table reorganizations and inspect all run online. Most ISW configuration parameters can be adjusted online and take effect immediately. Many are automatically adjusted by the database in response to changing conditions without DBA intervention. ISW has excellent facilities for space management. Containers can be added to a running ISW system to increase the amount of space available. Similarly containers can be removed from ISW, with automatic redistribution of the data into the remaining space. Index space can be reclaimed through online index reorganization. Some ISW technologies, such as MDC, help avoid the need for space reclamation with reorganization by changing the physical arrangement of the data.

Features to Minimize Unplanned Outages
Hardware failures Integration with TSA cluster manager Built-in redundancy can't be turned off Consistency bits Log mirroring Automatic mirroring of critical data files Support for RAID Fast recovery Continuous check pointing Parallel recovery Automatic recovery tuning Filtered recovery Dynamic debugging capability High availability Clustering / failover support Integrated with TSM Automatic client reroute Human and Application Errors Point-in-Time (POT) recovery Drop table recovery Miscellaneous Infinite active logging Online container operations A continuation of the previous chart, here are some more features that help you minimize outages – only this time we are focusing on unplanned outages. Many of these go unnoticed, like built-in redundancy checking and consistency bits. This is a feature that’s inside of ISW that you can’t see and can’t turn off (like in some other databases). This feature protects your data and improves your availability. Things like log mirroring and mirroring of other critical files protect against single points of failure which could cause database outages. Other technologies are focused on minimizing the duration of an outage once it has occurred. The recovery mechanism uses check pointing before a crash along with intelligent filtering afterwards to minimize the amount of log that needs to be examined during a recovery. Parallel recovery and automatic tuning exploit the hardware available to shorten the length of time that the recovery takes. ISW provides utilities to trace or debug problem conditions that can help your staff or IBM support to diagnose and fix problems quickly. ISW provides mechanisms to build automatic failover into clustered systems. Surviving machines can take over processing for failed machines to keep the warehouse up and running. ISW also offers some facilities to deal with damage to the data through human errors, such as dropping an important table, or application errors such as entering several incorrect transactions. Some of these are built into the database, such as the ability to restore a database to a point in time before the error or reverse accidental table drops. Others are available in optional packages, such as tools to easily and selectively reverse log entries of errant transactions . Other capabilities such as “infinite” logging allow ISW to send logs to archive storage to avoid filling up the available log space, even if there are still transactions open on the archived logs. Online container operations let you grow your disk storage to accommodate new data before your database runs out of room and stops.

OLAP Optimization Advisor
InfoSphere Warehouse will design the aggregates to support dimensional analysis for you using: Hybrid line Statistics Meta-data that describes the cubes Hierarchies, dimensions, measures, etc. Optimizes to understand impact to load times and performance trade-off The Cubing Services OLAP Optimization advisor recommends MQTs that can be created to improve the performance of OLAP queries. The InfoSphere Warehouse query optimizer transparently rewrites incoming queries and routes eligible queries to the appropriate summary tables for significantly faster query performance. These Cubing Services recommended summary tables can accelerate all SQL-based queries into the data warehouse, not just those queries submitted on behalf of Cubing Services. Note that these recommended MQTs are based on the OLAP metadata which complements the MQTs recommended by the design advisor which recommends MQTs based on workload. This way, we get two approaches to create the best MQTs for the customer.

Universal Cubing Services Access
Portals, Web Applications, Dashboards, Interactive Reports, Ad Hoc Analysis, Common Desktop Tools IBM Cognos 8 BI IBM DataQuant & DB2 QMF Microsoft Excel Cubeware Cockpit Universal Cube Access (ODBO, XMLA) Cubing Services allows the creation of warehouse-based data cubes, natively stored within the warehouse. This accelerates the operations of the tools your users want to use to analyze data. Any tools that support the industry standard ODBO or XMLA interfaces can natively access Cubing Services cubes. ODBO = OLE DB for OLAP. XMLA = XML for Analysis Cubing Services cubes are first class data providers to Cognos 8 BI, enabling the full suite of Cognos clients and applications to take advantage of these powerful warehouse-based data cubes, including but not limited to: Query and Reporting Studios Analysis Studio, for ad hoc analysis Dashboarding and Scorecarding Cognos Analytics for Excel (Cafe) In addition to Cognos, the following clients can also natively access Cubing Services cubes: Microsoft Excel - ODBO interface Cubeware Cockpit - ODBO interface DB2 Query Management Facility (QMF) Enterprise Edition V9 Fix Pack 8 – XMLA interface IBM DataQuant for Multiplatform V1.2 Fix Pack 4– XMLA interface IBM DataQuant for z/OS V1.2 Fix Pack 4 – XMLA interface IBM Alphablox And many others that support ODBO or XMLA InfoSphere Warehouse

InfoSphere Warehouse Data Mining
Data Mining Embedded into Applications and Processes SOA Processes BI Analytical Tools Web Analytical Apps Mining Visualizer SQL Interface DB2 InfoSphere Warehouse Enterprise-Level Data Mining High-Speed, In-Database Scoring Modeling Model Results Powerful, yet simple, data mining capabilities enable integrated mining of the data in the enterprise data warehouse. Standard data mining models (clustering, associations, classification, and prediction) are supported and can be developed via drag and drop in the Design Studio. These models also may be imported in industry standard Predictive Model Markup Language (PMML) format from third-party modeling tools. The data mining models can be executed in the production environment to provide real time scoring of data records. Additionally, rich presentation components are provided to enable visual analysis of data mining results. In-Database Data Mining SQL Scoring Functions Structured & Unstructured Data

InfoSphere Warehouse Text Analytics
Analyze and extract structured data from text Makes data available to normal reporting and analysis tools From customer call center records, claim forms, etc. Benefits Target specific information hidden within text Competitive edge by driving further business insight Drives a greater ROI for your applications Business value examples Better product categorization Early warning on customer attrition Fraud detection Product defect analysis Better customer profiling Simple text analysis capabilities for text columns stored in warehouse tables Pattern matching rules and simple linguistics Enhance existing reports and data mining with insights gleaned from text Simple rules and dictionary editor Many applications today capture vital information on products, customers, contracts, claims information, etc. in their applications but are unable to harness it because it is stored as free-form text. InfoSphere Warehouse Text Analytics can do linguistic analysis on the text enabling you to derive new attributes and dimensionality from your information. Once analyzed the information can be stored as normal structured relational data for further OLAP and data mining analysis using existing tools. For example, you may have a customer call center which records the call takers notes about customer conversations as large text fields. Text Analytics could analyze this looking for and extracting complaints about specific products, storing that information into a relational table. Data mining tools could then discover trends in complaint rates about products, common problem areas, or sudden new problems. This information can be used by the business to address the product issues and avoid future customer dissatisfaction.

InfoSphere Warehouse Design Studio
Leverage and extend InfoSphere Data Architect: Design and modify database physical models (schema & storage design, etc) Design and model OLAP objects Design and model warehouse transformation and mining flows Key Features: Database design, or reverse engineer an existing database or DDL (RDA) View/Modify the schema Compare/Sync DB objects Analyze design (best practices and dependencies), Validation DB2 Storage Modeling: Table Space, Buffer Pool, Partition Generate script & Deploy: on data models, and flow models Impact Analysis: on data models and flow models Delivering a single user interface, InfoSphere Warehouse leverages the Eclipse-based Rational Data Architect (RDA) modeling functions to form the InfoSphere Warehouse Design Studio, a development environment for BI solutions. The Design Studio integrates the following tasks in a unified graphical environment: Physical data modeling SQL-based warehouse construction OLAP cube modeling Data mining modeling Online Analytics design Using a single user interface, designers can connect to source and target databases, reverse-engineer physical data models, build SQL-based data flows and mining flows, set up OLAP cubes, and prepare applications for deployment to runtime systems. Within the Design Studio, you can use Eclipse-based technology to create, modify, and generate DDL for physical data models. A physical data model is a database-specific model that represents relational data objects (for example, tables, columns, primary keys, and foreign keys) and their relationships. You create physical data models for source and target databases and staging tables for data warehouse applications. Generated DDL statements can be deployed directly to a database server via the Database Explorer. You can also use Eclipsed-based technology to compare data objects or analyze impact and dependencies. You can compare objects or models and highlight the differences. Then you can copy changes between the compared resources. You can navigate and merge structural differences between data objects, merge properties changes between data objects, generate DDL for changes that you made, and export the structural differences to an XML file on the file system. You can also analyze the impact and dependencies of a data object to determine how the object affects or is affected by changes to other objects in the model. 71 71

What’s new in TPoX 2.0 TPoX 2.0 includes pervasive change to the benchmark TPoX 2.0 test results not comparable to previous versions of TPoX Data Generator Workload and WorkloadDriver TPoX V1.3 and Earlier TPoX 2.0 Based on Toxgene A single java based program 3rd party tool, lack of support Complete rewrite Slow (> 5 days for 1TB data) Fast (6 hours for 1TB data) Can’t generate dense account IDs for CUSTACC Account IDs are now dense Large amount of small XML files Small amount of larger files, each contains 50K XML documents TPoX V1.3 and Earlier TPoX 2.0 Workload description file in proprietary format, hard to read Workload description file in XML format, easy to read and create WorkloadDriver reads input documents from large amount of small files WorkloadDriver reads input documents from smaller amount of larger files, improved performance for reading XML input documents Update transaction U1, U5 and U6 select account for update based on customer ID Update transaction U1, U5 and U6 select account for update based on account ID Data Distribution TPoX V1.3 and Earlier TPoX 2.0 # of CUSTACC vs # of ORDER 1:5 1:10 XML document size range 1-20KB 1-23KB ACCOUNT IDs of customer Not dense Dense Total XML document size of “100GB” scale Slightly less than 100GB Slight larger than 100GB avg # of accounts per customer 1.5 2.0 Changes have improved performance of generating and consuming TPoX XML data in large scale TPoX benchmarks ! NOTE: please refer to TPoX V2.0 Release Note at for more detail

More information on XML data management in DB2 for Linux, UNIX, Windows and DB2 for z/OS
The speaker notes throughout this slide deck contain links to specific related pages in the DB2 Information Center. Beyond the official documentation, the largest chapter in the "DB2 pureXML Cookbook" is the one on "Developing XML Applications with DB2" which contains more details and application code samples for Java, .NET, C, COBOL, PL/1, PHP, and Perl programmers.

Scalable Data Management with DB2

Similar presentations

Presentation on theme: "Scalable Data Management with DB2"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Scalable Data Management with DB2

Similar presentations

Presentation on theme: "Scalable Data Management with DB2"— Presentation transcript:

Similar presentations

About project

Feedback