2. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Oracle Database In-Memory Tirthankar Lahiri Vice President Oracle Data Technologies and TimesTen Vineet Marwah Senior Director Oracle Data Technologies

3. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | $$$ Accelerate Mixed Workload OLTP Orders of Magnitude Faster Analytics No Changes to Applications Cost Effective Oracle Database In-Memory Goals 100x 3 $ 2x-10x Simple

4. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | What is a Controversy? “A discussion marked especially by the expression of opposing views” Merriam Webster

5. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Decades Long Controversy in Database Systems Row Format Column Format “A discussion marked especially by the expression of opposing views” Merriam Webster

6. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Until Now Must Choose One Format and Suffer Tradeoffs Row Format Databases vs. Column Format Databases Row  Transactions run faster on row format – Example: Insert or query a sales order – Fast processing for few rows, many columns Column  Analytics run faster on column format – Example : Report on sales totals by region – Fast accessing few columns, many rows SALES 6

7. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Oracle Database In-Memory: Dual Format Architecture BOTH row and column formats for same table Simultaneously active and consistent OLTP uses existing row format Analytics uses new In-Memory Column format 7 Existing Buffer Cache New In-Memory Format SALES Row Format Column Format

8. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | In-Memory Columnar Format Highly optimized compressed columnar format Pure in-memory format: Cheap to maintain – no logging or IO Allows efficient OLTP No changes to disk format Transparent to Applications Can be enabled for subset of database 8 SALES Pure In-Memory Columnar

9. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Selective Enabling In-Memory Columnar Storage Selectively enable in-memory storage New INMEMORY clause Applies to Tables, Partitions, Sub-Partitions, Materialized Views, Tablespaces Can also exclude unneeded columns Coming – automatic inmemory placement ALTER TABLE orders INMEMORY; CREATE TABLE sales …… PARTITION BY RANGE(date) (PARTITION p1 …… INMEMORY, (PARTITION p2 …… NO INMEMORY); ALTER TABLE accounts INMEMORY NO INMEMORY (photo); 9

10. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Populating the In-Memory Column Store CREATE TABLE orders (c1 number, c2 varchar(20), c3 number) INMEMORY PRIORITY CRITICAL; ALTER TABLE sales INMEMORY PRIORITY MEDIUM; ALTER TABLE accounts INMEMORY PRIORITY NONE; In-Memory objects are populated in the background Always accessible via buffer cache in the meantime Queries do not block for populate Default behavior - initiate populate on first access Optional pre-populate via PRIORITY sub-clause Initiates populate without waiting for queries CRITICAL > HIGH > MEDIUM > LOW Controls order (not speed) of populate 10

11. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | In-Memory Column Store Performance Optimizations Vector Processing Software on Silicon Operation pushdown In-Memory Storage Index Elimination of Analytic Indexes 11

12. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Vector Processing: Additional Advantage of Column Format Each CPU core scans only required columns SIMD vector instructions used to process multiple values in each instruction E.g. Intel AVX instructions with 256 bit vector registers Billions of rows/sec scan rate per CPU core Row format is millions/sec 12 Vector Register Load multiple region values Vector Compare all values in 1 instruction CPU Memory CA Example: Find all sales in state of CA > 100x Faster STATE

13. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | DBIM Software On Silicon SIMD Vector Instructions were originally designed for High Performance Computing and Graphics – not for Databases New SPARC M7 chip has 32 Database Acceleration Engines (DAX) on chip Like having 32 specialized cores for DBIM query processing Oracle Confidential – Highly Restricted13 SPARC M7 On-Chip DBIM Accelerator/Compressor Column Format Compressed MEMORY or L3 $ Row Format DB Up to 32 Concurrent Result Streams MEMORY or L3 $ Bit/Byte-packed, Padded, Indexed Vectors M7 In-Silicon Query Engines Up to 32 Concurrent DB Streams

14. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | Performance: Database In-Memory Acceleration Engines 14 DAX includes specialized query functions for predicates, conversions, set membership tests, etc. Independently process streams of columns: – E.g. find all values that match ‘California’ – 10x performance gains – Up to 170 billion rows per second Core Shared Cache Core DB Accel 32 Database Accelerators (DAX) SPARC M7 Oracle Confidential – Highly Restricted

15. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |Copyright © 2015, Oracle and/or its affiliates. All rights reserved. | Capacity: Database In-Memory Decompression Engines 15 32 OZIP Decompressors 2x Memory capacity Core Shared Cache Core DB OZIP Oracle Confidential – Highly Restricted Compression increases in-memory capacity FOR QUERY mode compresses column values  Fastest for queries - no decompression needed FOR CAPACITY mode compresses column bit patterns  Data must be decompressed prior to access  Uses custom Oracle Zip (OZIP): Superfast bit pattern decompressor DAX includes specialized OZIP decompression engine Run OZIP decompress at full memory speed, > 120 GB/sec Pipelines decompression and data processing (predicate evaluation, aggregation, comparisons, etc.) in hardware Doubles memory capacity with no performance penalty

16. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Operation Pushdown: Bloom Filter Bloom filter created on dimension scan Bloom filter pushdown: Filtering pushed down to fact scan Returns only rows that are likely to be join candidates Joins tables 10x faster 16 Example: Find total sales in outlet stores Sales Stores Store ID StoreID in 15, 38, 64 Type=‘Outlet’ Type Sum Store ID Amount Bloom Filter

17. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Operation Pushdown: Vector Group By Create (empty) in-memory report outline during dimension scan Push down report outline aggregation to fact scan Reduces complex aggregations to series of fast inmemory scans Reports run 10x faster Without predefined cubes 17 Example: Report sales of footwear in outlet stores Sales Stores Products In-Memory Report Outline Footwear Outlets $ $$ $ $$$ Footwear Sales Outlets

18. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | In-Memory Storage Index In-Memory Compression Units (IMCU) – unit of column store allocation ~ 0.5 million rows Per-column min/max values – Check predicates against min/max values – Skip IMCU if predicate not satisfied Eliminates accessing unnecessary IMCUs Eliminates predicate evaluation when all values pass Min $4000 Max $7000 Min $8000 Max $12000 Min $13000 Max $15000 Example: Find stores with sales greater than $10,000 18

19. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Complex OLTP is Slowed by Analytic Indexes Most Indexes in complex OLTP (e.g. ERP) databases are only used for analytic queries Inserting one row into a table requires updating 10-20 analytic indexes: Slow! Indexes only speed up predictable queries & reports Table 1 – 3 OLTP Indexes 10 – 20 Analytic Indexes 19

20. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | OLTP is Slowed Down by Analytic Indexes Insert rate decreases as number of indexes increases # of Fully Cached Indexes (Disk Indexes are much slower) 20

21. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Column Store Replaces Analytic Indexes Fast analytics on any columns Better for unpredictable analytics Less tuning & administration Column Store not persistent so update cost is much lower OLTP & batch run faster Table 1 – 3 OLTP Indexes In-Memory Column Store 21

22. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Schneider In-Memory Compression 13x 16x 19x Over 2 billion General Ledger Entries 22

23. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Schneider Speedup Across 1545 Queries 7x to 128x faster Million rows returned by query 2000M 300M 30M 5M 0.5M Buffer Cache IN-MEMORY 2 billion General Ledger Entries 1545 queries – Originally took 34 hours to complete – Combination of filter queries, aggregations and summations 23

24. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Schneider Transactions Speedup From 5x to 9x faster All Indexes Primary Index Only Primary Index Plus In-Memory Columns Data – Sales Accounts Main table has 1 Primary Key + 21 secondary indexes Test - 303 million transactions 24

25. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Schneider Storage and IO Reduction Over 70% reduction in storage usage due to analytic index removal Size in GBs SECONDARY INDEXES TABLES & PK INDEXES 25

26. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Scale-Out In-Memory Database to Any Size Scale-Out across servers to grow memory and CPUs DISTRIBUTE clause: by Partition, Sub-Partition, or Rowid Range In-Memory queries parallelized across servers to access local column data Direct-to-Wire InfiniBand protocol speeds messaging 26

27. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Scale-Out In-Memory with Fault Tolerance Ability to Duplicate IMCUs on another node Enabled via DUPLICATE subclause Application transparent Similar to storage mirroring Downtime eliminated by using duplicate after failure 27 Only Available on Engineered Systems

28. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Conclusion: The End of a Controversy Dual Format

29. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | In-Memory Database Technology Across Tiers Application TimesTen In-Memory Database Embeddable In-Memory Database for Application Tier Primary Usecase: Latency-critical custom OLTP  Microsecond Response Time Standalone Database or as Application-Tier Cache for Oracle Database Oracle Database In-Memory Dual-Format In-Memory Database Primary Usecase: Real Time Analytics on any source  Billions of Rows/Sec analytic data access Faster mixed-workload enterprise OLTP Storage-Tiering: Combines best of memory, flash, disk Transparent: packaged apps run with no changes In-Memory Row Store In-Memory Column Store Application Tier Database Tier 29Oracle Confidential – Highly Restricted

30. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Summary Dual Format Architecture – Fully consistent row and column format – Best of both worlds OLTP and Analytics performance. – Typically, row format (Buffer cache) memory < 10% of column format memory New In-Memory Column Format – In-memory only representation – Seamlessly built into Oracle Database Engine – Compatible with all Oracle Database features Cost Effective – Use in-memory for hot data, flash for intermediate data, disk for cold data 30

31. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |

33. Backup Slides

34. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | In-Memory Area: Composition Contains two subpools: – Pool of In Memory Compression Units (IMCUs) – Pool of Snapshot Metadata Units (SMUs) IMCUs contain column formatted data SMUs contain metadata and transactional information SMU IMCU In Memory Area Column Format Data Metadata 34

35. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Column Compression Unit (CU) Contiguous storage per column in an IMCU All CUs automatically store Min/Max values Multiple formats: – For example, Dictionary Compression: CU stores (smaller) dictionary IDs instead of full values – Additional compression schemes 2 0 2 1 2 0 0 BMW Audi BMW Cadillac BMW Audi Column value list VALUE ID Audi 0 BMW 1 Cadillac 2 Column CU Min: Audi Max: Cadillac 35 Dictionary

36. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | In-Memory Compression Unit (IMCU) Unit of column store allocation – Columnar representation of a large number of rows (e.g. 0.5 million) – Rows in one or more table extents – Variable size Contains contiguous runs for each column (column compression units) IMCU header Extent #13 Blocks 20-120 Extent #14 Blocks 82-182 Extent #15 Blocks 201-301 ROWID E MP ID NAME DEPT SALARY Column CUs 36

37. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | In-Memory Column Store Populate Populate: Initial creation of IMCU from Row Format Repopulate: Recreation of IMCU after modification – Threshold Repopulate, after exceeding change threshold – Trickle Repopulate, constant activity, any changed IMCU is a potential candidate (Initial) Populate Change threshold or trickle action SMU IMCU SMU DML operations Repopulate Workload transactions IMCU Row Format 37

38. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | In-Memory Column Store Compression IMCUs compressed during population Controlled by MEMCOMPRESS sub-clause Ascending order of compression levels: FOR DML (for heavily updated tables, slower for queries) FOR QUERY LOW (default: light compression and fastest) FOR QUERY HIGH (slightly more compression) FOR CAPACITY LOW (balances capacity and performance) FOR CAPACITY HIGH (maximizes capacity) Allows in-memory storage tiering ALTER MATERIALIZED VIEW mv1 INMEMORY MEMCOMPRESS FOR QUERY; CREATE TABLE history (Name varchar(20), Desc varchar(200)) INMEMORY MEMCOMPRESS FOR CAPACITY LOW; 38

39. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | In-Memory Column Store Compression IMCUs compressed during population Controlled by MEMCOMPRESS sub-clause Ascending order of compression levels: FOR DML (for heavily updated tables, slower for queries) FOR QUERY LOW (default: light compression and fastest) FOR QUERY HIGH (slightly more compression) FOR CAPACITY LOW (balances capacity and performance) FOR CAPACITY HIGH (maximizes capacity) Allows in-memory storage tiering 39 DML QUERY LOW QUERY HIGH CAPACITY LOW CAPACITY HIGH Hotter Partition s Colder Partitions

40. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | In-Memory Column Store Compression MEMCOMPRESS FOR QUERY LOW or HIGH –LOW: Fastest performance –HIGH: Slightly greater compression –Lightweight compression schemes –Queries run on compressed data –Faster than on uncompressed data ModeCompression FactorQuery Speed Typical Range Observed Max QUERY 2x-7xUp to 10xFastest CAPACITY LOW 4x-9xUp to 20xVery Fast CAPACITY HIGH 7x-12xUp to 30xFast Compression ratios can be highly inflated by choosing a bad uncompressed format, or reporting most compressible table. Our results are measured relative to Oracle’s efficient row format for customer data.. 40

41. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | In-Memory Column Store Compression MEMCOMPRESS FOR CAPACITY LOW –Balances throughput and capacity –Adds Oracle custom ZIP (OZIP) on top of COMPRESS FOR QUERY –World’s fastest decompressor, tuned for DB query performance –2x - 3x faster than LZO (standard for fast zip) –Further optimized on SPARC M7 via Software on Silicon 41 ModeCompression FactorQuery Speed Typical Range Observed Max QUERY 2x-6xUp to 10xFastest CAPACITY LOW 4x-9xUp to 20xVery Fast CAPACITY HIGH 7x-12xUp to 30xFast Compression ratios can be highly inflated by choosing a bad uncompressed format, or reporting most compressible table. Our results are measured relative to Oracle’s efficient row format for customer data..

42. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | In-Memory Column Store Compression MEMCOMPRESS FOR CAPACITY HIGH –Compress with emphasis on space-savings –Heavier weight decompression required before query processing –Trades off some performance for capacity –Extra 1.5-2x compression 42 ModeCompression FactorQuery Speed Typical Range Observed Max QUERY 2x-6xUp to 10xFastest CAPACITY LOW 4x-9xUp to 20xVery Fast CAPACITY HIGH 7x-12xUp to 30xFast Compression ratios can be highly inflated by choosing a bad uncompressed format, or reporting most compressible table. Our results are measured relative to Oracle’s efficient row format for customer data..

43. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Scale-Up for Maximum In-Memory Performance Scale-Up on large SMPs NUMA optimizations Parallel Execution SMP scaling removes overhead of distributing queries across servers Memory interconnect far faster than any network 43

44. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | M6-32 Big Memory Machine World’s largest SMP System 32 TB DRAM 32 Sockets 384 Cores 3072 processing threads 3 Terabyte/sec Bandwidth 44

45. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | Scale-Out In-Memory Database to Any Size Distribution allows in memory segments larger than single host memory Policy is automatic or user-specifiable Controlled by DISTRIBUTE subclause Distribute by rowid range Distribute by partition Distribute by sub-partition Distribute AUTO (default) 45 ALTER TABLE sales INMEMORY DISTRIBUTE BY PARTITION; ALTER TABLE COSTS INMEMORY DISTRIBUTE AUTO;

46. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | ORDERS Scale-Out: Distribute by Partition 46 Distribute by Partition (top- level partition for composite partitioned tables) Ideal for Hash Partitions Also for other partition types if uniformly accessed Allows in-memory partition- wise joins

47. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | ORDERS Scale-Out: Distribute by Sub-Partition 47 For composite partitions, can distribute by Sub-Partition Ideal for Hash Sub-Partitions Also for other sub-partition types if uniformly accessed Allows in-memory partition- wise joins

48. Copyright © 2014, Oracle and/or its affiliates. All rights reserved. | ORDERS Scale-Out: Distribute by Rowid Range 48 Distributes IMCUs by uniform hash on first rowid For non-partitioned tables Also for partitioned tables with skewed access across partitions Ensures uniform distribution of load across instances