1. REMINDER Check in on the COLLABORATE mobile app Something Old, Something New: Leveraging Oracle 12cs Information Lifecycle Management (ILM) Features for Improved Database Performance Session #187 Jim Czuprynski Zero Defect Computing, Inc. April 9, 2014

2. My Credentials 30+ years of database-centric IT experience Oracle DBA since 2001 Oracle 9i, 10g, 11g OCP and Oracle ACE Director > 100 articles on databasejournal.com and ioug.org Teach core Oracle DBA courses (Grid + RAC, Exadata, Performance Tuning, Data Guard) Regular speaker at Oracle OpenWorld, IOUG COLLABORATE, OUG Norway, and Hotsos Oracle-centric blog (Generally, It Depends)

3. Our Agenda The Persistence of Data Through Time Information Lifecycle Management (ILM) In-Database Archiving (IDA) Concepts Temporal Validity (TV) Concepts Temporal History: Old Feature, New Name Leveraging ILM ADO Features Q+A

4. The Persistence of Data Through Time Data needs to be retained for inordinately long periods of time Data mining / warehousing / trend analysis Regulatory and legal requirements Once data has been purged … its often needed again (usually immediately) Therefore, data purging doesnt happen nearly as often as it should Handling schema changes after archiving is a nightmare … … so, what if we never purged data at all?

5. Information Lifecycle Management (ILM) ILM encompasses three key feature sets: In-Database Archiving (IDA) stores active and inactive data within same table Temporal Validity (TV) stores past, current, and future data within same table Automatic Data Optimization (ADO) tiers data according to its age and usage as well as how much space it consumes

6. In-Database Archiving (IDA): A Cloaking Device

7. In-Database Archiving (IDA) Avoids unnecessary deletion of rows when they no longer contain valid data Activated via new ROW ARCHIVAL attribute of data segment During initial INSERT, each rows state is set to active by placing a zero (0) in the ORA_ARCHIVE_STATE hidden column A row can be marked as inactive by setting ORA_ARCHIVE_STATE to any non-zero value Unless the ORA_ ARCHIVE_STATE column is mentioned in query, a rows IDA status is invisible Inactive rows can be compressed via ILM

8. Activating IDA CREATE table oe.currency_conversions ( currency_from VARCHAR2(3) NOT NULL,currency_to VARCHAR2(3) NOT NULL,effect_dtm TIMESTAMP NOT NULL,conversion_fctr NUMBER(10,6) DEFAULT 0 ) TABLESPACE example; ALTER TABLE oe.currency_conversions ROW ARCHIVAL; Activates In- Database Archiving for an existing table CREATE table oe.currency_conversions ( currency_from VARCHAR2(3) NOT NULL,currency_to VARCHAR2(3) NOT NULL,effect_dtm TIMESTAMP NOT NULL,conversion_fctr NUMBER(10,6) DEFAULT 0 ) ROW ARCHIVAL TABLESPACE example; Activates In- Database Archiving immediately upon table creation

9. COL column_name FORMAT A30 HEADING "Column Name" COL segment_column_id FORMAT 9999 HEADING "Seg|Col|#" COL internal_column_id FORMAT 9999 HEADING "Int|Col|#" COL hidden_column FORMAT A08 HEADING "Hidden?" COL virtual_column FORMAT A08 HEADING "Virtual?" COL user_generated FORMAT A08 HEADING "User|Gen'd?" TTITLE "Metadata for Table OE.CURRENCY_CONVERSIONS|(from DBA_TAB_COLS)" SELECT column_name,segment_column_id,internal_column_id,hidden_column,virtual_column,user_generated FROM dba_tab_cols WHERE owner = 'OE' AND table_name = 'CURRCONV_PARTED' ORDER BY internal_column_id; TTITLE OFF Viewing IDA Metadata Metadata for Table OE.CURRENCY_CONVERSIONS (from DBA_TAB_COLS) Seg Int Col Col User Column Name # # Hidden? Virtual? Gen'd? ------------------------------ ----- ----- -------- -------- -------- CURRENCY_FROM 1 1 NO NO YES CURRENCY_TO 2 2 NO NO YES EFFECT_DTM 3 3 NO NO YES CONVERSION_FCTR 4 4 NO NO YES SYS_NC00005$ 5 5 YES NO NO ORA_ARCHIVE_STATE 6 6 YES NO NO Note the addition of two new hidden columns exclusively for IDA

10. Populating IDA Test Data DECLARE ctr NUMBER := 0; curfr CHAR(3); curto CHAR(3); effdt DATE; BEGIN FOR ctr IN 1..36000 LOOP effdt := (TO_DATE('12/31/2015','mm/dd/yyyy') - (ctr/10)); INSERT INTO oe.currconv_parted VALUES( DECODE(MOD(ctr, 6), 0,'USD', 1,'JPY', 2,'EUR', 3,'AUD', 4,'CHF', 5, 'GBP'),DECODE(MOD(ctr,36), 0,'JPY', 1,'EUR', 2,'AUD', 3,'CHF', 4,'GBP', 5,'CDN', 6,'EUR', 7,'AUD', 8,'CHF', 9,'GBP', 10,'CDN', 11,'USD', 12,'AUD', 13,'CHF', 14,'GBP', 15,'CDN', 16,'USD', 17,'JPY', 18,'CHF', 19,'GBP', 20,'CDN', 21,'USD', 22,'JPY', 23,'EUR', 24,'GBP', 25,'CDN', 26,'USD', 27,'JPY', 28,'EUR', 29,'AUD', 30,'CDN', 31,'USD', 32,'JPY', 33,'EUR', 34,'AUD', 35,'CHF'),effdt,ROUND(DBMS_RANDOM.VALUE(1,5),6) ); IF MOD(ctr, 5000) = 0 THEN COMMIT; END IF; END LOOP; COMMIT; EXCEPTION WHEN OTHERS THEN DBMS_OUTPUT.PUT_LINE('Fatal unexpected error has occurred: ' || SQLERRM ); END; / Generates 36,000 rows of historical test data for conversions between six key currencies SQL> UPDATE oe.currency_conversions ora_archive_state 10 SET ora_archive_state = 10 GBP WHERE (currency_from = 'GBP') GBP OR (currency_to = 'GBP'); COMMIT; Activates IDA for any rows that convert currencies from or to British Pounds

11. Activating IDA Cloaking row archival visibilityACTIVE SQL> ALTER SESSION SET row archival visibility = ACTIVE; COL currency_from FORMAT A08 HEADING "From|Currency" COL currency_to FORMAT A08 HEADING "To|Currency" COL effect_cdtm FORMAT A19 HEADING "Effective As Of:" COL conversion_fctr FORMAT 9999.999999 HEADING "Currency|Conversion|Factor" COL min_dtm FORMAT A19 HEADING "Minimum|Effective|Date" COL max_dtm FORMAT A19 HEADING "Maximum|Effective|Date" COL min_cvf FORMAT 9999.999999 HEADING "Minimum|Currency|Conversion|Factor" COL max_cvf FORMAT 9999.999999 HEADING "Maximum|Currency|Conversion|Factor" TTITLE "Active Currency Conversion Factors|(from OE.CURRENCY_CONVERSIONS)" SELECT currency_from,currency_to,TO_CHAR(MIN(effect_dtm), 'yyyy-mm-dd hh24:mi') min_dtm,TO_CHAR(MAX(effect_dtm), 'yyyy-mm-dd hh24:mi') max_dtm,MIN(conversion_fctr) min_cvf,MAX(conversion_fctr) max_cvf FROM oe.currconv_parted GROUP BY currency_from, currency_to ORDER BY currency_from, currency_to ; TTITLE OFF s Active Currency Conversion Factors (from OE.CURRENCY_CONVERSIONS) Minimum Maximum Minimum Maximum Currency Currency From To Effective Effective Conversion Conversion Currency Currency Date Date Factor Factor -------- -------- ------------------- ------------------- ------------ ------------ AUD CDN 2006-02-23 02:24 2015-12-29 12:00 1.004594 4.996108 AUD CHF 2006-02-24 07:12 2015-12-30 16:48 1.000100 4.999312 AUD EUR 2006-02-21 07:12 2015-12-27 16:48 1.005547 4.998198 AUD JPY 2006-02-21 21:36 2015-12-28 07:12 1.000219 4.994470 AUD USD 2006-02-22 12:00 2015-12-28 21:36 1.001565 4.999302 CHF AUD 2006-02-21 04:48 2015-12-27 14:24 1.001286 4.995967 CHF CDN 2006-02-23 14:24 2015-12-30 00:00 1.005144 4.989977... JPY EUR 2006-02-24 12:00 2015-12-30 21:36 1.012854 4.992390 JPY USD 2006-02-21 12:00 2015-12-27 21:36 1.000304 4.995134 USD AUD 2006-02-23 09:36 2015-12-29 19:12 1.010373 4.999434 USD CDN 2006-02-21 14:24 2015-12-28 00:00 1.011684 4.996535 USD CHF 2006-02-22 19:12 2015-12-29 04:48 1.004463 4.997040 USD EUR 2006-02-24 00:00 2015-12-30 09:36 1.000201 4.998452 USD JPY 2006-02-21 00:00 2015-12-27 09:36 1.010545 4.987463 25 rows selected. No results for British Pound appear … SQL> SELECT currency_from, currency_to, TO_CHAR(effect_dtm, 'yyyy-mm-dd hh24:mi'), conversion_fctr FROM oe.currency_conversions GBP WHERE (currency_from = 'GBP') GBP OR (currency_to = 'GBP') ORDER BY 1,2,3; no rows selected … and all British Pound entries are cloaked!

12. Deactivating IDA Cloaking row archival visibility SQL> ALTER SESSION SET row archival visibility = ALL; Active Currency Conversion Factors (from OE.CURRENCY_CONVERSIONS) Minimum Maximum Minimum Maximum Currency Currency From To Effective Effective Conversion Conversion Currency Currency Date Date Factor Factor -------- -------- ------------------- ------------------- ------------ ------------ AUD CDN 2006-02-23 02:24 2015-12-29 12:00 1.004594 4.996108 AUD CHF 2006-02-24 07:12 2015-12-30 16:48 1.000100 4.999312 AUD EUR 2006-02-21 07:12 2015-12-27 16:48 1.005547 4.998198 AUD GBP 2006-02-23 16:48 2015-12-30 02:24 18.002189 21.999171... EUR GBP 2006-02-23 04:48 2015-12-29 14:24 18.009543 21.997833 EUR JPY 2006-02-21 09:36 2015-12-27 19:12 1.003705 4.998172 EUR USD 2006-02-22 00:00 2015-12-28 09:36 1.000780 4.997188 GBP AUD 2006-02-21 16:48 2015-12-28 02:24 18.006136 21.977187 GBP CDN 2006-02-24 02:24 2015-12-30 12:00 18.003076 21.997785 GBP CHF 2006-02-21 02:24 2015-12-27 12:00 18.001375 21.995166 GBP EUR 2006-02-22 07:12 2015-12-28 16:48 18.018042 21.999391 GBP JPY 2006-02-22 21:36 2015-12-29 07:12 18.003179 21.995334 GBP USD 2006-02-23 12:00 2015-12-29 21:36 18.006870 21.998615... USD GBP 2006-02-22 04:48 2015-12-28 14:24 18.001940 21.999533 USD JPY 2006-02-21 00:00 2015-12-27 09:36 1.010545 4.987463 36 rows selected. Inactive Currency Conversion Factors From To Conversion Currency Currency Effective As Of: Factor -------- -------- ------------------- ------------ AUD GBP 2014-04-01 00:00.466400 CDN GBP 2014-04-01 00:00.505900 CHF GBP 2014-04-01 00:00.489300 EUR GBP 2014-04-01 00:00.788300 GBP AUD 2014-04-01 00:00 2.144300... GBP USD 2014-04-01 00:00 1.868200 JPY GBP 2014-04-01 00:00.004850 USD GBP 2014-04-01 00:00.535300 12 rows selected. Cloaking device … off.

13. Temporal Validity (TV) and Temporal History (TH)

14. Temporal Validity (TV) TV allows specification of time periods when data stored within a table will be actually considered valid PERIOD FORImplemented through new PERIOD FOR table attribute Specified TV dimension can then be populated as desired with both time-valid and non-time- valid data Non-time-valid data can be compressed until its no longer needed

15. Implementing TV: Examples DROP TABLE oe.product_pricing PURGE; CREATE TABLE oe.product_pricing ( product_id NUMBER(6) NOT NULL,unit_price NUMBER(10,6) DEFAULT 0,beg_dtm TIMESTAMP NOT NULL,end_dtm TIMESTAMP NOT NULL PERIOD FOR effective,PERIOD FOR effective (beg_dtm, end_dtm) ) TABLESPACE example; … or, if columns are specified during the tables creation, theyll be used as the new dimension columns DROP TABLE oe.product_pricing PURGE; CREATE TABLE oe.product_pricing ( product_id NUMBER(6) NOT NULL,unit_price NUMBER(10,6) DEFAULT 0,beg_dtm TIMESTAMP NOT NULL,end_dtm TIMESTAMP NOT NULL PERIOD FOR effective,PERIOD FOR effective ) TABLESPACE example; The new dimension can be specified without any predefined columns, and they will be created automatically …

16. TV: Populating Sample Data DECLARE ctr NUMBER := 0; begdt DATE := TO_DATE('01/01/2000','mm/dd/yyyy'); enddt DATE; BEGIN FOR ctr IN 1..180 LOOP enddt := ADD_MONTHS(begdt, 1); FOR pidx IN 1..10 LOOP INSERT INTO oe.pricing_parted VALUES( DECODE(MOD(pidx,10), 0,1726, 1,1729, 2,1733, 3,1734, 4,1737, 5,1801, 6,2056, 7,2058, 8,3399, 9,3400, 3515),ROUND(DBMS_RANDOM.VALUE(1,100),4),begdt,enddt ); END LOOP; begdt := enddt; END LOOP; COMMIT; EXCEPTION WHEN OTHERS THEN DBMS_OUTPUT.PUT_LINE('Fatal unexpected error has occurred: ' || SQLERRM ); END; / Adds 1,800 rows of randomized data for Temporal Validity testing

17. Leveraging TV Session-Level Filtering CONNECT / AS SYSDBA DBMS_FLASHBACK_ARCHIVEoe GRANT EXECUTE ON DBMS_FLASHBACK_ARCHIVE TO oe; valid time window GRANTs access to users to request a valid time window -- Summary of all product prices: SELECT product_id,TO_CHAR(MIN(beg_dtm), 'yyyy-mm-dd.hh24:mi:ss') start_dtm,TO_CHAR(MAX(end_dtm), 'yyyy-mm-dd.hh24:mi:ss') stop_dtm,MIN(unit_price) min_price,MAX(unit_price) max_price FROM oe.pricing_parted GROUP BY product_id ORDER BY product_id; -- Detail for a single product: SELECT product_id,unit_price,TO_CHAR(beg_dtm,'YYYY-MM-DD') beg_dte,TO_CHAR(end_dtm,'YYYY-MM-DD') end_dte FROM oe.pricing_parted WHERE product_id = 1726 ORDER BY product_id, beg_dtm; EXEC DBMS_FLASHBACK_ARCHIVE.ENABLE_AT_VALID_TIME(level => ALL'); Summary of Unit Pricing Statistics Within Product ID (from OE.PRICING_PARTED) Starting Ending Product Time Time Minimum Maximum # Entries Period Period Unit Price Unit Price ------- ---------- ------------------- ------------------- ----------- ----------- 1726 180 2000-01-01.00:00:00 2015-01-01.00:00:00 1.18 129.08 1729 180 2000-01-01.00:00:00 2015-01-01.00:00:00 1.08 99.29... 3399 180 2000-01-01.00:00:00 2015-01-01.00:00:00 1.79 98.31 3400 180 2000-01-01.00:00:00 2015-01-01.00:00:00 1.68 99.99 1800 rows selected. Detailed Product Pricing for 1726 (from OE.PRICING_PARTED) Starting Ending Product Unit Time Time # Price Period Period ------- ----------- ------------------- ------------------- 1726 85.26 2000-01-01.00:00:00 2000-02-01.00:00:00 1726 40.55 2000-02-01.00:00:00 2000-03-01.00:00:00... 1726 34.48 2014-10-01.00:00:00 2014-11-01.00:00:00 1726 34.24 2014-11-01.00:00:00 2014-12-01.00:00:00 180 rows selected. EXEC DBMS_FLASHBACK_ARCHIVE.ENABLE_AT_VALID_TIME(level => ASOF, query_time => (TO_TIMESTAMP('2009-01-31','yyyy-mm-dd))); Summary of Unit Pricing Statistics Within Product ID (from OE.PRICING_PARTED) Starting Ending Product Time Time Minimum Maximum # Entries Period Period Unit Price Unit Price ------- ---------- ------------------- ------------------- ----------- ----------- 1726 1 2009-01-01.00:00:00 2009-02-01.00:00:00 53.32 53.32 1729 1 2009-01-01.00:00:00 2009-02-01.00:00:00 40.75 40.75 1733 1 2009-01-01.00:00:00 2009-02-01.00:00:00 10.11 10.11 1734 1 2009-01-01.00:00:00 2009-02-01.00:00:00 81.29 81.29 1737 1 2009-01-01.00:00:00 2009-02-01.00:00:00 77.01 77.01 1801 1 2009-01-01.00:00:00 2009-02-01.00:00:00 77.18 77.18 2056 1 2009-01-01.00:00:00 2009-02-01.00:00:00 63.37 63.37 2058 1 2009-01-01.00:00:00 2009-02-01.00:00:00 5.86 5.86 3399 1 2009-01-01.00:00:00 2009-02-01.00:00:00 30.52 30.52 3400 1 2009-01-01.00:00:00 2009-02-01.00:00:00 99.64 99.64 Detailed Product Pricing for 1726 (from OE.PRICING_PARTED) Starting Ending Product Unit Time Time # Price Period Period ------- ----------- ------------------- ------------------- 1726 53.32 2009-01-01.00:00:00 2009-02-01.00:00:00 EXEC DBMS_FLASHBACK_ARCHIVE.ENABLE_AT_VALID_TIME(level => CURRENT'); Summary of Unit Pricing Statistics Within Product ID (from OE.PRICING_PARTED) Starting Ending Product Time Time Minimum Maximum # Entries Period Period Unit Price Unit Price ------- ---------- ------------------- ------------------- ----------- ----------- 1726 1 2014-03-01.00:00:00 2014-04-01.00:00:00 77.32 77.32 1729 1 2014-03-01.00:00:00 2014-04-01.00:00:00 42.89 42.89 1733 1 2014-03-01.00:00:00 2014-04-01.00:00:00 88.22 88.22 1734 1 2014-03-01.00:00:00 2014-04-01.00:00:00 51.49 51.49 1737 1 2014-03-01.00:00:00 2014-04-01.00:00:00 2.73 2.73 1801 1 2014-03-01.00:00:00 2014-04-01.00:00:00 1.88 1.88 2056 1 2014-03-01.00:00:00 2014-04-01.00:00:00 99.57 99.57 2058 1 2014-03-01.00:00:00 2014-04-01.00:00:00 78.32 78.32 3399 1 2014-03-01.00:00:00 2014-04-01.00:00:00 67.21 67.21 3400 1 2014-03-01.00:00:00 2014-04-01.00:00:00 3.16 3.16 Detailed Product Pricing for 1726 (from OE.PRICING_PARTED) Starting Ending Product Unit Time Time # Price Period Period ------- ----------- ------------------- ------------------- 1726 77.32 2014-03-01.00:00:00 2014-04-01.00:00:00

18. Temporal History (TH) Original name: Oracle Total Recall Later releases: Flashback Data Archive Intrinsic to Oracle 12cR1 Permits bi-temporal queries: Flashback Query uses AS OF clause Flashback Versions Query uses VERSIONS BETWEEN clause

19. Leveraging Bi-Temporal Queries 108.85108.0094.21119.1198.9051.5915.07 Changed on 04-05-2013 107.2598.90 Changed on 04-18-2013 95.23119.11 Changed on 04-30-2013 98.90 SCNSCN 01-1302-1303-1304-1305-1306-1307-1308-13 Time EXEC DBMS_FLASHBACK_ARCHIVE(ASOF,2013-04-01); SELECT product_id,unit_price,TO_CHAR(beg_dtm,'YYYY-MM-DD') beg_dte,TO_CHAR(end_dtm,'YYYY-MM-DD') end_dte FROM oe.pricing_parted AS OF PERIOD FOR effective TO_DATE('2013-04-15','yyyy-mm-dd') AS OF PERIOD FOR effective TO_DATE('2013-04-15','yyyy-mm-dd') WHERE product_id = 1726 ORDER BY product_id, beg_dtm; Flashback Query EXEC DBMS_FLASHBACK_ARCHIVE(ASOF,2013-04-01); SELECT product_id,unit_price,TO_CHAR(beg_dtm,'YYYY-MM-DD') beg_dte,TO_CHAR(end_dtm,'YYYY-MM-DD') end_dte FROM oe.pricing_parted VERSIONS PERIOD FOR effective PERIOD FOR effective BETWEEN TO_DATE('2013-04-08','yyyy-mm-dd') BETWEEN TO_DATE('2013-04-08','yyyy-mm-dd') AND TO_DATE('2013-04-30','yyyy-mm-dd') AND TO_DATE('2013-04-30','yyyy-mm-dd') WHERE product_id = 1726 ORDER BY product_id, beg_dtm; Flashback Versions Query

20. Leveraging ILM ADO Features

21. Automatic Data Optimization (ADO) Moves or compresses data based on observed usage patterns Leverages heat maps to determine how often data has been accessed Tracks exactly how data has been utilized DML versus query Random access versus table scan Usage patterns can be tracked at tablespace, segment, and even row level

22. Covering the Space-Time Continuum An ADO policy can be based on either space or time Space-based: Moves segment between tablespaces in different storage tiers Movement is based on fullness of initial tablespace Time-based: Compresses data more tightly within same object Data compressible at three levels: ROW STORAGE (row level within a segment) SEGMENT (one database object at segment level) GROUP (multiple database objects)

23. IDA: Implementing ILM Storage Tiering CREATE table oe.currconv_parted ( currency_from VARCHAR2(3) NOT NULL,currency_to VARCHAR2(3) NOT NULL,effect_dtm TIMESTAMP NOT NULL,conversion_fctr NUMBER(10,6) DEFAULT 0 ) PARTITION BY LIST(ora_archive_state) ( PARTITION currconv_history VALUES (10,20,30,40,50) TABLESPACE ado_cold_data ILM ADD POLICY COMPRESS FOR QUERY HIGH COMPRESS FOR QUERY HIGH SEGMENT SEGMENT AFTER 180 DAYS OF NO MODIFICATION AFTER 180 DAYS OF NO MODIFICATION,PARTITION currconv_current VALUES (0) TABLESPACE ado_hot_data ) ROW ARCHIVAL NOLOGGING PARALLEL 4; 180 days compressed HCC QUERY HIGH After 180 days of no further DML activity, the segment of the partition containing all inactive rows will be compressed using HCC QUERY HIGH compression

24. TV: Implementing ILM Storage Tiering CREATE TABLE oe.pricing_parted ( product_id NUMBER(6) NOT NULL,unit_price NUMBER(10,6) DEFAULT 0,beg_dtm TIMESTAMP NOT NULL,end_dtm TIMESTAMP NOT NULL,PERIOD FOR EFFECTIVE (beg_dtm, end_dtm) ) PARTITION BY RANGE(end_dtm) ( PARTITION opp_history VALUES LESS THAN (TO_DATE('2014-01-01','yyyy-mm-dd')) TABLESPACE ado_cold_data ILM ADD POLICY COMPRESS FOR QUERY HIGH COMPRESS FOR QUERY HIGH SEGMENT SEGMENT AFTER 180 DAYS OF NO ACCESS AFTER 180 DAYS OF NO ACCESS,PARTITION opp_current VALUES LESS THAN (MAXVALUE) TABLESPACE ado_warm_data ) NOLOGGING PARALLEL 4;

25. Over To You …

26. Please feel free to evaluate this session: Use the COLLABORATE14 Application! Session #187 Something Old, Something New: Leveraging Oracle 12cs Information Lifecycle Management (ILM) Features for Improved Database Performance If you have any questions or comments, feel free to: E-mail me at jczuprynski@zerodefectcomputing.com Follow my blog (Generally, It Depends): http://jimczuprynski.wordpress.com Connect with me on LinkedIn (Jim Czuprynski) Follow me on Twitter (@jczuprynski) Thank You For Your Kind Attention