Carl Dudley – University of Wolverhampton, UK Materialized Views Carl Dudley University of Wolverhampton, UK Oracle ACE Director UKOUG Committee

2 Carl Dudley – University of Wolverhampton, UK Introduction Working with Oracle since 1986 Oracle DBA - OCP Oracle7, 8, 9, 10 Oracle DBA of the Year – 2002 Oracle ACE Director Regular Presenter at Oracle Conferences Consultant and Trainer Technical Editor for a number of Oracle texts UK Oracle User Group Director Member of IOUC Day job – University of Wolverhampton, UK

3 Carl Dudley – University of Wolverhampton, UK Topics Materialized View Features and Syntax Query Rewrite Refreshing Materialized Views Nested Materialized Views Implementing Constraints via Materialized Views Fast and Complete Refresh Performance Multiple Materialized Views Estimating the Size of Materialized Views

4 Carl Dudley – University of Wolverhampton, UK Materialized View Features  Data Warehouses typically contain a few very large tables —Joins and aggregate queries can be very expensive  MVs often contain summary tables of aggregations of a table's data —Pre-calculated results enhance query performance —Usually defined as aggregate views and/or join views For example, a view containing sums of salaries by job within department  Sometimes act as replica snapshots of tables on remote databases —Not covered in this presentation

5 Carl Dudley – University of Wolverhampton, UK Materialized Views Features (continued)  Not normally accessed directly —Queries on base tables are automatically re-written to use MVs  MVs can be nested —MVs built on other materialized views  Transparent to SQL applications —Can be created or dropped without affecting the validity of SQL statements  Consume storage space  Contents must be updated whenever underlying detail tables are modified

6 Carl Dudley – University of Wolverhampton, UK Creating Materialized Views CREATE MATERIALIZED VIEW emv BUILD IMMEDIATE REFRESH COMPLETE ON DEMAND AS SELECT deptno,SUM(sal) FROM emp GROUP BY deptno;  This view is populated with data upon creation  To keep the view synchronised with emp, it must be completely refreshed —This will be done manually on demand (the default)  Queries against emp will not be re-written to use the view (default)

7 Carl Dudley – University of Wolverhampton, UK Creating Materialized Views (continued) CREATE MATERIALIZED VIEW emv BUILD DEFERRED REFRESH FAST ON COMMIT ENABLE QUERY REWRITE AS SELECT deptno,SUM(sal),COUNT(*),COUNT(sal) FROM emp GROUP BY deptno;  Not populated with data until first refresh action takes place —The first refresh must be a complete refresh  To keep the view synchronised, it will undergo fast refresh operations —Refresh happens every time a transaction on emp commits  Queries against emp will be re-written to use the view —Unless the parameter QUERY_REWRITE_ENABLED is set to false

8 Carl Dudley – University of Wolverhampton, UK Topics Materialized View Features and Syntax Query Rewrite Refreshing Materialized Views Nested Materialized Views Implementing Constraints via Materialized Views Fast and Complete Refresh Performance Multiple Materialized Views Estimating the Size of Materialized Views

9 Carl Dudley – University of Wolverhampton, UK Query Rewrite  The optimizer compares costs when deciding to use a materialized view —Query is rewritten to use the materialized view —Underlying base tables are not used and performance is normally improved  A few carefully chosen MVs can support a large range of queries

10 Carl Dudley – University of Wolverhampton, UK Query Rewrite  Transformation of a query based on tables to an equivalent query based on one or more materialized views  Several factors affect whether query rewrite will occur 1.Enabling or disabling query rewrite can be achieved via the CREATE or ALTER statement for materialized views REWRITE and NOREWRITE hints in individual SQL statements SELECT /* +NOREWRITE */... The REWRITE_OR_ERROR hint will prevent the processing of a statement if it is not rewritten 2.Rewrite integrity levels Some levels allow the use of stale data 3.Dimensions and constraints Outside the scope of this talk but details are in the Oracle docs

11 Carl Dudley – University of Wolverhampton, UK Controlling Accuracy of Query Rewrite with QUERY_REWRITE_INTEGRITY ALTER SESSION SET QUERY_REWRITE_INTEGRITY = ENFORCED|TRUSTED|STALE_TOLERATED; ENFORCED MV data must be fresh and any constraints used must be enabled and validated TRUSTED Trusts data conforms to constraints that are not validated STALE_TOLERATED Allows stale MVs to be used  Only the ENFORCED level guarantees totally accurate results —MVs will not be used if they are not refreshed after the base table changes

12 Carl Dudley – University of Wolverhampton, UK Checking on the MV Situation  DML on underlying objects causes NEEDS_COMPILE when REFRESH ON DEMAND is specified  A refresh or compile fixes the state SELECT mview_name,refresh_mode,staleness,compile_state FROM user_mviews MVIEW_NAME REFRESH_MODE STALENESS COMPILE_STATE CAL_MONTH_SALES_MV DEMAND NEEDS_COMPILE NEEDS_COMPILE FWEEK_PSCAT_SALES_MV DEMAND STALE VALID SUM_SALES_PSCAT_WEEK_MV COMMIT FRESH VALID ALTER MATERIALIZED VIEW cal_month_sales_mv COMPILE; SELECT mview_name,refresh_mode,staleness,compile_state FROM user_mviews; MVIEW_NAME REFRESH_MODE STALENESS COMPILE_STATE CAL_MONTH_SALES_MV DEMAND STALE VALID FWEEK_PSCAT_SALES_MV DEMAND STALE VALID SUM_SALES_PSCAT_WEEK_MV COMMIT FRESH VALID

13 Carl Dudley – University of Wolverhampton, UK Query Rewrite Performance CREATE MATERIALIZED VIEW sales_prod_mv BUILD IMMEDIATE ENABLE QUERY REWRITE AS SELECT prod_id,SUM(amount_sold) FROM sales GROUP BY prod_id; SELECT prod_id,SUM(amount_sold) FROM sales GROUP BY prod_id; Sample MV Sample query

14 Carl Dudley – University of Wolverhampton, UK Observing Query Rewrite  Queries may be rewritten to use the materialized view SELECT prod_id,SUM(amount_sold) FROM sales GROUP BY prod_id; Elapsed time : 0.03 seconds |Id | Operation | Name |Rows|Bytes | Cost(%CPU)| | 0| SELECT STATEMENT | | 10| 1872 | 3 (0)| | 1| MAT_VIEW REWRITE ACCESS FULL| SALES_PROD_MV| 72| 1872 | 3 (0)| |Id | Operation | Name |Rows | Bytes | Cost(%CPU) | | 0 | SELECT STATEMENT | | 73 | 657 | 1182 (17)| | 1 | HASH GROUP BY | | 73 | 657 | 1182 (17)| | 2 | TABLE ACCESS FULL| SALES | 932K| 8196K| 1038 (5)|  When the view is not present, the base table is scanned Elapsed time : seconds

15 Carl Dudley – University of Wolverhampton, UK Did Query Rewrite Occur and Why?  Query rewrite occurs transparently —Check for query rewrite AUTOTRACE dbms_mview.explain_rewrite  dbms_mview.explain_rewrite —Gives reasons WHY the rewrite did not occur The rules governing this are extremely complex —Populates the rewrite_table with information Which materialized views will be used Comparative costings of the rewritten and un-rewritten query —Each account must have its own rewrite_table Created by running the utlxrw script in the admin directory

16 Carl Dudley – University of Wolverhampton, UK Explaining the rewrite: Example One BEGIN DBMS_MVIEW.EXPLAIN_REWRITE ('SELECT prod_id,SUM(amount_sold) FROM sales GROUP BY prod_id'); END; / SELECT message,original_cost,rewritten_cost FROM rewrite_table; MESSAGE ORIGINAL_COST REWRITTEN_COST QSM-01009: materialized view,prod_sum_mv, matched query text  An efficient rewrite occurs because of the exact match of the query and the definition of the view —Most rewrites are more complex  Aliases can be used for MV columns to allow exact match with table columns CREATE MATERIALIZED VIEW emv (e_deptno, d_deptno) ENABLE QUERY REWRITE AS SELECT e.deptno, d.deptno...

17 Carl Dudley – University of Wolverhampton, UK Explaining the rewrite: Example Two BEGIN DBMS_MVIEW.EXPLAIN_REWRITE ('SELECT prod_id,AVG(amount_sold) FROM sales GROUP BY prod_id'); END; / SELECT message,original_cost,rewritten_cost FROM rewrite_table; MESSAGE ORIGINAL_COST REWRITTEN_COST QSM-01086: dimension(s) not present or not 0 0 used in ENFORCED integrity mode QSM-01065: materialized view, prod_sum_mv, 0 0 Cannot compute measure, AVG, in the query  Query rewrite cannot occur for the given reasons —No costs are calculated

18 Carl Dudley – University of Wolverhampton, UK Undocumented Query Rewrite Parameters KSPPINM KSPPSTVL _explain_rewrite_mode FALSE _query_rewrite_or_error FALSE _query_cost_rewrite TRUE _query_rewrite_2 TRUE _query_rewrite_1 TRUE _query_rewrite_fudge 90 _query_rewrite_expression TRUE _query_rewrite_jgmigrate TRUE _query_rewrite_fpc TRUE _query_rewrite_drj TRUE _query_rewrite_maxdisjunct 257 _query_rewrite_vop_cleanup TRUE _mmv_query_rewrite_enabled TRUE KSPPINM KSPPSTVL _bt_mmv_query_rewrite_enabled TRUE _pre_rewrite_push_pred TRUE _union_rewrite_for_gs YES_GSET_MVS _query_rewrite_setopgrw_enable TRUE _force_rewrite_enable FALSE _query_mmvrewrite_maxpreds 10 _query_mmvrewrite_maxintervals 5 _query_mmvrewrite_maxinlists 5 _query_mmvrewrite_maxdmaps 10 _query_mmvrewrite_maxcmaps 20 _query_mmvrewrite_maxregperm 512 _query_mmvrewrite_maxmergedcmaps 50 _query_mmvrewrite_maxqryinlistvals 500 _enable_query_rewrite_on_remote_objs TRUE COL ksppinm FORMAT a40 COL ksppstvl FORMAT a40 SELECT ksppinm,ksppstvl FROM x$ksppi i,x$ksppcv v WHERE i.indx = v.indx AND i.ksppinm like '%rewrite%‘ AND SUBSTR(ksppinm,1,1) = '_';

19 Carl Dudley – University of Wolverhampton, UK Topics Materialized View Features and Syntax Query Rewrite Refreshing Materialized Views Nested Materialized Views Implementing Constraints via Materialized Views Fast and Complete Refresh Performance Multiple Materialized Views Estimating the Size of Materialized Views

20 Carl Dudley – University of Wolverhampton, UK Refreshing Materialized Views  If data in base tables change, MVs need to be refreshed in one of two ways 1.On transaction commit ( REFRESH ON COMMIT ) MV kept up to date but performance impact on base table transactions 2. On demand ( REFRESH ON DEMAND ) Requires a procedure to be invoked The procedure can take an argument to specify the type of refresh COMPLETE - complete refresh by repopulating FAST - incrementally applies changes to the view FORCE - fast refresh if possible, if not it forces a complete refresh dbms_mview.refresh('emp_mv1')

21 Carl Dudley – University of Wolverhampton, UK Refreshing Materialized Views  Refresh can be accelerated with ATOMIC_REFRESH set to 'FALSE ' —MV is truncated rather then deleted (default value is TRUE on 10g and up) Less undo and redo but MV unavailable during refresh —Allows parallel refresh  Fast Refresh can be accelerated if MV logs are created 'WITH COMMIT SCN' —Snapshots do not have to be built during the refresh —Use is made of all_summap view —Can also set NOLOGGING for the materialized view logs

22 Carl Dudley – University of Wolverhampton, UK Materialized View Logs  Materialized view logs are required for fast refresh  Defined using CREATE MATERIALIZED VIEW LOG on the base table —Not created on the materialized view —Each inserted and deleted row on the base table creates a row in the log —An update may generate two rows in the log for old and new values  Normally need to specify the ROWID clause  Logs that support aggregate materialized views must also contain: 1.Every column in the table that is referenced in the materialized view 2.The INCLUDING NEW VALUES clause —Name of the log is mlog$_ —Specify SEQUENCE if the table is expected to have a mix of inserts/direct- loads, deletes, and updates CREATE MATERIALIZED VIEW LOG ON emp WITH ROWID (deptno,sal) INCLUDING NEW VALUES;

23 Carl Dudley – University of Wolverhampton, UK Tuning Materialized Views for Fast Refresh and Query Rewrite  dbms_advisor.tune_mview helps optimize CREATE MATERIALIZED VIEW statements  Shows how to implement materialized views and any required logs  The output statements for ‘ IMPLEMENTATION’ include: — CREATE MATERIALIZED VIEW LOG statements Creates any missing materialized view logs required for fast refresh — ALTER MATERIALIZED VIEW LOG FORCE statements Fixes any materialized view logs for maybe fast refresh —One or more CREATE MATERIALIZED VIEW statements Could suggest just adding required columns For example, add ROWID column for materialized join view and add aggregate column for materialized aggregate view

24 Carl Dudley – University of Wolverhampton, UK Tuning Materialized Views – Using the Advisor VARIABLE task_cust_mv VARCHAR2(30); VARIABLE create_mv_ddl VARCHAR2(4000); BEGIN :task_cust_mv := 'emp_mv1'; :create_mv_ddl := 'CREATE MATERIALIZED VIEW emp_mv1 REFRESH FAST ENABLE QUERY REWRITE AS SELECT deptno,sum(sal) FROM emp GROUP BY deptno'; DBMS_ADVISOR.TUNE_MVIEW(:task_cust_mv, :create_mv_ddl); END; /  Tasks can be deleted EXEC dbms_advisor.delete_task( ' emp_mv1 ' ) Requires ADVISOR privilege

25 Carl Dudley – University of Wolverhampton, UK Tuning Materialized Views – Examining the Output SELECT statement FROM user_tune_mview WHERE task_name= :task_cust_mv AND script_type='IMPLEMENTATION'; STATEMENT CREATE MATERIALIZED VIEW LOG ON "SCOTT"."EMP" WITH ROWID, SEQUENCE  "SAL","DEPTNO") INCLUDING NEW VALUES ALTER MATERIALIZED VIEW LOG FORCE ON "SCOTT"."EMP" ADD ROWID, SEQUENCE  "SAL","DEPTNO") INCLUDING NEW VALUES CREATE MATERIALIZED VIEW SCOTT.EMP_MV1 REFRESH FAST WITH ROWID ENABLE QUERY REWRITE AS SELECT SCOTT.EMP.DEPTNO C1, SUM("SCOTT"."EMP"."SAL") M1, COUNT("SCOTT"."EMP"."SAL") M2, COUNT(*) M3 FROM SCOTT.EMP GROUP BY  SCOTT.EMP.DEPTNO  For the view to be fast refreshed 1.A log is recommended because the view will be fast refreshed 2.The view must include a count of the aggregate column and also a count of the rows

26 Carl Dudley – University of Wolverhampton, UK Tuning Materialized Views – a Further Example BEGIN :task_cust_mv := 'emp_mv2'; :create_mv_ddl := 'CREATE MATERIALIZED VIEW emp_mv2 REFRESH FAST ENABLE QUERY REWRITE AS SELECT dept.deptno,dname,sum(sal) FROM emp,dept WHERE dept.deptno = emp.deptno GROUP BY dept.deptno,dname' DBMS_ADVISOR.TUNE_MVIEW(:task_cust_mv, :create_mv_ddl); END; /

27 Carl Dudley – University of Wolverhampton, UK Further Example Output SELECT statement FROM user_tune_mview WHERE task_name= :task_cust_mv AND script_type='IMPLEMENTATION'; STATEMENT CREATE MATERIALIZED VIEW LOG ON "SCOTT"."DEPT" WITH ROWID, SEQUENCE ("DEPTNO","DNAME") INCLUDING NEW VALUES ALTER MATERIALIZED VIEW LOG FORCE ON "SCOTT"."DEPT" ADD ROWID, SEQUENCE ("DEPTNO","DNAME") INCLUDING NEW VALUES CREATE MATERIALIZED VIEW LOG ON "SCOTT"."EMP" WITH ROWID, SEQUENCE ("SAL","DEPTNO") INCLUDING NEW VALUES ALTER MATERIALIZED VIEW LOG FORCE ON "SCOTT"."EMP" ADD ROWID, SEQUENCE ("SAL","DEPTNO") INCLUDING NEW VALUES CREATE MATERIALIZED VIEW SCOTT.EMP_MV2 REFRESH FAST WITH ROWID ENABLE QUERY REWRITE AS SELECT SCOTT.DEPT.DNAME C1, SCOTT.DEPT.DEPTNO C2, SUM("SCOTT"."EMP"."SAL") M1, COUNT("SCOTT"."EMP"."SAL") M2, COUNT(*) M3 FROM SCOTT.EMP, SCOTT.DEPT WHERE SCOTT.DEPT.DEPTNO = SCOTT.EMP.DEPTNO GROUP BY SCOTT.DEPT.DNAME, SCOTT.DEPT.DEPTNO

28 Carl Dudley – University of Wolverhampton, UK Materialized View Capabilities  dbms_mview.explain_mview is used to explain the capabilities (query rewrite, refresh modes) of an MV —The optional value '6' is a user supplied statement identifier  Places information in mv_capabilities_table —Has one row for each reason why a certain action is not possible —Needs to be built in each investigating account by executing the script utlxmv.sql found in the admin directory —The statement_id column can be used to fetch rows for a given MV EXECUTE dbms_mview.explain_mview ('emp_mv1','6')

29 Carl Dudley – University of Wolverhampton, UK The mv_capabilities_table EXECUTE dbms_mview.explain_mview('emp_mv1','6') SELECT capability_name,possible,msgtxt FROM mv_capabilities_table WHERE statement_id = '6'; CAPABILITY_NAME P MSGTXT PCT N REFRESH_COMPLETE Y REFRESH_FAST Y REWRITE Y PCT_TABLE N relation is not a partitioned table REFRESH_FAST_AFTER_INSERT Y REFRESH_FAST_AFTER_ONETAB_DML N SUM(expr) without COUNT(expr) REFRESH_FAST_AFTER_ONETAB_DML N COUNT(*) is not present in the select list REFRESH_FAST_AFTER_ANY_DML N mv log does not have sequence # REFRESH_FAST_AFTER_ANY_DML N see the reason why REFRESH_FAST_AFTER_ONETAB_DML is disabled REFRESH_FAST_PCT N PCT is not possible on any of the detail tables in the materialized view REWRITE_FULL_TEXT_MATCH Y REWRITE_PARTIAL_TEXT_MATCH Y REWRITE_GENERAL Y REWRITE_PCT N general rewrite is not possible or PCT is not possible on any of the detail tables PCT_TABLE_REWRITE N relation is not a partitioned table

30 Carl Dudley – University of Wolverhampton, UK Restrictions on Fast Refresh on MVs There are many more special case restrictions and rules  Joins… — ROWID s of all tables in FROM list must appear in the SELECT list of the query —Materialized view logs must exist on all tables involved in the join  Aggregates… —Only SUM, COUNT, AVG, STDDEV, VARIANCE, MIN and MAX are supported for fast refresh COUNT(*) must be included in the view —For each aggregate such as AVG(expr), the corresponding COUNT(expr) must be present

31 Carl Dudley – University of Wolverhampton, UK salessales_mv mlog$_sales materialized view Materialized view log build commit 62 Base table 62  REFRESH COMPLETE ON COMMIT does not require logs – Very unlikely to be used Fast Refresh on Commit

32 Carl Dudley – University of Wolverhampton, UK Bulk Loading – Impact on Fast Refresh 1.INSERT /*+ append */ 2.SQL*Loader DIRECT=TRUE  Improved performance as no data written to MV log —Much less redo and undo —Minimal tracking in all_sumdelta Summary information with ROWID ranges

33 Carl Dudley – University of Wolverhampton, UK Flexing the Materialized Views with ON PREBUILT TABLE  MVs cannot be ALTER ed —What if a column width change is required? Could make use of ON PREBUILT TABLE to avoid MV rebuild CREATE TABLE j_g AS SELECT job, COUNT(*) ct FROM emp GROUP BY job; CREATE MATERIALIZED VIEW j_g ON PREBUILT TABLE AS SELECT job, COUNT(*) ct FROM emp GROUP BY job; OBJECT_ID DATA_OBJECT_ID OBJECT_NAME OBJECT_TYPE J_G TABLE J_G MATERIALIZED VIEW DROP TABLE j_g; -- fails due to presence of MV DROP MATERIALIZED VIEW j_g; --succeeds but leaves table intact ALTER TABLE j_g MODIFY job VARCHAR2(40); CREATE MATERIALIZED VIEW j_g ON PREBUILT TABLE AS SELECT job, COUNT(*) ct FROM emp GROUP BY job; —Can also help when fixing 'out of sync' errors with logs and base tables ORA-12034: materialized view log younger than last refresh

34 Carl Dudley – University of Wolverhampton, UK Topics Materialized View Features and Syntax Query Rewrite Refreshing Materialized Views Nested Materialized Views Implementing Constraints via Materialized Views Fast and Complete Refresh Performance Multiple Materialized Views Estimating the Size of Materialized Views

35 Carl Dudley – University of Wolverhampton, UK Nested Materialized Views Base table 1Base table 2Base table 3 Join Materialized View Aggregate Materialized View 1 Aggregate Materialized View 2 Aggregate Materialized View 3 Aggregate Materialized View 4 Nested Views

36 Carl Dudley – University of Wolverhampton, UK prod_sales (master) Prod_name Cust_id Amt_sold Bolt Pin Bolt Refreshing Nested MVs  DBMS_MVIEW.REFRESH_DEPENDENT with NESTED = TRUE refreshes all views in a hierarchy that depend on one or more specified base tables sum_prod_sales (nested) Prod_name SUM(Amt_sold) Bolt Pin 9.23 Fresh products Prod_id Prod_name 10 Bolt 14 Pin Sales Prod_id Cust_id Amt_sold Fresh Stale Refresh nested view Prod_name SUM(Amt_sold) Bolt Pin 9.23 Stale sales Prod_id Cust_id Amt_sold Update the sales table Refresh the master Prod_name Cust_id Amt_sold Bolt Pin Bolt Bolt

37 Carl Dudley – University of Wolverhampton, UK Nested MVs in Query Rewrite 2. Every master object requires a log containing the ROWID CREATE MATERIALIZED VIEW master_nest_mv BUILD IMMEDIATE ENABLE QUERY REWRITE AS SELECT p.prod_id, p.prod_name, c.channel_id, c.channel_desc FROM sales s, products p, channels c WHERE s.prod_id = p.prod_id AND s.channel_id = c.channel_id; 1.Create a master materialized view CREATE MATERIALIZED VIEW LOG ON sales WITH ROWID; CREATE MATERIALIZED VIEW LOG ON products WITH ROWID; CREATE MATERIALIZED VIEW LOG ON channels WITH ROWID; 3. The master materialized view also requires a log containing ROWID, each column in the materialized view itself, and all new values CREATE MATERIALIZED VIEW LOG ON master_nest_mv WITH ROWID (prod_id, prod_name, channel_id, channel_desc) INCLUDING NEW VALUES;

38 Carl Dudley – University of Wolverhampton, UK Nested MVs in Query Rewrite (continued) — AUTOTRACE shows evidence of use but does not show the hierarchy |Id| Operation |Name |Rows|Bytes|Cost(%CPU)| Time | | 0|SELECT STATEMENT | | 225| 9450| 3 (0) |00:00:01| | 1|MAT_VIEW REWRITE ACCESS FULL|NESTED_PROD_CHAN_MV| 225| 9450| 3 (0) |00:00:01| CREATE MATERIALIZED VIEW nested_prod_chan_mv BUILD IMMEDIATE ENABLE QUERY REWRITE AS SELECT channel_desc, prod_name, COUNT(prod_id) FROM master_nest_mv GROUP BY channel_desc, prod_name; 4. Create the nested MV based on the master MV

39 Carl Dudley – University of Wolverhampton, UK Nested MVs in Query Rewrite (continued) SELECT message FROM rewrite_table; MESSAGE QSM-01151: query was rewritten QSM-01033: query rewritten with materialized view, NESTED_PROD_CHAN_MV QSM-01336: the materialized view you specified (NESTED_PROD_CHAN_MV) was not used to rewrite the query QSM-01033: query rewritten with materialized view, MASTER_NEST_MV BEGIN DBMS_MVIEW.EXPLAIN_REWRITE( 'SELECT c.channel_desc, p.prod_name FROM sales s, products p, channels c WHERE s.prod_id = p.prod_id AND c.channel_id = s.channel_id GROUP BY channel_desc, prod_name ','NESTED_PROD_CHAN_MV'); END; /

40 Carl Dudley – University of Wolverhampton, UK Topics Materialized View Features and Syntax Query Rewrite Refreshing Materialized Views Nested Materialized Views Implementing Constraints via Materialized Views Fast and Complete Refresh Performance Multiple Materialized Views Estimating the Size of Materialized Views

41 Carl Dudley – University of Wolverhampton, UK Foreign Keys Referencing Non-unique Columns  Problem :  Need to enforce a foreign key constraint on the occupation column in the empdep table based on data in the emp table —Requires a reference to a non-unique column ( job ) in the emp table  Reference a materialized view carrying only unique values of job EMPNO JOB SALESMAN 7500 MANAGER 7902 MANAGER 7566 CLERK 7934 SALESMAN JOB SALESMAN MANAGER CLERK ENAME OCCUPATION WOODS CLERK JOHNSON MANAGER COX CLERK PITT CLERK SPINK SALESMAN DRAPER MANAGER EMP EMP_V1 EMPDEP primary key foreign key

42 Carl Dudley – University of Wolverhampton, UK Enforcing Foreign Keys Without Primary keys CREATE MATERIALIZED VIEW LOG ON emp WITH ROWID,PRIMARY KEY,SEQUENCE(job) INCLUDING NEW VALUES; CREATE MATERIALIZED VIEW emp_v1 REFRESH FAST ON COMMIT ENABLE QUERY REWRITE AS SELECT job,COUNT(*),COUNT(job) FROM emp GROUP BY job; ALTER MATERIALIZED VIEW emp_v1 ADD PRIMARY KEY (job); ALTER TABLE empdep ADD CONSTRAINT empdep_emp_v1 FOREIGN KEY (occupation) REFERENCES emp_v1; UPDATE empdep SET occupation = 'X'; ORA-02291: integrity constraint (SCOTT.EMPDEP_EMP_V1) violated - parent key not found job is not unique in emp job is unique in MV empdep must have only jobs in the emp table

43 Carl Dudley – University of Wolverhampton, UK Enforcing Complex Constraints with Materialized Views  Employee's appraisal date must be within their department's appraisal period EMPNO ENAME JOB MGR HIREDATE SAL COMM DEPTNO APP_DATE SMITH CLERK DEC MAR ALLEN SALESMAN FEB APR WARD SALESMAN FEB APR JONES MANAGER APR MAR-11 DEPTNO DNAME LOC START_APP_PERIOD END_APP_PERIOD RESEARCH DALLAS 01-FEB MAY SALES CHICAGO 01-MAR JUN-11  MV which returns app_date s with departmental appraisal periods CREATE MATERIALIZED VIEW emp_dept_appr_mv REFRESH FAST ON COMMIT AS SELECT e.deptno,e.app_date e_app_date,d.start_app_period d_start_app_period,d.end_app_period d_end_app_period,e.ROWID e_rowid,d.ROWID d_rowid FROM emp e,dept d WHERE e.deptno = d.deptno constraint checked when changes to the emp table are committed

44 Carl Dudley – University of Wolverhampton, UK Enforcing Complex Constraints with Materialized Views (continued)  Prevent invalid appr_date s appearing in the view ALTER TABLE emp_dept_appr_mv ADD CONSTRAINT emp_dept_appr_mv_chk1 CHECK (e_app_date BETWEEN d_start_app_period AND d_end_app_period) DEFERRABLE; —Constraint is checked on commit rather than on update  Insert a row with an appr_date outside of the departmental period INSERT INTO emp (empno,deptno,app_date) VALUES(1, 20, '01-jan-2013');  Constraint is not enforced until the commit (and refresh) takes place SQL> COMMIT; ORA-12048: error encountered while refreshing materialized View "EMP_DEPT_APPR_MV" ORA-02290: check constraint (emp_dept_appr_mv_chk1) violated

45 Carl Dudley – University of Wolverhampton, UK Testing Complex Constraint Checking with Materialized Views  Could lead to large numbers of rows in the MV when constraint is repeatedly NOT violated —Hence, simply reverse the MV condition to prevent valid data in the view —The MV never collects any data Constraint prevents the invalid data But the MV logs are still populated! AND NOT (e.app_date BETWEEN d.start_app_period AND d.end_app_period)

46 Carl Dudley – University of Wolverhampton, UK Indexes on Materialized Views  A single unique index is often created on an MV when it is built —If the MV is a grouped (aggregate) MV, the index is a function-based Btree and has a name of I_SNAP$_ —Built on virtual RAW column(s) built into the MV —Number of columns in the index is equal to number of grouping columns Names are similar to sys_nc00003$ CREATE MATERIALIZED VIEW sales_mv AS SELECT p.prod_subcategory, t.week_ending_day, SUM(s.amount_sold) AS sum_amount_sold FROM sales s, products p, times t WHERE s.time_id = t.time_id AND s.prod_id = p.prod_id GROUP BY p.prod_subcategory, t.week_ending_day generates two hidden indexed columns in the MV

47 Carl Dudley – University of Wolverhampton, UK Indexes on Grouped Materialized Views  Values in the hidden columns SELECT sys_nc00004$,sys_nc00005$,prod_subcategory,week_ending_day FROM sales_mv; SYS_NC00004$ SYS_NC00005$ PROD_SUBCATEGORY WEEK_ENDING_DAY D C Cameras 01-FEB-98 4D6F6E69746F C Monitors 22-MAR B746F C Desktop PCs 08-MAR D636F C6010B Camcorders 11-JAN D636F C Camcorders 25-JAN F C Accessories 01-FEB F C Accessories 01-MAR F6D F00 77C6010B Home Audio 11-JAN F6D F00 77C Home Audio 22-MAR-98 : : : : —43 is the ASCII value of ‘C’ in hexadecimal

48 Carl Dudley – University of Wolverhampton, UK Indexes on Non-grouped Materialized Views  If base table has a PK and a one to one mapping of rows with the MV —A normal Btree index is built on the PK within the view Oracle recommends building an index on any ROWID column in the MV and a concatenated index on all key columns of the base tables on the MV (for fast refresh) —If PK on the base table is explicitly named (e.g PK_EMP) The normal Btree index on the MV is named PK_EMP1 and supports a constraint named PK_EMP1  If no primary key on the base table —No index is built on the MV Indexes are not built when the USING NO INDEX clause is used in MV creation

49 Carl Dudley – University of Wolverhampton, UK Topics Materialized View Features and Syntax Query Rewrite Refreshing Materialized Views Nested Materialized Views Implementing Constraints via Materialized Views Fast and Complete Refresh Performance Multiple Materialized Views Estimating the Size of Materialized Views

50 Carl Dudley – University of Wolverhampton, UK Managing Update and Refresh Operations No. of rows updated UpdateRefresh FastCompleteFastComplete 100:00.200:00.100:01.400: : :02.200: :00.500:00.100:00.200: :00.700:00.100:01.500: :06.500:00.300:01.600: :58.000:19.100:39.100: :35.700:22.300:53.500: :58.900:28.501:40.600: :52.902:47.712:40.800:07.2  Tests performed on a row sales table having one MV in fast or complete refresh mode —Updates performed in batch mode (single update statement) on base table

51 Carl Dudley – University of Wolverhampton, UK Updating Tables having Materialized Views Batch Update Performance on Table with Fast or Complete Refresh Materialized View 00: : : : : : : : : : : No of Rows Updated Time Taken (Minutes) Fast Complete  Fast refresh has the overhead of MV log maintenance

52 Carl Dudley – University of Wolverhampton, UK Fast and Complete Refresh Performance Performance of Fast Vs Complete Refresh 00: : : : : : : : : : : : : No of Rows Updated Time Taken to Refresh (Minutes) Fast Complete  MV is truncated during a complete refresh —Faster for large scale updates

53 Carl Dudley – University of Wolverhampton, UK Overheads on OLTP - Transaction Test  Repeated updates on a single row BEGIN FOR i in 1..n LOOP UPDATE sales SET amount_sold = amount_sold +.01 WHERE prod_id = 14 AND cust_id = AND time_id = '03-NOV-01' AND channel_id = 3; COMMIT; END LOOP; END;  sales table has an index to avoid full table scans masking update timings ALTER TABLE sales ADD CONSTRAINT pk_sales PRIMARY KEY (prod_id,cust_id,time_id,channel_id);

54 Carl Dudley – University of Wolverhampton, UK Transaction Overheads - Test  Routine timed with and without MV log on sales —Log was truncated after each test CREATE MATERIALIZED VIEW LOG ON sales WITH ROWID (cust_id,prod_id,channel_id,time_id,amount_sold,quantity_sold) INCLUDING NEW VALUES;  Timings also produced with and without the COMMIT statement

55 Carl Dudley – University of Wolverhampton, UK Timings With and Without MV log and COMMIT Number of iterations (n) Time taken (seconds) With commitWithout commit No MV log MV log present No MV log MV log present  Presence of MV log can result in up to 250% performance overhead

56 Carl Dudley – University of Wolverhampton, UK Contention Caused by Materialized Views  Applies to REFRESH ON COMMIT views in OLTP environments —Refresh takes place as part of the OLTP transaction —Aggregate MVs are prone to contention issues —Updates on base tables are 'condensed' into fewer rows in the MV —Causes locking and contention issues as transactions queue to update the grouped row in the MV EMPNO ENAME SAL DEPTNO CLARK KING MILLER SMITH ADAMS FORD SCOTT JONES WARD ALLEN MARTIN BLAKE TURNER JAMES CREATE MATERIALIZED VIEW m1 REFRESH FAST ON COMMIT AS SELECT deptno, SUM(sal),COUNT(*), COUNT(sal) FROM emp GROUP BY deptno; DEPTNO SUM(SAL) COUNT(*) COUNT(sal) Possible contention for locks of type JI

57 Carl Dudley – University of Wolverhampton, UK Topics Materialized View Features and Syntax Query Rewrite Refreshing Materialized Views Nested Materialized Views Implementing Constraints via Materialized Views Fast and Complete Refresh Performance Multiple Materialized Views Estimating the Size of Materialized Views

58 Carl Dudley – University of Wolverhampton, UK Using Multiple Materialized Views CREATE MATERIALIZED view prod_high_mv BUILD IMMEDIATE ENABLE QUERY REWRITE AS SELECT prod_id, SUM(amount_sold) FROM sales WHERE prod_id > 15 GROUP BY prod_id; CREATE MATERIALIZED view prod_low_mv BUILD IMMEDIATE ENABLE QUERY REWRITE AS SELECT prod_id, SUM(amount_sold) FROM sales WHERE prod_id <= 15 GROUP BY prod_id; PROD_ID SUM(AMOUNT_SOLD) PROD_ID SUM(AMOUNT_SOLD)  Two materialized views each covering only part of the base data —Table contains sales data for prod_id s from 10 to 19

59 Carl Dudley – University of Wolverhampton, UK Using Multiple Materialized Views Execution Plan | Id | Operation | Name |Rows | Bytes | Cost (%CPU)| Time | | 0 | SELECT STATEMENT | | 10 | 110 | 6 (50)| 00:00:01| | 1 | UNION-ALL | | | | | | | 2 | MAT_VIEW REWRITE ACCESS FULL| PROD_LOW_MV | 6 | 66 | 3 (0)| 00:00:01| | 3 | MAT_VIEW REWRITE ACCESS FULL| PROD_HIGH_MV| 4 | 44 | 3 (0)| 00:00:01| SELECT prod_id, SUM(amount_sold) FROM sales -- WHERE prod_id > 0 GROUP BY prod_id; A WHERE clause appears to be necessary for multiple MVs to be used in pre 11gR2

60 Carl Dudley – University of Wolverhampton, UK Overlapping Materialized Views CREATE MATERIALIZED VIEW prod_low_mv BUILD IMMEDIATE ENABLE QUERY REWRITE AS SELECT prod_id, SUM(amount_sold) FROM sales WHERE prod_id <= 15 GROUP BY prod_id; PROD_ID SUM(AMOUNT_SOLD) CREATE MATERIALIZED VIEW prod_high_mv BUILD IMMEDIATE ENABLE QUERY REWRITE AS SELECT prod_id, SUM(amount_sold) FROM sales WHERE prod_id >= 15 GROUP BY prod_id; PROD_ID SUM(AMOUNT_SOLD)

61 Carl Dudley – University of Wolverhampton, UK Overlapping Materialized Views  Duplicate row for prod_id = 15 is (correctly) not shown in query output SELECT prod_id, SUM(amount_sold) FROM sales WHERE prod_id > 0 GROUP BY prod_id ORDER BY prod_id; PROD_ID SUM(AMOUNT_SOLD)

62 Carl Dudley – University of Wolverhampton, UK Overlapping Materialized Views  But the plan shows a UNION-ALL operation across the two views —This is due to re-aggregation that is required across inline views |Id| Operation | Name | | 0| SELECT STATEMENT | | | 1| VIEW | | | 2| UNION-ALL | | | 3| MAT_VIEW REWRITE ACCESS FULL | PROD_HIGH_MV | | 4| MAT_VIEW REWRITE ACCESS FULL | PROD_LOW_MV | SELECT prod_id,SUM(sa) FROM (SELECT prod_id, SUM(amount_sold) sa FROM sales WHERE prod_id <= 15 GROUP BY prod_id UNION SELECT prod_id, SUM(amount_sold) sa FROM sales WHERE prod_id >= 15 GROUP BY prod_id) GROUP BY prod_id ORDER BY prod_id; Equivalent direct use of MVs generates exact same plan even when ALL is not specified

63 Carl Dudley – University of Wolverhampton, UK Multiple MVs with Different Aggregate Functions  The two views generate same groups —Different aggregate functions CREATE MATERIALIZED view prod_avg_mv BUILD IMMEDIATE ENABLE QUERY REWRITE AS SELECT prod_id, AVG(amount_sold) FROM sales GROUP BY prod_id; CREATE MATERIALIZED view prod_sum_mv BUILD IMMEDIATE ENABLE QUERY REWRITE AS SELECT prod_id, SUM(amount_sold) FROM sales GROUP BY prod_id; SELECT prod_id, SUM(amount_sold), AVG(amount_sold) FROM sales GROUP BY prod_id; | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | | 0 | SELECT STATEMENT | | 72 | 648 | 960 (3)| 00:00:12 | | 1 | HASH GROUP BY | | 72 | 648 | 960 (3)| 00:00:12 | | 2 | TABLE ACCESS FULL| SALES | 500K| 4394K| 940 (1)| 00:00:12 | —Neither view is used

64 Carl Dudley – University of Wolverhampton, UK Multiple MVs with Different Aggregate Functions - a Workaround  Encapsulate the views in the FROM clause —Results from views are hash-joined SELECT a.prod_id, a.sum_prod, b.avg_prod FROM (SELECT prod_id,SUM(amount_sold) sum_prod FROM sales GROUP BY prod_id) a, (SELECT prod_id,AVG(amount_sold) avg_prod FROM sales GROUP BY prod_id) b WHERE a.prod_id = b.prod_id; | Id | Operation | Name | Rows| Bytes| Cost (%CPU)| Time | | 0 | SELECT STATEMENT | | 72| 2664| 7 (15)| 00:00:01 | | 1 | HASH JOIN | | 72| 2664| 7 (15)| 00:00:01 | | 2 | MAT_VIEW REWRITE ACCESS FULL| PROD_SUM_MV | 72| 792| 3 (0)| 00:00:01 | | 3 | MAT_VIEW REWRITE ACCESS FULL| PROD_AVG_MV | 72| 1872| 3 (0)| 00:00:01 | —Both views are used

65 Carl Dudley – University of Wolverhampton, UK Topics Materialized View Features and Syntax Query Rewrite Refreshing Materialized Views Nested Materialized Views Implementing Constraints via Materialized Views Fast and Complete Refresh Performance Multiple Materialized Views Estimating the Size of Materialized Views

66 Carl Dudley – University of Wolverhampton, UK Estimating Size of Materialized Views  Materialized views can consume significant amounts of storage  Estimate size of MV with dbms_mview.estimate_mview_size —Estimates number of rows and bytes  Accuracy of the estimate procedure was tested on an original sales table of rows, joined with customers and products

67 Carl Dudley – University of Wolverhampton, UK The Oracle Block Block 17 Free Space Row 4 Block header Reserved for updates to existing rows 1 2 Row numbers  8K block —Row header occupies approximately 100 bytes —819 bytes of free space reserved for updates (default PCTFREE of 10%) —Inserted data can occupy around 7300 bytes

68 Carl Dudley – University of Wolverhampton, UK Estimate Routine The output from the routine is as follows : Number of rows is : Number of bytes is : DECLARE nr NUMBER; --number of rows nb NUMBER; --number of bytes BEGIN dbms_mview.estimate_mview_size (stmt_id => 'mv_est', select_clause => 'SELECT c.cust_id,c.cust_city,p.prod_id,p.prod_name FROM sales s, customers c, products p WHERE s.cust_id = c.cust_id AND s.prod_id = p.prod_id', num_rows => nr, num_bytes => nb); dbms_output.put_line('Number of rows is : '||TO_CHAR(nr)); dbms_output.put_line('Number of bytes is : '||TO_CHAR(nb)); END;

69 Carl Dudley – University of Wolverhampton, UK Accuracy of Estimate  On building the view, the number of rows was actually —This shows that the row estimate of is reasonably accurate  Its ROWID s were interrogated to find the actual block storage SELECT COUNT(COUNT(*)) “No. of Blocks” FROM mv_est GROUP BY dbms_rowid.rowid_relative_fno(ROWID),dbms_rowid.rowid_block_number(ROWID); No. of Blocks  If each block has around 7300 bytes of data, then the number of bytes is (7300*6170) —This is wildly different from the estimated value of

70 Carl Dudley – University of Wolverhampton, UK Checking the Dictionary EXEC dbms_stats.gather_table_stats('SH',' ') SELECT avg_col_len,column_name FROM user_tab_cols WHERE table_name IN ('CUSTOMERS','PRODUCTS') AND column_name IN ('CUST_ID','PROD_ID','CUST_CITY','PROD_NAME'); AVG_COL_LEN COLUMN_NAME CUST_ID 10 CUST_CITY 4 PROD_ID 26 PROD_NAME  So, on average, each row should consume (row overhead) bytes  The total number of occupied blocks should be (45+5)*918443/7300 = 6290 blocks —This is close to the result obtained from the actual ROWID s of the MV (6170)  Perhaps a more accurate value can be obtained from the dictionary

71 Carl Dudley – University of Wolverhampton, UK Results and Conclusions – Query Rewrite  MVs return queries in a fraction of the time  MVs are considered for rewrite even if they contain only part of the data required by the query  MVs are not considered for query rewrite if base table(s) are updated —Unless the session settings are altered to STALE_TOLERATED mode This emphasizes the need for refreshing the views  Multiple materialized views can be used to satisfy a single query — However if the aggregate materialized views do not include the same aggregate function then neither view will be considered  The estimate_mview_size procedure — Extremely accurate in estimating the number of rows an MV will contain — Not so accurate at predicting actual storage consumption

72 Carl Dudley – University of Wolverhampton, UK Results and Conclusions – Refresh Methods  REFRESH FAST —Efficient refresh performance when the numbers of rows affected by a single transaction was small —Due to the presence of the materialized view log —BUT if transaction rate is significant, could have significant effect  REFRESH COMPLETE ON DEMAND suits batch processing environments —Table updates are large and infrequent  REFRESH FAST ON COMMIT suits OLTP environments —WELL perhaps —Regular yet small table updates depending on real-time data

Carl Dudley – University of Wolverhampton, UK Materialized Views Carl Dudley University of Wolverhampton, UK Oracle ACE Director UKOUG Committee