40183 : Oracle XML DB Performance and Tuning Mark D. Drake Senior Product Manager Session id:

Introduction

Managing XML Content  XML Stored as XMLType  SQL functions allow XPath expressions to operate on XML content – extract(), extractValue(), updateXML(), existsNode()  Two modes of operation – Functional Evaluation – Query-Rewrite

Functional Evaluation  Works with all XMLType columns and Tables.  Parse the XML document and construct a DOM  Evaluate XPath expressions using DOM API  Based on proven XDK ‘C’ XML Parser and DOM  Most appropriate to operations on small numbers of small documents

XPath Re-write  Query and Update operations  Requires Schema based XMLType  Operates directly on the underlying Object Store  Relational database engine used to evaluate XPath expression – XPath Expression translated into equivalent SQL – Eliminates need to parse XML and construct DOM  Allows efficient operations on large collections of XML Documents

Collections

What is a collection  A Collection is a set of elements with the same name and similar structure Duel at Ichijoji Temple <Part Id=" " UnitPrice="29.95" Quantity="3"/ The Red Shoes

Defining a Collection  In XML Schema, Collections are defined using maxOccurs attribute <xs:complexType name="LineItemsType“ <xs:element name="LineItem“ type="LineItemType“ maxOccurs="unbounded"/>

Working with Collections  A Typical XML document contains one or more collections  Collections may be Nested – A collection may contain a collections  Query and Updating collections present the biggest challenge

Querying Collections  Sample Query on a Collection Get the Reference and Item Number of each LineItem that contains a reference to Part “ “ XPATH := select extractValue(p.object_value,'/PurchaseOrder/Reference') from PURCHASEORDER p, table (xmlsequence(extract(p.object_value,XPATH))) l

Querying Collections  Use the extract() function to get the set of nodes that match an XPath expression XPATH := select extractValue(object_value,'/PurchaseOrder/Reference') from PURCHASEORDER p, table (xmlsequence(extract(p.object_value,XPATH))) l

Querying Collections  The extract function will be evaluated for each document in the table.  An XMLType will be generated from each document that contains a node matching the XPath expression.  The XML Type will be contain an XML Fragment – The Fragment will consist of the set of nodes that match the specified XPATH expression.

Querying Collections  Use the xmlSequence function to generate a set of XMLType Objects from each fragment XPATH := select extractValue(object_value,'/PurchaseOrder/Reference') from PURCHASEORDER p, table (xmlsequence(extract(p.object_value,XPATH))) l

Querying Collections  xmlSequence will create a set of XMLType objects from the result of the extract.  Each set will contain one XMLType object for each top level node in the XML fragment.  Each XMLType will consist of a well-formed XML document.

Querying Collections  Use the TABLE function to convert the set of XMLType objects into a virtual table XPATH := select extractValue(p.object_value,'/PurchaseOrder/Reference') from PURCHASEORDER p, table (xmlsequence(extract(p.object_value,XPATH))) l  Join the virtual table with the base table to get required result set

Managing Collections Managing Collections

Collection Management  Four options for storing collections – BLOB Storage  SQL objects stored in a single BLOB Column – Nested Table Storage  SQL objects stored as rows in a Nested Table – CLOB Storage  XML text – XMLType Table Storage  XMLType objects stored in an XMLType Table

Why different storage models ?  Applications have different requirements – Querying within in a collection – Indexing within a collection – Updating within a collection  Choosing the correct storage model is vital

Collections and VARRAYS  VARRAY: SQL concept for managing collections  VARRAY can be collection of Scalars or collection of SQL Type – AUTHOR_V VARRAY(n) of VARCHAR2(40) – LINE_ITEM_V VARRAY(n) of LINE_ITEM_T  In XML DB the members of a collection are mapped to a SQL VARRAY.

LOB

LOB model  Default : No Annotation Required  Collection stored as VARRAY of SQL Objects  SQL Objects serialized into a single BLOB  Indexing using CtxXPath index  No partial updates – update requires re-writing entire BLOB  Query re-write for XPath expressions

LOB : Objects SQL> desc LINEITEMS_T LINEITEMS_T is NOT FINAL Name Null? Type SYS_XDBPD$ XDB.XDB$RAW_LIST_T LINEITEM LINEITEM_V

LOB : Storage SQL> describe LINEITEM_V LINEITEM_V VARRAY( ) OF LINEITEM_T LINEITEM_T is NOT FINAL Name Null? Type SYS_XDBPD$ XDB.XDB$RAW_LIST_T ITEMNUMBER NUMBER(38) DESCRIPTION VARCHAR2(256 CHAR) PART PART_T

LOB Storage Reference User Id … … LineItem ABANDA … … ABANDA-20.. LINEITEM_V ( LINEITEM_T(...), LINEITEM_T(...), LINEITEM_T(...), LINEITEM_T(...) )

LOB : Query Re-Write SELECT P.XMLDATA."Reference", L."ItemNumber" FROM PURCHASEORDER P, TABLE (P.XMLDATA."LineItems"."LineItem") L WHERE L."Part"."PartNumber" = ' '  Evaluated using SQL to process VARRAY of LINEITEM objects  Entire VARRAY has to be loaded into Memory

Nested Tables

Nested Table model  Annotation: xdb:storeVarrayAsTable=“true”  Collection converted into a set of SQL objects  Each SQL Object stored as a separate row in a Nested Table.  Nesting of Nested Tables for collections within collections  SQL Based Fragment access and update  Index with B-Tree, Functional and CtxXPath indexes

Nested Table : Objects SQL> desc LINEITEMS_T LINEITEMS_T is NOT FINAL Name Null? Type SYS_XDBPD$ XDB.XDB$RAW_LIST_T LINEITEM LINEITEM_V

Nested Table : Objects SQL> describe LINEITEM_V LINEITEM_V VARRAY( ) OF LINEITEM_T LINEITEM_T is NOT FINAL Name Null? Type SYS_XDBPD$ XDB.XDB$RAW_LIST_T ITEMNUMBER NUMBER(38) DESCRIPTION VARCHAR2(256 CHAR) PART PART_T SQL> describe LINEITEM_TABLE Name Null? Type SYS_XDBPD$ XDB.XDB$RAW_LIST_T ITEMNUMBER NUMBER(38) DESCRIPTION VARCHAR2(256 CHAR) PART PART_T

Nested Table : Storage Reference User Id … … LineItem ABANDA … … ABANDA-20.. ID ItemNumber Description Part Good Morn… … … 1 Uriah Hee… … … 21 Sisters … … 31 The Prince… … …

Nested Table : Query Re-Write SELECT P.XMLDATA."Reference", L.ITEMNUMBER FROM PURCHASEORDER P, LINEITEM_TABLE l WHERE L."Part."PartNumber"=' ' AND L.NESTED_TABLE_ID = SETID(P.XMLDATA."LineItems"."LineItem“ )  Evaluated using SQL to query contents of the Nested Table

Working with Nested Tables  xdb:storeVarrayAsTable=“true” is a “shotgun” approach – Every collection stored as a Nested Table – Nested Tables have ‘unfriendly’ names  Simple cases – Rename tables created by Schema Registration  Complex cases – Create tables manually using CREATE TABLE

Renaming Nested Tables  Query USER_NESTED_TABLES to find the direct descendant of the Parent Table SQL> select PARENT_TABLE_COLUMN, TABLE_NAME 2 from USER_NESTED_TABLES 3 where PARENT_TABLE_NAME = 'PURCHASEORDER' 4 / PARENT_TABLE_COLUMN TABLE_NAME "XMLDATA"."LINEITEMS"."LINEITEM" SYS_NTDfLwYKWcRxmyssvmCvRMqw== "XMLDATA"."ACTIONS"."ACTION" SYS_NTqxF5epLrSniXaAWq5A4Uig==

Renaming Nested Tables SQL> rename "SYS_NTDfLwYKWcRxmyssvmCvRMqw==" to LINEITEM_TABLE 2 / Table renamed. SQL> rename "SYS_NTqxF5epLrSniXaAWq5A4Uig==" to ACTION_TABLE 2 / Table renamed. SQL> select PARENT_TABLE_COLUMN, TABLE_NAME 2 from USER_NESTED_TABLES 3 where PARENT_TABLE_NAME = 'PURCHASEORDER' 4 / PARENT_TABLE_COLUMN TABLE_NAME "XMLDATA"."ACTIONS"."ACTION" ACTION_TABLE "XMLDATA"."LINEITEMS"."LINEITEM" LINEITEM_TABLE

Multi-Level Nested Tables  Collections within Collections result in Storage Models consisting of multiple levels of Nested Tables  Repeat the process using the Nested Table name as the PARENT_TABLE_NAME

Manual Table Creation  Can create Nested Tables as part of create Table  Need to know the name of the VARRAYs that will stored as Nested Tables  Allows flexibility, only use Nested Table Storage only where it adds value  Must use Nested Table storage with collections that contain an element stored as a CLOB.

Manual Table Creation Example create table PURCHASEORDER of XMLTYPE XMLSCHEMA " ELEMENT "PurchaseOrder" varray "XMLDATA"."ACTIONS"."ACTION" store as table ACTION_TABLE ( (primary key (NESTED_TABLE_ID, ARRAY_INDEX)) organization index overflow ) varray "XMLDATA"."LINEITEMS"."LINEITEM" store as table LINEITEM_TABLE ( (primary key (NESTED_TABLE_ID, ARRAY_INDEX)) organization index overflow )

CLOB

CLOB model  Annotation: xdb:SQLType=“CLOB”  Collection stored as XML text – Reduced complexity – Fast Ingestion and Retrieval of Collection – Lower memory and processing requirements  Index using CTXPATH index  No Optimization of Fragment Level operations – XPath based Functions evaluated via DOM API – Fetching Fragments requires Parsing – Updating Requires Parsing and Rewriting

CLOB : Objects SQL> describe LINEITEMS_T LINEITEMS_T is NOT FINAL Name Null? Type SYS_XDBPD$ XDB.XDB$RAW_LIST_T LINEITEM CLOB

CLOB : Storage Reference User Id … … LineItem ABANDA … … ABANDA-20.. Good … Uriah..

CLOB: Query Re-Write SELECT P.XMLDATA."Reference", FROM PURCHASEORDER P, TABLE ( XMLSEQUENCE ( EXTRACT ( VALUE(P), '/PurchaseOrder/LineItems/LineItem/' || ) ) ) l  Evaluated using by Parsing CLOB and using DOM API

XMLType Table

XMLType Table model  Annotation xdb:SQLInline=“false”  Also known as ‘out-of-line’ storage  Collection converted into a set of XMLType objects stored in a separate XMLType table.  Parent table contains a VARRAY of REF XMLTYPE that points to rows in the child table  Index using B-Tree, Functional and CtxXPath indexes  SQL Based Fragment access and update.  Must allow Schema Registration to create the Tables.

XMLType Table : Objects SQL> describe LINEITEMS_T LINEITEMS_T is NOT FINAL Name Null? Type SYS_XDBPD$ XDB.XDB$RAW_LIST_T LINEITEM XDB.XDB$XMLTYPE_REF_LIST_T

XMLType Table : Objects SQL> describe XDB.XDB$XMLTYPE_REF_LIST_T XDB.XDB$XMLTYPE_REF_LIST_T VARRAY( ) OF REF XMLTYPE SQL> describe LINEITEM_TABLE Name Null? Type TABLE of SYS.XMLTYPE ( XMLSchema " Element "LineItem“ ) STORAGE Object-relational TYPE "LINEITEM_T"

XMLType Table : Storage Reference User Id … … LineItem ABANDA … … ABANDA-20.. ItemNumber Description Part Good Morn… … … 1 Uriah Hee… … … 2 Sisters … … 3 The Prince… … … 4 Ref, Ref…

XMLType with Ref Table  Combine with xdb:storeVarrayAsTable=“true” to force VARRAY of REFS to be stored in a nested table  Each row in the nested table consists of SETID and a REF to a ROW in the child table  Scope the REFS in the child table to allow full optimization of queries rename "SYS_NTDfLwYKWcRxmyssvmCvRMqw==" to LINEITEM_REF_TABLE alter table LINEITEM_REF_TABLE add (scope for (COLUMN_VALUE) is LINEITEM_TABLE)

Scope REF of XMLType select PARENT_TABLE_COLUMN, TABLE_NAME from USER_NESTED_TABLES where PARENT_TABLE_NAME = 'PURCHASEORDER‘ / PARENT_TABLE_COLUMN TABLE_NAME "XMLDATA"."LINEITEMS"."LINEITEM" SYS_NTDfLwYKWcRxmyssvmCvRMqw== rename "SYS_NTDfLwYKWcRxmyssvmCvRMqw==" to LINEITEM_REF_TABLE / alter table LINEITEM_REF_TABLE add (scope for (COLUMN_VALUE) is LINEITEM_TABLE) / create index LINEITEM_REF_INDEX on LINEITEM_REF_TABLE (COLUMN_VALUE) /

ID XMLType Table : Storage Reference User Id … … LineItem ABANDA … … ABANDA-20.. ItemNumber Description Part Good Morn… … … 1 Uriah Hee… … … 2 Sisters … … 3 The Prince… … … Ref 1 1

XMLType Table : Query Rewrite SELECT P.XMLDATA."Reference", L.XMLDATA."Item" FROM PURCHASEORDER p, TABLE("P.XMLDATA."LineItems"."LineItem") R, LINEITEM_TABLE L WHERE L."Part"."PartNumber" = ' ' AND VALUE(R)= Ref(l)  Evaluated using SQL to query contents of the XMLType table

Relative FTP Insert performance

Un-Indexed Query Performance

Indexing Collections

 Collections stored as LOB or CLOB can only be indexed using CtxXPath Index  Collections stored as Nested Tables or Out- of-Line tables can be index using B-Tree Indexes – Indexes on collections are created on the nested table or Out-of-Line Table, not the Parent Table

Indexing with Nested Tables  Indexes on Nested Table are defined in terms of the SQL Type structure  Nested Tables contain a foreign key called NESTED_TABLE_ID  Consider adding NESTED_TABLE_ID to any secondary indexes

Nested Table: Secondary Index create index iPartNumberIndex on LINEITEM_TABLE l (l.PART.PART_NUMBER, l.NESTED_TABLE_ID) Query Plan showing use of Index | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | | 0 | SELECT STATEMENT | | 662 | 17M| 89 (2)| 00:00:02 | | 1 | NESTED LOOPS | | 662 | 17M| 89 (2)| 00:00:02 | |* 2 | INDEX UNIQUE SCAN | LINEITEM_TABLE_IOT | 82 | 370K| 6 (0)| 00:00:01 | |* 3 | INDEX RANGE SCAN | IPARTNUMBERINDEX | 33 | | 3 (34)| 00:00:01 | | 4 | TABLE ACCESS BY INDEX ROWID| PURCHASEORDER | 8 | 176K| 2 (50)| 00:00:01 | |* 5 | INDEX UNIQUE SCAN | SYS_C | 1 | | 1 (100)| 00:00:01 |

Indexing with XMLType Tables  Indexes on Out-Of-Line XMLType tables are created using XPath syntax  Index is defined in terms of the extractValue() functions  Index is not a function index. XPath re-write allows the index to be converted into Object-Relational SQL.  XPath is from the root element of the Out-of-Line table, not the root of the parent document  When storing REFs as nested table remember to index the nested table

XMLType Table: Secondary Index create index iPartNumberIndex on LINEITEM_TABLE l create index LINEITEM_REF_INDEX on LINEITEM_REF_TABLE (COLUMN_VALUE) Query Plan showing use of Index | Id | Operation | Name | Rows | Bytes | Cost (%CPU)| Time | | 0 | SELECT STATEMENT | | 4626 | 100M| 591 (2)| 00:00:08 | | 1 | NESTED LOOPS | | 4626 | 100M| 591 (2)| 00:00:08 | |* 2 | HASH JOIN | | 571 | 136K| 14 (15)| 00:00:01 | | 3 | TABLE ACCESS BY INDEX ROWID| LINEITEM_TABLE | 571 | 115K| 6 (17)| 00:00:01 | |* 4 | INDEX RANGE SCAN | IPARTNUMBERINDEX | 571 | | 2 (50)| 00:00:01 | | 5 | INDEX FAST FULL SCAN | LINEITEM_REF_INDEX | 8168 | 295K| 8 (13)| 00:00:01 | | 6 | TABLE ACCESS BY INDEX ROWID | PURCHASEORDER | 8 | 176K| 2 (50)| 00:00:01 | |* 7 | INDEX RANGE SCAN | SYS_C | 8 | | 1 (100)| 00:00:01 |

Indexed Query Performance

DOM Fidelity

 Multiple namespaces with Prefixes  XMLSchemaInstance attributes – xsi:nill, xsi:Type  Mixed Text, Substitution groups  Comments and Processing Instructions  Ordering of nodes in an all, choice  Empty Vs Missing Nodes  Default Values

Dom Fidelity  Requires additional processing and storage overhead  Specific meta data maintained for each instance – Meta data managed at the ‘Type’ Level in the SYS_XDBPD$ attribute – SYS_XDBPD$ is stored as a (in-line) LOB.

Do you need it ?  Can disable on a Type by Type basis using annotation xdb:maintainDOM=“false”  Not using Namespaces  No Comments or Processing Instructions  Do not care about empty Vs missing elements  Elements in an All and Choice will be ordered as per the Schema, not the instance  Not worried about defaults  Type does not allow Mixed text

Type defn with DOM Fidelity desc LINEITEMS_T LINEITEMS_T is NOT FINAL Name Null? Type SYS_XDBPD$ XDB.XDB$RAW_LIST_T LINEITEM LINEITEM_V desc LINEITEM_V LINEITEM_V VARRAY( ) OF LINEITEM_T LINEITEM_T is NOT FINAL Name Null? Type SYS_XDBPD$ XDB.XDB$RAW_LIST_T ITEMNUMBER NUMBER(38) DESCRIPTION VARCHAR2(256 CHAR) PART PART_T

xdb:maintainDOM <xs:attribute name="ItemNumber" type="xs:integer" xdb:SQLName="ITEMNUMBER" xdb:SQLType="NUMBER"/>

Type defn without DOM Fidelity desc LINEITEMS_T LINEITEMS_T is NOT FINAL Name Null? Type LINEITEM LINEITEM_V desc LINEITEM_V LINEITEM_V VARRAY( ) OF LINEITEM_T LINEITEM_T is NOT FINAL Name Null? Type ITEMNUMBER NUMBER(38) DESCRIPTION VARCHAR2(256 CHAR) PART PART_T

Dom Fidelity

Hierarchical Tables

 XMLType Tables can be Hierarchically Enabled – Table is linked to the Oracle XML DB repository – Each Row in the table corresponds to a Resource (document) in the XML DB repository – Row Level Security defined by associating an ACL with a resource – DML operations on the row reflected in the resource (Last Modified Date, Delete)

Hierarchical Considerations  Tables created by Schema Registration are Hierarchically Enabled – DML operations constrained by ACL – Table Level Triggers sync DML operations on base-table with XDB repository  Cannot bypass ACL security by accessing content via SQL  Delete of a row in a table will delete the corresponding Resource

ACL Management  XML DB architected to support systems with thousands or millions of ACLs  ACL based security automatically enforced by calling the sys_checkAcl() function  ACL check is based on ACLOID and OID  Dedicated ACL cache ensures that Oracle XML DB can locate ACLs very efficiently  ACLOID and OID are stored as hidden columns on an XMLType table

ACL Checking  ACL checking can affect query performance – Need ACLOID and OID in addition to other columns  ACL Based security can be disabled using dbms_xdbz.disable_hierarchy() – Turns off ACL Evaluation and repository synchronization  In some cases disabling the hierarchy can lead to significant performance benefits

A Q & Q U E S T I O N S A N S W E R S