Presentation on theme: "Limor Fledel-Vagman BI Practice Manager Oracle Israel OBIEE Data Federation."— Presentation transcript:
Limor Fledel-Vagman BI Practice Manager Oracle Israel OBIEE Data Federation
OBIEE Federation Vertical Federation – Integrating two or more disparate data source having different levels of granularity joined by one or more conformed dimensions – Example: Essbase Sales cube to relational Sales detail data (drill- through) Horizontal Federation – Integrating two or more disparate data source having the same level of granularity joined by one or more conformed dimensions – Example: Essbase Sales cube with HR relational database
LTS ??? Logical table sources (LTSs) are a key feature within the OBIEE semantic model
OBIEE Aggregate Navigation
Why Aggregate Navigation? Problem: Summary questions are slow – Example, All sales worldwide for last 2 years by quarter by family – This will potentially sum/group by millions of rows! – The disk spin time to scan all these rows will be very long Solution: ETL to batch build high level aggregates – Oracle BI EE has aggregate navigation to use them – Aggregates can be in the same database as the source – They can also be in a different relational source or Essbase – Often orders of magnitude performance improvement
Oracle BI EE Architecture Relational Tables BI Server Semantic Layer Metadata SQL
Oracle BI EE Aggregate Navigation Relational Detail Summary Semantic Layer Metadata Metadata describes aggregate mappings SQL At query time, BI Server queries the fastest source that has enough detail to satisfy the user request Load time aggregation Performance of highly summarized requests is dramatically improved BI Server
Federation Enables External Aggregates Relational Detail Summary Semantic Layer Metadata SQL Load time aggregation Summary SQL Aggregates can be in other federated sources BI Server
External Aggregates Can Be in Essbase Relational Detail Summary Essbase Semantic Layer Metadata Aggregate navigation can include multidimensional SQL MDX Load time aggregation BI Server
External Aggregates Can Be Oracle OLAP Relational Detail Summary Oracle OLAP AW Semantic Layer Metadata Aggregate navigation can include multidimensional SQL MDX Load time aggregation BI Server "This information is not a commitment to deliver any material, code, or functionality. The development, release, and timing of any features or functionality described remains at the sole discretion of Oracle"
How is Aggregate Navigation Set Up? Database Design time: – DBA defines aggregate stars at various grain combinations – Builds ETL to load the aggregates each night StarTimeProduct BaseDaySKU Agg 1MonthBrand Agg 2Qtr-
Mapping a Base-Level Physical Star Business Model and Mapping Layer Physical LayerMappings D0 Time D4 Product F0 Rev Base Measures D_Day D_Prod_SKU F_Bill
Adding a Moderately Aggregated Star Business Model and Mapping Layer D0 Time D4 Product F0 Rev Base Measures D_Day D_Prod_SKU F_Bill D_Month D_Brand F_Bill_agg2 Physical LayerMappings
Adding a Highly-Aggregated Star Business Model and Mapping Layer D0 Time D4 Product F0 Rev Base Measures D_Day D_Prod_SKU F_Bill D_Month D_Brand F_Bill_agg2 D_Qtr F_Bill_agg1 Physical LayerMappings
Essbase as an Aggregate Source Business Model and Mapping Layer D0 Time D4 Product F0 Rev Base Measures D_Day D_Prod_SKU F_Bill Physical LayerMappings Month Brand Billed Amt
Adding a Highly-Aggregated Star Business Model and Mapping Layer D0 Time D4 Product F0 Rev Base Measures D_Day D_Prod_SKU F_Bill D_Month D_Brand F_Bill_agg2 D_Qtr F_Bill_agg1 How does the query planner decide which FJC to use as its source for a given request? Physical LayerMappings
Selecting an Aggregate or Base Star at Query Time How does the query planner decide which star to use as its source for a given LSQL request? First, it eliminates sources that dont have enough detail to answer the question – Query grain < star grainstar not qualified – Query grain = star grainstar qualified – Query grain > star grainstar qualified Second, it estimates the fastest/prioritized source
How BI Server Knows the Grain of a Star At query time, the BI Server checks the available mappings to find which ones are qualified by grain
How Does It Know Which Star is Fastest? Number of elements Be sure to correctly populate the Dimension Levels Number of elements at this level OBIEE 10g
How Does It Know Which Star is Fastest? LTS Priority Group Order LTS Priority Group Order Lower value = higher priority LTS Priority Group becomes main decider in which LTS to use OBIEE 11g New Feature
Aggregate Persistence Feature Managing aggregates manually has high TCO Oracle BI EE automates creation of aggregates Leverages existing metadata Orders of magnitude performance boost; low TCO
Aggregate Persistence with Essbase & Oracle OLAP Detail Semantic Layer Metadata Admin Tool: Aggregate build automation wizard Define cube schema Create load rules & schedule Summary Essbase or AW BI Server Define agg navigation metadata "This information is not a commitment to deliver any material, code, or functionality. The development, release, and timing of any features or functionality described remains at the sole discretion of Oracle"
Administration Steps Design aggregate facts and grains Wizard to create metadata 1 Target database for create/load 3 2
Aggregate Persistence Wizard
Automates the creation of physical aggregate tables and their corresponding objects in the repository Use wizard to build script to generate aggregate tables. Script generates physical tables, repository objects, and mappings. Fact aggregate Dimension aggregates
Aggregate Persistence Wizard Steps 1.Open Aggregate Persistence Wizard. 2.Specify the file name and location. 3.Select business model and measures. 4.Select dimensions and levels 5.Select connection pool, container, and name. 6.Review aggregate definition. 7.View complete aggregate script. 8.Verify script is created. 9.Create and run a batch file. 10.Verify aggregates in the Physical layer. 11.Verify aggregates in the BMM layer. 12.Verify aggregates in the database. 13.Verify results in Answers.
1. Open Aggregate Persistence Wizard Select Tools > Utilities > Aggregate Persistence Wizard and click the Execute button.
2. Specify File Name and Location Specify a file and location where the output script should be saved.
3. Select Business Model and Measures Select business model. Select fact table. Select measures.
4. Select Dimensions and Levels Select corresponding aggregate dimensions and levels.
5. Select Connection Pool, Container, and Name Select the database object. Select the schema. Select the connection pool. Name the aggregate table.
6. Review Aggregate Definition View the aggregate definition.
7. View Complete Aggregate Script Confirmation of script creation and location Script
8. Verify that the Script Is Created Navigate to the directory where the file was saved and verify that the script was created as expected.
9. Create and Run a Batch File Create and run a batch file with the following format to drive the aggregate creation process: nqcmd Oracle BI Server command utility -d Oracle BI Server data source name -u Repository username -p Repository password -s Path to the create aggregate SQL script
10. Verify Aggregates in the Physical Layer Verify that the aggregates are created in the Physical layer of the repository as expected. Fact aggregate Dimension aggregates
11. Verify Aggregates in the BMM Layer Verify that the aggregates are created in the Business Model and Mapping layer of the repository as expected. Fact aggregate Dimension aggregate
12. Verify Aggregates in the Database Verify that the aggregates are created in the database.
13. Verify Results in Answers – Activate the aggregate tables: – Run a query in Answers: – Check the log and verify that the aggregate tables are accessed as expected:
Troubleshooting Aggregate Navigation If aggregate navigation is not working, the cause might be one of the following: – Aggregation content is not specified correctly for one or more sources. – Aggregate dimension sources are not physically joined to aggregate fact table sources at the same level. – Dimensional source does not exist at the same level as a fact table source. – Aggregate dimension sources do not contain a column that maps to the primary key of the dimension hierarchy level. – The number of elements is not specified correctly for dimension hierarchy levels.
Considerations Using aggregates comes with a price: – Additional time is required to build and load these tables. – Additional storage is necessary. Build only the aggregates you need: – Look at query patterns and build aggregates to speed up common queries that require summarized results. – Ensure that enough data is combined to offset the cost of building aggregates. – Monitor and adjust to account for changing query patterns.
Summary In this lesson, you should have learned how to: – Describe aggregate tables and their purpose in dimensional modeling – Model aggregate tables – Use the Aggregate Persistence Wizard to create aggregates
Using Partitions and Fragments
Business Challenge – Data is often partitioned into multiple physical sources for a single logical table. – Organizations need to seamlessly and efficiently access and process data from multiple sources to satisfy user requests. – Business applications must know where to go for what type of data and under what conditions.
Business Solution: Oracle BI Server – Oracle BI repository can be configured so that Oracle BI Server handles the navigation to the appropriate source. – Oracle BI Server seamlessly and efficiently accesses and processes data from multiple sources to satisfy user requests.
Partition – Is a database element that contains part of the data for a fact or a dimension – Combines with other data fragments as necessary – May be: Fact-based Value-based Level-based Complex
Partitioning by Fact – Data is partitioned by fact when different fact data is stored in different tables. – Example: Actual sales versus quota targets 1000 Product Sale 1100 Sales Rep 2000 Product Quota 1100 Sales Rep Actual salesQuota targets
Partitioning by Value – Data is partitioned by value when the data is split into separate tables according to the values of the data. – Example: Invoice data is stored separately for each region. 1000 Dollars Central1135293 RegionInvNbr Invoices for Central Region Invoices for West Region 200 Dollars West114444 RegionInvNbr
Partitioning by Level – Data is partitioned by level when the same facts are stored in separate tables at different levels of aggregation. – Example: Detailed sales data is summarized and stored by year and region. 19981010 19981001 19980105 Date 100001100 250001100 10000 Product Sale 1100 Sales Rep Sales detailed dataSales by year and region Central3000001999 200000 Total Dollars Central1998 RegionYear
Complex Partitioning – Data is partitioned using more than one technique. – Example: Invoice sales data is partitioned by value and level. Central25000199802 10000 Total Dollars Central199801 RegionMonth Invoices by month for CentralInvoices by year for Central 200000 Total Dollars Central1998 RegionYear Invoices by month for WestInvoices by year for West West350000199802 300000 Total Dollars West199801 RegionMonth 3000000 Total Dollars West1998 RegionYear
ABC Example: Fact-Based (Quota) Allow users to query for actual sales data and quota data in a single query. 1000 Product Sale 1100 Sales Rep 2000 Product Quota 1100 Sales Rep Actual salesQuota targets
ABC Example: Value-Based (Customer) Replace the current, single source for customer data with two value-based partitions. Clifton Lunch Name 1000 NewKey Customers with names starting with letters A–M Customers with names starting with letters N–Z Tongs Wok Name 1002 NewKey
ABC Example: Value-Based (Inventory) Build a business model for inventory data that is fragmented into multiple tables. Eight quarters of inventory data stored in eight separate tables
Implementation Steps – Import physical sources. – Create physical joins. – Add sources to the Business Model and Mapping layer. – Specify fragmentation content. – Test the results. New step
Specify Fragmentation Content – Use the Expression Builder to define the type of content that the fragment contains. – Set the flag to specify whether to combine this fragment with other data. Specifies that customer data from A–M is contained in this fragment Specifies combining this fragment with the other
Summary Identify reasons for segmenting data and implement it – benefit is promised! Question: The end user needs to know what data is contained in which partition before submitting a query. True or false?
Real-time BI Federated OLTP/EDW Reporting
Horizontal Federation for Real-Time BI Using horizontal fragmentation in OBIEE, we can map a single logical fact table to multiple LTSs. Physical fact table in our EDW called SALES_FACT Physical fact table in our source system to represent real-time data One logical fact table in the BMM called Sales Fact Realtime
Horizontal Federation for Real-Time BI Fact table mapping
Horizontal Federation for Real-Time BI Customer dimension mapping
Horizontal Federation for Real-Time BI Use variable as the threshold between reporting against the EDW schema and the source system schema
Horizontal Federation for Real-Time BI LTS mapping using threshold date
Horizontal Federation for Real-Time BI OBIEE does the logical union
Horizontal Federation for Real-Time BI When the BI Server has enough information to know that the entire result set will come from a single source, then the SQL will be issued against only one of the LTSs.
A Q &
Oracle by Example and Relevant Blogs Web tutorial :Oracle by Example Federating Essbase and Relational Data Sources in OBIEE http://www.artofbi.com/index.php/2009/07/federation-in-obiee-what-are-you-talking-about/ http://odtugspconference.com/Slides/OBIEE%20Integration%20with%20Essbase.pdf http://www.rittmanmead.com/2011/05/real-time-bi-federated-oltpedw-reporting/