2. OBIEE Performance and RPD Optimization techniques
Deb Bhatacharjee
Sr. Director
Anupam Bordia
Consulting Member of Technical Staff
Nathan Reynolds
Architect
PSR
September, 2014
This is a Title Slide with Picture slide ideal for including a picture with a brief title, subtitle and presenter information.
To customize this slide with your own picture:
Right-click the slide area and choose Format Background from the pop-up menu. From the Fill menu, click Picture and texture fill. Under Insert from: click File. Locate your new picture and click Insert.
To copy the Customized Background from Another Presentation on PC
Click New Slide from the Home tab's Slides group and select Reuse Slides.
Click Browse in the Reuse Slides panel and select Browse Files. Double-click the PowerPoint presentation that contains the background you wish to copy.
Check Keep Source Formatting and click the slide that contains the background you want.
Click the left-hand slide preview to which you wish to apply the new master layout.
Apply New Layout (Important): Right-click any selected slide, point to Layout, and click the slide containing the desired layout from the layout gallery.
Delete any unwanted slides or duplicates.
To copy the Customized Background from Another Presentation on Mac
Click New Slide from the Home tab's Slides group and select Insert Slides from Other Presentation…
Navigate to the PowerPoint presentation file that contains the background you wish to copy. Double-click or press Insert. This prompts the Slide Finder dialogue box.
Make sure Keep design of original slides is unchecked and click the slide(s) that contains the background you want. Hold Shift key to select multiple slides.
Apply New Layout (Important): Click Layout from the Home tab's Slides group, and click the slide containing the desired layout from the layout gallery.
Copyright © 2014, Oracle and/or its affiliates. All rights reserved. |
Oracle Confidential – Internal

3. This is a Safe Harbor Front slide, one of two Safe Harbor Statement slides included in this template.
One of the Safe Harbor slides must be used if your presentation covers material affected by Oracle’s Revenue Recognition Policy
To learn more about this policy,
For internal communication, Safe Harbor Statements are not required. However, there is an applicable disclaimer (Exhibit E) that should be used, found in the Oracle Revenue Recognition Policy for Future Product Communications. Copy and paste this link into a web browser, to find out more information.  
For all external communications such as press release, roadmaps, PowerPoint presentations, Safe Harbor Statements are required. You can refer to the link mentioned above to find out additional information/disclaimers required depending on your audience.
Oracle Confidential – Internal

4. Program Agenda 1
Best practices Aggregate persistence Caching Case studies Q&A 2 3 4 5
Oracle Confidential – Internal

5. Prerequisites Good understanding of OBIEE architecture
Some understanding of FA (Fusion Application) – BI Integration
Oracle Confidential – Internal

6. Program Agenda 1
Best practices Aggregate persistence Caching Case studies Q&A 2 3 4 5
Oracle Confidential – Internal

7. Query governor – Global
How do I prevent a BI Query from running more than 20 minutes and consuming most of the CPU?
How do I prevent a BI Query from fetching more than a million rows?
Order of precedence:
Global setting
Role based
Session based
Global setting in NQSConfig.ini:
DEFAULT_DB_MAXEXECTIME=600; # in seconds
DEFAULT_DB_MAXROWS=100000; # number of rows
Oracle Confidential – Internal

8. Query governor – Role based
Maximum governing value of all the available roles assigned to user
To set, one can open user/role in AdminTool and set Max rows fetched or time after which query should be aborted if not completed.
Manage -> Identity -> Application Roles -> Permissions -> Query Limits
Terminates query if it exceeds 10 minutes (not including fetch time) and/ or 100K rows
Time granularity: 1 minute
Oracle Confidential – Internal

9. Query governor – Role based
Oracle Confidential – Internal

10. Query governor – Session based
Set below variables in query:
OBIS_DB_MAXEXECTIME
OBIS_DB_MAXROWS
SET VARIABLE OBIS_DB_MAXEXECTIME=60, OBIS_DB_MAXROWS=100000: SELECT…
Oracle Confidential – Internal

11. Init Block - Defer Check if session init blocks are deferred:
Earlier row-wise init blocks could not be deferred
Non deferred session init block are executed at each login delaying login
Most of the session init blocks should be deferred. “Allow deferred execution” should be selected
Oracle Confidential – Internal

12. Init Block - Defer
Oracle Confidential – Internal

13. SQL Bypass SQL-bypass is the recommended approach for OTBI reports
Ensure all ADF queries are executed in SQL-bypass mode
To verify, check nqquery.log for:
<ADFQuery mode="SQLBypass" queryid=" " locale="en">
<ADFQuery queryid=" " locale="en"> # not using sql bypass
System would be slow if missing sql-bypass database
50x improvement observed if sql-bypass is used
Avoid projecting transient attributes, this results in BI-ADF broker skipping sql-bypass mode and hence queries are slower
Oracle Confidential – Internal

14. SQL Bypass
Oracle Confidential – Internal

15. OBIS – Query logs
In RPD, set session variable LOGLEVEL between 0 (no logging)-7 (extensive logging)
‘2’ is recommended (on production) to get the logical and physical sqls from the query logs, this setting has minimal performance impact
For detailed logging (not recommended for production instance), add variables
<variable id="LOGLEVEL" value="7"/>
<variable id="NQUIRE_INTERNAL_LOGGING" value="1"/>
under <ias-component id="coreapplication_obis1" inherit- environment="true"><environment>
in file ~orainst\config\OPMN\opmn\opmn.xml
Oracle Confidential – Internal

16. OBIS – Query logs
Oracle Confidential – Internal

17. OBIS – Server logs
Set system log level to 3 (to be set only during diagnostics)
Helps debugging issues related to init blocks
Time taken by LDAP server to authenticate
Time taken by each init block to execute
Example:
User ' ' spent milliseconds getting http response when 'getAuthenticatedUserWithLanguageAndProperties'.
User spent milliseconds at init block 'RPD Msgs’
Oracle Confidential – Internal

18. OBIS – Server logs
Oracle Confidential – Internal

19. OBIS Logs Setup Enterprise Manager (EM)
Log level of various BI components can be changed via EM.
Ensure that parameter centralConfigurationEnabled is set to “true” in file:~user_projects/domains/bifoundation_domain/config/fmwconfig/biee-domain.xml
Oracle Confidential – Internal

20. OBIS Logs Setup Enterprise Manager (EM)
Oracle Confidential – Internal

21. OBIS – Log inspection
Line/Hash value after “SQL Request” denotes logical hash value for the query
SQL Request, logical request hash:
SET VARIABLE LOGLEVEL=7:SELECT, "Fact - Sales Invoice Lines"."Gross Sales” FROM "Sales …”
Message to check if query is cacheable or not:
The logical query block fail to hits or seed the cache in subrequest level due to session variable or config file disable subrequest cache
Use Query log to verify cache hit:
The logical query hits the plan cache, plan
Oracle Confidential – Internal

22. OBIS – Log inspection
Find NQServer Execution plan (Do not confuse with SQL execution plan):
Execution plan:
ADF query executed in bypass mode:
<ADFQuery mode="SQLBypass" queryid=" " locale="en">
SQLBypass mode is skipped if a transient attribute is being used in query
Physical (Database) SQL Request:
Physical query follows statement where query is being sent to physical datasource, “Oracle Data Warehouse Connection“ in this case. Logical and Physical hash values are highlighted
Sending query to database named Oracle Data Warehouse (id: <<270614>>), connection pool named Oracle Data Warehouse Connection Pool, logical request hash , physical request hash 8245d55b:
Oracle Confidential – Internal

23. OBIS – Log inspection Query Status
Query Status: Successful Completion
Find number of rows returned by DB to NQServer:
Execution Node: <<27109>> DbGateway Exchange, Close Row Count = 11, Row Width = 64 bytes
Find response time for physical query execution:
Physical query response time (seconds), id <<270614>>
Oracle Confidential – Internal

24. OBIS – Log inspection
Find summary stats for physical and logical query execution:
Physical Query Summary Stats: Number of physical queries 1, Cumulative time 1.905, DB-connect time (seconds)
Logical Query Summary Stats: Elapsed time 2.305, Response time 2.304, Compilation time (seconds), Logical hash
Tie physical sql from nqquery.log to ORDBMS using physical hash value
Physical hash value is propagated to ORDBMS during query execution (it gets stored in action.v$sql)
select * from v$sql where action = 'c8dc78e8‘
Oracle Confidential – Internal

25. Configurations for better performance inspection
Parallel init block execution
NUM_INIT_BLOCK_THREADS_PER_USER
Parallel ADF VO query fetching
ADF_SQL_BYPASS_THREAD_RANGE
Recommend to set to same value as DB_GATEWAY_THREAD_RANGE.
Based on your workload, thread pools needs to be adjusted to capitalize on these features.
Limit login timeout (in seconds)
MAX_AUTHENTICATION_TIME = 120
INIT_BLOCK_LOG_TIME_THRESHOLD = 60
Oracle Confidential – Internal

26. Program Agenda 1
Best practices Aggregate persistence Caching Case studies Q&A 2 3 4 5
Oracle Confidential – Internal

27. What Makes Queries Run Slowly?
Problem
Causes
Large table
Complex transformations
Too many joins/ poor SQL
Scanning too many records on the disk
Returning too many records over the network
Interface & processing bottlenecks
Unsupported DB function
Cross-DB join of large table
Poor prompt design/ poor SQL
Connection pool settings
Logging
Configuration settings
Insufficient hardware

28. Why Aggregates? Pros Cons
Faster query response time usually by several order of magnitude
Efficient resource utilization
Aggregates can be in the same database as the source
Aggregates can be on a different relational source or Essbase
Cons
Creating aggregates is a resource intense exercise
Cannot be used for real time data

29. Oracle BI EE “Architecture”
Semantic Layer Metadata
BI Server
SQL
Tables
RDBMS

30. Aggregates can reside in a different database
Metadata describes aggregate mappings
Semantic Layer Metadata
BI Server
At query time, BI Server queries the fastest source that has enough detail to satisfy the user request
Performance of highly summarized requests is dramatically improved
SQL
SQL
Summary aggregates
Aggregates can reside in a different database
Detail
Load time aggregation
RDBMS

31. Steps using Admin Tool Design aggregate facts and grains
Wizard to create metadata 1
Target database for create/load 3 2

32. Program Agenda 1
Best practices Aggregate persistence Caching Case studies Q&A 2 3 4 5
Oracle Confidential – Internal

33. Query Plan Cache Saves CPU processing time by avoiding recompiling SQL
MAX_QUERY_PLAN_CACHE_ENTRIES # recommended value 1024
MAX_QUERY_PLAN_CACHE_ENTRY_SIZE # recommended 1 MB
Does not persist during server restart
Check query logs if query is being seeded in plan cache
The logical query seeds the plan cache
Check available RAM on the server and adjust above two parameters for your workload
Check compilation time from query logs
Logical Query Summary Stats: Elapsed time 9.958, Response time 9.958, Compilation time 0.956
Oracle Confidential – Internal

34. Result Cache Logical query results are cached by the server
Parameters to control number of rows per cache as well as number of cache entries allowed
[CACHE]
ENABLE = NO;
MAX_ROWS_PER_CACHE_ENTRY = ;
MAX_CACHE_ENTRY_SIZE = 20 MB;
MAX_CACHE_ENTRIES = 1000;
Performance boost for repeating queries
Shared dashboards normally get high hit rates
Ad hoc and highly dynamic use cases have high miss rate
Not a substitute for database tuning and aggregation
Cached results are automatically shared by users with similar security
Oracle Confidential – Internal

35. Result Cache Table level control (i.e., don’t cache real-time tables)
Use Cache Manager in Admin Tool (in online mode) to gather cache hit and miss number
You need to define seeding and purge strategy
Cache is persistent during server restart
May not be compatible from release to release, needs to be seeded after patching
Oracle Confidential – Internal

36. Cache – Statistics (programmatic)
OBISAvailableDiagnostics();
Lists all caches, result as well as plan cache
OBISDiagnostics(‘$Name_of_cache_from_above_step')
Result cache statistics parameter list:
QUERIES PER SEC
FAILED QUERIES PER SEC
NEW PREPARES
ROWS PER SEC
KB PER SEC
Plan cache statistics parameter list:
CAPACITY
TOTAL REQUESTS
AVG REQUESTS
AVG HITS
AVG MISS
Oracle Confidential – Internal

37. Cache Hits Caching is at the logical query layer, not physical layer
A query doesn’t need to be identical to get a hit
Cache results can be aggregated to satisfy a request
A more restrictive WHERE clause can hit
A subset of columns will hit
Expressions can hit if they use only cached columns
Cache entry currently contains
Database tables
Database Schemas
Connection pool parameters
Connection scripts
Physical Plan
Session variables (name/value) included in the plan
Cache detector will not hit if the above variable values does not match for incoming query
Oracle Confidential – Internal

38. Cache Strategy - Purge
Purge to avoid stale data inconsistent with source
Event Polling Tables allow integration with ETL
Timed purges
Manual purges
Programmatic purges via SQL calls
Purge plan cache, nqSQL Statement: call sapurgeplancache();
Purge data cache, nqSQL Statement: call sapurgeallcache();
Oracle Confidential – Internal

39. Cache Strategy - Seeding
Seeding ensures good performance for the first users of the day
iBots can run standard dashboards to seed them
Run seed queries that are easy to hit (see Admin Guide)
Simple columns (no expressions)
No WHERE clause
Oracle Confidential – Internal

40. Advanced Hit Detection
Multiple passes for more cache hits results in additional CPU usage
NQSConfig.INI Settings
USE_ADVANCED_HIT_DETECTION = YES;
Consider the following 3 queries
Q1: SELECT Markets.Region, Markets.District, Markets.Market, Products.Brand,
Periods."Year", "Sales Measures".Dollars FROM Paint
WHERE Periods."Year" IN ('1998', '1999')
Q2: SELECT Markets.District, Products.Brand, Periods."Year", "Sales Measures".Dollars
FROM Paint WHERE Periods."Year" = '2000'
Q3:SELECT Markets.District, Products.Brand, Periods."Year", "Sales Measures".Dollars
FROM Paint WHERE Periods."Year" = '1998‘
Q1 is cached; Q3 gets a cache-hit from Q1 with aggregate-rewrite;
Q1 and Q2 is cached; Q3 now has a cache miss because exact-match algorithm doesn’t match Q2 ;
Can be fixed with advanced cache hit detection with multi-pass algorithm
Oracle Confidential – Internal

41. Program Agenda 1
Best practices Aggregate persistence Caching Case studies Q&A 2 3 4 5
Oracle Confidential – Internal

42. Column size mismatch Inefficient memory utilization
Ensure that data model is efficient.
column taking 2,000 characters is not obvious (10 such columns)
Alternatively, modify RPD by creating opaque table (not the desired option)
Oracle Confidential – Internal

43. Use Init Block judiciously
Issue: BI Server memory is too high (40GB) in few minutes when large number of logins would occur
Cause:
Each login was invoking a init block which would fetch 68,000 rows adding up to 38 MB
OBIS memory would not come down after users have logged out or timed out
Large memory usage by OBIS was depriving other processes of memory. Only option to reclaim memory was to restart OBIS
Fix:
Enable caching for row-wise init block such that it gets executed only once for the life of the server. Cache lasts for the life of the server and cannot be refreshed like repository init block.
OBIS periodically calls MACompact() to release unwanted memory
Log Init Block query upon execution including its name
Consider: Cache lasts for the life of the server and cannot be periodically refreshed like repository Init Block.
Oracle Confidential – Internal

44. Use Init Block judiciously
Oracle Confidential – Internal

45. Nested REPLACE
Nested replace statement results in padding large amount of data since for each replace it would allocate maximum size (1500 characters in this case)
Query returns 200+ MB for 3 rows
Projection column is VARCHAR(1500)
For each replace statement padded size grows exponentially
This would occur with other concatenated functions like concat
‘cast’ the nested replace :
REPLACE(REPLACE(REPLACE(REPLACE("Sales - CRM Interactions and Customers Real Time"."Interaction Extension"."Extension Attribute Character 006",'U','Undecided'),'S','Successful'),'N','Not-Successful'),'I','Incomplete')
CAST(REPLACE(REPLACE(REPLACE(REPLACE("Sales - CRM Interactions and Customers Real Time"."Interaction Extension"."Extension Attribute Character 006",'U','Undecided'),'S','Successful'),'N','Not-Successful'),'I','Incomplete') AS VARCHAR(1500))
Parameter OBIS_MAX_FIELD_SIZE was introduced to govern the field size (default: 32K)
Oracle Confidential – Internal

46. Setting hints
Hints that you intend to pass to the end datasource (ORDBMS) can be set in RPD at connection or query level:
Execute on connect / disconnect
Execute before / after query
Upon disconnect if statement errors, connection is damaged and not returned to pool and counter not decremented
Eventually connections will be saturated and system will be frozen since no connections are available
Has been fixed
Oracle Confidential – Internal

47. Setting hints
Oracle Confidential – Internal

48. Table / VO Pruning
Use correct cardinality, 1:N results in more efficient SQL generation compared to (0,1):N
Tables used in query with (0,1):N cardinality
PER_LOCATION_DETAILS_F_TL  PER_LOCATION_DETAILS_F_VL  POR_REQUISITION_LINES_ALL  POR_REQUISITION_HEADERS_ALL  PO_LINE_LOCATIONS_ALL  PO_VERSIONS_INIT_SEQUENCE_V  FUN_ALL_BUSINESS_UNITS_V  GL_LEDGERS  FND_CURRENCIES_B  FND_CURRENCIES_TL 
Oracle Confidential – Internal

49. Table / VO Pruning
Oracle Confidential – Internal

50. Table/ VO Pruning
Unnecessary VO’s being used (VO join pruning not occurring)
Oracle Confidential – Internal

51. Filter rewrite
New style physical SQL generation (after filter rewrite)
SELECT
IDOF("BIQA_AW_REL"."C1Channel"."Primary"."All Channels"),
IDOF("BIQA_AW_REL"."C1 Product"."Standard"."Total"),
"BIQA_AW_REL"."C1Channel"."All Channels",
"BIQA_AW_REL"."C1Cube Units Sales Measures"."C1 Sales",
"BIQA_AW_REL"."C1Cube Units Sales Measures"."C1 Units",
FILTER("BIQA_AW_REL"."C1Cube Units Sales Measures"."C1 Sales" USING IDOF("BIQA_AW_REL"."C1Channel"."Primary"."Channel") IN (2, 4)) as "SalesCG1",
FILTER("BIQA_AW_REL"."C1Cube Units Sales Measures"."C1 Units" USING IDOF("BIQA_AW_REL"."C1Channel"."Primary"."Channel") IN (2, 4)) as "UnitsCG1",
FILTER("BIQA_AW_REL"."C1Cube Units Sales Measures"."C1 Sales" USING IDOF("BIQA_AW_REL"."C1Channel"."Primary"."Channel") IN (3, 4)) as "SalesCG2",
FILTER("BIQA_AW_REL"."C1Cube Units Sales Measures"."C1 Units" USING IDOF("BIQA_AW_REL"."C1Channel"."Primary"."Channel") IN (3, 4)) as "UnitsCG2"
FROM "BIQA_AW_REL"
ORDER BY 1
Old style logical SQL generation
SELECT
IDOF("BIQA_AW_REL"."C1Channel"."Primary"."All Channels"),
IDOF("BIQA_AW_REL"."C1 Product"."Standard"."Total"),
"BIQA_AW_REL"."C1Channel"."All Channels",
"BIQA_AW_REL"."C1Cube Units Sales Measures"."C1 Sales",
"BIQA_AW_REL"."C1Cube Units Sales Measures"."C1 Units"
FROM "BIQA_AW_REL"
UNION ALL
CAST(NULL AS INTEGER),
WHERE
IDOF("BIQA_AW_REL"."C1Channel"."Primary"."Channel") IN (2, 4)
IDOF("BIQA_AW_REL"."C1Channel"."Primary"."Channel") IN (3, 4)
ORDER BY 1
Oracle Confidential – Internal

52. Filter rewrite
WITH
SAWITH0 AS (select sum(T7209.UNITS) as c1,
sum(T7209.SALES) as c2,
T6641.TOTAL_CHANNEL_ID as c3,
T6716.TOTAL_CUSTOMER_ID as c4,
T7121.TOTAL_PRODUCT_ID as c5
from
PRODUCT_DIM T7121,
CHANNEL_DIM T6641,
CUSTOMER_DIM T6716,
SALES_FACT T7209
where ( T6641.CHANNEL_ID = T7209.CHANNEL_ID and T6716.SHIP_TO_ID = T7209.SHIP_TO_ID and T7121.ITEM_ID = T7209.ITEM_ID )
group by T6641.TOTAL_CHANNEL_ID, T6716.TOTAL_CUSTOMER_ID, T7121.TOTAL_PRODUCT_ID),
SAWITH1 AS (select sum(T7209.UNITS) as c1,
T6716.TOTAL_CUSTOMER_ID as c3,
T7121.TOTAL_PRODUCT_ID as c4
where ( T6641.CHANNEL_ID = T7209.CHANNEL_ID and T6716.SHIP_TO_ID = T7209.SHIP_TO_ID and T7121.ITEM_ID = T7209.ITEM_ID and (T6641.CHANNEL_ID in (2, 4)) and (T7209.CHANNEL_ID in (2, 4)) )
group by T6716.TOTAL_CUSTOMER_ID, T7121.TOTAL_PRODUCT_ID),
SAWITH2 AS (select sum(T7209.UNITS) as c1,
where ( T6641.CHANNEL_ID = T7209.CHANNEL_ID and T6716.SHIP_TO_ID = T7209.SHIP_TO_ID and T7121.ITEM_ID = T7209.ITEM_ID and (T6641.CHANNEL_ID in (3, 4)) and (T7209.CHANNEL_ID in (3, 4)) )
SAWITH3 AS (((select distinct 0 as c1,
from …
SAWITH3 D1
order by c1, c6, c7, c8
More efficient sql, execution results in single pass on fact table
WITH
SAWITH0 AS (select sum(T7209.UNITS) as c1,
sum(T7209.SALES) as c2,
sum(case when T6641.CHANNEL_ID in (2, 4) then T7209.SALES end ) as c3,
sum(case when T6641.CHANNEL_ID in (2, 4) then T7209.UNITS end ) as c4,
sum(case when T6641.CHANNEL_ID in (3, 4) then T7209.SALES end ) as c5,
sum(case when T6641.CHANNEL_ID in (3, 4) then T7209.UNITS end ) as c6,
T6641.TOTAL_CHANNEL_ID as c7,
T6716.TOTAL_CUSTOMER_ID as c8,
T7121.TOTAL_PRODUCT_ID as c9
from
PRODUCT_DIM T7121,
CHANNEL_DIM T6641,
CUSTOMER_DIM T6716,
SALES_FACT T7209
where ( T6641.CHANNEL_ID = T7209.CHANNEL_ID and T6716.SHIP_TO_ID = T7209.SHIP_TO_ID and T7121.ITEM_ID = T7209.ITEM_ID )
group by T6641.TOTAL_CHANNEL_ID, T6716.TOTAL_CUSTOMER_ID, T7121.TOTAL_PRODUCT_ID)
select distinct 0 as c1, …
SAWITH0 D1
order by c1, c2, c4, c5
Oracle Confidential – Internal

53. Filter rewrite
Convert a physical column CASE statement into a logical column definition based on filter function
Logical Query:SELECT Time.Quarter saw_0, Time.Week saw_1, rsum("Fact - RMW Opportunity"."Expected Revenue (000)") saw_3 FROM "Sales - Sales Revenue and Pipeline" WHERE (Time.Quarter = VALUEOF(CURRENT_QUARTER)) ORDER BY saw_0, saw_1
Old Physical Query:
select T "PER_NAME_QTR" as c1, T "PER_NAME_WEEK" as c2,
sum(case when T "LOST_FLG" = 'N' and T "SUMMARY_FLG" = 'Y' then T "REVN" end ) as c5
from "W_DAY_D" T264047, "W_REVN_F" T346644
where ( T "ROW_WID" = T "CLOSE_DT_WID" and T "PER_NAME_QTR" = '2004 Q 2' )
group by T "PER_NAME_QTR", T "PER_NAME_WEEK“ order by c1, c2
New Physical Query:
select T "PER_NAME_QTR" as c1, T "PER_NAME_WEEK" as c2, sum(T "REVN") as c5
from "W_DAY_D" T264047, "W_REVN_F" T346644
where ( T "LOST_FLG" = 'N' and T "PER_NAME_QTR" = '2004 Q 2' and T "ROW_WID" = T "CLOSE_DT_WID" and T "SUMMARY_FLG" = 'Y' )
group by T "PER_NAME_QTR", T "PER_NAME_WEEK“ order by c1, c2
Oracle Confidential – Internal

54. Missing descriptor id column
Inefficient SQL Plan due to missing Descriptor Id Column when using lookup
With OBIS cache ON, queries which include lookup do not perform because the plan generated by nqsserver is suboptimal.  
OBIS executes each lookup query and then stitches it together.  
For the query in question, we saw RT of 18+ sec with 7K+ rows being fetched for each lookup.   
Fix is to specify "Descriptor ID Column" for the lookup.  Query generation engine in that case creates a better plan by sending the complete query to database and RT is reduced to 7 sec. 
Oracle Confidential – Internal

55. Missing descriptor id column
Oracle Confidential – Internal

56. Program Agenda 1
Best practices Aggregate persistence Caching Case studies Q&A 2 3 4 5
Oracle Confidential – Internal

57. Oracle Confidential – Internal