1. Maximizing the Performance of the Oracle eBusiness Suite
Ahmed Alomari
Applications Performance Group
Oracle Corporation
June 14, 2005

2. Agenda Architecture Application Services Tier Network Performance
Forms
Reports
Concurrent Manager
Apache
Mod PL/SQL
Jserv/JVM
Portal
Discoverer
Network Performance
Data Server Tier
Application Tuning
11i10 Performance Improvements
Q & A

3. Java Server Pages (JSP)
Architecture
Portal
Real Application
Clusters (RAC)
Servlet Engine
JDBC
(Thin)
Java Server Pages (JSP)
http
AOLJ
Web Listener
Discoverer
TNS
(Net8)
PL/SQL Gateway
Data Server Tier
Clients
Reports
Forms
Concurrent Manager
Application Tier

4. Application Services Tier
Forms
Deploy with socket mode for internal users:
connectMode=socket (appsweb.cfg)
Enable Forms Dead Client Detection
Value specified in minutes
Terminates f60webmx processes for dead clients.
FORMS60_TIMEOUT=10
Enable Forms Abnormal Termination Handler
Do not set FORMS60_CATCHTERM

5. Application Services Tier
Forms
Upgrade to the latest Forms patch set (patch set 16)
Forms patchset 7 ( X) introduced a regression which affects scalability (bug ).
Results in Forms generating invalid SQL.
Fixed in Forms patchset 10 ( X).
Refer to MetaLink document for Forms patchset upgrades.

6. Application Services Tier
Forms – Cancel Query
Should not be enabled unless you are on Forms patchset 14 (Forms version x or higher).
Refer to MetaLink Note for instructions on how to enable and tune Cancel query related parameters.
Cancel Query increases middle-tier CPU as well as DB CPU.
To Disable Cancel Query
Set the Profile “FND: Enable Cancel Query” to No.

7. Application Services Tier
Forms
Ensure users are optimally utilizing the professional Forms interfaces.
Avoid Blind queries
Provide selective criteria in Find windows and LOVs.
Avoid opening and closing forms across transactions.
Users which need to switch responsibility in order to complete a business transaction should instead keep the relevant Forms open.

8. Application Services Tier
Forms (Tracing)
If the form is slow, or a specific flow in the Form is slow, first generate a complete SQL trace using the Trace menu in Help-Diagnostics.
If the SQL trace does not account for the majority of the response time experienced by the user, then generate a Forms Runtime Diagnostics (FRD) Trace.

9. Application Services Tier
Forms (Tracing – SQL Trace)

10. Application Services Tier
Forms (Tracing – FRD Trace)
Logon to Applications 11i with the following parameters appended to the URL: play=&record=collect&log=/tmp/oeform.trc
Navigate through the flows which you would like to trace.
Exit from Oracle Applications completely.
Review the trace file from the application server (/tmp/oeform.trc).

11. Application Services Tier
Reports
Ensure users provide selective parameters to the standard Reports.
Review the Report output:
Number of rows returned to the report.
Number of report pages.
SQL Trace can be generated at Request Submission Time (SRS form – Debug Options) or by setting the “Enable Trace” flag in the Concurrent Program Definition Form.

12. Application Services Tier
Reports (Tracing)
If the SQL trace for the report does not account for the elapsed time of the request, then generate a Reports trace by appending tracing parameters the to concurrent program command line.
The following are the trace options:
TRACE_ALL : log all possible trace information in the trace file. (DEFAULT)
TRACE_APP : log trace information on all the report objects in the trace file.
TRACE_BRK : list breakpoints in the trace file.
TRACE_ERR : list error messages and warnings in the trace file.
TRACE_PLS : log trace information on all the PL/SQL objects in the trace file.
TRACE_PRF : log performance statistics in the trace file.
TRACE_SQL : log trace information on all the SQL in the trace file.
TRACE_TMS : enter a timestamp for each entry in the trace file.

13. Application Services Tier
Reports (Tracing)
Enabling Reports Trace (pre 11i10)
Sign-on to Apps with System Administrator responsibility
Navigate to Concurrent -> Program -> Define
Query the concurrent program (report) which you want to trace.
Locate the “Options” field and enter the following values :
TRACEOPTS=<trace option> TRACEFILE=<path/file>
Save the concurrent program
Run the report and a trace file will be generated.
Remember to clear the trace options after running the report.
Enabling Reports Trace (11i10)
Use the Debug Options page at request submission.

14. Application Services Tier
Reports (Trace Output)
Report: /u02/appl/fnd/11.5.0/reports/US/FNDSCURS.rdf
Logged onto server:
Username:
23:59:57 APP ... ( Generic Graphical Object B_SECURITY_GROUP_NAME1
23:59:57 APP ... ) Generic Graphical Object B_SECURITY_GROUP_NAME1
23:59:57 APP ... ( Generic Graphical Object B_USER_NAME1
| Report Builder Profiler statistics |
TOTAL ELAPSED Time: seconds
Reports Time: seconds (53.34% of TOTAL)
ORACLE Time: seconds (46.65% of TOTAL)
UPI: seconds
SQL: seconds
TOTAL CPU Time used by process: seconds

15. Application Services Tier
Concurrent Manager
Avoid enabling an excessive number of standard or specialized managers.
Use specialization rules and work shifts to bind specific jobs to specific time windows.
Helps avoid scheduling resource intensive batch requests during peak activity.
For jobs which spawn parallel workers such as Auto Invoice or Payroll, set the sleep time of the Conflict Resolution Manager (CRM) to null (i.e. 10 seconds). The default value is 60 seconds.

16. Application Services Tier
Concurrent Manager
Database Resource Manager Integration
Define Resource Consumer Groups and associate specific concurrent programs with a resource group.
Allows an administrator to constrain the amount of CPU resources used by a concurrent program.
Resource Groups can also be defined for online users using the profile “FND: Resource Consumer Group”.

17. Application Services Tier
Concurrent Manager
Transaction Managers
Used for Inventory Transactions as well as other synchronous online processing.
Ensure enough transaction managers exist to service the request load.
Set the profile “Concurrent:Wait for Available TM” to 1 (second).
Set “TP:INV Transaction processing mode” to “On-line processing” for small inventory requests from the UI.
Set the sleep time on the transaction managers to a high number (e.g. 10 minutes).
Avoids constant polls to check for shutdown requests.
Transaction Managers can also be traced.

18. Application Services Tier
Concurrent Manager
Enabling Tracing
Program Level
Enable the trace option in the concurrent program definition form.
Request Level (new in 11i10)
Use the Debug Options button in the Request Submission Form
The profile “Concurrent: Allow Debugging” should be set to “Yes.”

19. Application Services Tier
Concurrent Manager
Enabling Trace at the Program Level

20. Application Services Tier
Concurrent Manager
Enabling Trace at the Request Level

21. Application Services Tier
Concurrent Manager
Enabling Trace at the Request Level

22. Application Services Tier
Apache
On UNIX, Apache is process based (httpd), and mods such as mod PL/SQL run within the process address space of the httpd processes.
On Windows, Apache is multi-threaded (Apache.exe).
Tune the number of processes and number of clients (httpds.conf):
MinSpareServers 5
MaxSpareServers 10
StartServers 5
MaxClients 512

23. Application Services Tier
Apache
Minimize levels of logging (httpds.conf):
LogLevel warn
SSLLogLevel warn
Enable Caching of non-HTML resources including images, style sheets, and Java script.
Caches content in the browser cache.
Reduces network round-trips (non SSL)
Reduces network bandwidth utilization (SSL)
Included in AutoConfig templates in (or higher).

24. Application Services Tier
Apache
Enable Caching of non-HTML resources (httpd.conf or apps.conf)
<Directory "<physical_path_corresponding_to_the_alias_/OA_HTML/>">
#enable the generation of the Expires header for files under /OA_HTML/
ExpiresActive On
#expire images one month after last client access
ExpiresByType image/gif "access plus 1 month"
#expire stylesheets one week after the last client access
ExpiresByType text/css "access plus 1 weeks"
#expire javascript libraries one day after the last client access
ExpiresByType text/javascript "access plus 1 days“
ExpiresByType application/x-javascript "access plus 1 day"
</Directory>

25. Application Services Tier
Apache
Review Apache Access Log file to ensure images are being satisfied from the browser cache.
Http code 200 (request for document)
Http code 304 (request for time stamp)
[23/Oct/2002:19:00: ] "GET /OA_MEDIA/FNDINVDT.gif HTTP/1.1"
[25/Oct/2002:14:38: ] "GET /OA_MEDIA/FNDINVDT.gif HTTP/1.1" 304 -
[25/Oct/2002:14:38: ] "GET /OA_MEDIA/FNDWATHS.gif HTTP/1.1"
[25/Oct/2002:14:38: ] "GET /OA_MEDIA/FNDREDPT.gif HTTP/1.1"
[25/Oct/2002:14:39: ] "GET /OA_HTML/OA.jsp?page=/oracle/apps/icx/por/rcv/pages/ReceivingHomePage&OAHP=ICXP
OR_MENU&OASF=ICXPOR_RCV_HOME_PAGE&dbc=ap107fam_ipdev11i&language_code=US&transactionid=70A5819F04C0F411 HTTP/1.1"
[25/Oct/2002:14:39: ] "GET /servlets/PoolMonitor?jvm HTTP/1.1"
[25/Oct/2002:14:41: ] "GET /OA_HTML/US/ICXINDEX_ipdev11i.htm HTTP/1.1" 304 -
[25/Oct/2002:14:41: ] "GET /OA_MEDIA/logo.gif HTTP/1.1" 304 -
[25/Oct/2002:14:41: ] "GET /OA_MEDIA/appslogo.gif HTTP/1.1" 304 -
[25/Oct/2002:14:41: ] "GET /OA_MEDIA/FNDJLFRL.gif HTTP/1.1" 304 -
[25/Oct/2002:14:41: ] "GET /OA_MEDIA/FNDINVDT.gif HTTP/1.1" 304 -
[25/Oct/2002:14:41: ] "GET /OA_MEDIA/FNDJLFRR.gif HTTP/1.1" 304 -

26. Application Services Tier
Apache Mod PL/SQL
Configure a dedicated mod PL/SQL Listener
Improves performance and scalability
Significantly reduces overall number of sessions/connections.
Reduces latency of web requests.
Improves cursor sharing.
Documented in “Oracle 9i Application Server Using the PL/SQL Gateway Release 1 (v )”

27. Application Services Tier
Apache Mod PL/SQL (dedicated listener)
1. For the main Listener running on Port 7000, edit the file $IAS_HOME/Apache/modplsql/cfg/plsql.conf as follows:
Disable the mod PL/SQL service from the main listener by commenting out the lines between the two Location parameters as follows:
# <Location /pls>
# SetHandler pls_handler
# Order deny,allow
# Allow from all
# </Location>
Comment out the following line as follows:
# LoadModule plsql_module /d1/ias/Apache/modplsql/bin/modplsql.so

28. Application Services Tier
Apache Mod PL/SQL (dedicated listener)
2. Configure the main listener to forward all mod_plsql requests to the dedicated mod PL/SQL listener by adding the following line:
ProxyPass /pls/
For the dedicated mod PL/SQL Listener running on Port 8888, configure each DAD to override the default CGI environment variables in order to allow redirects. Edit the file $IAS_HOME/Apache/modplsql/cfg/wdbsvr.app and add the following line for each DAD:
cgi_env_list=SERVER_NAME=mainhost.us.oracle.com,SERVER_PORT=7000,HOST=mainhost.us.oracle.com:7000

29. Application Services Tier
Apache Jserv / JVM
Minimize Jserv logging
log=true
log.channel.warning=true
log.file=/d1/ias/Apache/Jserv/logs/jserv_7000.log
Review error log files for exceptions.
Disable auto reload in production environments:
autoreload.classes=false
autoreload.file=false
Use Jserv Auto Load Balancing
Configure Multiple Zones.
Provides higher availability and improves scalability.

30. Application Services Tier
Apache Jserv / JVM
Upgrade to the latest JDK (1.4.2_08).
Do not disable hotspot or the JIT (i.e –DCOMPILER=NONE or –Xint).
Enable verbose GC in order to optimally tune the heap sizes based on the GC traffic.
Minor Collections
Major Collections
Elapsed Time of GC

31. Application Services Tier
Apache Jserv / JVM
Enable Verbose GC Output
Overload wrapper.bin with a shell wrapper in order to redirect stdout to a file.
Existing:
wrapper.bin=/apps/jdk/jdk1.4.2/bin/java
Change To:
Wrapper.bin=/apps/scripts/java.sh
java.sh:
=========
/apps/jdk/jdk1.4.2/bin/java -verbosegc $* >> /apps/logs/java.log

32. Application Services Tier
Apache Jserv / JVM (Verbose GC Output)
/apps/logs/java.log:
[GC 30460K->1369K(510848K), secs]
[Full GC 15135K->1686K(510848K), secs]
[GC 32123K->2131K(510848K), secs]
[GC 32595K->2130K(510848K), secs]
[GC 32593K->2171K(510848K), secs]
[GC 32635K->2419K(510848K), secs]
[GC 32881K->3157K(510848K), secs]
[GC 33620K->3197K(510848K), secs]
[GC 33661K->3218K(510848K), secs]
[GC 33674K->3309K(510848K), secs]
[GC 33769K->3532K(510848K), secs]
[GC 33983K->3784K(510848K), secs]
[GC 34248K->4056K(510848K), secs]
[GC 34520K->4404K(510848K), secs]
[GC 34868K->4828K(510848K), secs]
[GC 35292K->5242K(510848K), secs]

33. Application Services Tier
Apache Jserv / JVM
Utilize the verbose GC output to tune the JVM heaps (-Xmx and –Xms) accordingly.
Review the frequency of collections, especially major collections (i.e. Full GC).
Start with:
-Xms256M and –Xmx512M
-XX:NewSize=60M -XX:MaxNewSize=120M
Above settings have been incorporated in 11i10.

34. Application Services Tier
Apache Jserv / JVM
Process Identification (V$SESSION)
Specify the property –DCLIENT_PROCESSID in the JVM startup shell script.
Set automatically in 11i10.
Existing:
wrapper.bin=/apps/jdk1.4.2/bin/java
Change To:
Wrapper.bin=/apps/scripts/java.sh
java.sh:
=========
#!/bin/sh
/apps/jdk1.4.2/bin/java -verbosegc -DCLIENT_PROCESSID=$$ $* >> /d2/logs/java.log

35. Application Services Tier
Apache Jserv / JVM
Process Identification (V$SESSION)
Allows you to map the JDBC session from v$session to a particular JVM process.
Set automatically in 11i10
SID MACHINE PROCESS MODULE LOGON
41 aptier1.us.oracle.com JDBC Thin Client 12/02/04 14:26:43
42 aptier2.us.oracle.com JDBC Thin Client 12/02/04 14:27:01
43 aptier3.us.oracle.com JDBC Thin Client 12/02/04 14:28:15
44 aptier1.us.oracle.com JDBC Thin Client 12/02/04 14:29:17
aptier1{apps_a}-> ps -ef | grep 28806
apps_a :26:39 pts/20 0:00 /bin/sh ./java.sh
apps_a :26:40 pts/20 8:55 /jdk1.4.2/bin/../bin/sparc/native_threads/java

36. Application Services Tier
Apache Jserv / JVM
JDK is Certified with Apps.
Refer to MetaLink note for details on upgrading to the JDK
~15% performance improvement.
Parallel GC (-XX:+UseParNewGC)
Detail GC Statistics (-XX:+PrintGCDetails)
[GC [DefNew: 2047K->0K(2112K), secs] 2472K->424K(3520K), secs]
[GC [DefNew: 2047K->0K(2112K), secs] 2472K->424K(3520K), secs]
[GC [DefNew: 2047K->0K(2112K), secs] 2472K->424K(3520K), secs]
[GC [DefNew: 0K->0K(2112K), secs][Tenured: 20227K->227K(21416K), secs] 20227K->227K(23528K), secs]
[GC [DefNew: 0K->0K(2112K), secs][Tenured: 20227K->227K(21416K), secs] 20227K->227K(23528K), secs]
[GC [DefNew: 0K->0K(2112K), secs][Tenured: 20227K->227K(21416K), secs] 20227K->227K(23528K), secs]

37. Application Services Tier
Apache Jserv / JVM
JDK 1.4 provides new JVM monitoring tool (jvmstat)
Download from

38. Application Services Tier
Java Server Pages (JSPs)
Precompile the JSPs to avoid dynamic compilation.
Users experience poor performance for the initial page loads.
Potential deadlocks if multiple users attempt to compile the same JSP.
Potential JVM death due to OutOfMemoryException during concurrent compilation.
MetaLink Document provides the instructions and the patch reference for an automated script to perform the precompilation.
ojspCompile Script
With the latest version of ojspCompile, it takes 15 minutes to compile all the JSPs.

39. Application Services Tier
Java Server Pages (JSPs)
Use a separate JVM to perform the JSP compilation.
Automatically set in 11i10.
root.properties:
servlet.oracle.jsp.JspServlet.initArgs=translate_params=true,
unsafe_reload=false,
page_repository_root=/appl_top/115/common/html/jsp/pagecache,
alias_translation=true,developer_mode=false,
javaccmd=/usr/jdk142/bin/javac,send_error=true

40. Application Services Tier
Portal Tuning
Tune the number of content fetcher threads for PPE (poolSize). Default is 25.
Adjust the fetch request timeout (requesttime).
Adjust the fetch connection request timeout (stall).
zone.properties:
servlet.page.initArgs=poolSize=50
servlet.page.initArgs=requesttime=200
servlet.page.initArgs=stall=100

41. Application Services Tier
Portal Tuning
Ensure that the mod PL/SQL cache is enabled and sized appropriately:
$APACHE_TOP/modplsql/cfg/cache.cfg:
[PLSQL Cache]
enabled=yes
total_size=
cleanup_size=
cleanup_interval=86400 ;
[Cookie Cache]
total_size=
cleanup_size=
max_size = 0

42. Application Services Tier
Discoverer 4i Viewer Tuning
Disable Query Prediction by setting QPPEnable=0 in pref.txt.
Set ObjectsAlwaysAccessible=1 to avoid extra workbook SQL validation, and related dictionary SQL.
Set the Query Governor option “Limit retrieved query data to:” to 100 rows.
Set the option “After opening a worksheet:” to “Don’t run query.”
Ensure Custom workbooks define mandatory parameters.

43. Network Performance Network performance is often neglected.
Networks with poor latency or low bandwidth impacts user response time.
Can also impact concurrent programs if the network between the application tier and data server tier is not optimal.
A direct switch should be used between the application tier and data server tier
Latency should ideally be ~1-2ms per round-trip.

44. Network Performance
Use the ping command to determine the latency of requests. Vary the packet size to determine the average latency. Do not use the default ping packet size of 32 bytes.
D:\>ping aptierhost.us.oracle.com -l 2048
Pinging aptierhost.us.oracle.com [ ] with 2048 bytes of data:
Reply from : bytes=2048 time=371ms TTL=252
Reply from : bytes=2048 time=330ms TTL=252
Reply from : bytes=2048 time=361ms TTL=252
Reply from : bytes=2048 time=360ms TTL=252
D:\>ping aptierhost.us.oracle.com -l 512
Pinging aptierhost.us.oracle.com [ ] with 512 bytes of data:
Reply from : bytes=512 time=231ms TTL=252
Reply from : bytes=512 time=210ms TTL=252
Reply from : bytes=512 time=220ms TTL=252

45. Network Performance
Use the ping command to determine the latency between the application server and the data server. Use a packet size of 2K since this the default size for SQL*Net traffic.
aptierhost{applmgr}-2-> ping -s dbtierhost 2048
PING dbtierhost.us.oracle.com: 2048 data bytes
2056 bytes from dbtierhost.us.oracle.com ( ): icmp_seq=0. time=1. ms
2056 bytes from dbtierhost.us.oracle.com ( ): icmp_seq=1. time=1. ms
2056 bytes from dbtierhost.us.oracle.com ( ): icmp_seq=2. time=1. ms
2056 bytes from dbtierhost.us.oracle.com ( ): icmp_seq=3. time=1. ms
2056 bytes from dbtierhost.us.oracle.com ( ): icmp_seq=4. time=1. ms
2056 bytes from dbtierhost.us.oracle.com ( ): icmp_seq=5. time=1. Ms
---- dbtierhost.us.oracle.com PING Statistics----
6 packets transmitted, 6 packets received, 0% packet loss
round-trip (ms) min/avg/max = 1/1/1

46. Network Performance
Measure the sustained bandwidth on both the client network (i.e. desktops) as well as the application-tiers.
Use traceroute (or tracert) utility to determine number of hops and impact on latency.
Monitor network statistics including packet rates, error rates, collisions, etc.. using tools such as netstat –i.

47. Data Server Tier
Refer to the MetaLink document “Database Initialization Parameters and Configuration for Oracle Applications 11i”.
Ensure mandatory parameters are set correctly.
Optimally configure the buffer cache and shared pool as per the workload and the number of users:
[typical configuration for 2,000 Apps users]
db_block_buffers=800,000
shared_pool_size=2000M
A poorly sized buffer cache results in excessive buffer gets and physical I/O.
A poorly sized shared pool results in library cache and shared pool latch contention due to reloads and lack of space.

48. Data Server Tier Enable asynchronous I/O
Consider using Quick I/O or equivalent if using file system based files for the DB files.
Quick I/O improves performance and scalability by simulating raw devices, and avoiding file system cache traffic for the DB files.
Use locally managed temp files (uniform) for the temporary tablespace.
Install Stats Pack (spcreate.sql) and use spauto.sql to automate hourly snapshots.
Stats Pack reports can be used to trend transaction rates and business flows.

49. Data Server Tier Stats Pack Report (Instance Summary)
STATSPACK report for
DB Name DB Id Instance Inst Num Release Cluster Host
GSIAP gsi1ap YES agsidbs1
Snap Id Snap Time Sessions Curs/Sess Comment
Begin Snap: May-05 11:00:01 #######
End Snap: May-05 16:00:04 #######
Elapsed: (mins)
Cache Sizes (end)
~~~~~~~~~~~~~~~~~
Buffer Cache: 5,313M Std Block Size: K
Shared Pool Size: 2,864M Log Buffer: 1,024K

50. Data Server Tier Stats Pack Report (Instance Summary) Load Profile
~~~~~~~~~~~~ Per Second Per Transaction
Redo size: , ,266.16
Logical reads: , ,095.50
Block changes: ,
Physical reads: ,
Physical writes:
User calls:
Parses:
Hard parses:
Sorts:
Logons:
Executes: ,
Transactions:
% Blocks changed per Read: Recursive Call %:
Rollback per transaction %: Rows per Sort:

51. Data Server Tier Stats Pack Report (Instance Summary)
Instance Efficiency Percentages (Target 100%)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
Buffer Nowait %: Redo NoWait %:
Buffer Hit %: In-memory Sort %:
Library Hit %: Soft Parse %:
Execute to Parse %: Latch Hit %:
Parse CPU to Parse Elapsd %: % Non-Parse CPU:
Shared Pool Statistics Begin End
Memory Usage %:
% SQL with executions>1:
% Memory for SQL w/exec>1:
Top 5 Timed Events
~~~~~~~~~~~~~~~~~~ % Total
Event Waits Time (s) Ela Time
db file sequential read ,187, ,
CPU time ,
global cache cr request ,410, ,
db file scattered read ,184, ,
KJC: Wait for msg sends to complete ,722, ,

52. Data Server Tier Stats Pack Report (Wait Events) Avg
Total Wait wait Waits
Event Waits Timeouts Time (s) (ms) /txn
db file sequential read ,187, ,
global cache cr request ,410, , ,
db file scattered read ,184, ,
KJC: Wait for msg sends to c 10,722, , ,
row cache lock , , ,
wait for unread message on b , , ,
SQL*Net more data to client 6,021, ,
enqueue , , ,
buffer busy global CR , ,
buffer busy waits , ,
direct path read ,239, ,
inactive session , , ,
latch free , , ,
db file parallel read , ,
direct path write ,270, ,
log file sync , ,
SQL*Net message from dblink , ,

53. Data Server Tier
Review Top 5 Timed Events section in the Stats Pack reports to determine the overall instance utilization.
For latch free waits, latch contention is often a symptom due to a legitimate problem such as non-sharable SQL, sub-optimal SQL which performs full table or full index scans, dynamic object creation/removal, etc..
Review the latch Statistics section of the Stats Pack report to determine the hot latches.
Enable tracing for a few of the sessions waiting on the latch as well as the holder to determine the actual cause.

54. Data Server Tier Stats Pack Report (Expensive SQL) CPU Elapsd
Physical Reads Executions Reads per Exec %Total Time (s) Time (s) Hash Value
3,413, ,413,
Module: ALECDC
SELECT distinct ooha.order_number , fu.user_name , rac.cu
stomer_name , to_char(wctc.last_update_date, 'DD-MON-YY hh24:mi
') , DECODE(SUBSTR(wctc.comments, 1, 21),
'NOTE: This Order-Line', wctc.process_flag || ': manually scre
ened', DECODE(wctc.process_flag ,'101','101: host u
2,229, ,229,
select distinct pv.segment1 , nvl(pv.vendor_name_a
lt, pv.vendor_name) , ai.invoice_num , hr.name ,
from po_vendors pv ,
ap_invoices_all ai , ap_holds_all ah , hr_organi
zation_units hr where ai.payment_status_flag||''='Y' and
1,990, ,990,
Module: OKILOADDR
BEGIN OKI_REFRESH_PUB.REFRESH_ADDRS( :errbuf, :rc ); END;

55. Data Server Tier Monitor CPU and Memory Utilization Monitor paging
Monitor I/O statistics, including service times.
Review Stats Pack reports and correlate top SQL to the business flows.
Review expensive custom SQL

56. Data Server Tier Gathering Statistics
Do not gather statistics excessively on entire schemas or the entire database such as nightly or weekly.
Do not gather statistics on permanent objects during peak intervals.
Gathering statistics invalidates cursors
Gathering statistics requires dictionary and object level locks.
Plans are not likely to change if the data distribution has not changed.

57. Data Server Tier Gathering Statistics
For tables which are growing at a rapid rate, gather statistics only on those tables.
Use only FND_STATS or the Gather Schema and Gather Table Statistics Concurrent Programs
Do NOT USE the analyze nor dbms_stats command directly. It is not supported, and results in sub-optimal plans.
Review the table and index statistics for the objects which appear in the top SQL section of Stats Pack.

58. Data Server Tier Gathering Statistics Enhancements (11i10)
Gather Schema Statistics
Auto Gather option
Gather Statistics only on tables which have changed
Change threshold is user definable.
Utilizes the Table Monitoring feature. To enable schema monitoring:
SQL> exec fnd_stats.ENABLE_SCHEMA_MONITORING()
Auto List option
Lists the objects which have changed.
Maintain history of Statistics Collection
No Invalidate Option
Does not invalidate cursors.

59. Data Server Tier
Gathering Statistics Enhancements (11i10)

60. Data Server Tier Verifying the Statistics SQL> set serveroutput on
SQL> exec apps.fnd_stats.verify_stats ('ONT','OE_ORDER_LINES_ALL');
===================================================================================================
Table OE_ORDER_LINES_ALL
last analyzed sample_size num_rows blocks :
Index name last analyzed num_rows LB DK LB/key DB/key CF
OE_ORDER_LINES_N :
OE_ORDER_LINES_N :
OE_ORDER_LINES_N :
OE_ORDER_LINES_N :
OE_ORDER_LINES_N :
. . .
OE_ORDER_LINES_N :
OE_ORDER_LINES_N :
OE_ORDER_LINES_N :
OE_ORDER_LINES_N :
OE_ORDER_LINES_U :
Histogram Stats
Schema Table Name Status last analyzed Column Name
ONT OE_ORDER_LINES_ALL present :59 OPEN_FLAG

61. Data Server Tier Verifying the Statistics – Column Statistics
SQL> set serveroutput on
SQL> exec apps.fnd_stats.verify_stats ('ONT','OE_ORDER_LINES_ALL',0,TRUE);
===================================================================================================
Table OE_ORDER_LINES_ALL
last analyzed sample_size num_rows blocks :
Column Stats
Column name sample_size num_distinct num_nulls density last analyzed
ACCOUNTING_RULE_DURATION :51
ACCOUNTING_RULE_ID :34
. . .
BOOKED_FLAG :51
CANCELLED_FLAG :51
LINE_ID :34
LINE_NUMBER :34
LINE_SET_ID :34
OPTION_NUMBER :51
ORDERED_ITEM_ID :34
UPGRADED_FLAG :51
VEH_CUS_ITEM_CUM_KEY_ID :34
VISIBLE_DEMAND_FLAG :51

62. Data Server Tier Enable the Auto memory manager
Pga_aggregate_target can be set to tune the PGA for the entire instance.
Automatically tunes hash area and sort area sizes
Returns unused memory to the OS.
Enable System Managed Undo
No more ORA-1555s!!!
Set undo_retention to the length of the longest running request.
Utilize Auto Segment Tablespace Management
No need to manually set Freelist/freelist groups. A bitmap is used to automatically scale the segment related parameters.
Enable PL/SQL Native Compilation
Improves PL/SQL execution performance.
Improves scalability by removing a lot of stress off of the shared pool.

63. Data Server Tier New Tablespace Model for the eBusiness Suite:
Consolidates the number of Apps tablespaces from 400 tablespaces to 10.
Optimizes performance via the wide stripe model.
Ideal for customers migrating to RAC or migrating to raw devices.
Uniform extents, locally managed
No more fragmentation!
Migration script can be used to migrate a schema at a time.

64. Data Server Tier New Tablespace Model for the eBusiness Suite:
Objects are classified by object type and access patterns:
Transaction tables
Transaction indexes
Reference/SEED
Interface
Summary Management
Archive
NoLogging
Queue
Media
Temporary
Undo
System
Total = 12 Tablespaces

65. Data Server Tier Applications Tablespace Migration Utility
MetaLink note
ARU # is
Migration timings: (complete migration)
1.4 Terabyte customer DB
24 hours
New size of DB after migration (850 GB).
250 GB customer DB
4 hours
New size of DB after migration (150 GB).

66. Application Tuning
Review the MetaLink note “Recommended Performance Patches for the Oracle E-Business Suite”,
Recommended performance patches for all the modules and tech. stack components are consolidated in this note.

67. Application Tuning Workflow
When starting background engines via Concurrent Manager, set the Process Stuck parameter to No.
Start a separate background engine to handle stuck processes with a very low frequency such as once or twice a day.
Use deferred activities to improve online response times and facilitate asynchronous processing for flows such as Scheduling, PO Document Approval, etc..
Establish a regular frequency for purging Workflow runtime data (FNDWFPR - Purge Obsolete Workflow Runtime Data).

68. Application Tuning Order Management
OM Pack J contains numerous performance fixes including Pricing performance fixes.
Ensure that debugging is disabled:
OM: Debug Level, set to 0 for OFF
QP: Debug Mode, set to Request Viewer Off
WSH: Debug Enabled, set to No
WSH: Debug Level, set to 0 for OFF
Use the streamlined workflows.
Removes unnecessary activities and sub-processes.
Improves performance and amount of WF data significantly.
New streamlined flow in
Line Flow – Generic: Performance
Refer to MetaLink note

69. Application Tuning Order Management
For online order entry, use the new Quick Sales Order Form
Provides a checkbox “deferred pricing” which can be used to defer price calculation until the order is saved.
Also improves overall speed of order entry from a functional / navigational perspective.
Enable AutoScheduling
Use deferred activities
Tax Calculation
Credit Check

70. Application Tuning Payroll THREADS=25 CHUNK_SIZE = 20
Tune the number of threads, chunk size and the buffer sizes (pay_action_parameters):
THREADS=25
CHUNK_SIZE = 20
BAL BUFFER SIZE=550
EE BUFFER SIZE =550
RR BUFFER SIZE =550
COST BUFFER SIZE=550
Set threads to times the number of data server processors.
Set LOW_VOLUME=N to enable optimal plans.

71. Application Tuning OA Framework Applications
Refer to MetaLink note (Recommended Patches for Applications) and MetaLink note (Framework Roadmap).
If running FWK 5.7, ensure that you are running the latest rollup patch for 5.7H.
Refer to MetaLink note
Upgrade to the latest 9iR2 JDBC Drivers (MetaLink note )
Pool Monitor can be used to monitor the resource utilization
Application Module Pool
Memory Utilization
JVM Properties
Pre 11i10:
11i10: Login->Diagnostics->Show Pool Monitor.

72. Application Tuning
OA Framework Applications (Pool Monitor)

73. Application Tuning
OA Framework Applications (Pool Monitor)

74. Application Tuning (OA)
Tracing Framework Applications
Set profile FND : Diagnostics to Yes at user level
Login to Self Service as the above user
Click on Diagnostics icon at the top of page
Select ‘Set Trace Level’ and click Go
Select the desired trace level and click Save
Perform the activity that you want to trace
Exit application

75. Application Tuning (OA)
Tracing Framework Applications

76. Application Tuning AOLJ Connection Pool (dbc configuration)
$FND_TOP/secure/<hostname_sid>.dbc
Number of DB connections used by the Java based Self-Service Applications.
Start with a reasonable default such as 100 or 200.
FND_JDBC_MAX_CONNECTIONS=200
Tune FND_JDBC_MAX_CONNECTIONS as per the amount of user concurrency (per JVM).
AOLJ automatically decays idle connections and resizes the pool.

77. Application Tuning AOLJ Connection Pool (dbc configuration)
Disable sanity checks in production environments
FND_JDBC_USABLE_CHECK=false
FND_JDBC_PLSQL_RESET=false
Reduce the number of OXTA JDBC connections
In jserv.properties file, add the following entries
wrapper.bin.parameters=-DOXTAOutThreads=1
wrapper.bin.parameters=-DOXTAInPoolSize=1
If running multiple JVMs, the OXTA connections can be a significant portion of the total JDBC connections.
Automatically set in 11i10

78. Application Tuning AOLJ Connection Pool (dbc configuration)
AOLJ Supports the use of Services for RAC load balancing or failover.
Patch allows the use of service names in the configuration:
New dbc parameter APPS_JDBC_URL
Used in place of DB_HOST, DB_NAME, DB_PORT
Useful for automatic connection load balancing

79. Application Tuning AOLJ Connection Pool Monitoring
Use the JSP (AoljDbcPoolStatus.jsp) to monitor the connection pool status.
Lists DBC configuration parameter settings.
Connection Statistics
Locked Connections
Available Connections
Closed Connections
Allows drill-downs to determine source of the connection.

80. Application Tuning
Connection Pool Status (AoljDbcPoolStatus.jsp)

81. Application Tuning Purging
Implement a regular purge process including transient data as well as obsolete transactional data.
Archive system related entities prior to purge such as
FND_CONCURRENT_REQUESTS
ICX_SESSIONS
Purge old data before performing upgrades.

82. Application Tuning Purging Product Concurrent Program Name FND
Purge Concurrent Request and/or Manager Data
Purge "Sign on" Audit data
General Ledger
GL Archive and Purge
Assets
Mass Additions Purge Report
Purge accounting tables
Payables
Purge
Receivables
Archive and Purge
Order Management
Order Purge

83. 11i10 Performance Improvements
Order Management
Significant improvements in pricing and order management.

84. 11i10 Performance Improvements
Order Management
New Java Pricing Engine

85. 11i10 Performance Improvements
Inventory / Shipping
20% improvement in Pick Release.
25% improvement in Interface Trip Stop.
2X improvement in Ship Confirm.

86. 11i10 Performance Improvements
Workflow Improvements
Java Mailer
Parallel worker support
15,000 notifications per hour (per worker)
Workflow engine runtime
Improved Caching
Asset Import (5X)
Order Management (5%)
Integration with GSM
Business Event Listener
Mailer

87. 11i10 Performance Improvements
Payroll Improvements
New Run Balances Architecture (HRMS FP F)
Improves reporting performance significantly (~90%):
TRR – 95%+
Payroll Register – 90%+
US and Canadian Forms and Reports utilize the new Balance Reporting Architecture.
Custom reports which utilize the balance user exit also benefit from the new run balances arch.
Payroll run improves by 15%.
Suppression of zero and null balances provides an additional 10% performance improvement.

88. 11i10 Performance Improvements
TCA Customer Import
Completely rewritten.
Requires 9iR2 (or higher).
Scales well with parallel workers.
300K customers per hour per worker
1 million customers imported in one hour using 4 workers.

89. 11i10 Performance Improvements
TCA Customer Import

90. 11i10 Performance Improvements
11i10 Tracing Improvements
Ability to enable all levels of SQL trace as well as PL/SQL profiler from Forms, Framework based pages, and Concurrent Programs.
Help->About Oracle Applications shows performance information including time spent in the Form and time spent in the DB.
For concurrent programs, tracing can be enabled at the program definition level and at the request level.
Trace file identifier is automatically set to the Applications user name
prod11i_ora_2738_BASMITH.trc
New Report (PL/SQL Profiler)
Generates an HTML report of the PL/SQL profiler data.

91. 11i10 Performance Improvements
11i10 Tracing Improvements (PL/SQL Profiler Report)

92. 11i10 Performance Improvements
11i10 Tracing Improvements (PL/SQL Profiler Report)

93. 11i10 Performance Improvements
11i10 Tracing Improvements (Performance Info.)

94. 11i10 Performance Improvements
FND_STATS Improvements:
No invalidate option can be used to avoid invalidating cursors while gathering statistics.
Auto Gather has been added to allow statistics to be gathered only on those objects which have changed.
Change threshold is configurable.
Default is 10%.
Significantly reduces the time to gather statistics.
Improves manageability as FND_STATS automatically determines changed object list.
Maintain history of all gather stats requests.
Automatically determines degree of parallelism.

95. 11i10 Performance Improvements
Configuration Enhancements
Reviewed and optimized default settings for the entire stack via AutoConfig templates including:
Forms
JDK DB
Concurrent Manager
Discoverer
AOLJ DBC
Customer Sequence Cache settings are preserved across patches/upgrades if current setting exceeds standard default.
AutoConfig support for complex configurations including RAC, Load Balancing, Firewalls, etc..

96. 11i10 Performance Improvements
Significant reduction in patching downtime
Parallelized large table updates.
FNDLOAD Improvements
15% for small LDTs
Orders of magnitude for large LDTs.
Improvements to distributed AD.
Large indexes are now created in parallel.
APPS_MRC schema has been eliminated.
Optimized PL/SQL recompilation order for invalid objects.

97. MetaLink References Note #: Description: 123456.1
E-Business Suite Recommended Patch List
Upgrading Developer 6i with Oracle Applications 11i
Canceling Long Running Queries in Oracle Applications 11i
Upgrading Oracle JDBC Drivers with Oracle E-Business Suite 11i
Implementing and Using the JSP Precompiler
Database Initialization Parameters and Configuration for Oracle Applications 11i
Recommended Performance Patches for Oracle E-Business Suite
Using J2SE Version 1.4 with Oracle E-Business Suite 11i, Release or Higher
Oracle Applications Tablespace Model Release 11i - Tablespace Migration Utility
About OA Framework Mini-pack 11i.FWK.H
Oracle Applications Framework Release 11i Documentation Road Map

98. Q
&
Q U E S T I O N S
A N S W E R S A