Presentation on theme: "Internals of Concurrent Managers"— Presentation transcript:
1 Internals of Concurrent Managers UKOUG Conference Series Technology & E-Business Suite 2010Māris ElsiņšSenior Oracle Applications DBATieto Latvia,
2 Who I am? 8 years in IT Certificates Conferences 3 years – PL/SQL developer5 years – Oracle [Apps] DBA (started with and 8.1.7)Certificates10g OCM9i / 10g / 11g OCP11i Applications Database Administrator OCP11i System Administrator OCEConferencesUKOUG 2007/2008/2010LVOUG 2009/2010EMEA Harmony 2010Current employer – Tieto LatviaAll kinds of oracle DBA tasks - patching, upgrade, performance tuning, troubleshooting, planning and implementation of backup and recovery procedures, cross platform migration, etcPlanning of sydtem architecture, design and implementation of HA solutions (RAC, Data Guard, Custom cold failover)Implementation of system specific monitoring, automation of routine tasksTechnical project planning and coordination, management of team of DBAs
3 Purpose of this session Provide background knowledge for successful troubleshooting of concurrent request problemsWhat this presentation is about?Life cycle of a concurrent requestImplementation details of the most interesting phases of a life cycle of a concurrent requestInternals of Concurrent Managers and Conflict Resolution ManagersDB objects that can be used for troubleshootingWhy it is important?DBA has to be quick when solving real issuesKnowing the process before the problems happen decreases solving timeQuerying DB objects is faster then navigating through FormsKnowing the process is important when tuning the setup of Concurrent Managers
4 Most interesting DB objects FND_CONCURRENT_REQUESTS (FCR)FND_CONCURRENT_PROGRAMS (FCP)FND_CONCURRENT_QUEUES (FCQ)FND_CONCURRENT_PROCESSES (FCPROC)FND_CONCURRENT_PROGRAM_SERIAL (FCPS)FND_CONC_RELEASE_CLASSES (FCRC)FND_LOOKUPS (FL)FND_CONCURRENT_WORKER_REQESTS (FCWR)FND_CRM_HISTORY (FCH)FND_CONC_WAITING_REQUESTSSome examples how the tables are used will be uncovered in this presentation
5 Types of concurrent managers Internal Concurrent ManagerService ManagerConcurrent ManagerConflict Resolution ManagerInternal MonitorTransaction ManagerScheduler/Prereleaser Manager…Today the accent is on
6 Life cycle of a concurrent request INACTIVE (I)Disabled (U)On Hold (H)No Manager (M)PENDING (P)Normal (I)Standby (Q)Scheduled (P)Waiting (Z)RUNNING (R)Normal (R)Paused (W)Resuming (B)Terminating (T)COMPLETED (C)Normal (C)Error (E)Warning (G)Cancelled (D)Terminated (X)FND_LOOKUPS contains the values of Phase/Status Codes«Oracle® E-Business Suite System Administrator's Guide – Maintenance» describes the meaning of each phase and status
7 Typical life cycle of most requests Pending / ScheduledRequest is scheduled for the execution in futurePending / StandbyTime to execute the request has arrivedRequest waits for to be evaluated by the Conflict Resolution ManagerPending / NormalRequest is allowed to be executedRequest is waiting to be picked up by Concurrent ManagerRunning / RunningRequest is beeing executed by a concurrent managerCompleted / Normal
8 Phase/Status representation in tables FND_CONCURRENT_REQUESTSPHASE_CODESTATUS_CODEIs that so simple? NO!Phase = Inactive practically not usedInactive/Disabled: FCR.PHASE_CODE=‘P’, FCP.ENABLED_FLAG=‘N’Inactive/On Hold: FCR.PHASE_CODE=‘P’, FCR.HOLD_FLAG=‘Y’Inactive/No manager: FCR.PHASE_CODE=‘P’ and nonexistance of specific rows in FND_CONCURRENT_WORKER_REQESTS viewPending/Scheduled is actuallySTATUS_CODE=‘Q’+(FCR.REQUESTED_START_DATE>SYSDATE)Why so complicated?No need to update statuses too often
10 Schedule types As Soon as Possible/Once FCR.REQUESTED_START_DATE Periodically/On Specific Days FND_CONC_RELEASE_CLASSES (FCRC)Join with FCR on column RELEASE_CLASS_IDFCRC.DATE1 - “Start at” field in the formFCRC.DATE2 - “End at” field in the formFCRC.CLASS_TYPE – «P» for «Periodically», «S» for «On Specific days»FCRC.CLASS_INFO – actual schedule dataFCR.REQUESTED_START_DATE always contains the time of next execution, only this field used to determine when the request should be run by the concurrent manager
11 Periodic schedules FCRC.CLASS_INFO contents Values like «X:Y:Z» X – number of months/weeks/days/hours/minutes the request has to be rescheduled from prior run.Y – time units: “M” – months, “D” – days, “H” – hours, “N” – minutesZ – rescheduling type: «S» – from the start of the prior run, «C» – from the completion of the prior run.Samples30:N:S – Repeat every 30 minutes from the start of the prior run5:N:C – Repeat every 5 minutes from the completion of the prior run12:H:S – Repeat every 12 hours from the start of the prior run
12 «On Specific Days» schedules FCRC.CLASS_INFO contentsValues like «XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXYZZZZZZZ»X, Y, Z contains values 0 or 1, representing if the option is selected or notX – digit in each position represent dates 1 – 31Y – Last Day of monthZ – day of week, Su - SaSamples– Days of week: Sa– Dates: Days of week: Mo Tu We Th Fr– Last day of month
13 Reporting the schedules Easy to schedule requests, but hard to keep trackCheck if there are no obsolete schedules that can be removed or tunedMight be few repeting schedules of the same concurrent program can be consolidated into one
15 Conflict Resolution Manager (CRM) CRM resolves conflicts in execution of requests enforced by incompatibility rulesSimplified CRM workflowStartup – load all incompatibility rulesLoaded from FND_CONCURRENT_PROGRAM_SERIALStored in memory by FNDCRM (CRM binary executable)Each itereationIf FCQ.CONTROL_CODE=‘V’, reload incompatibility rules (?)Lock STATUS_CODE for all Pending/Normal and Pending/Scheduled requestsCheck each Pending request against the incompatibility rules and verify there are running concurrent managers that can process them (FND_CONCURRENT_WORKER_REQUESTS)«Release» Pending/Standby requests, which don’t break any rules«Return» Pending/Normal requests, which break some rules (?)Record statistics in FND_CRM_HISTORYSleep for «Sleep Seconods»
16 Interesting DB objects FND_CONCURRENT_WORKER_REQUESTSView returning mappings of requests and concurrent managers able to execute themDefinition of the view contains hardcoded IDs according to the Specialization Rules of Concurrent ManagersRebuilt by «Build Concurrent Request Queue View» request on change of Specilaization rulesFND_CRM_HISTORYGood information for tuning and troubleshootingRecords statistics for each CRM run, stotistics include:REQUESTS_EXAMINEDREQUESTS_STANDBYREQUESTS_RELEASEDREQUESTS_RETURNEDCan be joined with FCR to find out how many requests were released in each RCM execution iteration FCR.CRM_RELEASE_DATE between FCH.WORK_START AND FCH.WORK_ENDPurged by «Purge Concurrent Request and/or Manager Data», leaving 1 day of history (R12.1.3)
18 Why is my request still pending? Check the status!Pending/Scheduled – time for execution has not come yetPending/Normal –waiting to be picked up by concurrent managerThe request has just been releasedAll concurrent manager processes are busy executing requestsLong request execution queuePending/Standby – Waiting for CRM to be releasedSELECT WREQID, PHASE, STATUS, WHYFROM FND_CONC_WAITING_REQUESTSWHERE REQID = &REQUEST_ID
21 Building SQL for querying the requests queue Select R.Rowid From Fnd_Concurrent_Requests R Where R.Hold_Flag = 'N' And R.Status_Code = 'I' And R.Requested_Start_Date <= Sysdate And (R.Node_Name1 is null or (R.Node_Name1 is not null and FND_DCP.target_node_mgr_chk(R.request_id) = 1)) AND EXISTS (Select Null From Fnd_Concurrent_Programs P Where P.Enabled_Flag = 'Y' And R.Program_Application_Id = P.Application_Id And R.Concurrent_Program_Id = P.Concurrent_Program_Id From Fnd_Oracle_Userid O Where R.Oracle_Id = O.Oracle_Id AND EXISTS (Select Null From Fnd_Conflicts_Domain C Where P.Run_Alone_Flag = C.RunAlone_Flag And R.CD_Id = C.CD_Id)) …… And (P.Execution_Method_Code != 'S' OR (R.PROGRAM_APPLICATION_ID, R.CONCURRENT_PROGRAM_ID) IN ((0, 98), (0, 100), (0, 31721), (0, 31722), (0, 31757))) AND ((R.PROGRAM_APPLICATION_ID, R.CONCURRENT_PROGRAM_ID) NOT IN ((510, 40032), (510, 40033), (510, 42156), (510, 42157), (530, 43793), (530, 43794), (535, 42626), (535, 42627), (535, 42628)) AND ((R.REQUEST_CLASS_APPLICATION_ID IS NULL AND R.CONCURRENT_REQUEST_CLASS_ID IS NULL) OR (R.REQUEST_CLASS_APPLICATION_ID, R.CONCURRENT_REQUEST_CLASS_ID) NOT IN ((0, 2))))) ORDER BY NVL(R.priority, ), R.Priority_Request_ID, R.Request_IDExclusion specialization rules for concurrent programsExecution order for concurrent requestsImplementation of «Immediate» type executables (Subroutines)Exclusion Specialization rule for Request typeDistributed Concurrent Processing implementation«Run Alone» flag implementation with Conflict domainsQuery only «Pending/Normal» requests
22 Building SQL for querying the requests queue The query is built at the startup of the concurrent managerIt hardcodes all specialization rules for the managerAny changes to specialization rules force restart of the concurrent manager processes (and runs «Build Concurrent Request Queue View» concurrent program too)Be careful! It does restart automatically! What happens if there are long running requests?Use «request types»!
23 Locking the STATUS_CODE SELECT ... FROM fnd_concurrent_requests R, fnd_concurrent_programs P, fnd_application A, fnd_user U, fnd_oracle_userid O, fnd_conflicts_domain C, fnd_concurrent_queues Q, fnd_application A2, fnd_executables E, fnd_conc_request_arguments X WHERE R.Status_code = 'I' And ((R.OPS_INSTANCE is null) or (R.OPS_INSTANCE = -1) or (R.OPS_INSTANCE = decode(:dcp_on, 1, FND_CONC_GLOBAL.OPS_INST_NUM, R.OPS_INSTANCE))) And R.Request_ID = X.Request_ID(+) And R.Program_Application_Id = P.Application_Id(+) And R.Concurrent_Program_Id = P.Concurrent_Program_Id(+) And R.Program_Application_Id = A.Application_Id(+) And P.Executable_Application_Id = E.Application_Id(+) And P.Executable_Id = E.Executable_Id(+) And P.Executable_Application_Id = A2.Application_Id(+) And R.Requested_By = U.User_Id(+) And R.Cd_Id = C.Cd_Id(+) And R.Oracle_Id = O.Oracle_Id(+) And Q.Application_Id = :q_applid And Q.Concurrent_Queue_Id = :queue_id And (P.Enabled_Flag is NULL OR P.Enabled_Flag = 'Y') And R.Hold_Flag = 'N‘ …… And R.Requested_Start_Date <= Sysdate And (R.Enforce_Seriality_Flag = 'N' OR (C.RunAlone_Flag = P.Run_Alone_Flag And (P.Run_Alone_Flag = 'N' OR Not Exists (Select Null From Fnd_Concurrent_Requests Sr Where Sr.Status_Code In ('R', 'T') And Sr.Enforce_Seriality_Flag = 'Y' And Sr.CD_id = C.CD_Id)))) And Q.Running_Processes <= Q.Max_Processes And R.Rowid = :reqname And ((P.Execution_Method_Code != 'S' OR (R.PROGRAM_APPLICATION_ID, R.CONCURRENT_PROGRAM_ID) IN ((0, 98), (0, 100), (0, 31721), (0, 31722), (0, 31757))) AND ((R.PROGRAM_APPLICATION_ID, R.CONCURRENT_PROGRAM_ID) NOT IN ((510, 40032), (510, 40033), (510, 42156), (510, 42157), (530, 43793), (530, 43794), (535, 42626), (535, 42627), (535, 42628)) AND ((R.REQUEST_CLASS_APPLICATION_ID IS NULL AND R.CONCURRENT_REQUEST_CLASS_ID IS NULL) OR (R.REQUEST_CLASS_APPLICATION_ID, R.CONCURRENT_REQUEST_CLASS_ID) NOT IN ((0, 2))))) FOR UPDATE OF R.status_code NoWait
24 Locking the STATUS_CODE Query for locking the STATUS_CODE reimplements the same validation criteria to make sure situation has not changedAll processes of a concurrent manager use the same query to fetch the «cache size» number of requestsAs more processes of the same manager are run, as higher the competition for requests («ORA-00054: resource busy and acquire with NOWAIT specified», or 0 rows updated by the query if the status has been changed already)As higher the competition, as faster runs out the list of cached request ids for each managerAs sooner the list of cached queries runs out, as more often FCR is queriedWe want to query FCR as saldom as possibleNot hard to get to point where FCR queries are TOP SQLs in DBEven more important if you have RACThe key is to minimize the number of concurrent manager processesCache size and sleep seconds have some effect
25 Cache size and Sleep seconds sleep time of Conflict Resolution Manager affects how soon the request will be passed to executionare spent only when no requests are in pending/normal status in FCRshould be chosen based on max time the request is allowed to spend in the queueMax time 20s and 5 managers? == 100s sleep seconds?«Cache size»Large cache sizes make changes of request priorities less effective (do you use different priorities)Small cache size is OK for Long-running requests queueLarger cache sizes are OK for Short-running requests queues that have few concurrent manager instances.Large cache size increases the number of failed attempts to lock the status code.
27 How to find processes and sessions? select r.request_id req_id, r.phase_code p, r.status_code s, (select node_name || ':' from applsys.fnd_concurrent_processes cp where concurrent_process_id = r.controlling_manager) || r.os_process_id cp_process, gi.INSTANCE_NAME || ':' || ss.sid || ',' || ss.serial# inst_sid_serial#, gi.HOST_NAME || ':' || pp.spid db_process, ss.program, ss.status, ss.sql_id || ':' || ss.sql_child_number sql_id_chld, ss.event, ss.WAIT_TIME, ss.STATE from applsys.fnd_concurrent_requests r, gv$session ss, gv$process pp, gv$instance gi, applsys.fnd_concurrent_processes cp where request_id = &request_id and ss.audsid(+) = r.oracle_session_id and pp.inst_id(+) = ss.inst_id and pp.addr(+) = ss.paddr and gi.inst_id(+) = ss.inst_id and cp.concurrent_process_id(+) = r.controlling_managerFor completed requests CP_PROCESS field is still visible
28 Completed / Normal… …and also the presentation is Completed/Normal
29 Where to get more information? OTN - Oracle E-Business Suite System Administrator's Guide Documentation Set– ER Diagrams and information about the DB Objects– Comment and ask the questions, I will answer!Thank you!?
30 Māris Elsiņš Senior Oracle Applications DBA Tieto Latvia,
Your consent to our cookies if you continue to use this website.