We think you have liked this presentation. If you wish to download it, please recommend it to your friends in any social system. Share buttons are a little bit lower. Thank you!
Presentation is loading. Please wait.
Published byToby Warmington
Modified over 2 years ago
© 2010 Tieto Corporation Internals of Concurrent Managers UKOUG Conference Series Technology & E-Business Suite 2010 Māris Elsiņš Senior Oracle Applications DBA Tieto Latvia, Maris.Elsins@tieto.com
© 2010 Tieto Corporation Who I am? 8 years in IT 3 years – PL/SQL developer 5 years – Oracle [Apps] DBA (started with 11.5.7 and 8.1.7) Certificates 10g OCM 9i / 10g / 11g OCP 11i Applications Database Administrator OCP 11i System Administrator OCE Conferences UKOUG 2007/2008/2010 LVOUG 2009/2010 EMEA Harmony 2010 Current employer – Tieto Latvia All kinds of oracle DBA tasks - patching, upgrade, performance tuning, troubleshooting, planning and implementation of backup and recovery procedures, cross platform migration, etc Planning of sydtem architecture, design and implementation of HA solutions (RAC, Data Guard, Custom cold failover) Implementation of system specific monitoring, automation of routine tasks Technical project planning and coordination, management of team of DBAs 22010-11-29
© 2010 Tieto Corporation Purpose of this session Provide background knowledge for successful troubleshooting of concurrent request problems What this presentation is about? Life cycle of a concurrent request Implementation details of the most interesting phases of a life cycle of a concurrent request Internals of Concurrent Managers and Conflict Resolution Managers DB objects that can be used for troubleshooting Why it is important? DBA has to be quick when solving real issues Knowing the process before the problems happen decreases solving time Querying DB objects is faster then navigating through Forms Knowing the process is important when tuning the setup of Concurrent Managers 32010-11-29
© 2010 Tieto Corporation Most interesting DB objects Objects FND_CONCURRENT_REQUESTS (FCR) FND_CONCURRENT_PROGRAMS (FCP) FND_CONCURRENT_QUEUES (FCQ) FND_CONCURRENT_PROCESSES (FCPROC) FND_CONCURRENT_PROGRAM_SERIAL (FCPS) FND_CONC_RELEASE_CLASSES (FCRC) FND_LOOKUPS (FL) FND_CONCURRENT_WORKER_REQESTS (FCWR) FND_CRM_HISTORY (FCH) FND_CONC_WAITING_REQUESTS Some examples how the tables are used will be uncovered in this presentation 42010-11-29
© 2010 Tieto Corporation Types of concurrent managers Internal Concurrent Manager Service Manager Concurrent Manager Conflict Resolution Manager Internal Monitor Transaction Manager Scheduler/Prereleaser Manager … Today the accent is on Concurrent Manager Conflict Resolution Manager 52010-11-29
© 2010 Tieto Corporation Life cycle of a concurrent request INACTIVE (I) Disabled (U) On Hold (H) No Manager (M) PENDING (P) Normal (I) Standby (Q) Scheduled (P) Waiting (Z) RUNNING (R) Normal (R) Paused (W) Resuming (B) Terminating (T) COMPLETED (C) Normal (C) Error (E) Warning (G) Cancelled (D) Terminated (X) 62010-11-29 FND_LOOKUPS contains the values of Phase/Status Codes «Oracle® E-Business Suite System Administrator's Guide – Maintenance» describes the meaning of each phase and status
© 2010 Tieto Corporation Typical life cycle of most requests Pending / Scheduled Request is scheduled for the execution in future Pending / Standby Time to execute the request has arrived Request waits for to be evaluated by the Conflict Resolution Manager Pending / Normal Request is allowed to be executed Request is waiting to be picked up by Concurrent Manager Running / Running Request is beeing executed by a concurrent manager Completed / Normal 72010-11-29
© 2010 Tieto Corporation Phase/Status representation in tables FND_CONCURRENT_REQUESTS PHASE_CODE STATUS_CODE Is that so simple? NO! Phase = Inactive practically not used Inactive/Disabled: FCR.PHASE_CODE=P, FCP.ENABLED_FLAG=N Inactive/On Hold: FCR.PHASE_CODE=P, FCR.HOLD_FLAG=Y Inactive/No manager: FCR.PHASE_CODE=P and nonexistance of specific rows in FND_CONCURRENT_WORKER_REQESTS view Pending/Scheduled is actually STATUS_CODE=Q+(FCR.REQUESTED_START_DATE>SYSDATE) Why so complicated? No need to update statuses too often 82010-11-29
© 2010 Tieto Corporation Pending / Scheduled 92010-11-29
© 2010 Tieto Corporation Schedule types As Soon as Possible/Once FCR.REQUESTED_START_DATE Periodically/On Specific Days FND_CONC_RELEASE_CLASSES (FCRC) Join with FCR on column RELEASE_CLASS_ID FCRC.DATE1 - Start at field in the form FCRC.DATE2 - End at field in the form FCRC.CLASS_TYPE – «P» for «Periodically», «S» for «On Specific days» FCRC.CLASS_INFO – actual schedule data FCR.REQUESTED_START_DATE always contains the time of next execution, only this field used to determine when the request should be run by the concurrent manager 102010-11-29
© 2010 Tieto Corporation Periodic schedules FCRC.CLASS_INFO contents Values like «X:Y:Z» X – number of months/weeks/days/hours/minutes the request has to be rescheduled from prior run. Y – time units: M – months, D – days, H – hours, N – minutes Z – rescheduling type: «S» – from the start of the prior run, «C» – from the completion of the prior run. Samples 30:N:S – Repeat every 30 minutes from the start of the prior run 5:N:C – Repeat every 5 minutes from the completion of the prior run 12:H:S – Repeat every 12 hours from the start of the prior run 112010-11-29
© 2010 Tieto Corporation «On Specific Days» schedules FCRC.CLASS_INFO contents Values like «XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXYZZZZZZZ» X, Y, Z contains values 0 or 1, representing if the option is selected or not X – digit in each position represent dates 1 – 31 Y – Last Day of month Z – day of week, Su - Sa Samples 000000000000000000000000000000000000001 – Days of week: Sa 111111111000000000000000000000000111110 – Dates: 1 2 3 4 5 6 7 8 9. Days of week: Mo Tu We Th Fr 000000000000000000000000000000010000000 – Last day of month 122010-11-29
© 2010 Tieto Corporation Reporting the schedules Easy to schedule requests, but hard to keep track Check if there are no obsolete schedules that can be removed or tuned Might be few repeting schedules of the same concurrent program can be consolidated into one 132010-11-29
© 2010 Tieto Corporation Pending / Standby 142010-11-29
© 2010 Tieto Corporation Conflict Resolution Manager (CRM) CRM resolves conflicts in execution of requests enforced by incompatibility rules Simplified CRM workflow Startup – load all incompatibility rules 1.Loaded from FND_CONCURRENT_PROGRAM_SERIAL 2.Stored in memory by FNDCRM (CRM binary executable) Each itereation 1.If FCQ.CONTROL_CODE=V, reload incompatibility rules (?) 2.Lock STATUS_CODE for all Pending/Normal and Pending/Scheduled requests 3.Check each Pending request against the incompatibility rules and verify there are running concurrent managers that can process them (FND_CONCURRENT_WORKER_REQUESTS) 1.«Release» Pending/Standby requests, which dont break any rules 2.«Return» Pending/Normal requests, which break some rules (?) 4.Record statistics in FND_CRM_HISTORY 5.Sleep for «Sleep Seconods» 152010-11-29
© 2010 Tieto Corporation Interesting DB objects FND_CONCURRENT_WORKER_REQUESTS View returning mappings of requests and concurrent managers able to execute them Definition of the view contains hardcoded IDs according to the Specialization Rules of Concurrent Managers Rebuilt by «Build Concurrent Request Queue View» request on change of Specilaization rules FND_CRM_HISTORY Good information for tuning and troubleshooting Records statistics for each CRM run, stotistics include: REQUESTS_EXAMINED REQUESTS_STANDBY REQUESTS_RELEASED REQUESTS_RETURNED Can be joined with FCR to find out how many requests were released in each RCM execution iteration FCR.CRM_RELEASE_DATE between FCH.WORK_START AND FCH.WORK_END Purged by «Purge Concurrent Request and/or Manager Data», leaving 1 day of history (R12.1.3) 162010-11-29
© 2010 Tieto Corporation Graphing FND_CRM_HISTORY 172010-11-29
© 2010 Tieto Corporation Why is my request still pending? Check the status! Pending/Scheduled – time for execution has not come yet Pending/Normal –waiting to be picked up by concurrent manager The request has just been released All concurrent manager processes are busy executing requests Long request execution queue Pending/Standby – Waiting for CRM to be released SELECT WREQID, PHASE, STATUS, WHY FROM FND_CONC_WAITING_REQUESTS WHERE REQID = &REQUEST_ID --5820658 182010-11-29
© 2010 Tieto Corporation Pending / Normal 192010-11-29
© 2010 Tieto Corporation Workflow of «Concurrent Managers» 202010-11-29
© 2010 Tieto Corporation Building SQL for querying the requests queue Select R.Rowid From Fnd_Concurrent_Requests R Where R.Hold_Flag = 'N' And R.Status_Code = 'I' And R.Requested_Start_Date <= Sysdate And (R.Node_Name1 is null or (R.Node_Name1 is not null and FND_DCP.target_node_mgr_chk(R.request_id) = 1)) AND EXISTS (Select Null From Fnd_Concurrent_Programs P Where P.Enabled_Flag = 'Y' And R.Program_Application_Id = P.Application_Id And R.Concurrent_Program_Id = P.Concurrent_Program_Id AND EXISTS (Select Null From Fnd_Oracle_Userid O Where R.Oracle_Id = O.Oracle_Id AND EXISTS (Select Null From Fnd_Conflicts_Domain C Where P.Run_Alone_Flag = C.RunAlone_Flag And R.CD_Id = C.CD_Id)) … And (P.Execution_Method_Code != 'S' OR (R.PROGRAM_APPLICATION_ID, R.CONCURRENT_PROGRAM_ID) IN ((0, 98), (0, 100), (0, 31721), (0, 31722), (0, 31757))) AND ((R.PROGRAM_APPLICATION_ID, R.CONCURRENT_PROGRAM_ID) NOT IN ((510, 40032), (510, 40033), (510, 42156), (510, 42157), (530, 43793), (530, 43794), (535, 42626), (535, 42627), (535, 42628)) AND ((R.REQUEST_CLASS_APPLICATION_ID IS NULL AND R.CONCURRENT_REQUEST_CLASS_ID IS NULL) OR (R.REQUEST_CLASS_APPLICATION_ID, R.CONCURRENT_REQUEST_CLASS_ID) NOT IN ((0, 2))))) ORDER BY NVL(R.priority, 999999999), R.Priority_Request_ID, R.Request_ID 212010-11-29 Query only «Pending/Normal» requests Distributed Concurrent Processing implementation «Run Alone» flag implementation with Conflict domains Implementation of «Immediate» type executables (Subroutines)Exclusion specialization rules for concurrent programsExclusion Specialization rule for Request typeExecution order for concurrent requests
© 2010 Tieto Corporation Building SQL for querying the requests queue The query is built at the startup of the concurrent manager It hardcodes all specialization rules for the manager Any changes to specialization rules force restart of the concurrent manager processes (and runs «Build Concurrent Request Queue View» concurrent program too) Be careful! It does restart automatically! What happens if there are long running requests? Use «request types»! 222010-03-26
© 2010 Tieto Corporation Locking the STATUS_CODE SELECT... FROM fnd_concurrent_requests R, fnd_concurrent_programs P, fnd_application A, fnd_user U, fnd_oracle_userid O, fnd_conflicts_domain C, fnd_concurrent_queues Q, fnd_application A2, fnd_executables E, fnd_conc_request_arguments X WHERE R.Status_code = 'I' And ((R.OPS_INSTANCE is null) or (R.OPS_INSTANCE = -1) or (R.OPS_INSTANCE = decode(:dcp_on, 1, FND_CONC_GLOBAL.OPS_INST_NUM, R.OPS_INSTANCE))) And R.Request_ID = X.Request_ID(+) And R.Program_Application_Id = P.Application_Id(+) And R.Concurrent_Program_Id = P.Concurrent_Program_Id(+) And R.Program_Application_Id = A.Application_Id(+) And P.Executable_Application_Id = E.Application_Id(+) And P.Executable_Id = E.Executable_Id(+) And P.Executable_Application_Id = A2.Application_Id(+) And R.Requested_By = U.User_Id(+) And R.Cd_Id = C.Cd_Id(+) And R.Oracle_Id = O.Oracle_Id(+) And Q.Application_Id = :q_applid And Q.Concurrent_Queue_Id = :queue_id And (P.Enabled_Flag is NULL OR P.Enabled_Flag = 'Y') And R.Hold_Flag = 'N … … And R.Requested_Start_Date <= Sysdate And (R.Enforce_Seriality_Flag = 'N' OR (C.RunAlone_Flag = P.Run_Alone_Flag And (P.Run_Alone_Flag = 'N' OR Not Exists (Select Null From Fnd_Concurrent_Requests Sr Where Sr.Status_Code In ('R', 'T') And Sr.Enforce_Seriality_Flag = 'Y' And Sr.CD_id = C.CD_Id)))) And Q.Running_Processes <= Q.Max_Processes And R.Rowid = :reqname And ((P.Execution_Method_Code != 'S' OR (R.PROGRAM_APPLICATION_ID, R.CONCURRENT_PROGRAM_ID) IN ((0, 98), (0, 100), (0, 31721), (0, 31722), (0, 31757))) AND ((R.PROGRAM_APPLICATION_ID, R.CONCURRENT_PROGRAM_ID) NOT IN ((510, 40032), (510, 40033), (510, 42156), (510, 42157), (530, 43793), (530, 43794), (535, 42626), (535, 42627), (535, 42628)) AND ((R.REQUEST_CLASS_APPLICATION_ID IS NULL AND R.CONCURRENT_REQUEST_CLASS_ID IS NULL) OR (R.REQUEST_CLASS_APPLICATION_ID, R.CONCURRENT_REQUEST_CLASS_ID) NOT IN ((0, 2))))) FOR UPDATE OF R.status_code NoWait 232010-11-29
© 2010 Tieto Corporation Locking the STATUS_CODE Query for locking the STATUS_CODE reimplements the same validation criteria to make sure situation has not changed All processes of a concurrent manager use the same query to fetch the «cache size» number of requests As more processes of the same manager are run, as higher the competition for requests («ORA-00054: resource busy and acquire with NOWAIT specified», or 0 rows updated by the query if the status has been changed already) As higher the competition, as faster runs out the list of cached request ids for each manager As sooner the list of cached queries runs out, as more often FCR is queried We want to query FCR as saldom as possible Not hard to get to point where FCR queries are TOP SQLs in DB Even more important if you have RAC The key is to minimize the number of concurrent manager processes Cache size and sleep seconds have some effect 242010-11-29
© 2010 Tieto Corporation Cache size and Sleep seconds «Sleep seconds» sleep time of Conflict Resolution Manager affects how soon the request will be passed to execution are spent only when no requests are in pending/normal status in FCR should be chosen based on max time the request is allowed to spend in the queue Max time 20s and 5 managers? == 100s sleep seconds? «Cache size» Large cache sizes make changes of request priorities less effective (do you use different priorities) Small cache size is OK for Long-running requests queue Larger cache sizes are OK for Short-running requests queues that have few concurrent manager instances. Large cache size increases the number of failed attempts to lock the status code. 252010-11-29
© 2010 Tieto Corporation Running / Normal 262010-11-29
© 2010 Tieto Corporation How to find processes and sessions? select r.request_id req_id, r.phase_code p, r.status_code s, (select node_name || ':' from applsys.fnd_concurrent_processes cp where concurrent_process_id = r.controlling_manager) || r.os_process_id cp_process, gi.INSTANCE_NAME || ':' || ss.sid || ',' || ss.serial# inst_sid_serial#, gi.HOST_NAME || ':' || pp.spid db_process, ss.program, ss.status, ss.sql_id || ':' || ss.sql_child_number sql_id_chld, ss.event, ss.WAIT_TIME, ss.STATE from applsys.fnd_concurrent_requests r, gv$session ss, gv$process pp, gv$instance gi, applsys.fnd_concurrent_processes cp where request_id = &request_id and ss.audsid(+) = r.oracle_session_id and pp.inst_id(+) = ss.inst_id and pp.addr(+) = ss.paddr and gi.inst_id(+) = ss.inst_id and cp.concurrent_process_id(+) = r.controlling_manager 272010-11-29 For completed requests CP_PROCESS field is still visible
© 2010 Tieto Corporation Completed / Normal… …and also the presentation is Completed/Normal 282010-11-29
© 2010 Tieto Corporation Where to get more information? OTN - Oracle E-Business Suite System Administrator's Guide Documentation Set http://etrm.oracle.com – ER Diagrams and information about the DB Objectshttp://etrm.oracle.com http://appsdbalife.wordpress.com – Comment and ask the questions, I will answer!http://appsdbalife.wordpress.com 292010-11-29 Thank you! ?
© 2010 Tieto Corporation Māris Elsiņš Senior Oracle Applications DBA Tieto Latvia, Maris.Elsins@tieto.com
Copyright © 2007, Oracle. All rights reserved. Managing Concurrent Requests.
Log Shipping, Mirroring, Replication and Clustering Which should I use? That depends on a few questions we must ask the user. We will go over these questions.
ESRI User Conference 2004 ArcSDE. Some Nuggets Setup Performance Distribution Geodatabase History.
Oracle9i Performance Tuning Chapter 1 Performance Tuning Overview.
1. S318417: OAUG SysAdmin SIG Angelo Rosado, Oracle Senior Product Manager Kenneth Baxter, Oracle Strategy Product Manager Biju Mohan, Oracle Principal.
Top 7 performance bottlenecks in Payments and Treasury Banking Applications: QA practitioners viewpoint Seetha Gurunathan Infosys Limited (NASDAQ: INFY)
Oracle Business Intelligence Foundation – Testing and Deploying OBI Repository.
Process Architecture Process Architecture - A portion of a program that can run independently of and concurrently with other portions of the program. Some.
SQL Server - Review Managing the System. Objectives Understand the importance of maintenance plans. Check for database consistency. Understand how.
1 Introduction to Database Systems. 2 Database and Database System / A database is a shared collection of logically related data designed to meet the.
Oracle SQL Developer: Unit Testing, Tuning and Other Advanced Features Kris Rice Senior Director of Development, Database Tools.
Sofia, Bulgaria | 9-10 October SQL Server 2005 High Availability for developers Vladimir Tchalkov Crossroad Ltd. Vladimir Tchalkov Crossroad Ltd.
SQL Server DBA Online TrainingSQL Server DBA Online Training.
Michelle Malcher PepsiCo Session # For the DBA Manager – Understanding Oracle and DBAs.
Informix IDS Administration with the New Server Studio 4.0 By Lester Knutsen My experience with the beta of Server Studio and the new Informix database.
CSCI 150 Database Applications Chapter 1 – Getting Started.
Hibernate Persistence. What is Persistence Persist data to database or other storage. In OO world, persistence means persist object to external storage.
Triggers A Quick Reference and Summary BIT 275. Triggers SQL code permits you to access only one table for an INSERT, UPDATE, or DELETE statement. The.
3-1 4 Oracle Data Integrator First Project – Simple Transformations: One source, one target.
9 Copyright © 2009, Oracle. All rights reserved. Managing Data Concurrency.
Enhancing Application Performance Root Causes and Quick Solutions.
Complete Weekly Timesheets Select work. Add hours and comments Tell Mgr if ETC=0 and need more time. Tell Mgr if using someone else’s ETC. End week, change.
FireRMS SQL Audit, Archiving & Purging Presented by Laura Small FireRMS Quality Assurance.
10/25/2001Database Management -- R. Larson Data Administration and Database Administration University of California, Berkeley School of Information Management.
Denise Luther Senior IT Consultant Practical Technology Enablement with Enterprise Integrator.
Powered by. Motus is an integrated web-based and mobile software platform that supports field work management and execution, giving employees necessary.
37 Copyright © 2007, Oracle. All rights reserved. Module 37: Executing Workflow Processes Siebel 8.0 Essentials.
Presentation Date Top Down Performance Management with OEM Grid Control Or how I learned to stop worrying and love OEM Grid Control 10/1/2010 John Darrah.
Basic Navigation in Oracle R12 BY: Muhammad Irfan.
SSIS Over DTS Sagayaraj Putti (139460). 5 September What is DTS? Data Transformation Services (DTS) DTS is a set of objects and utilities that.
ITEC474 INTRODUCTION. Course Objectives In this course you learn to: Identify various components of the Oracle architecture. Start up and shut down an.
10/5/1999Database Management -- R. Larson Data Administration and Database Administration University of California, Berkeley School of Information Management.
Oracle9i Performance Tuning Chapter 11 Advanced Tuning Topics.
9 Chapter 7 Transaction Management and Concurrency Control.
ICIS-NPDES Plugin Design Preview Webinar ICIS-NPDES Full Batch OpenNode2 Plugin Project Presented by Bill Rensmith Windsor Solutions, Inc. 3/15/2012.
Oracle Business Intelligence Foundation - Commonly Used Features in Repository.
8 Copyright © 2005, Oracle. All rights reserved. Managing Data.
The Relational Model1 Transaction Processing Units of Work.
IS 4420 Database Fundamentals Chapter 12: Data and Database Administration Leon Chen.
CERN Physics Database Services and Plans Maria Girone, CERN-IT
Does Change Management Include Patches? Joel Howard, RingMaster Software Northern California OAUG San Ramon 2004.
Digital Edge Solutions Overview Services – Application Support.
LoadRunner SE Guide 김범수 한국비지네스써비스 ( 주 )
Best Practices for Supporting Oracle Hyperion EPM and Business Intelligence Solutions Mitra Veluri Senior Principal Technical Support Engineer David Valociek.
Oracle Hyperion Financial Data Quality Management Considerations for a scaled, expedited and integrated approach on data quality NCOAUG – Aug 15, 2008.
© 2010 Tieto Corporation Surviving the Crisis with the Help of Oracle Database Resource Manager UKOUG Conference Series Technology & E-Business Suite 2010.
Performance Testing - LR. 6/18/20162 Contents Why Load Test Your Web Application ? Functional vs. Load Web Testing Web-Based, Multi-Tiered Architecture.
SG SCM with MKS scmGalaxy Author: Rajesh Kumar
Making the System Operational
1 Informatica Productivity Pack Save Time and Money while Increasing the Quality of Your PowerCenter Deployment Louis Hausle.
© 2017 SlidePlayer.com Inc. All rights reserved.