Presentation is loading. Please wait.

Presentation is loading. Please wait.

© 2010 Tieto Corporation Surviving the Crisis with the Help of Oracle Database Resource Manager UKOUG Conference Series Technology & E-Business Suite 2010.

Similar presentations


Presentation on theme: "© 2010 Tieto Corporation Surviving the Crisis with the Help of Oracle Database Resource Manager UKOUG Conference Series Technology & E-Business Suite 2010."— Presentation transcript:

1 © 2010 Tieto Corporation Surviving the Crisis with the Help of Oracle Database Resource Manager UKOUG Conference Series Technology & E-Business Suite 2010 Māris Elsiņš Senior Oracle Applications DBA Tieto Latvia,

2 © 2010 Tieto Corporation Who I am? 8 years in IT 3 years – PL/SQL developer 5 years – Oracle [Apps] DBA (started with and 8.1.7) Certificates 10g OCM 9i / 10g / 11g OCP 11i Applications Database Administrator OCP 11i System Administrator OCE Conferences UKOUG 2007/2008/2010 LVOUG 2009/2010 EMEA Harmony 2010 Current employer – Tieto Latvia All kinds of oracle DBA tasks - patching, upgrade, performance tuning, troubleshooting, planning and implementation of backup and recovery procedures, cross platform migration, etc. Planning of system architecture, design and implementation of HA solutions (RAC, Data Guard, Custom cold failover) Implementation of system specific monitoring, automation of routine tasks Technical project planning and coordination, management of team of DBAs

3 © 2010 Tieto Corporation I come from Latvia! Latvia is a small country… …with a huge economic crisis. Thats where the topic comes from! Surviving the Crisis with the Help of Oracle Database Resource Manager

4 © 2010 Tieto Corporation Contents Presentation will contain: Short Introduction to the Resource Manager DEMO Evaluation of Performance impact The Goal is: Not to read the documentation for you… …but to show why and when the resource manager can be very useful

5 © 2010 Tieto Corporation The problem OS doesnt care enough about DB sessions/processes according to what business requires Assigns the same priority for all processes CPU resources are equally distributed among all processes Inability to manage DB-specific resources Parallel processes Active sessions Sessions holding latches Undo usage Context switching overhead when many processes running Problems start when theres not enough CPU gor everyone CPU starvation can be hard to recover from (snowball effect) CPU starvation makes online troubleshooting hard to do

6 © 2010 Tieto Corporation Some possible problem scenarios Some samples of problems running reports cause too much load on the OLTP system One of the sessions allocate all parallel query slaves therefore other sessions cant use any of them Application support team runs heavy queries to anayze the data leaving less resources for business processing Too wide search criteria cause hangs in the search form 3 of 8 CPU cores are idle, my query runs without parallel execution, I could use the idle CPUs to provide results faster. Users dont log out and leave idle sessions Sessions with incomplete transactions have locked some rows and other sessions have stuck therefore. Does Oracle Resource Manager solve all these problems? Some samples of problems running reports cause too much load on the OLTP system One of the sessions allocate all parallel query slaves therefore other sessions cant use any of them Application support team runs heavy queries to anayze the data leaving less resources for business processing Too wide search criteria cause hangs in the search form 3 of 8 CPU cores are idle, my query runs without parallel execution, I could use the idle CPUs to provide results faster. Users dont log out and leave idle sessions Sessions with incomplete transactions have locked some rows and other sessions have stuck therefore. Does Oracle Resource Manager solve all these problems?

7 © 2010 Tieto Corporation The concept of the Resource manager Resource manager Included in Oracle EE licnese Allows prioritization of sessions according to the defined business requirements Allows defining the guaranteed amount of allocated resources for each type of sessions (consumer group) Resources not used by higher priority sessions, can be used by lower priority sessions Prioritization archieved by changing the process states to running/sleeping Resource manager does not solve the «lack of CPU resources» problem, it just controls there execution queue Resource manager uses some resources too, the 3rd part of the presentation will estimate the overhead

8 © 2010 Tieto Corporation Which side is the RM enabled for? active sessions all doing CPU work in both cases Check Running/Sleeping process states!

9 © 2010 Tieto Corporation The concept of the Resource manager Consumer group Set of sessions having similar requirements for server resources Resources are allocated to the consumer group, not individual sessions dba_rsrc_consumer_groups Directives Rules that define resource allocation to the consumer group dba_rsrc_plan_directives Resource plan Set of directives defining the distribution of resources among consumer groups dba_rsrc_plans

10 © 2010 Tieto Corporation Features of Resource Manager CPU resource allocation to consumer groups by ratio or percentage Limit of the degree of parallelism active session pool Automated change of consumer group if session has used or is estimated to use the defined amount of resources (CPU, Est CPU, IO_MB, IO_REQ) CPU, Est CPU CPU, Est CPU, IO_MB, IO_REQ Limit of estimated execution time Limit size of undo used by uncommitted sessions Termination of idle sessions Termination of idle blocking sessions L0 70% CPU _ORACLE_BACKGROUND_GROUP_ hidden consumer group for background processes Instance caging /CPU_COUNT + resource plan/ Max CPU Utilization limit

11 © 2010 Tieto Corporation Setting up the resource manager Create the resource consumer groups, resource plan and directives Use DB Console or EM Grid Control Use API (DBMS_RESOURCE_MANAGER package) Define rules to assign sessions to particular consumer groups Connect time ( ORACLE_USER, SERVICE_NAME, CLIENT_OS_USER, CLIENT_PROGRAM, CLIENT_MACHINE) Run time ( MODULE_NAME, MODULE_NAME_ACTION, SERVICE_MODULE, SERVICE_MODULE_ACTION) Enable resource plan ALTER SYSTEM SET resource_manager_plan=ALL_OK'; Excellent documentation! Oracle® Database Administrator's Guide 11g Release 2 (11.2) 27 Managing Resource Allocation with Oracle Database Resource Manager

12 © 2010 Tieto Corporation Important to keep in mind 11g has resource manager enabled by default for maintenance tasks (can cause problems after upgrade to 11g) MAINTENANCE_WINDOW_GROUP scheduler window group DEFAULT_MAINTENANCE_PLAN plan Waitevent «rsmgr: cpu quantum» for sleeping processes Common mistake is to forget about v$session.state WAITING – session is currently waiting WAITED KNOWN TIME – session is running on CPU WAITED SHORT TIME – session is running on CPU (same warning is valid for other wait events too)

13 © 2010 Tieto Corporation DEMO

14 © 2010 Tieto Corporation DEMO on SAMPLE_PLAN3 SYS_GROUP 5% L1 CPU ONLINE3 95% L1 CPU Switch the session to LONG3 group if the execution takes longer then 30 seconds OTHER_GROUPS 100% L2 CPU KILL blocking session if its idle for longer then 90 seconds LONG3 20% L1 CPU KILL session inactive for longer then then 20 seconds

15 © 2010 Tieto Corporation Evaluation of performance impact

16 © 2010 Tieto Corporation Evaluation of performance impact Test platforms on RHEL 4.4, 4Cores + HT, RAID5 Local HDD, 6G RAM on HP-UX v11.31, vPAR on ia64 hp superdome server SD32B, 4Cores, NetApp storage (could have affected the test results), 32G RAM Concept Identically repeatable workload Testing is the only activity (no schedules, jobs, awr snapshots, etc.) Each test repeated 3 times, average execution time recorded DBs configured with ASYNC and DIRECT IO (FS cache does not affect testing) DB Buffer cache flushed before every IO test Goal Identify if enabling the resource manager causes visible impact on performance

17 © 2010 Tieto Corporation Tests «CPU bottleneck» on RHEL with 8, 24, 48 processes No RM plan in use Simple RM plan in use, processes evenly distributed among CGs Complex RM plan in use, processes evenly distributed among CGs «CPU bottleneck» on HP-UX with 4, 12, 24, 48 processes No RM plan in use Simple RM plan in use, processes evenly distributed among CGs Complex RM plan in use, processes evenly distributed among CGs «IO bottleneck» on RHEL with 24 processes No RM plan in use Complex RM plan in use «IO bottleneck» on HP-UX with 24 processes No RM plan in use Complex RM plan in use

18 © 2010 Tieto Corporation Testing with simple RM plan

19 © 2010 Tieto Corporation Testing with complex RM plan

20 © 2010 Tieto Corporation «CPU bottleneck» ---noramloadcpu.sql--- set verify off feed off timing on serverout on time on declare n number; x number:= *&1; z number; date1 date; begin select sysdate into date1 from dual; DBMS_APPLICATION_INFO.SET_MODULE(module_name => 'TEST', action_name => '&2'); for n in x+1..x loop z:=sqrt(n); end loop; dbms_output.put_line(to_char(date1,'HH24:MI:SS')||' : &&2 : &&1'); end; / sqlplus 1 Online_group & sqlplus 2 Batch_group & sqlplus 3 Bug_Maint_group & sqlplus 4 Users_group & sqlplus 5 Postman_group & sqlplus 6 Mail_Maint_group & sqlplus 7 Online_group & sqlplus 8 Batch_group & sqlplus 9 Bug_Maint_group & sqlplus 10 Users_group & sqlplus 11 Postman_group & sqlplus 12 Mail_Maint_group & sqlplus 13 Online_group & sqlplus 14 Batch_group & sqlplus 15 Bug_Maint_group & sqlplus 16 Users_group & sqlplus 17 Postman_group & sqlplus 18 Mail_Maint_group & sqlplus 19 Online_group & sqlplus 20 Batch_group & sqlplus 21 Bug_Maint_group & sqlplus 22 Users_group & sqlplus 23 Postman_group & sqlplus 24 Mail_Maint_group & echo " " wait echo " " Practically no IO Each process spinning on CPU

21 © 2010 Tieto Corporation «CPU bottleneck» results RHEL 4.4 and HP-UX v11.31 and

22 © 2010 Tieto Corporation «CPU bottleneck» results WITHOUT RESOURCE MANAGER :28:05 : LOW : 14 Elapsed: 00:00: :28:05 : LOW : 19 Elapsed: 00:00: :28:06 : HIGH : 43 Elapsed: 00:00: :28:07 : HIGH : 46 Elapsed: 00:00: :28:07 : HIGH : 41 Elapsed: 00:00: :28:05 : LOW : 13 Elapsed: 00:00: :28:15 : HIGH : 39 Elapsed: 00:00: :28:09 : HIGH : 35 Elapsed: 00:00: :28:22 : HIGH : 38 Elapsed: 00:00: :28:19 : HIGH : 34 Elapsed: 00:00: :28:30 : LOW : 4 Elapsed: 00:00: :28:33 : HIGH : 47 Elapsed: 00:00: :28:34 : HIGH : 25 Elapsed: 00:00: :28:37 : LOW : 20 Elapsed: 00:00: :28:41 : HIGH : 29 Elapsed: 00:00: :28:44 : LOW : 12 Elapsed: 00:00: :28:45 : LOW : 15 Elapsed: 00:00: :28:48 : LOW : 11 Elapsed: 00:00: :28:50 : LOW : 6 Elapsed: 00:00: :28:55 : LOW : 17 Elapsed: 00:00: :28:54 : HIGH : 33 Elapsed: 00:00: :28:50 : HIGH : 30 Elapsed: 00:00: :28:59 : LOW : 5 Elapsed: 00:00: :29:03 : HIGH : 37 Elapsed: 00:00: :29:08 : LOW : 23 Elapsed: 00:00: :29:02 : HIGH : 28 Elapsed: 00:00: :29:06 : LOW : 22 Elapsed: 00:00: :29:13 : HIGH : 45 Elapsed: 00:00: :29:17 : LOW : 1 Elapsed: 00:00: :29:15 : LOW : 18 Elapsed: 00:00: :29:20 : HIGH : 40 Elapsed: 00:00: :29:24 : LOW : 24 Elapsed: 00:00: :29:27 : HIGH : 44 Elapsed: 00:00: :29:33 : HIGH : 31 Elapsed: 00:00: :29:37 : LOW : 8 Elapsed: 00:00: :29:36 : LOW : 21 Elapsed: 00:00: :29:37 : LOW : 16 Elapsed: 00:00: :29:43 : LOW : 7 Elapsed: 00:00: :29:45 : HIGH : 36 Elapsed: 00:00: :29:50 : HIGH : 32 Elapsed: 00:00: :29:51 : LOW : 2 Elapsed: 00:00: :29:53 : HIGH : 42 Elapsed: 00:00: :29:56 : LOW : 9 Elapsed: 00:00: :29:57 : HIGH : 26 Elapsed: 00:00: :30:03 : LOW : 3 Elapsed: 00:00: :30:05 : LOW : 10 Elapsed: 00:00: :29:59 : HIGH : 27 Elapsed: 00:00: :30:08 : HIGH : 48 Elapsed: 00:00: real 2m17.292s WITH RESOURCE MANAGER :24:50 : HIGH : 25 Elapsed: 00:00: :24:54 : HIGH : 42 Elapsed: 00:00: :24:54 : HIGH : 29 Elapsed: 00:00: :24:53 : HIGH : 39 Elapsed: 00:00: :24:55 : HIGH : 48 Elapsed: 00:00: :25:04 : HIGH : 36 Elapsed: 00:00: :25:04 : HIGH : 30 Elapsed: 00:00: :25:03 : HIGH : 28 Elapsed: 00:00: :25:11 : HIGH : 38 Elapsed: 00:00: :25:11 : HIGH : 44 Elapsed: 00:00: :25:12 : HIGH : 35 Elapsed: 00:00: :25:19 : HIGH : 46 Elapsed: 00:00: :25:20 : HIGH : 40 Elapsed: 00:00: :25:21 : HIGH : 34 Elapsed: 00:00: :25:23 : HIGH : 45 Elapsed: 00:00: :25:23 : HIGH : 32 Elapsed: 00:00: :25:22 : HIGH : 31 Elapsed: 00:00: :25:26 : HIGH : 47 Elapsed: 00:00: :25:23 : HIGH : 33 Elapsed: 00:00: :25:25 : HIGH : 37 Elapsed: 00:00: :25:29 : HIGH : 26 Elapsed: 00:00: :25:28 : HIGH : 27 Elapsed: 00:00: :25:31 : HIGH : 41 Elapsed: 00:00: :25:49 : HIGH : 43 Elapsed: 00:00: :24:50 : LOW : 24 Elapsed: 00:01: :24:51 : LOW : 21 Elapsed: 00:02: :24:50 : LOW : 22 Elapsed: 00:02: :24:53 : LOW : 8 Elapsed: 00:02: :24:53 : LOW : 5 Elapsed: 00:02: :25:05 : LOW : 3 Elapsed: 00:01: :25:17 : LOW : 2 Elapsed: 00:01: :25:28 : LOW : 7 Elapsed: 00:01: :25:17 : LOW : 9 Elapsed: 00:01: :25:19 : LOW : 6 Elapsed: 00:01: :25:19 : LOW : 4 Elapsed: 00:01: :25:01 : LOW : 10 Elapsed: 00:02: :25:03 : LOW : 1 Elapsed: 00:02: :25:22 : LOW : 20 Elapsed: 00:01: :24:55 : LOW : 16 Elapsed: 00:02: :24:55 : LOW : 11 Elapsed: 00:02: :25:29 : LOW : 18 Elapsed: 00:01: :25:10 : LOW : 15 Elapsed: 00:01: :25:12 : LOW : 17 Elapsed: 00:01: :25:09 : LOW : 23 Elapsed: 00:01: :25:21 : LOW : 12 Elapsed: 00:01: :25:11 : LOW : 19 Elapsed: 00:01: :25:25 : LOW : 13 Elapsed: 00:01: :25:15 : LOW : 14 Elapsed: 00:01: real 2m15.818s First Last First Last

23 © 2010 Tieto Corporation «IO bottleneck» ---prepare data--- create table MEL_IO_TEST as select cast(rownum as number)x, lpad('a',400,'a') y from dual connect by level <= ; insert into MEL_IO_TEST select * from MEL_IO_TEST; commit; create index MEL_IO_TEST_IDX on MEL_IO_TEST(x); --- ioloadtest.sql--- set verify off feed off timing on serverout on time on declare nn number; date1 date; begin select sysdate into date1 from dual; DBMS_APPLICATION_INFO.SET_MODULE(module_name => 'TEST', action_name => '&3'); select /*+ index(a MEL_IO_TEST_IDX)*/avg(length(y)+length(x)) into nn from MEL_IO_TEST a where x between /&2*(&1-1) and /&&2*&&1; dbms_output.put_line(to_char(date1,'HH24:MI:SS')||' : &&3 : &&1 : &&2'); end; / sqlplus sqlplus 1 24 Online_group & sqlplus 2 24 Batch_group & sqlplus 3 24 Bug_Maint_group & sqlplus 4 24 Users_group & sqlplus 5 24 Postman_group & sqlplus 6 24 Mail_Maint_group & sqlplus 7 24 Online_group & sqlplus 8 24 Batch_group & sqlplus 9 24 Bug_Maint_group & sqlplus Users_group & sqlplus Postman_group & sqlplus Mail_Maint_group & sqlplus Online_group & sqlplus Batch_group & sqlplus Bug_Maint_group & sqlplus Users_group & sqlplus Postman_group & sqlplus Mail_Maint_group & sqlplus Online_group & sqlplus Batch_group & sqlplus Bug_Maint_group & sqlplus Users_group & sqlplus Postman_group & sqlplus Mail_Maint_group & echo " " wait echo " "

24 © 2010 Tieto Corporation «IO bottleneck» results RHEL 4.4 and Average execution time: No RM: s Complex RM: s HP-UX v11.31 and Average execution time: No RM: 36,30s Complex RM: 37,42s

25 © 2010 Tieto Corporation «IO bottleneck» results WITHOUT RESOURCE MANAGER :23:47 : HIGH : 23 : 64 Elapsed: 00:02: :23:47 : LOW : 1 : 64 Elapsed: 00:02: :23:47 : LOW : 2 : 64 Elapsed: 00:02: :23:47 : HIGH : 24 : 64 Elapsed: 00:03: :23:47 : LOW : 12 : 64 Elapsed: 00:03: :23:47 : LOW : 8 : 64 Elapsed: 00:03: :23:47 : LOW : 10 : 64 Elapsed: 00:03: :23:47 : LOW : 11 : 64 Elapsed: 00:03: :23:47 : LOW : 9 : 64 Elapsed: 00:03: :23:47 : HIGH : 16 : 64 Elapsed: 00:03: :23:47 : HIGH : 14 : 64 Elapsed: 00:03: :23:47 : HIGH : 15 : 64 Elapsed: 00:03: :23:47 : LOW : 7 : 64 Elapsed: 00:03: :23:47 : HIGH : 13 : 64 Elapsed: 00:03: :23:47 : HIGH : 20 : 64 Elapsed: 00:03: :23:47 : HIGH : 19 : 64 Elapsed: 00:03: :23:47 : LOW : 5 : 64 Elapsed: 00:03: :23:47 : LOW : 4 : 64 Elapsed: 00:03: :23:47 : HIGH : 22 : 64 Elapsed: 00:03: :23:47 : HIGH : 18 : 64 Elapsed: 00:03: :23:47 : LOW : 6 : 64 Elapsed: 00:03: :23:47 : HIGH : 17 : 64 Elapsed: 00:03: :23:47 : LOW : 3 : 64 Elapsed: 00:03: :23:47 : HIGH : 21 : 64 Elapsed: 00:03: real 3m48.504s WITH RESOURCE MANAGER :32:14 : LOW : 7 : 64 Elapsed: 00:02: :32:14 : LOW : 2 : 64 Elapsed: 00:02: :32:15 : HIGH : 24 : 64 Elapsed: 00:03: :32:15 : HIGH : 23 : 64 Elapsed: 00:03: :32:14 : HIGH : 13 : 64 Elapsed: 00:03: :32:15 : HIGH : 16 : 64 Elapsed: 00:03: :32:14 : LOW : 9 : 64 Elapsed: 00:03: :32:15 : HIGH : 22 : 64 Elapsed: 00:03: :32:14 : HIGH : 19 : 64 Elapsed: 00:03: :32:14 : LOW : 10 : 64 Elapsed: 00:03: :32:14 : LOW : 11 : 64 Elapsed: 00:03: :32:15 : HIGH : 21 : 64 Elapsed: 00:03: :32:15 : LOW : 3 : 64 Elapsed: 00:03: :32:15 : HIGH : 20 : 64 Elapsed: 00:03: :32:14 : LOW : 8 : 64 Elapsed: 00:03: :32:14 : LOW : 6 : 64 Elapsed: 00:03: :32:15 : HIGH : 17 : 64 Elapsed: 00:03: :32:14 : HIGH : 14 : 64 Elapsed: 00:03: :32:14 : LOW : 12 : 64 Elapsed: 00:03: :32:14 : LOW : 1 : 64 Elapsed: 00:03: :32:15 : HIGH : 15 : 64 Elapsed: 00:03: :32:15 : HIGH : 18 : 64 Elapsed: 00:03: :32:14 : LOW : 4 : 64 Elapsed: 00:03: :32:14 : LOW : 5 : 64 Elapsed: 00:03: real 3m47.104s RM does not help if the system is IO-bound

26 © 2010 Tieto Corporation Summary Very useful if CPU utilization close to 100% Ensures most critical sessions get the resources Reduces overall load on the server by setting least important processes to «sleep» Resource Manager does not solve the problem – just allows prioritization Performance overhead is practically invisible, (could depend on platform) – test yourself! Included in EE license No downtime needed to implement it, setup can be changed dynamically Can be used as a short-term workaround of some performance problems. Some bugs can show up as additional code layer introduced – test it!

27 © 2010 Tieto Corporation Thank you! ? – Comment my blog and Ill answer!

28 © 2010 Tieto Corporation Māris Elsiņš Senior Oracle Applications DBA Tieto Latvia,


Download ppt "© 2010 Tieto Corporation Surviving the Crisis with the Help of Oracle Database Resource Manager UKOUG Conference Series Technology & E-Business Suite 2010."

Similar presentations


Ads by Google