Presentation is loading. Please wait.

Presentation is loading. Please wait.

Mastering Oracle Real Application Clusters Performance Tuning at Verizon Wireless S311852

Similar presentations

Presentation on theme: "Mastering Oracle Real Application Clusters Performance Tuning at Verizon Wireless S311852"— Presentation transcript:


2 Mastering Oracle Real Application Clusters Performance Tuning at Verizon Wireless S311852
Ian Remedios Ph.D. Director, Global Product Management Oracle Advanced Customer Services

3 The following is intended to outline our general product direction
The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle. 3

4 Oracle Advanced Customer Services
Dedicated to the continual operational improvement of Oracle solutions and to maximizing the value of Oracle investments. Solution Lifecycle Management Services Database and Application Management Services Industry specific Solution Support Centers Remote and On-Site Expert Services 4 4 4

5 What Differentiates Advanced Customer Services
Financial Services Telecommunications Aerospace and Defense Public Sector Broad Industry Presence Custom Support Packages 2,500+ World Class Experts Remote and Onsite Services for unique and complex environments Breadth of Products & Services High Renewal Rates Existing customers expand services Strong References High Customer Loyalty Industry Leadership Industry/Analyst/Media recognition for operational excellence 1 5

6 Advanced Customer Services Four levels of annual services to meet specific business and budget requirements Solution Support Center Business Critical Assistance Performance Optimization Services Virtual Center of Excellence with designated experts Dedicated Hotline Proactive Onsite Support Prioritized Service Requests Personalized Portal Escalation Management Service Delivery Manager Priority Service Advanced Support Assistance Proactive Onsite Support Access to team of service engineers Prioritized Service Requests Personalized Portal Escalation Management Service Delivery Manager Customers can customize their solutions by choosing from more than 50 individual products, tools and expert services Prioritized Service Requests Personalized Portal Escalation Management Service Delivery Manager Escalation Management Service Delivery Manager Personalized Support Faster Problem Resolution Proactive Problem Avoidance Continual Operational Improvement 6

7 RAC: The Cluster Database
Network Users Centralized Management Console Interconnect No Single Point Of Failure High Speed Switch or Interconnect Clustered Database Servers Shared Cache What RAC is and how it is architected? Oracle RAC is a cluster database with a shared cache architecture that overcomes the limitations of traditional shared-nothing and shared-disk approaches to provide a highly scalable and available database solution for all your business applications. Oracle RAC provides the foundation for enterprise grid computing. Starting at the bottom, RAC is a shared disk architecture which means we have a single database to manage. All nodes in the cluster must see the disk. We require a shared disk subsystem which usually means a SAN (Storage Area Network) or a Network attached Storage (NAS). Oracle supports many servers clustered together with Oracle Clusterware. Up to 100 servers are supported in the cluster. How many nodes should you have in your cluster? It is up to you but think about the availability, if you only have 2 servers, then a node failure will affect 50% of the users, you will lose 50% of processing power. IF you have 10 nodes, then only 10% of users are affected, 10% of your processing power. RAC was architected to have many servers support applications so that the loss of one server does not affect the application throughput. With RAC database, each node in the cluster will have an instance to the database. We say we have shared cache since if data is in memory for instance 1 and another instance needs it, it is faster to ship the block from memory then to ship it from disk to memory. Hub or Switch Fabric Storage Area Network Mirrored Disk Subsystem Drive and Exploit Industry Advances in Clustering

8 Oracle RAC Architecture
public network VIP1 VIPn Service Service Node n Node1 Listener Listener instance 1 instance n ASM ASM cluster interconnect Oracle Clusterware Oracle Clusterware Operating System Operating System This graphic shows the major components of a RAC cluster. As you can see each server node has it’s own operating system and Oracle binaries, including the necessary Oracle clusterware, and runs one or more Oracle instances connected to a single common database on the shared storage. It’s important to note that as of 10g, Oracle provides all the necessary clusterware to manage cluster membership and inter-node communication. No other software is required. With Oracle providing all this functionality, end users are ensured of optimal integration with all supporting database features. It also significantly simplifies support issues. Availability in the architecture is increased with the addition of cluster nodes. Three node clusters are common where customers require high availability. In the event of a node failure the RAC clusterware determines the new cluster membership and notifies the database of a change. Surviving nodes continue to process transactions while beginning on-line recovery of the failed-nodes’ transactions from the Redo Logs on the shared storage. (No information needs be recovered from the lost node.) The time it takes to complete recovery is a function of the activity at the time of failure and tuning parameters set by the Database Administrator. From the application side, users can be automatically re-connected to another node in the cluster by implementing Oracle’s Fast Application Notification (FAN) and Fast Connection Failover (FCF). Lets take a closer look at the clusterware … shared storage Redo / Archive logs all instances Managed by ASM Database / Control files RAW Devices OCR and Voting Disks

9 Create reference RAC systems Create Production clusters
RAC Deployment Cycle Test users Testing and Staging 1 2 Create reference RAC systems Stage gold images Production We cover the entire RAC deployment lifecycle across 4 phases: The software is installed by EM on a reference node. It is patched and tested there. Once tested, the software is sucked into the Software Library as a gold image The gold image is used to create production RAC clusters Finally when load increases, one can extend the cluster to additional nodes via EM 1 4 3 5 Scale down Scale up RAC cluster Create Production clusters

10 RAC Related Service Offerings by Advanced Customer Services
Configuration Performance Patch Assessments Upgrade Assistance Patching Assistance Onsite Assistance Backup & Recovery Review Assisted Services Remote Monitoring Escalation Management Managed Services Data Migration Advisory Storage Server Configuration Database Machine Assisted and Managed Services Exadata V1 and V2 1 10

11 Assessment URL:
New: Oracle Customer Success Assessment Get more value from your Oracle investment with Customer Services 15min Online Survey on 5 Domains Strategy Process Technology People Governance Personalized Benchmark Study Compare your results to peers Advice on 25 good practice areas Recommended actions to take Oracle services to assist in practice improvements Navigate Oracle’s service catalog Complete portfolio of services mapped to IT lifecycle (ITIL) on Oracle Customer Success Assessment This online assessment is designed to help share Oracle Customer Services best practices across 5 domains: Strategy, Process, People, Technology and Governance to help customers get the maximum value out of their Oracle investments. These best practices have been developed through years of experience managing successful customers by Oracle’s highly skilled Services experts. Based on the customer’s answers, a personalized benchmark study is compiled comparing results against peers, recommended practices and actions, and services for practice improvement. The report provides a high-level summary of which services to take advantage of in Premier Support, as well as other value added services. For a full list of services covering the IT Lifecycle, service descriptions and content, customers are encouraged to visit services pages with new navigation and filter capabilities making it easier to find the right services based on our customer’s IT lifecycle needs. Assessment URL: 11

12 Advanced Customer Services
For More Information Advanced Customer Services or

13 Mastering Oracle Real Application Clusters Performance Tuning At Verizon Wireless
Session Id: S311852 Syamal Bandyopadhyay

14 Mastering Oracle Real Application Clusters Performance Tuning at Verizon Wireless
Agenda: Verizon Wireless Business Requirements for RAC RAC Implementation / Deployment Architecture Methodology To Proactively Detect Performance Issues Techniques / Tips in Resolving Performance Issues Application Performance Score Card: A 360 Degree View Conclusions Q/A

15 Verizon Wireless Business Requirements for RAC
High Availability Scalability Reduced IT cost Application performance meets or beats non-RAC deployment performance

16 RAC Implementation / Deployment Architecture

17 Oracle - RAC (Real Application cluster)
RAC Deployment DB CONFIGURATION V (64 Bit) DB Size 1.5 TB 4 Oracle Instances Data Guard Flashback Application: Business Critical Customer Facing 24 X 7 X 365 30000 Concurrent Users 30000 Middleware - Websphere Oracle - RAC (Real Application cluster) SERVER CONFIGURATION Sun M5000 8 Quad Core CPUs 64 GB RAM Solaris 10 (64 Bit) Platform Components Symantec SFRAC 5.0 Hitachi Storage Shareplex VZWPROD1

18 Disaster Recovery Strategy
Runs on 2 Data Centers 2 databases identical structure VZWPROD1 and VZWPROD2 Oracle’s Data Guard for DR Physical Data Guard with Maximum Availability VZWDR1 and VZWDR2 Data Center 1 Data center 2 Middleware Middleware VZWDR2 VZWPROD2 VZWPROD1 VZWDR1 Oracle – Data Guard (physical)

19 Real Time Data Replication
Bi-directional real time data replication More than 100 Tables Replicate data using Shareplex User Data center 1 Data center 2 VZWPROD2 VZWPROD1 2 – Way Data Replication using Shareplex (Quest Software)

20 Application Load Distribution
Data Guard – Preferred Node 4 Shareplex – Preferred Node 3 Application-1 – Preferred Node 1 & 2 Application-2 – Preferred Node 3 & 4 Application-3 – Preferred Node 3 & 4 Middleware Appl-2, Appl-3, Shareplex Appl-1 Appl-1 Appl-2, Appl-3, Data Guard Node 1 Node 2 Node 3 Node 4 VZWPROD1

21 Deployment Architecture
2 Data Centers 2 Production Databases – 4 Instances 2 Disaster Recovery Databases – 4 Instances Real Time Data Replication using Shareplex Oracle Oracle Data Guard SUN M5000 Servers Symantec SFRAC 5.0 Hitachi Storage Data center 2 Data Center 1 Middleware Middleware 30000 VZWDR2 VZWPROD2 VZWDR1 VZWPROD1 Oracle – Data Guard (physical) Shareplex

22 Methodology To Proactively Detect Performance Issues

23 Performance Challenges For RAC Implementation
Concern regarding meeting current level of application response time Inserts are taking significantly longer time Increased response time for both selects and updates

24 SQL Response Time – Key Components
SQL Response Time Equation: Non RAC: Response time = CPU time + Wait time (IO wait + Queue time) RAC: Response time = CPU time + Wait time (IO wait + Queue time + Cluster Wait Time)

25 Cluster Wait Time Time to access the blocks/data from the cache of the partner instance(s) More the # of blocks to access greater the wait time is Inefficient SQLs make cluster wait time worse Inadequate indexes increase cluster wait time Avoid / Minimize Block transfer among RAC instances Access Paths causing high rate of block transfer: Full Table Scans Index Full Scans Index Fastfull Scans index Skip Scans

26 Methodology to Improve Performance
Collect non-RAC production performance stats for all SQLs Collect RAC test performance stats for all SQLs Collect performance stats from GV$SQL table Compare RAC test performance stats with pre-RAC stats Identify SQLs and database objects having performance issues Analyze the performance data to detect the root cause of unacceptable performance Tune the database objects

27 Table To Collect Performance Data
CREATE TABLE SQL_PERF_DATA ( COLLECTION_TS DATE, data collection timestamp INDICATOR VARCHAR2(12 BYTE), test type e. g . test1 SQLID VARCHAR2(13 BYTE), sql_id of the SQL OWNER VARCHAR2(30 BYTE), parsing schema name of the sql INSTID NUMBER, RAC instance id EXCNT NUMBER, # of execution of the SQL ELAPT NUMBER, elapsed time (ms) of the sql CPUT NUMBER, cpu time (ms) of the sql CWT NUMBER, cluster wait time (ms) of the sql LIO NUMBER, # of buffer gets per execution of the sql PHYIO NUMBER, # of disk reads per execution of the sql ROWCNT NUMBER, # rows in result set per execution SQLFULLTEXT CLOB SQL full text ) STORAGE

28 SQL Script To Collect Performance Data
insert into sql_perf_data1(collection_ts, indicator, sqlid, owner, instid, excnt,elapt, cput, cwt, lio, phyio, rowcnt,sqlfulltext) select sysdate, 'ractst01', sql_id, parsing_schema_name, inst_id, executions, elapsed_time/1000/executions, cpu_time/1000/executions, cluster_wait_time/1000/executions, buffer_gets/executions, disk_reads/executions , rows_processed/executions, sql_fulltext from gv$sql where executions > 0 and (elapsed_time/1000)/(decode(executions,0,1,executions)) > 1 and parsing_schema_name in ( 'APPL1' , 'APPL2')

29 Sample Performance Data Collection
Non-RAC Production: RAC Test: indicator sqlid owner instid excnt elapt cput cwt lio phyio rowcnt sqlfulltext rac-test01 bh9kgy575a8j7 APPL1 1 1,250,058 34.6 1.3 25.9 32.1 2.4 13.2 INSERT into…… 2 927,829 33.0 24.6 29.6 2.3 18.5 2hacpd8v3fbsa APPL3 4 2,290,539 28.4 23.4 26.7 1.0 UPDATE ….. 88xdp6maq6pvs 5,005 19,118.2 517.2 14,186.1 10,414.8 319.3 SELECT ….. Note: Times are in milliseconds per execution

30 Performance Data Comparison SQL Script
Compares the performance stats between any 2 collections of data (e.g. production vs. test; test1 vs. test2, etc.) select a.owner, a.sqlid,a.instid Rinstid, a.excnt Rexcnt, a.elapt Relapt, (a.cput) Rcput, (a.lio) Rlio, (a.phyio) Rphyio, a.cwt Rcwt, (a.rowcnt) Rrowcnt, (a.elapt - b.elapt) "RAC - PROD" , (a.elapt /b.elapt) "RAC over PROD" , b.excnt Pexcnt, (b.elapt) Pelapt, (b.cput) Pcput, (b.lio) Plio, (b.phyio) Pphyio, b.cwt Pcwt, (b.rowcnt) Prowcnt,b.sqlid Psqlid, b.sqlfulltext from sql_perf_data a, sql_perf_data b where a.indicator = 'ractst01' and b.indicator = 'prode' and a.sqlid = b.sqlid order by (a.elapt /b.elapt) desc

31 Performance Data Comparison Result Set
RAC Performance Stats (RAC Elapsed time / Production Elapsed time) (RAC Elapsed time –Production Elapsed time) RAC Performs Better No RAC Performs worse Yes > 1 Non-RAC Performance Stats Note: Times are in milliseconds per execution

32 Cluster Wait Time Data Collection Methodology
V$SQL view contains cluster_wait_time for every sql Calculate cluster_wait_time per execution of the sql Calculate percentage of cluster_wait_time over elapsed_time per execution Focus on SQLs having: High cluster_wait_time per execution High % of cluster_wait_time over elapsed_time SQLs with high execution frequency

33 Cluster Wait Time Data Collection Script
select sql_id sqlid, parsing_schema_name appl, inst_id instid, executions excnt, cluster_wait_time/1000/executions cwt, ((cluster_wait_time/1000/executions)/( elapsed_time/1000/executions)) "CWT over ELAPT" , elapsed_time/1000/executions elapt, cpu_time/1000/executions cput from gv$sql where executions > 0 and (elapsed_time/1000)/(decode(executions,0,1,executions)) > and parsing_schema_name in ( 'APPL1') order by ((cluster_wait_time/1000/executions)/( elapsed_time/1000/executions)) desc

34 Sample Cluster Wait Time Data
Note: Times are in milliseconds per execution

35 SQL To Capture Database Objects Having Full Table Scans
select b.table_name, b.num_rows,a.frequency, b.owner from dba_tables b, (select object_name, count(*) frequency from gv$sql_plan where operation = 'TABLE ACCESS' and options like '%FULL%' and object_owner in ('SCHEMA_NAME1', 'SCHEMA_NAME2') group by object_name) a where a.object_name = b.table_name order by 4, 2 desc Sample Data: TABLE_NAME NUM_ROWS FREQUENCY OWNER ORDER_DETAILS 4 SCHEMA_NAME1 ORDER_HEADER 6

36 SQL to Capture SQL_ID doing Full Table Scan
SQL to capture SQL_ID doing Full Table Scans select object_name, sql_id from gv$sql_plan where operation = 'TABLE ACCESS' and options like '%FULL%' and object_owner in ('SCHEMA_NAME1', 'SCHEMA_NAME2') order by object_name Sample data: OBJECT_NAME SQL_ID ORDER_DETAILS 4a6svn2r2wamc 88sqhdvtp0fsd ORDER_HEADER ggtf49yp69p2t

37 Oracle Automated Workload Repository (AWR)
Analyzed AWR AWR contains significant amount of RAC related stats Each RAC Instance has its own AWR Very helpful performance stats for Global Cache blocks sent and received Interconnect Traffic Volume – for each instance Database Objects incurring Global Cache Buffer Busy Waits Database Objects having Consistent Read (CR) blocks received waits Database Objects with Current Blocks received waits

38 Additional Oracle Dynamic Performance Views
GV$ACTIVE_SESSION_HISTORY GV$SESSION_WAIT These views contain tremendous amount of performance related stats Identified the database objects with frequent GLOBAL CACHE (gc_%) waits

39 Techniques & Tips for Resolving Performance Issues

40 Cluster Wait Time Reduction Techniques
Table / Index Changes: Introduce partition / sub-partition – Hash Partition where feasible Add: freelist / freelist group Increase INITRANS Index Changes: Eliminate if possible Modify inefficient indexes Global Hash partition Make local where feasible Use Multiple Block Size, especially for indexes Reduce full tablescans, index fastfull scans, index full scans, index skip scans Tune SQLs / database objects to reduce the # of logical / physical io

41 Table / Index Change : Example

42 Application Performance Score Card: A 360 Degree View

43 Performance Data Collection : By Applications
Collect performance stats for each Application Compare RAC test results with non-RAC Compare non-RAC to RAC production stats SQL to capture the performance stats: select parsing_schema_name "appluser", sum(executions) "exec cnt", sum(elapsed_time/1000)/sum(executions) "elap", sum(cluster_wait_time/1000)/sum(executions) "cwt", sum(cpu_time/1000)/sum(executions) "cpu", sum(buffer_gets)/sum(executions) "log io", sum(disk_reads)/sum(executions) "phy io", sum(rows_processed)/sum(executions) "row/exec" from gv$sql where executions > 0 and (elapsed_time/1000)/(decode(executions,0,1,executions)) > 0 and parsing_schema_name in ( 'APPL1', 'APPL2', 'APPL3') group by parsing_schema_name order by parsing_schema_name

44 Performance Data Comparison: Final Score Card

45 Met Goals of RAC Implementation at Verizon Wireless
Achieved the goals: High Availability Scalability Reduced IT cost Application performance meets or beats pre-RAC performance Concern regarding meeting current level of application response time… Met the required performance Inserts are taking significantly longer time… Dramatic reduction of the elapsed time for inserts Increased response time for both selects and updates… Improved to meet the goals

46 Conclusions Identify SQLs and Database Objects having High Cluster Wait Time Cluster Wait Time must be reduced Use the Techniques to reduce Cluster Wait Time Partition / Sub-partition tables Partition indexes (preferably Hash) Use Freelist Group Use Freelist Increase initrans Remove unnecessary Indexes Reduce Full Table Scans, Index Full Scans, Index Fastfull Scans, Skip Index scans Use Multiple block size (2K, 4K, 8K, etc.)


Download ppt "Mastering Oracle Real Application Clusters Performance Tuning at Verizon Wireless S311852"

Similar presentations

Ads by Google