I Copyright © 2007, Oracle. All rights reserved. Introduction

Copyright © 2007, Oracle. All rights reserved. I - 2 Overview This course is designed for anyone interested in implementing a Real Application Clusters (RAC) database. Although coverage is general, most of the examples and labs in this course are Linux based. Knowledge of and experience with Oracle Database 10g architecture are assumed. Lecture material is supplemented with hands-on practices.

Copyright © 2007, Oracle. All rights reserved. I - 3 Course Objectives In this course, you: Learn the principal concepts of RAC Install the RAC components Administer database instances in a RAC and ASM environment Manage services Back up and recover RAC databases Monitor and tune performance of a RAC database Administer Oracle Clusterware

Copyright © 2007, Oracle. All rights reserved. I - 4 Typical Schedule TopicsLessonsDay Introduction and installation I, 1, 2, 31 RAC administration and tuning4, 5, 62 7, 83 Advanced topics9, , 125 Workshop: Cloning5

Copyright © 2007, Oracle. All rights reserved. I - 5 A History of Innovation Nonblocking queries OPS RAC Automatic Storage Management Automatic Workload Management Low-cost commodity clusters Resource manager Data Guard Enterprise Grids Oracle Clusterware Grid Control

Copyright © 2007, Oracle. All rights reserved. I - 6 What Is a Cluster? Interconnected nodes act as a single server. Cluster software hides the structure. Disks are available for read and write by all nodes. Operating system is the same on each machine. Node Disks Private Interconnect Clusterware on each node Public network

Copyright © 2007, Oracle. All rights reserved. I - 7 Oracle Real Application Clusters Multiple instances accessing the same database One instance per node Physical or logical access to each database file Software-controlled data access Instances spread across nodes Database files Interconnect Shared cache

Copyright © 2007, Oracle. All rights reserved. I - 8 Benefits of Using RAC High availability: Surviving node and instance failures Scalability: Adding more nodes as you need them in the future Pay as you grow: Paying for only what you need today Key grid computing features: –Growth and shrinkage on demand –Single-button addition of servers –Automatic workload management for services

Copyright © 2007, Oracle. All rights reserved. I - 9 Clusters and Scalability SMP modelRAC model Cache CPU Cache CPU Memory Cache coherency SGA BGP SGA BGP Shared storage Cache fusion BGP (background process)

Copyright © 2007, Oracle. All rights reserved. I - 10 Levels of Scalability Hardware: Disk input/output (I/O) Internode communication: High bandwidth and low latency Operating system: Number of CPUs Database management system: Synchronization Application: Design

Copyright © 2007, Oracle. All rights reserved. I - 11 Scaleup and Speedup Original system 100% of task Cluster system scaleup Up to 200% of task Up to 300% of task Time Hardware Time Cluster system speedup Time/2 Hardware Time Hardware 100% of task

Copyright © 2007, Oracle. All rights reserved. I - 12 Speedup/Scaleup and Workloads WorkloadSpeedupScaleup OLTP and InternetNoYes DSS with parallel queryYes Batch (mixed)PossibleYes

Copyright © 2007, Oracle. All rights reserved. I - 13 I/O Throughput Balanced: Example FC-switch HBA1HBA2 HBA1HBA2 HBA1HBA2 HBA1HBA2 Disk array 1 Disk array 2 Disk array 3 Disk array 4 Disk array 5 Disk array 6 Disk array 7 Disk array 8 Each machine has 2 CPUs: 2  200 MB/s  4 = 1600 MB/s Each machine has 2 HBAs: 8  200 MB/s = 1600 MB/s Each switch needs to support 800 MB/s to guarantee a total system throughput of 1600 MB/s. Each disk array has one 2-Gbit controller: 8  200 MB/s = 1600 MB/s

Copyright © 2007, Oracle. All rights reserved. I - 14 Performance of Typical Components Throughput Performance ComponentTheory (Bit/s)Maximal Byte/s HBA½ Gbit/s100/200 Mbytes/s 16 Port Switch8  2 Gbit/s1600 Mbytes/s Fibre Channel2 Gbit/s200 Mbytes/s Disk Controller 2 Gbit/s200 Mbytes/s GigE NIC 1 Gbit/s80 Mbytes/s Infiniband10 Gbit/s890 Mbytes/s CPU200–250 MB/s

Copyright © 2007, Oracle. All rights reserved. I - 15 Complete Integrated Clusterware Management APIs Hardware/OS kernel Connectivity Membership Messaging and locking Volume Manager file system Applications Cluster control Event Services System Management Applications/RAC Services framework Cluster control/Recovery APIs Automatic Storage Management Messaging and locking Membership Connectivity Event Services 9i RAC10g Oracle Clusterware

Copyright © 2007, Oracle. All rights reserved. I - 16 Necessity of Global Resources 1008 SGA1SGA SGA1SGA SGA1SGA SGA1SGA Lost updates! 12 34

Copyright © 2007, Oracle. All rights reserved. I - 17 Global Resources Coordination LMON LMD 0 LMS x DIAG … LCK 0 CacheGRD Master GES GCS LMON LMD 0 LMS x DIAG … Cache LCK 0 GRD Master GES GCS Node 1 Instance 1 Node n Instance n Cluster Interconnect Global resources Global Enqueue Services (GES) Global Cache Services (GCS) Global Resource Directory (GRD)

Copyright © 2007, Oracle. All rights reserved. I - 18 Global Cache Coordination: Example Node 1 Instance 1 Node 2 Instance 2 … Cache Cluster GCS 4 No disk I/O LMON LMD 0 LMS x … LCK 0 Cache 1009 DIAG LMON LMD 0 LMS x LCK 0 DIAG Block mastered by instance 1 Which instance masters the block? Instance 2 has the current version of the block.

Copyright © 2007, Oracle. All rights reserved. I - 19 Write to Disk Coordination: Example Node 1 Instance 1 Node 2 Instance 2 Cache Cluster GCS 45 Only one disk I/O LMON LMD 0 LMS x LCK 0 DIAG LMON LMD 0 LMS x LCK 0 DIAG … … Cache 1009 Need to make room in my cache. Who has the current version of that block? Instance 2 owns it. Instance 2, flush the block to disk. Block flushed, make room

Copyright © 2007, Oracle. All rights reserved. I - 20 Dynamic Reconfiguration Node 1 Instance 1 masters R1 granted R21, 3 1, 2, 3 Node 2 Instance 2 masters R3 granted R41, 2 2, 3 Node 3 Instance 3 masters R5 granted R61, 2, 3 2 Node 1 Instance 1 masters R1 granted R21, 3 Node 2 Instance 2 masters R3 granted R41, 2 2, 3 Node 3 Instance 3 masters R5 granted R61, 3 R33R41 Reconfiguration remastering

Copyright © 2007, Oracle. All rights reserved. I - 21 Object Affinity and Dynamic Remastering Node 1 Node 2 Instance 2 Instance 1 Object Read from disk GCS message to master Messages are sent to remote node when reading into cache. Node 1 Node 2 Instance 2 Instance 1 No messages are sent to remote node when reading into cache. Before dynamic remastering After dynamic remastering

Copyright © 2007, Oracle. All rights reserved. I - 22 Global Dynamic Performance Views Retrieve information about all started instances Have one global view for each local view Use one parallel slave on each instance Node 1 Instance 1 Node n Instance n Cluster V$INSTANCE GV$INSTANCE

Copyright © 2007, Oracle. All rights reserved. I - 23 Additional Memory Requirement for RAC Heuristics for scalability cases: –15% more shared pool –10% more buffer cache Smaller buffer cache per instance in the case of single-instance workload distributed across multiple instances Current values: SELECT resource_name, current_utilization,max_utilization FROM v$resource_limit WHERE resource_name like 'g%s_%'; SELECT * FROM v$sgastat WHERE name like 'g_s%' or name like 'KCL%';

Copyright © 2007, Oracle. All rights reserved. I - 24 Efficient Internode Row-Level Locking Node 1 Instance 1 Node 2 Instance 2 UPDATE Node 1 Instance 1 Node 2 Instance 2 UPDATE Node 1 Instance 1 Node 2 Instance 2 UPDATE Node1 Instance 1 Node 2 Instance 2 COMMIT No block-level lock

Copyright © 2007, Oracle. All rights reserved. I - 25 Parallel Execution with RAC Execution slaves have node affinity with the execution coordinator but will expand if needed. Execution coordinator Parallel execution server Shared disks Node 4 Node 1Node 2Node 3

Copyright © 2007, Oracle. All rights reserved. I - 26 RAC Software Principles Node 1 Instance 1 Oracle Clusterware CRSD & RACGIMON OCSSD & OPROCD EVMD LMON LMD 0 LMS x DIAG … Cache Node n Instance n Cluster Applications VIP, ONS, EMD, Listener Oracle Clusterware CRSD & RACGIMON OCSSD & OPROCD EVMD ASM, DB, Services, OCR Applications VIP, ONS, EMD, Listener ASM, DB, Services, OCR Global management: SRVCTL, DBCA, EM Cluster interface LCK 0 LMON LMD 0 LMS x DIAG … Cache LCK 0 Global resources

Copyright © 2007, Oracle. All rights reserved. I - 27 RAC Software Storage Principles Node 1 Instance 1 CRS_HOME Local storage ORACLE_HOME Node n Instance n Local storage … Voting files OCR files Node 1 Instance 1 Local storage Node n Instance n Local storage … Shared storage Voting files OCR files ORACLE_HOME Permits rolling patch upgrades Software is not a single point of failure. Shared storage CRS_HOME ASM_HOME CRS_HOME ORACLE_HOME ASM_HOME

Copyright © 2007, Oracle. All rights reserved. I - 28 RAC Database Storage Principles Node 1 Instance 1 … Node n Instance n Online redo log files for instance 1 Online redo log files for instance n Undo tablespace files for instance 1 Undo tablespace files for instance n Archived log files Local storage Shared storage Data files Flash recovery area files Change tracking file SPFILE Control files Temp files TDE Wallet Archived log files

Copyright © 2007, Oracle. All rights reserved. I - 29 RAC and Shared Storage Technologies Storage is a critical component of grids: –Sharing storage is fundamental. –New technology trends Supported shared storage for Oracle grids: –Network Attached Storage –Storage Area Network Supported file storage for Oracle grids: –Raw volumes –Cluster file system –ASM

Copyright © 2007, Oracle. All rights reserved. I - 30

Copyright © 2007, Oracle. All rights reserved. I - 31 Oracle Cluster File System Is a shared disk cluster file system for Linux and Windows Improves management of data for RAC by eliminating the need to manage raw devices Provides open solution on the operating system side Can be downloaded from OTN: – (Linux) – oracle10g/index.html (Windows)

Copyright © 2007, Oracle. All rights reserved. I - 32 Automatic Storage Management Provides the first portable and high-performance database file system Manages Oracle database files Contains data spread across disks to balance load Provides integrated mirroring across disks Solves many storage management challenges ASM File system Volume manager Operating system Application Database

Copyright © 2007, Oracle. All rights reserved. I - 33 CFS or Raw? Using CFS: –Simpler management –Use of OMF with RAC –Single Oracle software installation –Autoextend Using raw: –Performance –Use when CFS not available –Cannot be used for archivelog files –ASM eases work

Copyright © 2007, Oracle. All rights reserved. I - 34 Typical Cluster Stack with RAC High-speed Interconnect: Gigabit Ethernet UDP Proprietary Database shared storage Interconnect Servers Oracle Clusterware RAC Linux, UNIX, Windows ASM RAC Linux Windows RAW RAC Linux Windows OCFS RAC AIX, HP-UX, Solaris ASM OS C/W RAW OS CVM CFS

Copyright © 2007, Oracle. All rights reserved. I - 35 RAC Certification Matrix 1.Connect and log in to 2.Click the Certify tab on the menu frame. 3.Click the “View Certifications by Product” link. 4.Select Real Application Clusters. Then click Submit. 5.Select the correct platform and click Submit.

Copyright © 2007, Oracle. All rights reserved. I - 36 RAC and Services Up/down/LBA events notification engine Listeners RAC instances Application server ERPCRM ERP CRM Stop or start service connections. Modify service to instance mapping. Service connections Backup Priority Alerts Tuning Connection load balancingService availability aware Oracle Clusterware Run-time connection load balancing Service location transparency Restart failed components

Copyright © 2007, Oracle. All rights reserved. I - 37 Available Demonstrations RAC scalability and transaction throughput RAC speedup and parallel queries Use TAF with SELECT statements

Copyright © 2007, Oracle. All rights reserved. I - 38