Oracle10g RAC Service Architecture Overview of Real Application Cluster Ready Services, Nodeapps, and User Defined Services

Overview Service Architecture Cluster Ready Services (CRS) Nodeapps User defined Services ASM Services Internally Managed Services Monitoring Services

RAC Service Architecture Oracle 10g RAC features a service based architecture This is an improvement over 9i RAC in several ways Increased flexibility Increased manageability Improvements in High Availability Enables 10g Grid Deployment In general, Oracle 10g features increased usage of services to organize and manage Oracle functionality and associated application workloads. This is especially true for Real Application Clusters. With a service based architecture, Oracle RAC becomes more automated and self-managing than ever. In addition, a service based architecture makes it possible to implement Oracle 10g Grids.

RAC Services and High Availability Oracle Services facilitate high availability of databases and related applications If key database resources become unavailable (network, storage, etc.): Instances and Services will be relocated to another node The “failed” node will be rebooted By default, after any server boot-up, Oracle attempts to restart all services on the node 10g RAC Cluster Ready Services make it much easier than previous versions to implement a High Availability architecture. By default, Cluster Ready Services will start or restart automatically on any node when possible, without administrative intervention. If it is not possible to start or continue services on a node, whether because of resource limitations or outright loss of a node, all services will be cleanly relocated to another available node in the cluster.

Cluster Ready Services Manage the RAC Cluster Several Different Services OracleCRSService Oracle CSService OracleEVMService OraFenceService Required for RAC installation Installed in its own CRS_HOME Cluster Ready Services require a separate install before the main Oracle software installation. CRS is installed into a separate home directory from the standard Oracle Home directory. Once CRS is installed and services are active, it is possible to perform a full Real Application Clusters software installation.

CRS Basics Used to manage RAC Only one set of CRS Daemons per system Multiple instances share the same CRS CRS runs as both root and Oracle users CRS must be running before RAC can start It is important to note that multiple databases can be run from a single RAC installation. In the case of multiple RAC databases, multiple unrelated instances will run on a single node. Each database can be managed separately. However, all instances on a single node will share a common set of CRS daemons.

CRS Management Started automatically Can stop and start manually Start the OracleCRSService Stop the OracleCRSService Uses the voting disk and OCR (Oracle Cluster Repository) Requires 3 network addresses Public Private Virtual Public CRS can be started and stopped manually by accessing the OracleCRSService in Services. However, this is usually not necessary, due to the automatic behavior of CRS. CRS utilizes two types of “quorum” files on shared disk. These are the voting disk and the Oracle Cluster Repository. These are usually small files (< 1 GB), and are usually stored on raw disk. CRS requires three separate networks: a public interface on each node, a private interface on each node, and a virtual interface on each node. The public interface and virtual interface may optionally be placed on separate networks, or on the same subnet. However, the private interface must be completely separate from the other networks. It is a Best Practice to place the private interconnect on separate switches from the other networks. The private interconnect implements Cache Fusion over a Gigabit Ethernet network. Large volumes of messages may be sent across the private network. Since the UDP protocol is used, the private interconnect is extremely sensitive to collisions. This is why the private interconnect network must be isolated.

CRS Services OracleCRSService OracleCSService OracleEVMService Cluster Ready Services Daemon OracleCSService Oracle Cluster Synchronization Service Daemon OracleEVMService Event Manager Daemon OraFenceService Process Monitor

Cluster Ready Services Daemon OracleCRSService Runs as Administrator user Automatically restarted Manages Application Resources Starts, stops and fails-over application resources Maintains the OCR (Oracle Cluster Repository) Keeps state information in the OCR The Cluster Ready Services Daemon is the only RAC daemon that is run as root. It is responsible for maintaining basic state information in the Oracle Cluster Repository. The CRSD is responsible for rebooting failed cluster nodes, among other responsinilities.

Oracle Cluster Synchronization Service Daemon OracleCSService Runs as Administrator user Maintains the heartbeat (failure causes system reboot) Provides Node Membership Group Access Basic Cluster Locking Can integrate with 3rd party clustering products or run standalone OracleCSService also works with non-RAC systems The Oracle Cluster Synchronization Service Daemon is responsible for maintaining the basic “heartbeat” of the cluster. Although the private interconnect is polled to determine system health, it is not the ultimate heartbeat mechanism. The “Voting Disk” on shared storage is used to hold current status information for each node. In this case, the heartbeat is transmitted over the Storage Area Network, usually implemented as the Fibre Channel protocol on fiber optic cables. In addition to providing the heartbeat mechanism, OracleCSService is responsible for node membership, group access, and basic cluster locking.

Event Manager Daemon OracleEVMService Runs as Administrator user Restarts on failure Generates Events Starts the racgevt thread to invokes Server Callouts The Event Manager Daemon is responsible for generating messages when significant events occur. In addition to internal use of event messages, event messages may be accessed for programmatic use ny making use of Server Callouts in scripts and programs.

Process Monitor OraFenceService Runs as Administrator user Locked in memory to monitor the cluster Provides I/O fencing OraFenceService periodically monitors cluster status, and can reboot the node if a problem is detected An OraFenceService failure results in Oracle Clusterware restarting the node The OraFenceService performs its check, stops running, and if the wake up is beyond the expected time, then OraFenceService resets the processor and reboots the node.

RACG RACG is a behind-the-scenes process (or thread) that extends clusterware to support Oracle-specific requirements and complex resources. Runs server callout scripts when FAN events occur. Runs as processes (or threads), not as a service (racgmain.exe, racgimon.exe)

Cluster Ready Services Management Log Files OracleCRSService %ORA_CRS_HOME%\log\hostname\crsd Oracle Cluster Registry (OCR) %ORA_CRS_HOME%log\hostname\ocr OracleEVMService %ORA_CRS_HOME%\log\hostname\evmd OracleCSService %ORA_CRS_HOME%log\hostname\cssd RACG %ORA_CRS_HOME%log\hostname\racg

Nodeapp Services Nodeapps are a standard set of Oracle application services that are automatically launched for RAC Virtual IP (VIP) Oracle Net Listener Global Services Daemon (GSD) Oracle Notification Service (ONS) Nodeapp services run on each node Can be relocated to other nodes through the virtual IP Nodeapps are a set of Oracle applications that run automatically on each node. These include applications that were also significant with 9i RAC, including the Oracle Net Listener and the Global Services Daemon. In addition, the Oracle Notification Service and the Virtual IP service are also run as nodeapps. The VIP service is especially important. It allows services to be rapidly relocated to other nodes.

VIP (Virtual IP) Creates a virtual IP address used by the Listener The virtual IP address fails over between nodes Multiple virtual IP addresses can exist on the same system (during failover) Independent of the Oracle Instance Potential Problem if more than one database per node The VIP service is used to offer a consistent IP address for Oracle Net communications and applications connecting to the database, even if all services are relocated to another node. In case of a service relocation, multiple virtual IPs may be hosted on a single node. When multiple databases are hosted on a single RAC system, relocating the VIP service to move the instance and service resources for one database will also cause all other RAC instances and services hosted on the node to also be relocated.

Global Services Daemon (GSD) The daemon which executes SRVCTL commands GSD receives requests from SRVCTL to execute administrative tasks, such as startup or shutdown The command is executed locally on each node, and the results are sent back to SRVCTL. The daemon is installed on the nodes by default. It is important that you do not kill this process and it should not be deleted. The Global Services Daemon is used to execute administrative tasks for RAC. An Oracle Database Administrator may use srvctl commands to interface with the GSD. The GSD is a required component of RAC Nodeapps, and is installed by default on each node. As of Oracle 10g, it is no longer necessary to use the gsdctl utility to start and stop the Global Services Daemon. It is preferred to use srvctl to start and stop all nodeapps.

Listener Server-side component of Oracle Net Listens for incoming client connection requests Manages the traffic to the server; when a client requests a network session with a server, the listener actually receives the request and brokers the client request If the client's information matches the listener's information, then the listener grants a connection to the server. The listener is a process that resides on the server whose responsibility is to listen for incoming client connection requests and manage the traffic to the server. The listener brokers the client request, handing off the request to the server. Every time a client (or server acting as a client) requests a network session with a server, a listener receives the actual request. If the client's information matches the listener's information, then the listener grants a connection to the server. As of Oracle 10g, it is no longer necessary to use the lsnrctl utility to start and stop the listener. It is preferred to use srvctl to start and stop all nodeapps.

Oracle Notification Service (ONS) The Oracle Notification Service is installed automatically on each RAC node as a Node Application ONS starts automatically with each boot ONS uses a simple push/subscribe method to publish event messages to all RAC nodes with active ONS daemons The Oracle Notification service extends the functionality of the Event Manager Daemon to push event messages to multiple targets. This includes all other RAC nodes running ONS. In addition, mid-tier nodes may also run ONS, in order to receive and respond to event messages. This enables the Fast Application Notification system, which can be utilized for high availability and load balancing of mid-tier applications.

ONS and Fast Application Notification ONS can be configured to run on nodes hosting client or mid-tier applications ONS is the key component of Fast Application Notification (FAN) Can be utilized to extend RAC high availability and load balancing to mid-tier applications Independent of True Application Failover Less reliance on network configuration

User Defined Services User defined, named services may be created to manage database resources that are associated with application workloads One or more database instances may be mapped to a single service A database instance may be assigned to one or more services The Automated Workload Repository may be used to monitor Service metrics

User Defined Services and Failover Services can be defined with preferred and alternate instances A service may be assigned to start on preferred instances The same service may have alternate instances assigned for failover If multiple services are assigned for the same database, the preferred and alternate instance assignments may be different for each service User defined services can be defined to manage application workloads. For example, a service may be assigned for an ERP application and a separate service may be assigned for a Reporting application, both working on the same four node cluster. Two nodes could be assigned as the primary nodes for the ERP application, with the other two nodes as the primary nodes for the Reporting application. The other two nodes for each application may be defined as Alternate nodes for that application workload, to be used in case of failover.

Automatic Storage Management Services Automatic Storage Management (ASM) is a storage option for creating and managing databases ASM operates like a Logical Volume Manager between the physical storage and the database. A small, automatically managed Oracle database instance is created on each node (if ASM is chosen as a storage option) ASM instances start automatically as Oracle services Automatic Storage management is one option for configuring Oracle Storage. ASM acts like a Logical Volume Manager (LVM) that sits between the physical storage and the database. ASM automates all file creation, file naming, and file placement operations. ASM may be used to perform data striping and disk mirroring, if desired. ASM is implemented as a small Oracle instance on each RAC node. The ASM instances start automatically as servcies, and can be managed with the same tools as all other services (i.e. srvctl).

Internally Managed Services When the Global Services Daemon is started as a part of the Node Applications, it in turn launches key internally managed services The Global Cache Service manages Cache Fusion and in-memory data buffers The Global Enqueue Service manages inter-instance locking and RAC recovery GCS and GES show up as OS processes or threads, but GSD is the only service that can be externally controlled GCS and GES together manage a set of “virtual” tables in memory, called the Global Resouce Directory

Global Cache Service (GCS) The controlling process that implements Cache Fusion. Manages the status and transfer of data blocks across the buffer caches of all instances. Tightly integrated with the buffer cache manager to enable fast lookup of resource information in the Global Resource Directory. Maintains the block mode for blocks in the global role. Employs various background processes (or threads) such as the Global Cache Service Processes (LMSn) and Global Enqueue Service Daemon (LMD). This directory is distributed across all instances and maintains the status information about resources including any data blocks that require global coordination.

Global Enqueue Service Monitor (LMON) Background process that monitors the entire cluster to manage global resources. Manages instance and process expirations and recovery for GCS. Handles the part of recovery associated with global resources Note: LMON-provided services are also known as cluster group services (CGS). Note: In a cluster database, local locks are called local enqueues.

Global Resource Directory The data structures associated with global resources. It is distributed across all instances in a cluster. Global Cache Service and Global Enqueue Service maintain the Global Resource Directory to record information about resources and enqueues held globally. The Global Resource Directory resides in memory and is distributed throughout the cluster to all nodes. In this distributed architecture, each node participates in managing global resources and manages a portion of the Global Resource Directory. Note: Enqueues are shared memory structures that serialize access to database resources. Enqueues are local to one instance if Real Application Clusters is not enabled. When you enable RAC, enqueues can be global to a database. The Global Resource Directory is like a “vitual” set of tables that reside only in memory.

Monitoring RAC Services %ORA_CRS_HOME%\bin\crs_stat NAME=ora.rac1.gsd TYPE=application TARGET=ONLINE STATE=ONLINE NAME=ora.rac1.oem NAME=ora.rac1.ons Example crs_stat output: NAME=ora.rac1.gsd TYPE=application TARGET=ONLINE STATE=ONLINE NAME=ora.rac1.oem NAME=ora.rac1.ons NAME=ora.rac1.vip NAME=ora.rac2.gsd NAME=ora.rac2.oem NAME=ora.rac2.ons NAME=ora.rac2.vip

Monitoring RAC Services Creating a tabular report: %ORA_CRS_HOME%\bin\crs_stat -t Name Type Target State Host -------------------------------------------------------------------------------------------------- ora.rac1.gsd application ONLINE ONLINE rac1 ora.rac1.oem application ONLINE ONLINE rac1 ora.rac1.ons application ONLINE ONLINE rac1 ora.rac1.vip application ONLINE ONLINE rac1 ora.rac2.gsd application ONLINE ONLINE rac2 ora.rac2.oem application ONLINE ONLINE rac2 ora.rac2.ons application ONLINE ONLINE rac2 ora.rac2.vip application ONLINE ONLINE rac2 The crs_stat –t option provides output in a tabular format.

Review What advantages does a service based architecture offer? What four services comprise Cluster Ready Services? Nodeapps consists of which four applications? True or False: a database instance may be assigned to multiple services

Summary Service Architecture Cluster Ready Services (CRS) Nodeapps OracleCRSService OracleCSService OracleEVMService OraFenceService Nodeapps VIP Listener GSD ONS User defined Services ASM Services Internally managed services Global Cache Service Global Enqueue Service Global Resource Directory Monitoring Services