1. Oracle Cache Fusion – In Operation

2. Agenda Cache Fusion What is it? Cache Coherency Vs. Cache Fusion
Key Components and terminology
Cache Fusion in operation
Lock Mastering & Resource Affinity
Type of Contentions
Cache Fusion – I
Cache Fusion – II
Examples
Instance Crash Recovery in RAC
Key Components in a Instance crash
I Pass recovery
II Pass recovery

3. Cache Fusion – What is it?

4. What is it?
Oracle introduced the framework of sharing data using private interconnects between the nodes, which was used only for messaging purposes in previous versions. This protocol is Cache Fusion. Data blocks are shipped throughout the network similar to messages, reducing the most expensive component of data transfer, disk I/O, to data sharing.
According to the manual:
Process that implement Cache Fusion. It maintains the block mode for blocks in the global role. It is responsible for block transfers between instances. The Global Cache Service employs various background processes such as the Global Cache Service Processes (LMSn) and Global Enqueue Service Daemon (LMD).
A diskless cache coherency mechanism in Oracle Real Application Clusters that provides copies of blocks directly from a holding instance's memory cache to a requesting instance's memory cache.

5. Cache Coherency According to Manual
The synchronization of data in multiple caches so that reading a memory location through any cache will return the most recent data written to that location through any other cache. Sometimes called cache consistency.
Can We say its something to maintain the resource (block) status, If so, the following two together provides the same for us.
GCS (Global Cache Services)
GES (Global Enqueue Services)
In the name of
Global Resource Directory

6. Now both together ………
The GCS manages all types of data blocks. Cache coherency is maintained through the GCS by requiring that instances acquire a resource (lock or enqueue on a block) cluster-wide before modifying or reading a database block. The GCS is used to synchronize global cache access, allowing only one instance to modify a block at any single point in time. The GCS, through the RAC wide Global Services Directory, ensures that the status of data blocks cached in any mode in the cluster is globally visible and maintained.
Oracle’s RAC has multi-versioning architecture. This multi-versioning architecture distinguishes between current data blocks and one or more consistent read (CR) versions of a block. A current block contains changes for all committed and yet-to-be-committed transactions. A consistent read (CR) version of a block represents a consistent snapshot of the data at a previous point in time. A data block can reside in many buffer caches under the auspices of shared resources.
In Oracle9i RAC, applying rollback segment information to current blocks produces consistent read versions of a block. Both the current and consistent read blocks are managed by the GCS.
To transfer data blocks among database caches, buffers are shipped by means of the high speed IPC interconnect. Disk writes are only required for cache replacement. A past image (PI) of a block is kept in memory before the block is sent if it is a dirty (modified) block. In the event of failure, Oracle reconstructs the current version of the block by reading the PI blocks.

7. Background Process and their roles
LMSx – Lock Monitor Services (GCS)
Primarily responsible for shipping the blocks across buffers
Provides/creates a CR image whenever there is cross instance call for a dirtyblcok
LMS must also check constantly with the LMD background process (or our GES process) to get the lock requests placed by the LMD process.
Parameter: GCS_SERVER_PROCESS upto 36 as of 10.2, Min. cpu_count/2
LMON – Lock Monitor Process (GES)
LMON Processes manages the global locks & resources.
Reconfiguration of locks & resources when an instance joins or leaves the cluster are handled by LMON ( During reconfiguration LMON generate the trace files)
LMON also provides cluster group services.
LMD – Lock Manager Daemon
LMD process performs global lock deadlock detection local and remote . (GES)
Also monitors for lock conversion timeouts.
Basically maintains the lock queues, traverse through the GES structures
LCK – Lock Process
Manages instance resource requests & cross instance calls for shared resources.
During instance recovery,it builds a list of invalid lock elements and validates lock elements.
DIAG – Diagnostic Daemon
Oracle 10g - this one new background processes ( New enhanced diagnosability framework). Regularly monitors the health of the instance. Also checks instance hangs & deadlocks.

8. History of Cache Fusion
Oracle Release
Feature
Description
Prior to 8.1.5
OPS
OPS used disk-based pings
8.1.5
Cache Fusion I or Consistent Read Server
Consistent read version of the block is transferred over the interconnect 9i
Cache Fusion II (write/write cache fusion)
Current version of the block is transferred over the interconnect
10g R1
Oracle Cluster Ready Services (CRS)
CRS eliminates the need for third-party clusterware, though it can be used
10g R2
Oracle CRS for High Availability
CRS provides high availability for non-Oracle applications 8

9. Key Components in Cache Fusion
Ping
The transfer of a data block from one instance’s buffer cache to another instance’s buffer cache is known as a ping. Whenever an instance needs a block, it sends a request to the lock master to obtain a lock in the desired mode. If another lock resides on the same block, the master will ask the current holder to downgrade/release the current lock., this process is known as a blocking asynchronous trap (BAST). When an instance receives a BAST it downgrades the lock as soon as possible. However, before downgrading the lock, it might have to write the corresponding block to disk. This operation sequence is known as disk ping or a hard ping.
CR Fabrication
When ever there is Consistent read request from any other instance, the holding instance (LMS) has to create a Consistent read image by applying the undo information to the Current Block. Since CR fabrication is I/O expensive which requires a undo into the buffer and apply the undo image etc.
Past Image (PI) Blocks
PI blocks are copies of blocks in the local buffer cache. Whenever an instance has to send a block it has recently modified to another instance, it preserves a copy of that block, marking it as PI. An instance is obliged to keep Pls until that block is written to the disk by the current owner of the block. Pls are discarded after the latest version of the block is written to disk. When a block is written to disk and is known to have global role, indicating the presence of Pls in other instances’ buffer caches, Global Cache Services (GCS) informs the instance holding the Pls to discard the Pls. With Cache Fusion, a block is written to disk to satisfy checkpoint requests and so on, not to transfer the block from one instance to another via disk.
Lock Mastering
The memory structure where GCS keeps information about a data block (and other sharable resources) usage is known as the lock resource. The responsibility of tracking locks is distributed among all the instances and the required memory also comes from the participating instances’ System Global Area (SGA). Due to this distributed ownership of the resources, a master node exists for each lock resource. The master node maintains complete information about current users and requestors for the lock resource. The master node also contains information about the Pls of the block.
Cross Instance communication will be happen with the help of messages. There are two types of Messages…
BAST:- Blocking Asynchronous Trap Message, which is to acquire a exclusive lock or upgrade of lock which published immediately to master site.
AAST: Acquisition Asynchronous Trap Message, which is acknowledge the master instance, of acquired/downgraded , which is put in queue.

10. Resource Affinity and Dynamic remastering
Each block is mastered in any one of the instance at any given point of time
Resource Master can be changed based on frequency of the block that is requested by other instances
For a period of 10 Mins if an instance request 50 times for a particular resource the requested instance become the master. This is called resource affinity
- Block Mastering
In Oracle 9.2
documentation describes dynamic remastering
not implemented in code
In Oracle 10.1
work at data file level
very high threshold so difficult to test
does occur on some customer sites
may cause LMON process to crash in
bug patch available
fixed in /
In Oracle 10.2
works at object level
thresholds are relatively low.
Object re mastering is recorded in V$GCSPFMASTER_INFO

11. Cache Fusion- Possible Types of Contention
Contention of a resource occurs when two or more instances want the same resource. If a resource such as a data block is being used by an instance and is needed by another instance at the same time, a contention occurs. There are three types of contention for data blocks:
Read/Read contention   Read/read contention is never a problem because of the shared disk system. A block read by one instance can be read by other instances without the intervention of GCS.
Write/Read contention   Write/read contention was addressed in Oracle 8i by the consistent read server. The holding instance constructs the CR block and ships the requesting instance using interconnects.
Write/Write contention   Write/write contention is addressed by the Cache Fusion technology. Since Oracle 9i, cluster interconnect is used in some cases to ship data blocks among the instances that need to modify the same data block simultaneously.

12. Prior to Cache Fusion (before 8.1.5)
Write/read contention before Cache Fusion
1) An instance sends a message to the lock manager requesting a shared lock on the block.
2) Following are the possibilities in the global cache:
If there is no current user for the block, the lock manager grants the shared lock to the requesting instance.
If another instance has an exclusive lock on that block, the lock manager asks the owning instance to downgrade the lock.
3) Based on the result, either of the following can happen:
If the lock is granted, the requesting instance reads the block from the disk.
The owning instance writes the block to disk and disowns the lock on that resource.
4) The owning instance also informs the lock manager and the requesting instance that it has released the lock.
5) The requesting instance now has the lock granted. The lock manager updates the IDLM with the new holders of that resource.
6) The requesting instance reads the block from disk. If it is looking for the previous version of the block, it can proceed with the regular CR block fabrication in its own buffer cache.
Write/read contention before Cache Fusion

13. Cache Fusion – I aka Consistent Read Server
1) An instance sends a message to the lock manager requesting a shared lock on the block.
2) Following are the possibilities in the global cache:
If there is no current user for the block, the lock manager grants the shared lock to the requesting instance.
If another instance has an exclusive lock on that block, the lock manager asks the owning instance to build a CR copy and ship it to the requesting instance.
3) Based on the result, either of the following can happen:
If the lock is granted, the requesting instance reads the block from disk.
The owning instance creates a CR version of the buffer in its own buffer cache and ships it to the requesting instance over the interconnect.
4) The owning instance also informs the lock manager and the requesting instance that it has shipped the block.
5) The requesting instance now has the lock granted. The lock manager updates the IDLM with the new holders of that resource.
While making a CR copy needed by another instance, the holding instance may refuse to do so if
It doesn’t find any of the needed blocks in its buffer cache. It will not perform a disk read to make a CR copy for another instance.
It is repeatedly asked to send a CR copy of the same block. After sending CR copies of the same block four times, it will voluntarily relinquish the lock, write the block to disk, and let other instances get the block from disk. The number of copies it will serve before it does so is governed by the hidden parameter _fairness_threshold.
While downgrading the lock and writing the block to disk, the holding instance will send a copy of the block it has constructed so far, and the requesting instance will continue
Write/Read contention - CR Block Transfer in Cache Fusion
Oracle Introduced a background process called BSP (Block Server process) makes the CR fabrication at the holder’s cache and ships the CR version of the block across the interconnect

14. Still need to address Write/Write Contention
1) An instance sends a message to the lock manager requesting an exclusive lock on the block.
2) Following are the possibilities in the global cache:
a) If there is no current user for the block, the lock manager grants the exclusive lock to the requesting instance.
b) If another instance has an exclusive lock on that block, the lock manager asks the owning instance to release the lock.
3) Based on the result, either of the following can happen:
a) If the lock is granted, the requesting instance reads the block from the disk.
b) The owning instance writes the block to disk and disowns the lock on that resource.
4) The owning instance also informs the lock manager and the requesting instance that it has released the lock.
5) The requesting instance now has the exclusive lock. It can read the block from disk and proceed with the intended changes to the block.
6) The lock manager updates the resource directory with the new information about the holder.
Often, the requesting instance will find an active transaction in the active transaction list of the block, and it might need the corresponding undo block header and undo data block to clean out the block. Thus, multiple disk I/Os might be needed before the requesting instance can modify the block.
Write / Write Contention before Cache Fusion – II (before 9i)

15. So now – Cache Fusion – II or Write/Write Cache Fusion
1) The instance sends a message to the lock manager to request an exclusive lock on the block.
2) Following are the possibilities in the global cache based on whether any other user is holding the lock:
a) If there is no current user for the block, the lock manager grants the exclusive lock to the requesting instance.
b) If another instance has an exclusive lock on that block, the lock manager asks the owning instance to release the lock.
3) Based on the result, either of the following can happen:
4) If the lock is granted, the requesting instance reads the block from the disk.
5) The owning instance sends the current block to the requesting instance via the interconnect. To guarantee recovery in the event of instance death, the owning instance writes all the redo records generated for the block to the online redolog file. It will keep a past image of the block and inform the master instance that it has sent the current block to the requesting instance.
6) The lock manager updates the resource directory with the current holder of the block.
Cache Fusion current block transfer (from 9i r2 )

16. Buffer States In Cache Fusion
Mode/Role
Local
Global
Null: N NL NG
Shared: S SL SG
Exclusive: X XL XG
SL   When an instance has a resource in SL form, it can serve a copy of the block to other instances and it can read the block from disk. Since the block is not modified, there is no need to write to disk.
XL   When an instance has a resource in XL form, it has sole ownership and interest in that resource. It also has the exclusive right to modify the block. All changes to the blocks are in its local buffer cache, and it can write the block to disk. If another instance wants the block, it will contact the instance via GCS.
NL   A NL form is used to protect consistent read blocks. If a block is held in SL mode and another instance wants it in X mode, the current instance will send the block to the requesting instance and downgrade its role to NL.
SG   In SG form, a block is present in one or more instances. An instance can read the block from disk and serve it to other instances.
XG   In XG form, a block can have one or more Pls, indicating multiple copies of the block in several instances’ buffer caches. The instance with the XG role has the latest copy of the block and is the most likely candidate to write the block to disk. GCS can ask the instance with the XG role to write the block to disk or to serve it to another instance.
NG   After discarding Pls when instructed by GCS, the block is kept in the buffer cache with NG role. This serves only as the CR copy of the block.

17. Example 1: Reading a Block from Disk
Instance C requested for Shared lock to Master D with SL0 Assumes no one got access. An Master checks in GRD reverts to C get the same from Disk with an SCn and acquired a Shared Lock (SL0)

18. Example 2: Reading a Block from the Cache

19. Example 3: Getting a (Cached) Clean Block for Update
1) Instance A sends an exclusive lock request, Lock Req (X, A), to master instance D.
2) The lock master at D knows that the block may be available at instance B in SCUR mode and C in CR mode. It also sends a ping message to the shared lock holders. The most recent access was at instance B, and instance D sends a BAST message to instance B.
3) Instance B sends the block to instance A over the interconnect and closes its shared (SCUR) lock. The block may still be in its buffer to be used as CR; however, it releases any lock on it.
4) Instance A now has the block in exclusive mode and sends an assume message to instance D. Instance A holds the lock in XL0 mode.
5) Instance A modifies the block in its buffer cache. The changes are not committed and the block has not yet been written to disk; therefore, the SCN on disk remains at

20. Example 4: Getting a (Cached) Modified Block for Update and Commit
1) Instance C sends an exclusive lock request, Lock Req (X, C), to master instance D.
2) The lock master at D knows that instance A holds an exclusive lock on the block and hence sends a ping message to instance A.
3) Instance A sends the dirty block to instance C over the interconnect. It downgrades the lock on the buffer from XCUR to NULL. It keeps a PI version of the block and disowns any lock on that buffer. Before shipping the block, instance A has to create a PI image and flush any pending redo for the block changed. The block mode on instance A is now NG1.
4) Instance C sends a message to instance D indicating it now has the block in exclusive mode. The block role G indicates that the block is in global mode and if it needs to write the block to disk it must coordinate it with the other instances that have past images of that block. Instance C modifies the block and issues a commit. This takes the SCN to
Note:- Row level locks and block locksoperate independently in the RACenvironment. In thisexample, we update the same block in twodifferentinstances, but not the same row. Refer tofor adetailed discussion of Oracle’s implementation of row levellocking.Row locks (known as enqueue type TX) and GCSbuffer locks (knownas enqueue type BL) can run inparallel for a transaction.

21. Example 5: Commit the Previously Modified Block and Select the Data
Now instance A issues a commit to release the row level locks held by the transaction and flush the redo information to the redolog files. Global locks on the resources remain the same.
1) Instance A wants to commit the changes. Commit operations do not require any synchronous modifications to the data block.
2) The lock status remains the same as the previous state and change vectors for the commits are written to the redologs.

22. Example 6: Write the Dirty Buffers to Disk Due to Checkpoint
Instance B sends a write request to master D with the necessary SCN.
The lock master at D knows that the most recent copy of the block may be available at instance C and hence sends a message to instance C asking to write.
instance B writes the dirty blocks from the buffer cache due to a checkpoint.
1) Instance C initiates a disk write and writes a BWR into the redolog file.
2) Instance C gets the write notification that the write is completed.
3) Instance C notifies the master that the write is completed.
4) On receipt of the notification from C, master instance D tells all PI holders to discard their Pls, and the lock at instance C writes the modified block to the disk.
5) All instances that have previously modified this block will also have to write a BWR. The write request by instance C has now been satisfied, and instance C can now proceed with its checkpoint as usual.

23. Example 7: Master Instance Crash
The master instance D crashes.
The Global Resource Directory (GRD) is frozen momentarily and the resources held by master instance D will be equally distributed in the surviving nodes.

24. Example 7: What Alert log says abt reconfiguration…….
List of nodes:
0 1 2
Global Resource Directory frozen
* dead instance detected - domain 0 invalid = TRUE
Communication channels reestablished
* domain 0 valid = 0 according to instance 0
Wed Jun 21 23:22:
Master broadcasted resource hash value bitmaps
Non-local Process blocks cleaned out
LMS 0: 0 GCS shadows cancelled, 0 closed
LMS 2: 0 GCS shadows cancelled, 0 closed
LMS 3: 0 GCS shadows cancelled, 0 closed
LMS 1: 0 GCS shadows cancelled, 0 closed
Set master node info
Submitted all remote-enqueue requests
Dwn-cvts replayed, VALBLKs dubious
All grantable enqueues granted
LMS 0: 2189 GCS shadows traversed, 332 replayed
LMS 2: 2027 GCS shadows traversed, 364 replayed
LMS 3: 2098 GCS shadows traversed, 364 replayed
LMS 1: 2189 GCS shadows traversed, 343 replayed
Submitted all GCS remote-cache requests
Fix write in gcs resources
Reconfiguration complete

25. Crash Recovery – Key Components
Redo Threads and Streams
Redo Records and Change Vectors
Checkpoints
Thread Checkpoint or Local Checkpoint
Database Checkpoint or Global Checkpoint
Incremental Checkpoint
Bounded Recovery
Block Written Record (BWR)
Past Image (PI)
Checkpoints and PI
I Pass Recovery
II Pass Recovery
Merge Threads

26. Cache Fusion - Crash Instance Recovery
The steps for GRD reconfiguration are as follows:
Instance death is detected by the cluster manager.
Requests for PCM locks are frozen.
Enqueues are reconfigured and made available.
DLM recovery.
GCS (PCM lock) is remastered.
Pending writes and notifications are processed.
The steps for I Pass recovery are as follows:
The instance recovery (IR) lock is acquired by SMON.
The recovery set is prepared and built. Memory space is allocated in the SMON Program Global Area (PGA).
SMON acquires locks on buffers that need recovery.
II Pass recovery steps are as follows:
II Pass is initiated. The database is partially available.
Blocks are made available as they are recovered.
The IR lock is released by SMON. Recovery is complete.
The system is available.

27. Example 8: Select the Rows from Instance A
1) Instance A sends a shared lock request, Lock Req (S, D), to master instance C.
2) Master instance C knows that the most recent copy of that block may be in instance C (which is the current master) and asks the holder to ship the CR block to instance A.
3) Instance C ships the CR block to instance A over the interconnect.

28. Just for a clear understanding……
Its time to play ……

29. Cross Instance Consistent Read
segment 5 slot 18: state: 10 wrap#: 4E7 dba:
Session 15
LMS0
Session 27
SELECT runs,wickets FROM score WHERE team = 'ENG';
Build read consistent version of block 42
UPDATE score SET runs = runs + 6 WHERE team = 'ENG';
UPDATE score SET runs = runs + 4 WHERE team = 'ENG';
UPDATE score SET runs = runs + 2 WHERE team = 'ENG';
Undo Header
slot 0
col1: ENG
col2: 340
col3: 1
slot 1
col1: AUS
col2: 99
col3: 10
ITL1
Data Block 42
xid: E7
uba:
col2: 344
uba:
col2: 350
col2: 352
uba:
Data Block 42 (copy)
uba: -
slot 0
col1: ENG
col2: 340
col3: 1
slot 1
col1: AUS
col2: 99
col3: 10
ITL1
Data Block 42
xid: E7
uba:
col2: 344
uba:
col2: 350
col2: 352
uba:
slot 0
col1: ENG
col2: 340
col3: 1
slot 1
col1: AUS
col2: 99
col3: 10
ITL1
Data Block 42
xid: E7
uba:
col2: 344
uba:
col2: 350
col2: 352
uba:
Data Block 42 (copy)
uba: -
slot 0
col1: ENG
col2: 340
col3: 1
slot 1
col1: AUS
col2: 99
col3: 10
ITL1
Data Block 42
xid: E7
uba:
col2: 344
uba:
col2: 350
col2: 352
uba:
ITL1
seq: 530 irb 12
xid: E7
xid: E7
uba:
uba:
uba: -
uba:
uba:
uba: 12
uba: -
slot 0
block 42 slot 0
col3: 340
5.1
col1: ENG
col2: 350
col2: 340
col2: 344
col2: 340
col2: 350
col2: 344
col2: 352 13
uba
col3: 1
block 42 slot 0
col3: 344
5.1
slot 1 14
col1: AUS
block 42 slot 0
col3: 350
5.1
uba
col2: 99
col3: 10
Data Block 42 (copy)
Data Block 42
Undo Block

30. Commited Block – Block on Disk
UPDATE score SET runs = 200 WHERE team = 'ENG';
Session15
LMS0
Session27
22:9
22:10
AUS 99
ENG 205
AUS 99
ENG 205
AUS 99
ENG 199
UPDATE score SET runs = 204 WHERE team = 'ENG';
ENG 199
ENG 205
ENG 200
ENG 204
ENG 200
UPDATE score SET runs = 205 WHERE team = 'ENG';
ENG 204
COMMIT;
Block 42
Undo Block
AUS 99
ENG 205
AUS 99
ENG 199
SELECT runs FROM score WHERE team = 'ENG';
Instance 1
Instance 2

31. Committed Block – Block on Buffer Cache
UPDATE score SET runs = 200 WHERE team = 'ENG';
Session15
LMS0
Session27
22:9
22:10
AUS 99
ENG 205
AUS 99
ENG 205
AUS 99
ENG 199
UPDATE score SET runs = 204 WHERE team = 'ENG';
ENG 199
ENG 205
ENG 200
ENG 204
ENG 200
UPDATE score SET runs = 205 WHERE team = 'ENG';
ENG 204
COMMIT;
Block 42
Undo Block
AUS 99
ENG 199
SELECT runs FROM score WHERE team = 'ENG';
Instance 1
Instance 2
STOP

32. Uncommitted Block – Block in Buffer cache
UPDATE score SET runs = 200 WHERE team = 'ENG';
Session15
LMS0
Session27
22:10
AUS 99
ENG 199
AUS 99
ENG 205
AUS 99
ENG 199
AUS 99
ENG 199
AUS 99
ENG 205
UPDATE score SET runs = 204 WHERE team = 'ENG';
ENG 199
ENG 204
ENG 199
ENG 200
ENG 205
ENG 200
ENG 204
ENG 200
UPDATE score SET runs = 205 WHERE team = 'ENG';
ENG 204
Block 42 Copy
Block 42
Undo Block
AUS 99
ENG 199
SELECT runs FROM score WHERE team = 'ENG';
Instance 1
Instance 2

33. Uncommitted Block – On Disk
UPDATE score SET runs = 200 WHERE team = 'ENG';
Session15
LMS0
Session27
22:10
AUS 99
ENG 199
ENG 200
ENG 204
ENG 205
ENG 199
ENG 200
ENG 204
AUS 99
ENG 199
AUS 99
ENG 199
ENG 200
ENG 204
ENG 205
ENG 199
ENG 200
ENG 204
UPDATE score SET runs = 204 WHERE team = 'ENG';
ENG 199
ENG 200
ENG 199
ENG 204
ENG 204
ENG 200
ENG 205
ENG 200
UPDATE score SET runs = 205 WHERE team = 'ENG';
ENG 204
Block 42
Undo Block
AUS 99
ENG 199
AUS 99
ENG 199
ENG 200
ENG 204
ENG 205
In this example, Instance 1 is attempting to read a block which has been updated by Instance 2. .The data block has been flushed to disk by Instance 2 from its buffer cache before Instance 1 initiates the read operation.
As the data block is no longer in the buffer cache of Instance 2, Instance 1 is responsible for building the read-consistent version.
Instance1 therefore reads the data block from disk. However, in this example, the undo is still the buffer cache of Instance 2. Therefore, in order to build the read-consistent image of the block, Instance 1 must request the relevant undo block(s) from Instance 2. The LMS process on Instance 2 is responsible for serving the undo blocks to Instance 1. When Instance 1 receives the undo block(s) it will apply them locally to build the consistent-read version of the data block.
In this example, there is only one undo block in the buffer cache of Instance 2. Therefore it is only necessary to serve one undo block to Instance 1. However, if, for example, the data block had contained a sequence number which had been updated many times without committing, there could be many undo blocks in the buffer cache of Instance 2 which contained undo information for the data block. It would be necessary to serve all those undo blocks to Instance 1 which contained changes since the start of the transaction on Instance 2.
SELECT runs FROM score WHERE team = 'ENG';
SEE SLIDE NOTES FOR ADDITIONAL INFORMATION
Instance 1
Instance 2

34. Q & A

35. References:-
Oracle 10g Real Application Clusters handbook – K Gopalkrishnan
Julian Dyke – RAC Presentation
Oracle 10g RAC Administrators Guide