1. 1 RAC Internals Julian Dyke Independent Consultant Web Version juliandyke.com © 2007 Julian Dyke

2. juliandyke.com © 2007 Julian Dyke 2 Agenda  Transactions in RAC  Cross Instance Consistent Reads

3. juliandyke.com © 2007 Julian Dyke 3 Introduction

4. juliandyke.com © 2007 Julian Dyke 4 System Change Number  In RAC clusters SCN must be maintained across all nodes in cluster  SCN propagation scheme differs according to version  In Oracle 9.2 and below defaults to Lamport algorithm  Lamport or SCN Scheme 2 in alert.log  SCN piggy-backed on GCS/GES messages  Recorded in redo log  Default delay of 7 seconds  In Oracle 10.1 and above uses a new algorithm  SCN Scheme 3 in alert.log  Broadcast on commit  Apparently no delay

5. juliandyke.com © 2007 Julian Dyke 5 MAX_COMMIT_PROPAGATION_DELAY  Prior to Oracle 10.2  Default value is 700 centiseconds (7 seconds)  Specifies maximum time taken for a COMMIT on one node to be reflected on other nodes in the cluster  For some applications, value must be set to 0 (Broadcast on commit) including:  E-Business suite  SAP  In Oracle 10.2 and above default value is 0

6. juliandyke.com © 2007 Julian Dyke 6 LMS Background Processes  LMS background processes:  Implement cache fusion  Serve both consistent and current versions of blocks in cache of local instance to other instances  Maintain local part of Global Resource Directory  Minimum of 1 LMS process per instance  Maximum is version dependent  Oracle 9.210  Oracle 10.120  Oracle 10.236  Prior to Oracle 10.1, could be configured using _lm_lms parameter  In Oracle 10.1 and above, initial number of LMS processes specified by gcs_server_processes parameter

7. juliandyke.com © 2007 Julian Dyke 7 LMS Background Processes  Each LMS background process manages a set of blocks  Determined by hash function based on number of LMS background processes  Consequently  a block will always be handled by the same LMS process  Number of blocks served recorded in  Session / System statistics  V$CR_BLOCK_SERVER

8. juliandyke.com © 2007 Julian Dyke 8 Cross Instance Consistent Read UPDATE score SET runs = runs + 4 WHERE team = 'ENG'; slot 0col1: ENGcol2: 340col3: 1slot 1col1: AUScol2: 99col3: 10block 42 slot 0 col3: 340 5.1 block 42 slot 0 col3: 344 5.1 ITL1 block 42 slot 0 col3: 350 5.1 uba 800777.530.13 uba 800777.530.12 uba: - seq: 530 irb 12 12 13 14 xid: 0005.018.4E7 segment 5 slot 18: state: 10 wrap#: 4E7 dba: 00800777 UPDATE score SET runs = runs + 6 WHERE team = 'ENG'; UPDATE score SET runs = runs + 2 WHERE team = 'ENG'; Undo Header Data Block 42Undo Block 800777 xid: 0005.018.4E7 uba: 800777.530.12 col2: 344 uba: 800777.530.13 col2: 350col2: 352 uba: 800777.530.14 Session 27LMS0 Instance 2Instance 1 Session 15 SELECT runs,wickets FROM score WHERE team = 'ENG'; Build read consistent version of block 42 slot 0 col1: ENG col2: 340 col3: 1 slot 1 col1: AUS col2: 99 col3: 10 ITL1 Data Block 42 xid: 0005.018.4E7 uba: 800777.530.12 col2: 344 uba: 800777.530.13 col2: 350col2: 352 uba: 800777.530.14 slot 0 col1: ENG col2: 340 col3: 1 slot 1 col1: AUS col2: 99 col3: 10 ITL1 Data Block 42 xid: 0005.018.4E7 uba: 800777.530.12 col2: 344 uba: 800777.530.13 col2: 350col2: 352 uba: 800777.530.14 Data Block 42 (copy) uba: 800777.530.13 col2: 350 uba: 800777.530.12 col2: 344 uba: - col2: 340 slot 0 col1: ENG col2: 340 col3: 1 slot 1 col1: AUS col2: 99 col3: 10 ITL1 Data Block 42 xid: 0005.018.4E7 uba: 800777.530.12 col2: 344 uba: 800777.530.13 col2: 350col2: 352 uba: 800777.530.14 Data Block 42 (copy) uba: 800777.530.13 col2: 350 uba: 800777.530.12 col2: 344 uba: - col2: 340 slot 0 col1: ENG col2: 340 col3: 1 slot 1 col1: AUS col2: 99 col3: 10 ITL1 Data Block 42 xid: 0005.018.4E7 uba: 800777.530.12 col2: 344 uba: 800777.530.13 col2: 350col2: 352 uba: 800777.530.14 Data Block 42 (copy) uba: 800777.530.13 col2: 350 uba: 800777.530.12 col2: 344 uba: - col2: 340

9. juliandyke.com © 2007 Julian Dyke 9 V$CR_BLOCK_SERVER Column NameData TypeDescription CR_REQUESTSNUMBER# CR Blocks served to other instances CURRENT_REQUESTSNUMBER# Current Blocks served to other instances DATA_REQUESTSNUMBER# Data Blocks served to other instances UNDO_REQUESTSNUMBER# Undo Blocks served to other instances TX_REQUESTSNUMBER# Undo Segment Headers served to other instances CURRENT_REQUESTSNUMBER# requests requiring no changes to blocks served PRIVATE_REQUESTSNUMBER# requests requiring changes for requesting transaction only ZERO_RESULTSNUMBER# requests requiring changes for zero-XID transactions only DISK_READ_RESULTSNUMBER# requests requiring requesting instance to read block from disk FAIL_RESULTSNUMBER# requests failing - requesting instance must reissue request FAIRNESS_DOWN_CONVERTSNUMBER# times receiving instance has downgraded an X lock FAIRNESS_CLEARSNUMBER# times fairness counter was cleared FREE_GC_ELEMENTSNUMBER# times request received and X-lock had no buffers FLUSHESNUMBER# times log flushes by LMS process(es) FLUSHES_QUEUEDNUMBER# flushes queued by LMS process(es) FLUSH_QUEUE_FULLNUMBER# times flush queue was full FLUSH_MAX_TIMENUMBERmaximum time for flush LIGHT_WORKSNUMBER# times light works rule was invoked ERRORSNUMBER# times error signalled by LMS process

10. juliandyke.com © 2007 Julian Dyke 10 Light Works Rule  In theory, once a block has been written to disk, the LMS process will not attempt to read it again when responding to a consistent read request  Light Works Rule  Prevents LMS processes from going to disk when responding to CR requests for data, undo or undo segment blocks  Can prevent LMS process from completing its response to a CR request

11. juliandyke.com © 2007 Julian Dyke 11 GC Read Committed Block Instance 1Instance 2 22:10 AUS 99 ENG 199 Block 42Undo Block SELECT runs FROM score WHERE team = 'ENG'; UPDATE score SET runs = 200 WHERE team = 'ENG'; UPDATE score SET runs = 204 WHERE team = 'ENG'; UPDATE score SET runs = 205 WHERE team = 'ENG'; AUS 99 ENG 199 ENG 200ENG 204ENG 205 ENG 199ENG 200ENG 204 COMMIT; 22:9 Session15Session27LMS0 AUS 99 ENG 205 Committed Block - Data Block on disk AUS 99 ENG 205 AUS 99 ENG 205 STOP

12. juliandyke.com © 2007 Julian Dyke 12 GC Read Committed Block Instance 1Instance 2 22:10 AUS 99 ENG 199 Block 42Undo Block SELECT runs FROM score WHERE team = 'ENG'; UPDATE score SET runs = 200 WHERE team = 'ENG'; UPDATE score SET runs = 204 WHERE team = 'ENG'; UPDATE score SET runs = 205 WHERE team = 'ENG'; AUS 99 ENG 199 ENG 200ENG 204ENG 205 ENG 199ENG 200ENG 204 COMMIT; 22:9 Session15Session27LMS0 AUS 99 ENG 205 AUS 99 ENG 205 Committed Block - Data Block in buffer cache STOP

13. juliandyke.com © 2007 Julian Dyke 13 GC Read Uncommitted Block  Uncommitted changes MUST be flushed to the redo log before the LMS process can ship a consistent block to another instance  Reading process must wait until redo log changes have been written to redo log by LMS process  Bad for standard RAC databases  Reads must wait for redo log writes  Worse for extended / stretch RAC clusters  Increased latency of cross site disk communications

14. juliandyke.com © 2007 Julian Dyke 14 GC Read Uncommitted Block  For each block on which a consistent read is performed, a redo log flush must first be performed  Number of redo log flushes is recorded in the FLUSHES column of V$CR_BLOCK_SERVER  Redo log flush time  is recorded in the gc cr block flush time statistic for the LMS process  will increase time taken to serve consistent block  will increase time taken to perform consistent read  If LMS processes become very busy, consistent reads will experience high wait times e.g. for a full table scan gc cr multi block request

15. juliandyke.com © 2007 Julian Dyke 15 GC Read Uncommitted Block Instance 1Instance 2 22:10 AUS 99 ENG 199 Block 42Undo Block SELECT runs FROM score WHERE team = 'ENG'; UPDATE score SET runs = 200 WHERE team = 'ENG'; UPDATE score SET runs = 204 WHERE team = 'ENG'; UPDATE score SET runs = 205 WHERE team = 'ENG'; AUS 99 ENG 199 ENG 200ENG 204ENG 205 ENG 199ENG 200ENG 204 Session15Session27LMS0 AUS 99 ENG 205 AUS 99 ENG 205 ENG 204ENG 200ENG 199 Block 42 Copy AUS 99 ENG 199 AUS 99 ENG 199 Uncommitted Block - Data Block in buffer cache STOP

16. juliandyke.com © 2007 Julian Dyke 16 GC Read Uncommitted Block Instance 1Instance 2 22:10 AUS 99 ENG 199 Block 42Undo Block UPDATE score SET runs = 200 WHERE team = 'ENG'; UPDATE score SET runs = 204 WHERE team = 'ENG'; UPDATE score SET runs = 205 WHERE team = 'ENG'; AUS 99 ENG 199 ENG 200ENG 204ENG 205 ENG 199ENG 200ENG 204 Session15Session27LMS0 Uncommitted Block - Data Block on disk SELECT runs FROM score WHERE team = 'ENG'; AUS 99 ENG 199ENG 200ENG 204ENG 205 AUS 99 ENG 205 AUS 99 ENG 199ENG 200ENG 204ENG 205 AUS 99 ENG 205 AUS 99 ENG 199ENG 200ENG 204ENG 205 AUS 99 ENG 205 ENG 199 ENG 200 ENG 204 ENG 199 ENG 200 ENG 204 ENG 200ENG 199 STOP SEE SLIDE NOTES FOR ADDITIONAL INFORMATION

17. juliandyke.com © 2007 Julian Dyke 17 Consistent Reads in RAC  If possible, blocks will always be read from the cache of another instance  Undo blocks will be flushed to disk more frequently when:  All columns are updated  Indexed columns are updated  Single rows inserted  as opposed to using array inserts  Transactions are regularly rolled back  Rows locked using SELECT FOR UPDATE  Data blocks will be flushed to disk more frequently when:  Most transactions are read-only

18. juliandyke.com © 2007 Julian Dyke 18 Consistent Reads in RAC  Consistent read response times in RAC can be reduced by:  Avoid reading uncommitted blocks on remote nodes  Partitioning  Limiting number of rows per block  Specifying SCN  Minimizing size of transactions on remote nodes  Must retain ACID properties  May be possible to use application logic to synchronize writes and reads  Increasing number of LMS processes on remote node  Should be added dynamically by kernel  Also by obvious hardware changes such as  reducing latency of interconnect  increasing disk speed

19. juliandyke.com © 2007 Julian Dyke 19 Thank you for your interest For more information and to provide feedback please contact me My e-mail address is: info@juliandyke.com My website address is: www.juliandyke.com