Presentation is loading. Please wait.

Presentation is loading. Please wait.

Gray & Reuter: Resource Manager 1 Resource Managers 9:00 11:00 1:30 3:30 7:00 Overview Faults Tolerance T Models Party TP mons Lock Theory Lock Techniq.

Similar presentations

Presentation on theme: "Gray & Reuter: Resource Manager 1 Resource Managers 9:00 11:00 1:30 3:30 7:00 Overview Faults Tolerance T Models Party TP mons Lock Theory Lock Techniq."— Presentation transcript:

1 Gray & Reuter: Resource Manager 1 Resource Managers 9:00 11:00 1:30 3:30 7:00 Overview Faults Tolerance T Models Party TP mons Lock Theory Lock Techniq Queues Workflow Log ResMgr CICS & Inet Adv TM Cyberbrick Files &Buffers COM+ Corba Replication Party B-tree Access Paths Groupware Benchmark MonTueWedThurFri Jim Gray Microsoft, Gray @ Andreas Reuter International University,

2 Gray & Reuter: Resource Manager 2 Whirlwind Tour: The Actors Resource managers – provide ACID objects (transactional objects) – Use log manager to record changes – Use transaction manager to coordinate multi-RM changes – Use communication manager to make transactional RPCs Transaction Manager Log Manager Log Objects Resource Managers Objects Resource Managers Volatile Storage Durable Storage Volatile Storage Durable Storage Communication Manager Transaction Manager Log Manager Communication Manager Log

3 Gray & Reuter: Resource Manager 3 Whirlwind Tour: the Application Verbs TRIDBegin_Work(context *); /* begin a transaction*/ BooleanCommit_Work(context *);/* commit the transaction*/ voidAbort_Work(void); /* rollback to savepoint zero*/ savepoint Save_Work(context *); /* establish a savepoint*/ savepoint Rollback_Work(savepoint); /*return to savept (savept 0 = abort)*/ BooleanPrepare_Work(context *); /* put transaction in prepared state*/ context Read_Context(void); /* return current savepoint context*/ TRIDChain_Work(context *); /* end current and start next trans*/ TRIDMy_Trid(void); /* return current transaction identifier*/ TRID Leave_Transaction(void); /*set process trid null, return current id*/ Boolean Resume_Transaction(TRID); /* set process trid to desired trid*/ enum tran_status { ACTIVE, PREPARED, ABORTING, COMMITTING, ABORTED, COMMITTED }; tran_status Status_Transaction(TRID); /* transaction identifier status*/

4 Gray & Reuter: Resource Manager 4 Whirlwind Tour Types Of Transaction Executions Shaded stuff is undone Save Persistent Begin Action Save Action Save Action Save Action Commit A Simple Commit A Simple Abort Begin Action Save Action Save Action Save Action Rollback Action Save Action A Partial Rollback Begin Action Save Action Save Action Save Action Rollback A Persistent Transaction Surviving A System Restart Begin Action Save Action Restart Action Save Action Commit

5 Gray & Reuter: Resource Manager 5 Whirlwind Tour: the TRID Flow Call graph: who calls whom. TRIDs flow on all such calls. Application is typically root. RM can be an application (use a transactional RM to store state) Application Servers Resource Managers Resource Managers Transaction Application Servers

6 Gray & Reuter: Resource Manager 6 Whirlwind tour Normal (no failure) Transaction Execution TM generates the TRID at Begin_Work(). Coordinates Commit, RM joins work, generates log records, allows commit

7 Gray & Reuter: Resource Manager 7 WW tour: The Resource Manger view Resource Manager resource manager's own service interface rmCall(...) transaction management other resource managers rmCall(...) TP monitor administrative functions and callbacks to install, start, and schedule a resource manager response invocation callbacks (depends on application) Save Prepare Commit UNDO REDO Checkpoint Transaction Manager functions callbacks Identify SaveWork RollbackWork Join StatusTransaction Leave Resume

8 Gray & Reuter: Resource Manager 8 WW tour: The Resource manager view BooleanSavepoint(LSN *);/* invoked at tran Save_Work(). Returns RM vote */ BooleanPrepare(LSN *);/* invoked at phase_1. Return vote on commit*/ voidCommit();/* called at commit ¯2 */ voidAbort();/* called at failed commit ¯2 or abort */ voidUNDO(LSN);/* Undo the log record with this LSN*/ voidREDO(LSN);/* Redo the log record with this LSN*/ BooleanUNDO_Savepoint(LSN);/* Vote TRUE if can return to savepoint*/ voidREDO_Savepoint(LSN);/* Redo a savepoint. */ voidTM_Startup(LSN);/* TM restarting. Passes RM ckpt LSN*/ LSNCheckpoint(LSN * low_water); /* TM checkpointing, Return RM ckpt LSN, set low water LSN*/ Boolean Join_Work(RMID, TRID); /* Become part of a transaction*/

9 Gray & Reuter: Resource Manager 9 WW Tour: The Transaction Manager Transaction rollback. coordinates transaction rollback to a savepoint or abort rollbacks can be initiated by any participant. Resource manager restart. If an RM fails and restarts, TM presents checkpoint anchor & RM undo/redo log System restart. TM drives local RM recovery (like RM restart) TM resolves any in-doubt distributed transactions Media recovery. TM helps RM reconstruct damaged objects by providing archive copies of object + the log of object since archived. Node restart. Transaction commit among independent TMs when a TM fails.

10 Gray & Reuter: Resource Manager 10 WW Tour: When a Transaction Aborts At transaction rollback TM drives undo of each RM joined to the transaction Can be to savepoint 0 (abort) or partial rollback.

11 Gray & Reuter: Resource Manager 11 WW tour: the Transaction Manager at Restart/Recovery At restart, TM reading the log drives RM recovery. Single log scan. Single resolver of transactions. Multiple logs possible, but more complex/more work.

12 Gray & Reuter: Resource Manager 12 End of Whirl-Wind Tour

13 Gray & Reuter: Resource Manager 13 Resource Manager Concepts: Undo Redo Protocol

14 Gray & Reuter: Resource Manager 14 Resource Manager Concepts: Transaction UNDO Protocol declare cursor for transaction_log select rmid, lsn/* a cursor on the transaction's log */ fromlog/* it returns the resource manager name*/ wheretrid = :trid/* and record id (log sequence number)*/ descending lsn;/* and returns records in LIFO order*/ void transaction_undo(TRID trid)/* Undo the specified transaction. */ { int sqlcode;/* event variables set by sql */ open cursor transaction_log;/* open an sql cursor on the trans log*/ while (TRUE)/* scan trans log backwards & undo each*/ { /* fetch the next most recent log rec */ fetch transaction_log into :rmid, :lsn; /* */ if (sqlcode != 0) break; /* if no more, trans is undone, end loop*/ rmid.undo(lsn);/* tell RM to undo that record */ }/* tell RM to undo that record */ close cursor transaction_log;/* Undo scan is complete, close cursor */ }; /* return to caller */ If UNDO to savepoint, the UNDO stops at desired savepoint

15 Gray & Reuter: Resource Manager 15 Resource Manager Concepts: Restart REDO Protocol Note: REDO forwards, UNDO backwards void log_redo(void)/* */ {declare cursor for the_log /* declare cursor from log start forward */ selectrmid, lsn/* gets RM id and log record id (lsn) */ fromlog/* of all log records. */ ascending lsn;/* in FIFO order */ open cursor the_log;/* open an sql cursor on the log table */ while (TRUE)/* Scan log forward& redo each record. */ { fetch the_log into :rmid, :lsn;/* fetch the next log record */ if (sqlcode != 0) break; /* if no more, then all redone, end loop */ rmid.redo(lsn);}/* tell RM to redo that record */ close cursor the_log; /* Redo scan complete, close cursor */ };/* return to caller */

16 Gray & Reuter: Resource Manager 16 Idempotence F(F(X)) == F(X): Needed in case restart fails (and restarts) Redo(Redo(old_state,log), log) = Redo(new_state,log) = new_state Undo(Undo(new_state,log), log) = Undo(old_state,log) = old_state Old State New State log record undo redo

17 Gray & Reuter: Resource Manager 17 Testable State: Can Tell If It Happened. IF operation not idempotent AND state not testable THEN recovery is impossible ELSE for F in {UNDO, REDO}: not testable: WHILE (! ACK) F(F(X)) testable: WHILE ( not desired state) {F(x)}

18 Gray & Reuter: Resource Manager 18 Real Operations: Can Not Be Undone Defer operations until commit is assured. Perform as part of Phase 2 of commit If must undo for some reason, generate compensation log record to be processed by some higher authority.

19 Gray & Reuter: Resource Manager 19 Example: Communications Session RM Ops are idempotent (sequence numbers) and testable (sequence numbers)

20 Gray & Reuter: Resource Manager 20 Kinds of Logging Physical: Keep old and new value of container (page, file,...) Pro: Simple Allows recovery of physical object (e.g. broken page) Con:Generates LOTS of log data Logical: Keep call params such that you can compute F(x), F -1 (x) Pro: Sounds simple Compact log. Con: Doesn't work (wrong failure model). Operations do not fail cleanly.

21 Gray & Reuter: Resource Manager 21 Sample Physical LOG RECORD Ordinary sequential insert is OK. Update of sorted (B-tree) page: update LSN update page space map update pointer to record insert record at correct spot (move 1/2 the others) Essentially writes whole page (old and new). 16KB log records for 100-byte updates. struct compressed_log_record_for_page_update /* */ { int opcode;/* opcode will say compressed page update*/ filename fname;/* name of file that was updated */ longpageno;/* page that was updated */ longoffset;/* offset within page that was updated */ longlength;/* length of field that was updated */ charold_value[length];/* old value of field */ charnew_value[length];/* new value of field */ };/* */

22 Gray & Reuter: Resource Manager 22 Sample Physical LOG RECORD Very compact. Implies page update(s) for record (may be many pages long). Implies index updates (many be many indices on base table) struct logical_log_record_for_insert/* */ { int opcode;/* opcode will says insert */ filename fname;/* name of file that was updated */ longlength;/* length of record that was updated*/ charrecord[length];/* value record */ };/* */

23 Gray & Reuter: Resource Manager 23 The trouble with Logical Logging Logical logging needs to start UNDO/REDO with an action-consistent state. No half completed operations. for example: insert (table, record) ALL or NONE of the indices should be updated when logical UNDO/REDO is invoked. Problem: Failure model is Page & Message action consistency (Lampson /Sturgis model of Chapter 3). Actions can fail due to: Logic: e.g. duplicate key. Limit: ran out of space Contention: deadlock Media: broken page or session System: computer failure/restart

24 Gray & Reuter: Resource Manager 24 Making Logical Logging Work: Shadows Keep old copy of each page Reset page to old copy at abort (no undo log) Discard old copy at commit. Handles all online failures due to: Logic: e.g. duplicate key. Limit: ran out of space Contention: deadlock Problem: forces page locking, only one updater per page. What about restart? Need to atomically write out all changed pages.

25 Gray & Reuter: Resource Manager 25 Making Logical Logging Work: Shadows Perform same shadow trick at disc level. Keep shadow copy of old pages. Write out new pages. In one careful write, write out new page root. Makes update atomic

26 Gray & Reuter: Resource Manager 26 Shadows Pro: Simple Not such a bad deal with non-volatile ram Con: page locking extra space extra overhead (for page maps) extra IO declusters sequential data

27 Gray & Reuter: Resource Manager 27 Compromise Physio-Logical Logging Physio-Logical Logging Physical to a "page" (physical container) Logical within a "page". Keep old and new value of container (page, file,...) Pro: Simple Allows recovery of physical object (e.g. broken page) Con: Generates LOTS of log data

28 Gray & Reuter: Resource Manager 28 Logical vs Physio-logical Logging Note: physical log records would be bigger for sorted pages.

29 Gray & Reuter: Resource Manager 29 Physiological Logging Rules Complex operations are a sequence of simple operations on pages and messages. Each operation is constructed as a mini-transaction: lock the object in exclusive mode transform the object generate an UNDO-REDO log record record log LSN in object unlock the object. Action Consistent Object: When object semaphore free, no ops in progress. Log-Consistency: contains log records of all complete page/msg actions.

30 Gray & Reuter: Resource Manager 30 Physiological Logging Rules Online Operation - Only Need the Fix Rule Each operation is structured as a mini-transaction. Each operation generates an UNDO record. No page operation fails with the semaphore set. (exception handler must clean up state and UNFIX any pages). Then Rollback can be physical to a page/session/container and logical within page/session/container.

31 Gray & Reuter: Resource Manager 31 Physiological Logging Rules Restart Operation - Need WAL and F@C Need Page-Action consistent disc state. Pages are action consistent. Committed actions can be redone from log. Uncommitted actions can be undone from log. WAL: Write Ahead Log Write undo/redo log records before overwriting disc page Only write action-consistent pages Force-Log-At-Commit Make transaction log records durable at commit.

32 Gray & Reuter: Resource Manager 32 Physiological Logging Rules WAL and F@C WAL: Write Ahead Log write page: get page semaphore copy page give page semaphore /* avoids holding semaphore during IO */ Force_log(Page(LSN)) /*WAL logic, probably already flushed*/ Write copy to disc. WAL gives idempotence and testability. Force-Log-At-Commit At commit phase 1: Force_log(transaction.max_lsn)

33 Gray & Reuter: Resource Manager 33 WAL & F@C in Pictures VVlsn Volatile Page Versions Volatile Log Records VLlsn PVlsn Persistent Page Versions Durable Log Records DLlsn Time online: VVlsn = VLlsn restart: DLlsn <= VVlsn PVlsn <= DLlsn Commit: commit_lsn <= DLlsn At restart all volatile memory is reset and must be reconstructed from persistent memory. restart: PVlsn <= DLlsn commit_lsn <= DLlsn PVlsn DLlsn FIX, WAL and F@C assure these assertions

34 Gray & Reuter: Resource Manager 34 The One Bit Resource Manager Manages an array of transactional bits (the free space bit map). i = get_bit();/* gets a free bit and sets it*/ give_bit(i);/* returns a free bit (when transaction commits)*/

35 Gray & Reuter: Resource Manager 35 The Bitmap and Its Log Records The Data Structure struct { /* layout of the one-bit RM data structure*/ LSNlsn;/* page LSN for WAL protocol*/ xsemaphoresem; /* semaphore regulates access to the page */ Booleanbit[BITS];/* page.bit[i] = TRUE => bit[i] is free*/ } page;/* allocates the page structure*/ The Log Records struct/* log record format for the one-bit RM*/ { int index;/* index of bit that was updated*/ Booleanvalue;/* new value of bit[index]*/ } log_rec;/* log record used by the one-bit RM*/ const int rec_size = sizeof(log_rec); /*size of the log record body.*/

36 Gray & Reuter: Resource Manager 36 Page and Log Consistency for 1-Bit RM Data dirty if reflects an uncommitted transaction update Otherwise, data is clean. Page Consistency: No clean free bit has been given to any transaction. Every clean busy bit was given to exactly one transaction. Dirty bits locked in X mode by updating transactions. The page.lsn reflects most recent log record for page. Log Consistency: Log contains a record for every completed mini-transaction update to the page.

37 Gray & Reuter: Resource Manager 37 give_bit() get_bit() & give_bit(i) temporarily violate page consistency. Mini-transaction holds semaphore while violating consistency. Makes page & log mutually consistent before releasing sem. => each mini-transaction observes a consistent page state. void give_bit(int i)/* free a bit */ { if (LOCK_GRANTED==lock(i,LOCK_X,LOCK_LONG,0)) /* Lock bit*/ { Xsem_get(&page.sem);/* get page sem*/ page.bit[i] = TRUE; /* free the bit*/ log_rec.index = i;/* generate log rec*/ log_rec.value = TRUE;/*saying bit is free*/ page.lsn = log_insert(log_rec,rec_size); /*write log rec&update lsn*/ Xsem_give(&page.sem);}/* page consistent */ else /* if lock failed, caller doesn't own bit, */ Abort_Work();/* in that case abort caller's trans*/ return; };/* */

38 Gray & Reuter: Resource Manager 38 get_bit() int get_bit(void) /* allocate a bit to and returns bit index*/ { inti;/* loop variable*/ Xsem_get(&page.sem);/* get the page semaphore*/ for ( i = 0; i<BITS; i++);/* loop looking for a free bit*/ {if (page.bit[i])/* if bit is free, may be dirty (so locked)*/ {if (LOCK_GRANTED =lock(i,LOCK_X,LOCK_LONG,0));/* lock bit*/ { page.bit[i] =FALSE;/* got lock on it, so it was free*/ log_rec.value = FALSE;/* generate log rec describing update*/ log_rec.index = i;/* */ page.lsn = log_insert(log_rec,rec_size); /* write log rec&update lsn*/ Xsem_give(&page.sem);/* page now consistent, give up sem*/ return i; }/* return to caller*/ };/* else lock bounce so bit dirty*/ };/* try next free bit, */ Xsem_give(&page.sem);/* if no free bits, give up semaphore */ Abort_Work();/* abort transaction*/ return -1;};/* returns -1 if no bits are available. */

39 Gray & Reuter: Resource Manager 39 Compensation Logging Undo may generate a log record recording undo step Makes Page LSN monotonic Similar technique was used for Communication Manager (session sequence number was monotonic)

40 Gray & Reuter: Resource Manager 40 1-bit RM UNDO Callback void undo(LSN lsn) /* undo a one-bit RM operation */ { int i;/* bit index*/ Booleanvalue;/* old bit value from log rec to be undone*/ log_rec_header header;/* buffer to hold log record header*/ rec_size = log_read_lsn(lsn,header,0,log_rec,big); /* read log rec*/ Xsem_get(&page.sem);/* get the page semaphore*/ i = log_rec.index;/* get bit index from log record*/ value = ! log_rec.value;/* get complement of new bit value*/ page.bit[i] = value;/* update bit to old value*/ log_rec.value= value;/* make a compensation log record*/ page.lsn = log_insert(log_rec,rec_size); /* log it and bump page lsn*/ Xsem_give(&page.sem);/* free the page semaphore*/ return; }/* */

41 Gray & Reuter: Resource Manager 41 1-bit RM Checkpoint Callback LSN checkpoint(LSN * low_water) /* copy 1-page RM state to persistent store*/ { Xsem_get(&page.sem);/* get the page semaphore*/ *low_water = log_flush(page.lsn);/* WAL force up to page lsn, and*/ /* set low water mark*/ write(file,page,0,sizeof(page));/* write page to persistent memory*/ Xsem_give(&page.sem);/* give page semaphore*/ return NULLlsn; }/* return checkpoint lsn (none needed)*/

42 Gray & Reuter: Resource Manager 42 1-bit RM REDO Callback void redo( LSN lsn) /* redo an free space operation */ { int i;/* bit index*/ Booleanvalue;/* new bit value from log rec to be redone*/ log_rec_header header;/* buffer to hold log record header*/ rec_size = log_read_lsn(lsn,header,0,log_rec,big); /* read log record*/ i = log_rec.index;/* Get bit index*/ lock(i,LOCK_X,LOCK_LONG,0);/* get lock on the bit (often not needed)*/ Xsem_get(&page.sem);/* get the page semaphore*/ if (page.lsn < lsn) /* if bit version older than log record*/ { value= log_rec.value;/* then redo the op. get new bit value*/ page.bit[i] = value;/* apply new bit value to bit*/ page.lsn = lsn; }/* advance the page lsn*/ Xsem_give(&page.sem);/* free the page semaphore*/ return; };/* */

43 Gray & Reuter: Resource Manager 43 1-BIT Rm Noise Callbacks Boolean prepare(LSN * lsn)/* 1-bit RM has no phase 1 work*/ {*lsn = NULLlsn; return TRUE ;};/* */ void Commit(void )/* Commit release locks &*/ { unlock_class(LOCK_LONG, TRUE, MyRMID()); }; /* return*/ void Abort(void )/* Abort release all locks &*/ { unlock_class(LOCK_LONG, TRUE, MyRMID()); }; /* return */ Boolean savepoint((LSN * lsn) /* no work to do at savepoint*/ {*lsn = NULLlsn; return TRUE ;};/* */ void UNDO_savepoint(LSN lsn) /* rollback work or abort transaction*/ {if (savepoint == 0) /* if at savepoint zero (abort)*/ unlock_class(LOCK_LONG, TRUE, MyRMID()); /* release all locks*/ }; /* */

44 Gray & Reuter: Resource Manager 44 Summary Model: Complex actions are a page/message action sequence. LSN: Each page carries an LSN and a semaphore. ReadFix: Read acts semaphore in shared mode. WriteFix: Update actions get semaphore in exclusive mode, generate one or more log records covering the page, advance the page LSN to match highest LSN give semaphore WAL: log_flush (page.LSN) before overwriting persistent page F@C: force all log records up to the commit LSN at commit Compensation Logging: Invalidate undone log record with a compensating log record. Idempotence via LSN: page LSN makes REDO idempotent

45 Gray & Reuter: Resource Manager 45 Two Phase Commit Getting two or more logs to agree Getting two or more RMs to agree Atomically and Durably Even in case one of them fails and restarts. The TM phases Prepare. Invoke each joined RM asking for its vote. Decide. If all vote yes, durably write commit log record. Commit. Invoke each joined RM, telling it commit decision. Complete. Write commit completion when all RM ACK.

46 Gray & Reuter: Resource Manager 46 Centralized Case of Two Phase Commit Each participant: (TM &RM) goes through a sequence of states These generate log records Null Active AbortingAborted Prepared CommittingCommitted

47 Gray & Reuter: Resource Manager 47 Examples CommittedAbortedbegin DO rm1DO rm1DO rm2DO rm2 prepare rm2 {locks} UNDO rm2 commit { rm1, rm2}UNDO rm2 completeUNDO rm1 UNDO begin { rm1, rm2} complete

48 Gray & Reuter: Resource Manager 48 Transitions in Case of Restart Active state not persistent, others are persistent For both TM and RM. Log records make them persistent (redo) TM tries to drive states to the right. (to committed, aborted)

49 Gray & Reuter: Resource Manager 49 Successful two phase commit Message/Call flow from TM to each RM joined to transaction If TM and RM share the same log, the RM FORCE can piggyback on the TM FORCE One IO to commit a transaction (less if commit is grouped)

50 Gray & Reuter: Resource Manager 50 Abort Two Phase Commit If RM sends "NO" or no response (timeout), TM starts abort. Calls UNDO of each trans log record May stop at a savepoint. At begin_trans it calls ABORT() callback of each joined RM

51 Gray & Reuter: Resource Manager 51 Distributed two phase commit Tracking joined TMs -- the communications manager helps Much as TRPC helps in the local case. Root TM owes a Prepare/Commit/Abort message to each joined TM. Joined TM does "local" commit.

52 Gray & Reuter: Resource Manager 52 Full Transaction State Diagram Next section explains how these states are implemented.

53 Gray & Reuter: Resource Manager 53 Summary of Resource Manager Concepts DO/UNDO/REDO Idempotent, Testable, Real operations Logical vs Physical logging Shadows to make logical logging work Physiological logging Fix, WAL, Force-at-commit Page/Message/Log consistency RM callbacks (the 1-bit resource manager) Join, Prepare, Commit, Abort, UNDO, REDO,.... Restart REDO/UNDO Two phase commit (RM story is simple).

Download ppt "Gray & Reuter: Resource Manager 1 Resource Managers 9:00 11:00 1:30 3:30 7:00 Overview Faults Tolerance T Models Party TP mons Lock Theory Lock Techniq."

Similar presentations

Ads by Google