1. Files and Buffer Manager Chapter 15

2. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 2 Abstractions Provided by the File Manager n Device independence: The file manager turns the large variety of external storage devices, such as disks (with their different numbers of cylinders, tracks, arms, and read/write heads), ram-disks, tapes, and so on, into simple abstract data types. n Allocation independence: The file manager does its own space management for storing the data objects presented by the client. It may store the same objects in more than one place (replication).

3. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 3 Abstractions Provided by the File Manager n Address independence: Whereas objects in main memory are always accessed through their addresses, the file manager provides mechanisms for associative access. Thus, for example, the client can request access to all records with a specified value in some field of the record. Support for associative access comes in many flavors, from simple mechanisms yielding fast retrieval via the primary key up to the expressive power of the SQL select statement.

4. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 4 External Storage vs. Main Memory n Capacity: Main memory is usually limited to a size that is some orders of magnitude smaller than what large databases need. n Economics: External storage holds large volumes of data at reasonable cost. n Durability: Main memory is volatile. External storage devices such as magnetic or optical disks are inherently durable and therefore are appropriate for storing persistent objects. After a crash, recovery starts with what is found in durable storage.

5. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 5 External Storage vs. Main Memory n Speed: External storage devices are some orders of magnitude slower than main memory. As a result, it is more costly, both in terms of latency and in terms of pathlength, to get data from external storage to the CPU than to load data from main memory. n Functionality: Data cannot be processed directly on external storage: they can neither be compared nor modified out there.

6. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 6 The Storage Pyramid main memory online external storage near line (archive) storage typical capacity Electronic RAM and bulk storage Magnetic / optical disks Automated archives (e.g. optical disk jukeboxes, tape robots, etc.) current data stale data

7. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 7 Interfacing to External Memory: Read-Write Mapping

8. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 8 Interfacing to External Memory: File Mapping

9. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 9 Interfacing to External Memory: Single-Level Storage External Storage File A File B File C File D Main Memory File A File B File CFile D Virtual memory Explicit mapping

10. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 10 Locality and Cacheing The movement of data through the pyramid is guided by the principle of locality: n Locality of active data: Data that have recently been referenced will very likely be referenced again. n Locality of passive data: Data that have not been referenced recently will most likely not be referenced in the future.

11. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 11 Levels of Abstraction in a File and Database Manager main memory online external memory nearline external memory DBMS Application database access modules databaseb uffer mgr. logging recovery media and file manager archive manager Transaction programs Tuple management, associative access Buffer management File management Archive management manages tuple oriented access block oriented access device oriented access setoriented access Application sort, join,... read, write

12. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 12 Operations of the Basic File System STATUS create(filename, allocparmp) STATUS delete(filename) STATUS open(filename, ACCESSMODE, FILEID); STATUS close(FILEID) STATUS extend(FILEID, allocparmp) STATUS read(FILEID, BLOCKID, BLOCKP) STATUS readc(FILEID, BLOCKID, blockcount, BLOCKP) STATUS write(FILEID, BLOCKID, BLOCKP) STATUS writec(FILEID, BLOCKID, blockcount, BLOCKP)

13. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 13 Mapping Files To Disk

14. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 14 Issues in Managing Disk Space n Initial allocation: When a file is created, how many contiguous slots should be allocated to it? n Incremental expansion: If an existing file grows beyond the number of slots currently allocated, how many additional contiguous blocks should be assigned to that file? n Reorganization: When and how should the free space on the disk be reorganized?

15. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 15 Extent-Based Allocation

16. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 16 Buddy Systems

17. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 17 Simple Mapping of Relations To Disks

18. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 18 A Usual Way of Mapping of Relations To Disks

19. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 19 Principles of the Database Buffer

20. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 20 Design Options for the Buffer Manager n Buffer per file: Each file has its own private buffer pool.. n Buffer per page size: In systems with different page (and block) sizes, there is usually at least one buffer for each page size. n Buffer per file type: There are files like indices, which are accessed in a significantly different way from other files. Therefore, some systems dedicate buffers to files depending on the access pattern and try to manage each of them in a way that is optimal for the respective file organization.

21. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 21 Logic of the Buffer Manager n Search in buffer: Check if the requested page is in the buffer. If found, return the address F of this frame to the caller. n Find free frame: If the page is not in the buffer, find a frame that holds no valid page. n Determine replacement victim: If no such frame exists, determine a page that can be removed from the buffer (in order to reuse its frame).

22. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 22 Logic of the Buffer Manager n Write modified page: If replacement page has been changed, write it. n Establish frame address: Denote the start address of the frame as F. n Determine block address: Translate the requested PAGEID P into a FILEID and a block number. Read the block into the frame selected. n Return: Return the frame address F to the caller.

23. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 23 Synchronization in the Buffer

24. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 24 What the Buffer Manager Does for Synchronization n Sharing: Pages are made addressable to all processes that run the database code. n Semaphore protection: Each requestor gets the address of a semaphor protecting the page. n Durable storage: The access modules inform the buffer manager if their page access has resulted in an update of the page; the actual write operation, however, is issued by the buffer manager, probably at a time when the update transaction is long gone.

25. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 25 The Interface to the Buffer Manager typedef struct {PAGEIDpageid; /* id of page in file */ PAGEPTRpageaddr; /* base addr. in buffer */ intindex; /* record within page */ semaphore *pagesem; /* pointer to the sem. */ Booleanmodified; /* caller modif. page */ Booleaninvalid; /* destroyed page */ } BUFFER_ACC_CB, *BUFFER_ACC_CBP; /* control block for buffer access */

26. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 26 The Need for Fix and Unfix

27. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 27 The Fix-Use-Unfix Protocol I n FIX: The client requests access to a page using the bufferfix interface. n USE: The client uses the page and the pointer to the frame containing the page will remain valid. n UNFIX: The client explicitly waives further usage of the frame pointer; that is, it tells the buffer manager that it no longer wants to use that page.

28. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 28 The Fix-Use-Unfix Protocol II

29. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 29 Structure of the Buffer Manager

30. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 30 Logging and Recovery from the Buffer Manager's Perspective I TransactionBufferDatabaseRemark running committed AB A B OK; old state in DB BAAB BAAB database corrupted BABA conflicting view on TA AB A B BAAB BAAB BABA OK; Read-only TA DB not in new state database corrupted OK; new state in DB

31. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 31 Logging and Recovery from the Buffer Manager's Perspective II state of transaction TA aborted committed state of page A in database old new result of recovery using operation log wrong tuple might be deleted old inverse operation succeeds operation succeeds duplicate of tuple is inserted

32. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 32 The Log and Page LSNs

33. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 33 Different Buffer Management Policies n Steal policy: When the buffer manager needs space, it can decide to replace dirty pages. n No-Steal policy: Pages can be replaced only if they are clean. n Force policy: At end of transaction, all modified pages are forced to disk in a series of synchronous write operations. n No-Force policy: No modified page is forced during commit. REDO log records are written to the log.

34. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 34 The Problem of Hotspot Pages bufferpool durable storage log TA1 TA2 TA3 TA4TA5 TA6 TA7 TA8 force The dotted arrows indicate an update of the page by the respective transaction.The arrows at 45 degrees indicate the forced writing of the page during commit processing.The downward arrows indicate the writing of log records for the respective transaction. update operations page A

35. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 35 The Basic Checkpoint Algorithm n Quiesce: Delay all incoming update DML calls until all fixes with exclusive semaphores have been released. n Flush the buffer: Write all modified pages. n Log the checkpoint: Write a record to the log, saying that a checkpoint has been generated. n Resume normal operation: The bufferfix requests for updates that have been delayed in order to take the checkpoint can now be processed again.

36. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 36 The Case for Indirect Checkpointing

37. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 37 The Indirect Checkpointing Algorithm n Record TOC: Log the list of PAGEIDs. n Compare with prev. ckpt: See if any modified pages have not been replaced since last ckpt. n Force lazy pages: Schedule the writing of those pages during the next checkpoint interval. n Low-water mark: Find the LSN of the oldest still-volatile update; write it to the log. n Write Checkpoint done record n Resume normal operation

38. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 38 Further Possibilities for Optimization n Pre-flushing can be performed by an asynchronous process that scans the buffer for "old" modified pages. Writing is done under semaphore protection. n Pre-fetching can, among other things, be used to make restart more efficient. If page reads are logged one can use the recent checkpoint plus the log to prime the bufferpool, i.e. it will look almost exactly like at the moment of the crash.

39. Jim Gray, Andreas Reuter Transaction Processing - Concepts and Techniques WICS August 2 - 6, 1999 39 Further Possibilities for Optimization Transaction scheduling and buffer management can take hints from the query optimizer: n This relation will be scanned sequentially. n This is a sequential scan of the leaves of a B- tree. n This is the traversal of a B-tree, starting at the root. n This is a nested-loop join, where the inner relation is scanned in physically sequential order.