1. Chapter 11.3 File System Implementation

2. 11.2 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Chapter 11.3 : File System Implementation Chapter 11.3 : File System Implementation Chapter 11.1 File-System Structure File-System Implementation Directory Implementation Chapter 11.2 Allocation Methods Chapter 11.3. Free-Space Management Recovery Log-Structured File Systems

3. 11.3 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Free-Space Management We’ve talked about file space structure and file space implementation. Now: how free space is managed? What space is available, how much, and where are the free blocks are located. In other words, we need some kind of storage map. Keeping track of free space requires a free-space list. Free space list is a list of unallocated disk blocks Free-space list: a list? Some other kind of structure? Let’s consider a couple of alternative structures used to manage free disk.

4. 11.4 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Bit Vector Approach – used for space allocation Bit vector (used to represent n blocks) is one approach. Each block is represented by a singe bit: 1  block is available; 0  block is in use (allocated) … 012n-1 bit[i] =  0  block[i] free 1  block[i] occupied Downside: to be efficient in searching, the bit map must be kept in primary memory. For small disks, there is not a problem, but for larger disks (say a 40GB disk with 1KB blocks) a bit map of 5MB is needed!! A sample bit vector might appear as 0011110011111100110011 …. Very simple approach but very efficient in finding the first free block. Too, a number of instruction sets contain instructions for bit manipulation

5. 11.5 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Linked List Approach used for Allocation This approach links all free space together via a linked list. All we really need in memory (cached) is a pointer to the first free block. All blocks then contain a pointer to the next free block It is easy, however, to quickly see inefficiencies… To allocate a lot of disk, we must read each block. This is an incredible amount of I/O and is very inefficient from a performance perspective! If only a small amount of storage is necessary, as is usually the case, the linked list approach to disk allocation may be reasonably efficient. Very often only a single block is requested. FAT approach uses this approach – each entry in the FAT points to the next block. With the linked list approach, we also cannot get contiguous space easily

6. 11.6 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Implementing the Bit-Mapped Approach Clearly, we need to protect these structures very closely. : We can implement the bit-mapped approach with a pointer to this free list Bit map itself should be kept on disk – as we have mentioned. Must ensure that the copy in memory and disk do not differ.  Cannot allow for block[i] to have a situation where bit[i] = 1 in memory and bit[i] = 0 on disk Solution:  Set bit[i] = 1 in disk  Allocate block[i]  Set bit[i] = 1 in memory

7. 11.7 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Implementation of Linked List and Similar Schemes Implementation of Linked List and Similar Schemes First Block: Maintain pointer to first free block in memory (preferably in cache) simple to program time-consuming to execute Each link points to next link Grouping – Store addresses of the first n free blocks in the first free block, where the first n-1 blocks are actually free, but where the last block contains addresses of another n free blocks. This greatly improves performance over the standard linked list. Counting – we often ‘know’ several contiguous blocks may be allocated / freed at the same time – particularly when space is allocated using a contiguous allocation scheme or via clustering. If this is the case, we can keep the address of the first free block and the number of free contiguous blocks that follow the first block. As a result, the free space list could contains a disk address and a count of blocks ‘starting’ at that spot – thus shortening the list considerably. Perhaps you can see problems with this. Consider the following:

8. 11.8 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Linked Free Space List on Disk Shows that address of first free block points to the disk area, where other blocks are linked from one to the next. I think this begs the question: Are we generating fragments when we keep track of available disk space this way? This takes us back to worst fit, best fit, first fit, etc….

9. 11.9 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Recovery I cannot say enough about Recovery. It is absolutely essential in any non-trivial computing environment. Recover is oftentimes easy (but time consuming), and Disks do fail for any number of reasons – power losses, surges, bad sectors on disk, dust, weather, and even malicious intent. There is nothing more sacred in the world of computers than our data. Programs can usually be reproduced, people replaced; but safeguarding our data and having it consistent is critical. Backup and Recovery – two topics – are often not covered in detail in academic environments. But rest assured, in a production environment, these activities and procedures constitute a daily activity and involve planning and established procedures. I cannot emphasize these topics enough!

10. 11.10 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Consistency Checking – Recovery During operations, directory information is kept both in memory and on disk – and it is usually more ‘current’ in memory. If the directory is kept in cache, it is usually not written back to memory every time it is updated. This would negate the performance gains of cache. Systems can and do crash at the absolute worst times. When this happens, files, directories, caches, buffers, etc. can easily be left in a very inconsistent state. Operating systems such as Unix and MS-DOS provide systems programs to run consistency checks when needed. These compare directory data with data blocks on disk and attempt to repair any inconsistencies these programs may find.

11. 11.11 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Consistency Checking – Recovery Types of Allocation are the Key: The degree of success in running these system-supplied programs is largely dependent upon the type of allocation (contiguous, linked, indexed) and the implementation of the free-space management routines and structures used to allocate disk. Sometimes broken links can be repaired; sometimes not.  Loss of directory entries in an indexed organization can be disastrous, because the blocks are linked from one to the next.  If link is broken, we have big trouble. Can easily lose a subtree! Interestingly, Unix caches directory entries for reads but any writes that cause any kind of space allocation are done synchronously. This simply means that the allocation of space is completed before the write takes place. Can still have a problem if crash occurs during this process too. But we always try to minimize and localize any problems.

12. 11.12 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Backup and Restore As mentioned in previous materials, all viable computing environments back up data from disk periodically to other media – either other disks, mag tapes, etc. We can then restore from these backups if needed. Often times directory information is used to develop the backup. For files / directories not changed since last backup, backup not needed. Scheduling Backups: depends on lots of things… May do periodic full backups. May do incremental backups – all files / directories changed usually overnight. May simply copy to another medium all files changed since “day n.” Since the volume of data is often large, files may be blocked big time! Often too, backed up is done to very low cost, high volume tapes.

13. 11.13 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Backup and Restore We also have combinations of these: Every so often a full backup. More frequently, an incremental backup. A restore can start restoring from the last full backup and then adding the incremental backups. But there are always procedures in place within computer operations that specify back up and recover procedures.

14. 11.14 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Log Structured File Systems – Motivation (1 of 2) Typically major database systems use what are termed: log-based recovery algorithms as part of their environment and operation. As it turns out, we use the same techniques as part of our consistency checking approach for backup and recovery when needed. As usual, the recovery process is interested in capturing all common operations that would take place in, say, creating a file: a number of very basic operations, and changes to several key data structures, that are used to maintain the integrity of the file / database.

15. 11.15 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Log Structured File Systems – Motivation (2 of 2) Certainly in creating a database or file: directory structure contents will be modified file control blocks (or inodes) will be allocated and descriptions developed, data itself are modified and corresponding data structures used to house free block counts (bit maps; linked lists, etc.) will be modified. All or some of these data structures can very easily be corrupted if a system crash occurs somewhere in this process – before these structures can be made to be consistent.

16. 11.16 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Recovery in Log-Structured File Systems We have spoken about backup and recovery and special system programs & procedures generally available to assist in re-establishing files and directories that are both consistent and current. It is important to note that in some cases, problems (inconsistencies) may not be recoverable. Sometimes human intervention is required and the system may simply be unavailable until a special recovery of some sort is established. Often some of these recoveries can be both automated and manual.

17. 11.17 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Recovery in Log-Structured File Systems One solution is to incorporate a log-based recovery approach, which captures changes written sequentially to a log. Some systems call this ‘journaling.’ It often takes the form of writing: Old transaction: New transaction… to some backup medium. Each set of operations for performing a specific task is usually called a transaction. Once changes are written to such a log, they are considered ‘committed, and the system call that writes these changes to the log may then return to the user process. But be careful: The problem is that the file system itself may not be updated yet (and normally isn’t) as updates to the file itself take place asynchronously. Once a committed transaction is completed, it can be removed from the log file. (It is recommended that the log be separated from the file system itself and perhaps on a different drive – in case the drive goes down.)

18. 11.18 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Restoring Consistency If the file system crashes, all remaining transactions in the log (committed but likely not updated) must still be accommodated, if any are present. Even though they were committed by the operating system, they may likely have not been effected in the file itself. Difficulties arise if the transaction itself was aborted before the system crashed or worse, right in the middle of completing the write to the file system. A partial ‘completion’ to the file system must be undone in order to arrive at a consistent data point.

19. 11.19 Silberschatz, Galvin and Gagne ©2005 Operating System Concepts Practical Considerations It is easy to see that these types of circumstances can be nightmares. The reality is that they do occur and, of course, at the worst possible times. Using reliable data structures coupled with carefully designed procedures – both automated and manual – are often needed to recover files and databases. If all goes well and we have periodic backups within a 24 hour period (full, incremental, specific applications,or whathaveyou) computer operations may be smooth running. But these nightmares can and do occur!

20. End of Chapter 11.3