1. Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials– 8 th Edition Chapter 10: File System Implementation

2. 10.2 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition Chapter 10: File System Implementation File-System Structure File-System Implementation Directory Implementation Allocation Methods Free-Space Management Efficiency and Performance Recovery NFS

3. 10.3 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition Objectives To describe the details of implementing local file systems and directory structures To describe the implementation of remote file systems To discuss block allocation and free-block algorithms and trade-offs

4. 10.4 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition File-System Structure File structure Logical storage unit Named collection of related information that may be manipulated as a unit by operations including open, close, create, destroy, copy, rename, list, etc. File system resides on secondary storage (disks) Provided user interface to storage, mapping logical to physical Provides efficient and convenient access to disk by allowing data to be stored, located, retrieved easily Disk provides in-place rewrite and random access I/O transfers performed in blocks of sectors (usually 512 bytes) File control block – storage structure consisting of information about a file Device driver controls the physical device File system organized into layers

5. 10.5 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition Layered File System

6. 10.6 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition File System Layers I/O control layer consists of device drivers and interrupt handlers to transfer information between main memory and the disk system. Device drivers can be thought of as a translator:  Input : high-level commands, “retrieve block 123”  Output : low-level, hardware-specific instructions Basic file system issues generic commands to the appropriate device driver: To read and write physical blocks on the disk. For example, “read drive1, cylinder 72, track 2, sector 10, into memory location 1060” Also manages memory buffers and caches (allocation, freeing, replacement)  Buffers hold data before the transfer of a disk block can occur.  Caches hold frequently used file-system metadata to improve performance. File-organization module understands files, logical blocks, and physical blocks Translates logical block # to physical block # Free-space manager of the file-organization module manages free space and disk allocation.  Tracks unallocated blocks and provides these blocks to the file-organization module.

7. 10.7 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition File System Layers (Cont.) Logical file system manages metadata information Metadata includes all file-system structure except for the actual data Manages directory structure Translates file name into file number, file handle, location by maintaining file-control blocks, FCB (inodes in most Unix) FCB contains information about the file, including ownership, permissions, and location of the file contents. Also responsible for protection and security Layering structure useful for reducing complexity and redundancy, but adds overhead and can decrease performance Each file system has its own logical file system and file-organization modules. Logical layers can be implemented by any coding method according to OS designer

8. 10.8 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition File-System Implementation We have system calls ( Open() or close() ) at the API level, but how do we implement a file system? Several on-disk and in-memory structures are used to implement a file system File-system operations Boot control block contains info needed by system to boot OS from that volume Needed if volume contains OS, usually first block of volume Volume control block (superblock, master file table) contains volume details Total # of blocks, # of free blocks, block size, free block pointers or array Directory structure organizes the files Names and inode numbers, master file table Per-file File Control Block (FCB) contains many details about the file inode number, permissions, size, dates

9. 10.9 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition A Typical File-Control Block

10. 10.10 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition In-Memory File System Structures The in-memory information is used for both file-system management and performance improvement via caching. Loaded at mount time, updated during file-system operations, and discarded at dismount Mount table storing file system mounts, mount points, file system types Plus buffers hold data blocks from secondary storage Open returns a file handle for subsequent use Data from read eventually copied to specified user process memory address

11. 10.11 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition In-Memory File System Structures and Implementation File system structures provided by operating systems (a) Opening a file (b) Reading a file

12. 10.12 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition Partitions and Mounting Partition can be a volume containing a file system (“cooked”) or raw – just a sequence of blocks with no file system Boot block can point to boot volume or boot loader, set of blocks that contain enough code to know how to load the kernel from the file system Or a boot management program for multi-os booting Root partition contains the OS, other partitions can hold other OSes, other file systems, or be raw Mounted at boot time Other partitions can mount automatically or manually At mount time, file system consistency checked Is all metadata correct?  If not, fix it, try again  If yes, add to mount table, allow access

13. 10.13 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition Virtual File Systems Virtual File Systems (VFS) on Unix provide an object-oriented way of implementing file systems VFS allows the same system call interface (the API) to be used for different types of file systems by allowing transparent access Separates file-system-generic operations from implementation details Implementation can be one of many file systems types, or network file system  Implements vnodes which hold inodes or network file details Then dispatches operation to appropriate file system implementation routines The API is to the VFS interface, rather than any specific type of file system

14. 10.14 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition Schematic View of Virtual File System

15. 10.15 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition Virtual File System Implementation For example, Linux VFS has four object types: inode object : represents individual file file object : represents an open file superblock object : represents entire file system dentry object : represents individual directory entry VFS defines set of operations on the objects that must be implemented Every object has a pointer to a function table  function table has addresses of the actual functions (routines) that implement the defined operations for that particular object

16. 10.16 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition Directory Implementation Linear list of file names with pointer to the data blocks Simple to program Time-consuming to execute  Linear search time  Could keep ordered alphabetically via linked list or use B+ tree Hash Table – linear list with hash data structure Decreases directory search time Collisions – situations where two file names hash to the same location Difficulties are its generally fixed size and the dependence of the hash function on that size Only good if entries are fixed size, or use chained-overflow hash table

17. 10.17 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition Allocation Methods - Contiguous An allocation method refers to how disk blocks are allocated for files: Contiguous allocation – each file occupies set of contiguous blocks Best performance in most cases Simple – only starting location (block #) and length (number of blocks) are required Problems include finding space for file, knowing file size, external fragmentation, need for compaction off-line (downtime) or on-line

18. 10.18 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition Contiguous Allocation of Disk Space

19. 10.19 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition Allocation Methods - Linked Linked allocation – each file is a linked list of disk blocks Directory contains a pointer to the first and last blocks of the file. File ends at nil pointer Each block contains pointer to the next block No compaction, external fragmentation Free space management system called when new block needed Improve efficiency by collect blocks into multiples, called clusters, and allocating clusters rather than blocks.  Improves disk throughput, but increases internal fragmentation Reliability can be a problem Locating a block can take many I/Os and disk seeks because the disk blocks may be scattered anywhere on the disk. FAT (File Allocation Table) as an important variation in MS-DOS and OS/2 A section of disk at the beginning of volume contains FAT,  Each entry for each disk block is indexed by block number Directory and FAT work together FAT entry like a linked list, but results in a significant number of disk head seeks  Cache FAT! New block allocation simple

20. 10.20 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition Linked Allocation

21. 10.21 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition File-Allocation Table

22. 10.22 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition Allocation Methods - Indexed Indexed allocation Linked allocation should track the pointers to the blocks scattered with the blocks all over the disk and must be retrieved in order. Indexed allocation solves by bringing all the pointers together into on location: the index block Each file has its own index block(s) of all the pointers together  An array of disk-block addresses Logical view of an index block index table

23. 10.23 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition Example of Indexed Allocation index block

24. 10.24 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition Indexed Allocation (Cont.) Need index table and waste space Random access Dynamic access without external fragmentation, but have pointer overhead of index block

25. 10.25 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition Combined Scheme: UNIX UFS (4K bytes per block, 32-bit addresses) Note: More index blocks than can be addressed with 32-bit file pointer UNIX file’s inode with 15 pointers of the index block inode for a file in Unix - 12 Direct blocks - 1 Single indirect block - 1 Double indirect block - 1 Triple indirect block

26. 10.26 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition Performance Before selecting an allocation method, we need to determine how the systems will be used. Best method depends on file access type Contiguous allocation requires only one access to get a disk block. Linked allocation by keeping the address of the next block in memory  read the block directly  Fine for sequential access, not random  For direct access, requires i disk reads for accessing ith block Some system: direct-access files by using contiguous allocation and sequential-access files by using linked allocation Declare access type at creation  select either contiguous or linked Indexed allocation is more complex Single block access could require 2 block accesses:  index block reads then data block read Clustering can help improve throughput, reduce CPU overhead

27. 10.27 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition Free-Space Management File system maintains free-space list to track available blocks/clusters (Using term “block” for simplicity) Bit vector or bit map (n blocks) … 012n-1 bit[i] =  1  block[i] free 0  block[i] occupied Calculation of the first free block number from the first non-0 word: (number of bits per word) * (number of 0-value words) + offset of first 1 bit * CPUs have instructions to return offset within word of first “1” bit, the first free block

28. 10.28 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition Free-Space Management (Cont.) Bit map requires extra space to manage free space Example: block size = 4KB = 2 12 bytes disk size = 2 40 bytes (1 terabyte) n = 2 40 /2 12 = 2 28 blocks (or 256 M Block)  2 25 bytes (32 MB) needed for Bit map if clusters of 4 blocks -> 8 MB for Bit map Easy to get contiguous files by looking at bit map Another scheme with linked free-space list for free block list Not efficient to traverse the entire list Cannot get contiguous space easily No waste of space No need to traverse the entire list (if # of free blocks recorded)

29. 10.29 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition Linked Free Space List on Disk

30. 10.30 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition Free-Space Management (Cont.) Grouping Modify linked list to store address of next n-1 free blocks in first free block, plus a pointer to next block that contains free-block-pointers Counting Because space is frequently contiguously used and freed, with contiguous-allocation allocation, extents, or clustering  Keep address of first free block and count of following free blocks  Free space list then has entries containing addresses and counts

31. 10.31 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition Recovery Consistency checking – compares data in directory structure with data blocks on disk, and tries to fix inconsistencies Can be slow and sometimes fails Use system programs to back up data from disk to another storage device (magnetic tape, other magnetic disk, optical) Recover lost file or disk by restoring data from backup

32. 10.32 Silberschatz, Galvin and Gagne ©2011 Operating System Concepts Essentials – 8 th Edition Log Structured File Systems Log structured (or journaling) file systems record each metadata update to the file system as a transaction All transactions are written to a log A transaction is considered committed once it is written to the log (sequentially) Sometimes to a separate device or section of disk However, the file system may not yet be updated The transactions in the log are asynchronously written to the file system structures When the file system structures are modified, the transaction is removed from the log If the file system crashes, all remaining transactions in the log must still be performed Faster recovery from crash, removes chance of inconsistency of metadata