1. cs431-cotter1 File Systems Tanenbaum Chapter 4 Silberschatz Chapters 10, 11, 12

2. cs431-cotter2 Essential requirements for long-term information storage: It must be possible to store a very large amount of information. The information must survive the termination of the process using it. Multiple processes must be able to access the information concurrently. File Systems Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

3. cs431-cotter3 File Structure None: –File can be a sequence of words or bytes Simple record structure: –Lines –Fixed Length –Variable Length Complex Structure: –Formatted documents –Relocatable load files Who decides?

4. cs431-cotter4 Think of a disk as a linear sequence of fixed-size blocks and supporting reading and writing of blocks. Questions that quickly arise: How do you find information? How do you keep one user from reading another’s data? How do you know which blocks are free? File Systems Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

5. cs431-cotter5 Figure 4-1. Some typical file extensions. File Naming Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

6. cs431-cotter6 File Access Methods Sequential Access –Based on a magnetic tape model –read next, write next –reset Direct Access –Based on fixed length logical records –read n, write n –position to n –relative or absolute block numbers

7. cs431-cotter7 Figure 4-2. Three kinds of files. (a) Byte sequence. (b) Record sequence. (c) Tree. File Structure Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

8. cs431-cotter8 Figure 4-3. (a) An executable file. (b) An archive. File Types Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

9. cs431-cotter9 Figure 4-4a. Some possible file attributes. File Attributes Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

10. cs431-cotter10 The most common system calls relating to files: File Operations Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 Append Seek Get Attributes Set Attributes Rename Create Delete Open Close Read Write

11. cs431-cotter11 Figure 4-5. A simple program to copy a file. Example Program Using File System Calls (1) Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639...

12. cs431-cotter12 Figure 4-5. A simple program to copy a file. Example Program Using File System Calls (2) Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

13. cs431-cotter13 Directory Structure Collection of nodes containing information on all files F1 F2 F3 F4 F5

14. cs431-cotter14 Information in a Device Directory File name: File Type: Address: Current Length Maximum Length Date Last accessed (for archiving) Date Last updated (for dumping) Owner ID Protection information

15. cs431-cotter15 Directory Operations Search for a file Create a file Delete a file List a directory Rename a file Traverse the file system

16. cs431-cotter16 Objectives for a Directory System Make it efficient –It should be easy to locate a file quickly Make file (and directory) naming convenient –Allow 2 users to have the same name for different files –Allow the same file to have more than 1 name Allow logical grouping of files –All word processing files together –All c++ files together –etc.

17. cs431-cotter17 Alternative Directory Structures Single-Level Directory Issues: –Naming –Grouping cat bo a test data mail cont hex word calc

18. cs431-cotter18 Alternative Directory Structures Two-Level Directory User1 User2 User3

19. cs431-cotter19 Tree-Structured Directory

20. cs431-cotter20 Figure 4-8. A UNIX directory tree. Path Names Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

21. cs431-cotter21 File System Structure A data structure on a disk that holds files Disk Storage attributes: –Allows direct access to any given block of information –Supports re-writing in place (copy block, modify, rewrite) Organized in Layers: –Application Programs –Logical file system –file-organization module –basic file system –I/O control –devices

22. cs431-cotter22 File System Organization Devices –Peripheral equipment- disk drives, tapes, etc. I/O Control –device drivers and interrupt handlers –Communicates directly with peripheral devices –Starts and closes out I/O request Basic File System –Requires knowledge of physical device (track, cylinder, head, etc.) –Manages data transfer at block level (buffering, placement)

23. cs431-cotter23 File System Organization File-organization Module –Maps logical block structure of file to physical block structure of disk –Includes free space manager Logical file system –Manages file directory information –Provides security, protection Applications –Maintains basic data about the file –Allows users to access file information

24. cs431-cotter24 File System Structure Access to File –file control block (generic) –file descriptor (UNIX) –file handle (Windows) File System Requirements –boot mechanism –file system information –file information –file data

25. cs431-cotter25 Allocation Methods Contiguous Allocation Each file occupies a set of contiguous blocks on the disk. Number of blocks needed identified at file creation –May be increased using file extensions Advantages: –Simple to implement –Good for random access of data Disadvantages –Files cannot grow –Wastes space

26. cs431-cotter26 Contiguous Allocation 0123 4 56789 10 11 12 1314 15 1617 1819 20 212223 24 25 26 27 28 29 30 31 3233 34 FileA FileB FileC FileE FileD File Allocation Table File NameStart BlockLength FileA FileB FileC FileD FileE 23 95 188 302 263 FileA

27. cs431-cotter27 Allocation Methods Linked Allocation Each file consists of a linked list of disk blocks. Advantages: –Simple to use (only need a starting address) –Good use of free space Disadvantages: –Random Access is difficult ptrdataptrdataptrdata Null data

28. cs431-cotter28 Linked Allocation 01234 56789 1011121314 1516171819 2021222324 2526272829 3031323334 FileB File Allocation Table File NameStart BlockEnd... FileB28... 1

29. cs431-cotter29 Linked Allocation 01234 56789 1011121314 1516171819 2021222324 2526272829 3031323334 FileB File Allocation Table File NameStart BlockEnd... FileB 28... 1

30. cs431-cotter30 Allocation Methods Indexed Allocation Collect all block pointers into an index block. Advantages: –Random Access is easy –No external fragmentation Disadvantages –Overhead of index block Index Table

31. cs431-cotter31 Indexed Allocation 1 8 3 14 28 01234 56789 1011121314 1516171819 2021222324 2526272829 3031323334 File Allocation Table File Name Index Block Jeep24

32. cs431-cotter32 Indexed Allocation 1 8 3 14 28 01 2 34 56789 1011121314 1516171819 20212223 24 2526272829 3031323334 File Allocation Table File Name Index Block Jeep24

33. cs431-cotter33 UNIX File System File Types Regular File Directory Block-oriented Device Character-oriented Device Symbolic Link

34. cs431-cotter34 UNIX File Ownership read write execute owner group others 1 0 00 00 1 1 1 (for directories: read, write, search) -rwx------ 1 rsmith is 785 Feb 11 1994 send_exe drwxr-xr-x 2 rsmith is 512 Apr 16 13:49 sysmon_server

35. cs431-cotter35 File Access Model open -- read -- write -- close fdesc = open (*filename, r / w / a / +); n = read (fdesc, buff, 24); n = write (fdesc, buff, 24); lseek (fdesc, 100L, L_SET); (L_SET, L_CUR, L_END) close (fdesc);

36. cs431-cotter36 Concurrent File Access From separate processes –OPEN generates a new file descriptor. –Allows independent access to the same file. –No inherent blocking in access From parent / child processes –fork duplicates all active file descriptors. –all processes access a common “current” pointer File locking –flock, lockf

37. cs431-cotter37 UNIX File Sharing - unrelated processes - fd flags ptr fd 0 process table entry fd flags ptr fd 0 process table entry File Table file status flags current file offset v-node ptr file status flags current file offset v-node ptr V-node Table v-node info I-node info file size

38. cs431-cotter38 UNIX File Sharing - related processes - fd flags ptr fd 0 process table entry fd flags ptr fd 0 process table entry file status flags current file offset v-node ptr File Table file status flags current file offset v-node ptr V-node Table v-node info I-node info file size

39. cs431-cotter39 direct blocks UNIX i-node mode owners(2) timestamps(3) size block count single indir triple indir double indir data

40. cs431-cotter40 direct blocks UNIX i-node mode owners(2) timestamps(3) size block count single indir triple indir double indir data ::::

41. cs431-cotter41 direct blocks UNIX i-node mode owners(2) timestamps(3) size block count single indir triple indir double indir data :::: :::: :::: ::::

42. cs431-cotter42 I-node File Sizes 1 KB block2KB block4KB block Direct12 k bytes24 k bytes48 k bytes Single indirect 256 k bytes1 M bytes4 M bytes Double indirect 64 M bytes512 M bytes 2 G bytes Triple indirect 16 G bytes256 G bytes 2 T bytes

43. cs431-cotter43 Figure 4-20. Percentage of files smaller than a given size (in bytes). Disk Space Management Block Size (1) Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

44. cs431-cotter44 Disk Drive Layout (simplified) Disk Drive partition I-list directory blocks and data blocks I-node Block Group Partition Block group

45. cs431-cotter45 Sample filesystem: Creating the directory “testdir” i-node 1267 i-node 2549 i-list directory block

46. cs431-cotter46 i-node 1267 i-node 2549 i-list directory block 1267 Sample filesystem: Creating the directory “testdir”

47. cs431-cotter47 i-node 1267 i-node 2549 i-list directory block 2549testdir 1267 Sample filesystem: Creating the directory “testdir”

48. cs431-cotter48 i-node 1267 i-node 2549 i-list directory block 2549testdir 1267 Sample filesystem: Creating the directory “testdir”

49. cs431-cotter49 i-node 1267 i-node 2549 i-list directory block directory block 2549 1267 2549testdir 1267 Sample filesystem: Creating the directory “testdir”

50. cs431-cotter50 Sample filesystem after creating the directory “testdir” i-node 1267 i-node 2549 i-list directory block directory block 2549 1267 2549testdir 1267

51. cs431-cotter51 Viewing directory structure ls –ai / –2. –: –1730home ls –ai /home –1730. –2.. –: –199rcotter ls –ai /home/rcotter –199. –1730.. –: –241224cs431 ls –ai ~/cs431 –241224. –199.. –: –245436exec.cpp

52. cs431-cotter52 Shared Files (1) Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639 Figure 4-16. File system containing a shared file. ln –s shared_file new_link

53. cs431-cotter53 Figure 4-17. (a) Situation prior to linking. (b) After the link is created. (c) After the original owner removes the file. Shared Files (2) Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

54. cs431-cotter54 Linux File System Structure Linux uses a Virtual File System (VFS) –Defines a file object –Provides an interface to manipulate that object Designed around OO principles –File system object –File object –Inode object (index node) Primary File System - ext2fs –Supports (or maps) several other systems (MSDOS, NFS (network drives), VFAT (W95), HPFS (OS/2), etc.

55. cs431-cotter55 Virtual Filesystem A kernel software layer that handles all system calls related to a standard UNIX filesystem. Supports: –Disk-based filesystems IDE Hard drives (UNIX, LINUX, SMB, etc.) SCSI Hard drives floppy drives –Network filesystems remotely connected filesystems –Special filesystems /proc

56. cs431-cotter56 VF Example inf = open (“/floppy/test”, O_RDONLY, 0); outf = open (“/tmp/test”, O_WRONLY|O_CREATE|O_TRUNC, 0600); do { cnt = read(inf, buf, 4096); write (outf, buf, cnt); } while (cnt); close (outf); close (inf);

57. cs431-cotter57 Virtual file system Model Superblock object –metadata about a mounted filesystem inode object –general information about a specific file file object –information about process and open file interaction dentry object –directory entry information

58. cs431-cotter58 Virtual Filesystem and Processes

59. cs431-cotter59 Ext2fs History Minux extfs ext2fs - 1994

60. cs431-cotter60 Ext2fs Blocks 1k default 2k, 4k possible choice based on file sizes

61. cs431-cotter61 Ext2fs Disk Data Structures Each Block Group contains: –A copy of the Super Block –A copy of the group of block group descriptors –A data block bitmap (for this group) –An inode bitmap (for this group) –A group of inodes (for this group) –Data blocks Boot Block Block Group 0Block Group N Super Block Group Descriptors Data block Bitmap Inode Bitmap Inode Table Data Blocks

62. cs431-cotter62 Disk Data Structures SuperBlock –Contains metadata about the block group –# free blocks, –# free inodes, –block size, –times (last mount, last write), –check status, –blocks to pre-allocate, –alignments, etc. (~38 items)

63. cs431-cotter63 Disk Data Structures Block Groups –Block group limited to 8 * block size by data block bitmap. –1k block limited to 8,192 blocks or 8 mbytes per block group. 4k block = 128 mbytes Group descriptor - 24 bytes per group –block numbers for bitmaps, start of tables, etc.

64. cs431-cotter64 Disk Data Structures Inode Table –Inodes - 128 bytes –File type and access rights –owner –length –timestamps (last access, last change, etc.) –hard links counter –pointers to data blocks, –etc.

65. cs431-cotter65 Disk Data Structures Items cached in memory on filesystem mount –Superblock - always cached –Group descriptor - always cached –Block bitmap - fixed limit - most recent only –Inode bitmap - fixed limit - most recent only –Inode - dynamic –Data block - dynamic

66. cs431-cotter66 File Blocks Grouped Physically and logically Inodes distributed throughout disk Data blocks allocated close to inode Pre-allocation of free blocks to minimize frag.

67. cs431-cotter67 Summary Features Fast Symbolic Links pre-allocation of data blocks file-updating strategy immutable files or append only files

68. cs431-cotter68 Journal File Systems Designed to speed file system recovery from system crashes. Traditional systems use an fs recovery tool (e.g. fsck) to verify system integrity As file system grows, recovery time grows. Journal File Systems track changes in meta-data to allow rapid recovery from crashes

69. cs431-cotter69 Journal File System Uses a version of a transaction log to track changes to file system directory records. If a system crash occurs, the transaction log can be reviewed back to a check point to verify any pending work (either undo or redo). For large systems, recovery time goes from hours or days to seconds.

70. cs431-cotter70 Journal File System - Examples XFS –Commercial port by SGI (IRIX OS) –Based on 64 bit system (ported to 32 bit) –Supports large systems 2TB -> 9 million TB –Uses B+trees for improved performance –Journals file system meta-data –Supports Quotas, ACLs. –Supports filesystem extents (contiguous blocks)

71. cs431-cotter71 Journal File System - Examples JFS –Commercial Port by IBM. Used in OS/2 –64 bit system ported to 32 bits. –Journal support of meta-data. –System Size 2TB -> 32PB –Built to scale on SMP architectures –Uses B+trees for improved performance –Supports use of extents.

72. cs431-cotter72 Journal File System - Examples ReiserFS –Designed originally for Linux (32 bit system) –Supports file systems to 2TB -> 16TB –Journal support of meta-data –Btree structure supports large file counts –Supports block packing for small files –No fixed inode allocation - more flexible.

73. cs431-cotter73 Journal File System - Examples Ext3fs –Simple extension of ext2fs that adds journaling –All virtual file system operations are journaled. –Other limitations of ext2fs remain.

74. cs431-cotter74 Journal File System - Examples ext4fs –Next generation of file system development, incorporating improvements and changes to ext3fs. –Journalling filesystem Improvements is journal checksumming –Larger file systems – 1 MTB (exabyte) Larger files – 16 TB –Delayed block allocation, multi-block allocator Allows files to be written as contiguous sequences of blocks: improves read performance of streaming data files. Poses the risk of lost data if system crashes before data is written. –Use of extents Large (<= 128 MB) contiguous physical blocks.

75. cs431-cotter75 Free Space Management Bit Vector management One bit for each block –0 = free; 1 = occupied Use bit manipulation commands to find free block Bit vector requires space –block size = 4096 = 2 12 –disk size = 1 gigabyte = 2 30 –bits = 2 (30-12) = 2 18 = 32k bytes 01.....110

76. cs431-cotter76 Free Space Management Bit vector (advantages): –Easy to find contiguous blocks Bit vector (disadvantages): –Wastes space (bits allocated to unavailable blocks) Issues: –Must keep bit vector on disk (reliability) –Must keep bit vector in memory (speed) –cannot allow memory != disk (memory = 1, disk = 0)

77. cs431-cotter77 Free Space Management Linked List management –Use linked list to identify free space Advantages: –no wasted space Disadvantages: –harder to identify contiguous space. Issues: –Must protect pointer to free list

78. cs431-cotter78 Free Space Management Grouping of blocks Boot block File system descriptor File descriptors

79. cs431-cotter79 Figure 4-29. (a) I-nodes placed at the start of the disk. (b) Disk divided into cylinder groups, each with its own blocks and i-nodes. Reducing Disk Arm Motion Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

80. cs431-cotter80 Figure 4-22. (a) Storing the free list on a linked list. (b) A bitmap. Keeping Track of Free Blocks (1) Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

81. cs431-cotter81 Figure 4-24. Quotas are kept track of on a per-user basis in a quota table. Disk Quotas Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

82. cs431-cotter82 Backups to tape are generally made to handle one of two potential problems: Recover from disaster. Recover from stupidity. File System Backups (1) Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

83. cs431-cotter83 Figure 4-25. A file system to be dumped. Squares are directories, circles are files. Shaded items have been modified since last dump. Each directory and file is labeled by its i-node number. File System Backups (2) Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

84. cs431-cotter84 Figure 4-26. Bitmaps used by the logical dumping algorithm. File System Backups (3) Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

85. cs431-cotter85 Figure 4-27. File system states. (a) Consistent. (b) Missing block. (c) Duplicate block in free list. (d) Duplicate data block. File System Consistency Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

86. cs431-cotter86 Figure 4-28. The buffer cache data structures. Caching (1) Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

87. cs431-cotter87 Some blocks, such as i-node blocks, are rarely referenced two times within a short interval. Consider a modified LRU scheme, taking two factors into account: Is the block likely to be needed again soon? Is the block essential to the consistency of the file system? Caching (2) Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

88. cs431-cotter88 Example File Systems CD-ROM –ISO 9660 –Rock Ridge –Joliet MS-DOS

89. cs431-cotter89 Figure 4-30. The ISO 9660 directory entry. The ISO 9660 File System Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

90. cs431-cotter90 Rock Ridge extension fields: PX - POSIX attributes. PN - Major and minor device numbers. SL - Symbolic link. NM - Alternative name. CL - Child location. PL - Parent location. RE - Relocation. TF - Time stamps. Rock Ridge Extensions Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

91. cs431-cotter91 Joliet extension fields: Long file names. Unicode character set. Directory nesting deeper than eight levels. Directory names with extensions Joliet Extensions Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

92. cs431-cotter92 Figure 4-31. The MS-DOS directory entry. The MS-DOS File System (1) Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

93. cs431-cotter93 Figure 4-32. Maximum partition size for different block sizes. The empty boxes represent forbidden combinations. The MS-DOS File System (2) Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

94. cs431-cotter94 Summary File Structures Directory Structures File System Implementation File System Management Example File Systems

95. cs431-cotter95 Questions Two primary access methods are used to manage files. Discuss these two methods, including which types of physical devices each is normally associated with. What is the difference between a tree structured file directory and an acyclic graph directory? Please give an example Operating System used today that uses each. Discuss 3 ways in which files might be allocated to file stores (disks). What are the advantages or disadvantages of each? How might a file system manage its free space so that it can easily assign space for a new file when needed? Give at least 2 approaches. What can an OS designer to improve the efficiency of a file system? What factors should be considered? How do those affect the efficiency of the system? Discuss at least 2 mechanisms that might be used to ensure the recovery of information stored in a file system.