1. COSC 3407: Operating Systems Lecture 18: Disk Management and File System

2. This lecture… u Implementing file system abstraction u Comparison among disk management algorithms

3. File System Components u Disk management: how to arrange collection of disk blocks into files u Naming: user gives file name, not track 50, platter 5, etc. u Protection: keep information secure u Reliability/durability: when system crashes, lose stuff in memory, but want files to be durable. Physical Reality File System Abstraction Block oriented Byte oriented Physical sector #’s Named files No protection Users protected from each other Data might be corrupted Robust to machine failures if machine crashes

4. User vs. System View of a File u User’s view: – Durable data structures – executable com file (static data region, relocation table, code, etc.) – Memory-mapped files: operations – read/write to mem – Serialization (also pointer swizzling, marshalling) u Systems’ view (system call interface): – Collection of bytes (UNIX) u System’s view (inside OS): – Collection of blocks – a block is a logical transfer unit, while a sector is the physical transfer unit. – Block size >= sector size; – in UNIX, block size is 4KB.

5. Translating from user to system view u What happens if user says: give me bytes 2 – 12? – a. Fetch block corresponding to those bytes – b. Return just the correct portion of the block u What about: write bytes 2 – 12? – a. Fetch block – b. Modify portion – c. Write out block u Everything inside file system is in whole size blocks. u For example, getc, putc => buffers 4096 bytes, even if interface is one byte at a time. u From now on, file is collection of blocks.

6. File Access Methods 1. Sequential access – – bytes read in order (give me the next X bytes, then give me next) – Almost all file access are of this flavor – example: editor writes out new file, compiler reads in file, etc. 2. Random access – – read/write element out of middle of array (give me bytes i – j) – Less frequent, but still important. For example, virtual memory backing file: page of memory stored in file – Want this to be fast – don’t want to have to read all bytes to get to the middle of the file – examples: data set for demand paging, databases 3. Content-based access – – “find me 100 bytes starting with “TOM” – Example: employee records – once you find the bytes, increase my salary by a factor of 2 – Many file systems don’t provide this; instead, database is built on top to index content (requires efficient random access)

7. File Usage Patterns 1. Most files are small (for example,.login,.c files) 2. Large files use up most of the disk space 3. Large files account for most of the bytes transferred to/from disk u Bad news: need everything to be efficient. – Need small files to be efficient, since lots of them. – Need large files to be efficient, since most of the disk space, most of the I/O due to them

8. Disk Management Policies u Basic entities on a disk: – File: user-visible group of blocks arranged sequentially in logical space – Directory: user-visible index mapping names to files u Access disk as linear array of blocks. Two Options: – Identify blocks as vectors [cylinder, surface, sector]. » Blocks numbered in cylinder-major order, so that adjacent numbered blocks can be accessed without seeks or rotational delay. » Track 0, surface 0, sector 0, 1,... | surface 1, sector 0, 1... |... | track 1, surface 1, sector 0... » Not used much anymore. 1 0 2 3 7 8 9 10

9. Disk Management Policies – Logical Block Addressing (LBA). Every block has integer address from zero up to max number of cylinders. – Controller translates from address  physical position » First case: OS/BIOS must deal with bad blocks » Second case: hardware shields OS from structure of disk u Need way to track free disk blocks – Link free blocks together  too slow today – Use bitmap to represent free space on disk u Need way to structure files: File Header (file descriptor) – which blocks are associated with each file within the logical file structure – Optimize placement of files disk blocks to match access and usage patterns

10. Disk Management Policies u Caching: every OS today keeps a cache of recently used disk blocks in memory, to avoid having to go to disk. u Common to all organizations. u For now, assume no cache, and add it later. u How do we organize file on disk? u Goals: – Maximize sequential performance – Easy random access to file – Easy management of file (growth, truncation, etc)

11. Contiguous allocation u User says in advance how big file will be (disadvantage) u Search bit map (using best fit/first fit) to locate space for file u File header (or file descriptor) contains: – First sector/LBA in file – File size (# of sectors) u Pros & cons: – + Fast sequential access – + Easy random access – – External fragmentation – – Hard to grow files u Example: IBM OS/360 file a (base=1,len=3) file b (base=5,len=2) what happens if file c needs 2 sectors???

12. Linked files u Each block, pointer to next on disk (Alto, TOPS-10, DOS FAT) u File header: pointer to first block on disk u Pros & Cons: – + Can grow files dynamically – + Free list managed same as file – – Sequential access: seek between each block – – – – Random access: horrible – – Unreliable (lose block, lose rest of file) file a (base=1) file b (base=5) how do you find the last block in a?

13. Example: DOS FS (simplified) u Used linked list; but instead of embedding links in pages, they used a separate structure called the File Allocation Table (FAT). u The FAT has an entry for each block on the disk and the entries corresponding to the blocks of a particular file are linked up. free FAT (16-bit entries) eof 1 3 4... Directory (5) a: 6 b: 2 file a 6 4 3 file b 2 1 0 1 2 3 4 5 6

14. FAT discussion u Space overhead of FAT is trivial: – 2 bytes / 512 byte block = ~.4% (Compare to Unix) u Reliability: how to protect against errors? – Create duplicate copies of FAT on disk. – State duplication a very common theme in reliability u Bootstrapping: where is root directory? – Fixed location on disk: FAT (opt) FAT root dir … FAT (MS-DOS)

15. Indexed files (Nachos, VMS) u User declares max file size; u system allocates a file header to hold an array of pointers big enough to point to file size number of blocks. u Pros & Cons: – + Can easily grow up to space allocated for descriptor – + Random access is fast – – Clumsy to grow file bigger than table size – – Still lots of seeks: blocks can be spread all over the disk, so sequential access is slow. File header Null Disk blocks

16. Multilevel indexed (UNIX 4.1) u Key idea: efficient for small files, but still allow big files u File header contains 13 pointers – fixed size table, pointers not all equivalent – the header is called an “inode” in UNIX u First 10 are pointers to data blocks. – If file is small enough, some pointers will be NULL.

17. Multilevel indexed (UNIX 4.1) File header Disk blocks 1 2 3 11 12 13 1 2 11 256 266 256 267 256

18. Multilevel indexed (UNIX 4.1) u What if you allocate 11th block? – Pointer to an indirect block – a block of pointers to data blocks. – Gives us 256 blocks, + 10 (from file header) = 1/4 MB u What if you allocate a 267th block? – Pointer to a doubly indirect block – a block of pointers to indirect blocks (in turn, block of pointers to data blocks). – Gives us about 64K blocks => 64MB u What if want a file bigger than this? – One last pointer – what should it point to? – Instead, pointer to triply indirect block – block of pointers to doubly indirect blocks (which are...)

19. Multilevel indexed (UNIX 4.1) u Thus, file header is: – First 10 data block pointers – point to one block each, so 10 blocks – 11 indirect block pointer – points to 256 blocks – 12 doubly indirect block pointer – points to 64K blocks – 13 triply indirect block pointer – points to 16M blocks 1. Bad news: Still an upper limit on file size ~ 16 GB. 2. Pointers get filled in dynamically: need to allocate indirect block only when file grows > 10 blocks. – If small file, no indirection needed. 3. How many disk accesses to reach block #23? (2)

20. Multilevel indexed (UNIX 4.1) u How about block # 5? (1) u How about block # 340? (3) u UNIX Pros & Cons: – + Simple (more or less) – + Files can easily expand (up to a point) – + Small files particularly cheap and easy – – Very large files, spend lots of time reading indirect blocks – – Lots of seeks

21. DEMOS u OS for Cray-1, mid to late 70’s. u File system approach corresponds to segmentation. u Cray-1 had 12 ns cycle time, so CPU:disk speed ratio about the same as today (a few million instructions = 1 seek). u Idea: reduce disk seeks by using contiguous allocation in normal case, but allow flexibility to have non- contiguous allocation.

22. DEMOS u File header: table of base & size (10 “block group” pointers) u Each “block group” – a contiguous region of blocks u Are 10 block group pointers enough? – No. If need more than 10 block groups, set flag in file header: BIGFILE. Base size On disk File header

23. DEMOS u Each table entry now points to an indirect block group – a block group of pointers to block groups. u Can get huge files this way: Suppose 1000 blocks in a block group, each block 8K size, each entry 8 byte => (10 ptrs  1024 groups/ptr  1000 blocks/group)*8K =80 GB max file size u How do you find an available block group? – Use bit map to find block of 0’s Base size File header On disk Indirect block group Block group

24. DEMOS u Pros & cons: – + Fast sequential access – + Easy to find free block groups – + Free areas merge automatically – – When disk fills up: » a. No long runs of blocks (fragmentation) » b. High CPU overhead to find free block u Solution: – Don’t let disk get full – keep portion in reserve – Free count = # blocks free in bitmap. – Normally, don’t even try allocate if free count = 0. – Change this to: don’t allocate if free count < reserve u Why do this? – Tradeoff: pay for more disk space, get contiguous allocation u How much of a reserve do you need? – In practice, 10% seems like enough.

25. UNIX BSD 4.2 u Exactly the same as BSD 4.1 (same file header and triply indirect blocks), except incorporated some ideas from DEMOS: – Uses bitmap allocation in place of free list – Attempt to allocate files contiguously – 10% reserved disk space – Skip sector positioning u Problem: when you create a file, don’t know how big it will become (in UNIX, most writes are by appending to the file). u So how much contiguous space do you allocate for a file, when it’s created?

26. UNIX BSD 4.2 u In Demos, power of 2 growth: once it grows past 1 MB, allocate 2MB, etc. u In BSD 4.2, just find some range of free blocks, put each new file at the front of a different range. u When need to expand a file, you first try successive blocks in bitmap, then choose new range of blocks. u Also, rotational delay can cause a problem, even with sequential files.

27. Dealing with rotational delay u Read one block, do processing, and read next block. u In the meantime, disk has continued turning: missed next block! Need 1 revolution/block! u If have to wait for entire rotation, problem! – Go from reading at disk bandwidth, to reading one sector every rotation. Skip Sector Track Buffer (Holds complete track)

28. Dealing with rotational delay u Solution 1 – Skip sector positioning (“interleaving:) » Place the blocks from one file on every other block of a track: give time for processing to overlap rotation u Solution 2 – Read ahead/disk track buffers – read next block right after first, even if application hasn’t asked for it yet. » This could be done either by OS (read ahead) or by disk itself (track buffers). » Many disk controllers have internal RAM that allows them to read a complete track u Important Aside: Modern disks+controllers do many complex things “under the covers” – Track buffers, elevator algorithms, bad block filtering

29. How do we actually access files? u All information about a file contained in its file header – UNIX calls this an “inode” » Inodes are global resources identified by index (“inumber”) – Once you load the header structure, all the other blocks of the file are locatable u Question: how does the user ask for a particular file? – One option: user specifies an inode by a number (index). » Imagine: open(“14553344”) – Better option: specify by textual name » Have to map name  inumber – Another option: Icon » This is how Apple made its money. Graphical user interfaces. Point to a file and click.

30. How do we actually access files? u Naming: The process by which a system translates from user-visible names to system resources – In the case of files, need to translate from strings (textual names) or icons to inumbers/inodes – For global file systems, data may be spread over globe  need to translate from strings or icons to some combination of physical server location and inumber

31. Directories u Hierarchical name space: Files named by ordered set (e.g.: /programs/p/list)

32. Directories u Directories: a special type of relation – Just a table of (file name, inumber) pairs – Question: how is the relation stored? » Directories often stored just like files » Can store inumber for directories or files in other directories – Question: how is the directory structured? » Needs to be quickly searchable!