1. © Janice Regan, CMPT 300, May 2007 0 CMPT 300 Introduction to Operating Systems File systems

2. © Janice Regan, CMPT 300, May 2007 1 What is a ‘file’?  A file is  an abstraction to describe stored information.  A logical unit of information created by a process.  A file has (from users point of view)  A name (usually conveys meaning about the contents of the file) May have an extension to denote the type of contents or associated application (that created or uses the file).  Stored information  A size  Information on ownership

3. © Janice Regan, CMPT 300, May 2007 2 File systems  Files should be  Persistent Exists regardless of if processes are using it  shareable between processes  Have a consistent and clearly defined structure (how they are stored) This is important to the OS that manages the files

4. File naming  Naming rules are dependent on the OS.  Naming rules specify  Characters legal within the name  Maximum / minimum number of characters  Whether name is case sensitive  Structure of name (are there extensions, etc.)  Are file names fixed length or variable length  … © Janice Regan, CMPT 300, May 2007 3

5. 4 Describing file structure  Field: data element  may have fixed length or variable length  Record: a collection of related fields  fixed or variable length  Size or number of fields may vary  File: a series of bytes OR a collection of similar records  Treated as a single entity, access given to file not part of file  May be created or destroyed  Reference using a ‘name’ and possibly an ‘extension’  Most modern Oses use series of bytes model (provides flexibility)  Database: a collection of related records

6. File types  Most OSs support more than one type of file  Regular files that contain user information in either ASCII or binary formats. May have internal structure inside the list of bytes making up the file.  Directory files (directories) used by the system to manage and maintain the file structure  Special files: character files to model serial I/O devices, block files to manage disks  Some OSs have strongly typed files, makes doing unorthodox things difficult © Janice Regan, CMPT 300, May 2007 5

7. 6 Operations on files  To be useful a files will need to be accessed.  Provide functions to access each file  Create, Delete  Open, Close  Read, Write, Seek  Get / Set attributes (permissions, access history)  The details of how to find the file and how to access the file should be hidden from the user (as part of the OS)

8. © Janice Regan, CMPT 300, May 2007 7 Reading files  Users may want to  Read/Write entire file  Read/Write next record/byte in file (sequential access)  Remove next record/byte in file  Insert a new record/byte in the file  Read/Write a particular record/byte or group of records/bytes in the file (random access using Seek)  Difficultly of implementing these operations may depend on the organization of the file (e.g. fixed vs variable length records vs byte stream) and the organization of the file system

9. File attributes  Information describing properties of the file  Writable?  Owner, Creator, who may access  Sequential or Random access allowed  Record structure (if any)  File locked? File hidden? File a system file?  ASCII, binary, application specific binary  When was file created, last modified, last used © Janice Regan, CMPT 300, May 2007 8

10. 9 File management system  System software to provide I/O services to users  Meet needs of user, access and organize files, providing standardized interface Each user can create, modify, delete their own files and have controlled access to files of other users Each user may control access by others to their files Each user should be able to organize their files for efficient use, and refer to them by symbolic names  Verify validity of files, minimize lost/damaged data  Optimize, were possible, the throughput and system usage for I/O to files  Provide support for a variety of devices

11. © Janice Regan, CMPT 300, May 2007 10 Simplest: Monolithic File organization  A single directory (folder) containing all the files used by the system and the user  Not practical on modern systems  Multiple users may have filename conflicts  Too hard to find files, too many files root myfile1myfileN

12. © Janice Regan, CMPT 300, May 2007 11 User directory: 2 level File organization  A single directory (folder) per user containing all the files used by that user  Prevents file name conflicts between users  Too hard to find files, too many files, filename conflicts for one user may still occur root myfile1myfileN user1 myfile1myfileN User2

13. © Janice Regan, CMPT 300, May 2007 12 Hierarchy: multi level File organization  A directory tree for each user. Each directory  Has exactly one parent. (except root)  Contains any number of subdirectories or files  Each directory in the tree contains files with unique names. (the same names may be reused in other directories)  Prevents file name conflicts between users  Allows each user to organize their files easily root myfile1myfileN user1 myfile1myfileN User2 myfile1myfileN proj1proj31

14. © Janice Regan, CMPT 300, May 2007 13 File management system  System software to provide I/O services to users  Meet needs of user, access and organize files and directories, providing standardized interface Each user can create, modify, delete their own files and directories and have controlled access to files of other users Each user may control access by others to their files Each user should be able to organize their files for efficient use, and refer to them by symbolic names  Verify validity of files, minimize lost/damaged data  Optimize, were possible, the throughput and system usage for I/O to files  Provide support for a variety of devices

15. © Janice Regan, CMPT 300, May 2007 14 Storing a single file  Assume that the file management portion of the OS can provide us with a pointer to the beginning of the first block in a particular file  How should we store the file?  Store the entire file as one contiguous unit  Divide the available space into equal sized blocks Store the file in N consecutive blocks, filling N-1 blocks and partially filling the final block Store the file in N blocks (not required to be consecutive), filling N-1 blocks and partially filling the final block

16. © Janice Regan, CMPT 300, May 2007 15 Issues to consider:  If we append new files to the end of the file system the disk will eventually fill up  After the disk is full (or perhaps earlier) subsequent files must be fitted into spaces left by files that have been deleted

17. © Janice Regan, CMPT 300, May 2007 16 More Issues to consider  If files are stored contiguously (either in one piece or in successive disk blocks)  File size must be know to determine which available spaces (series of empty blocks) are big enough  Large files may not fit into available spaces  Space at between the end of an inserted file and the next file may be too small to be usable (part of a block or too small to hold a file)  Files may not able to grow beyond the number of block originally allocated to them because the following blocks are already occupied by the next file

18. © Janice Regan, CMPT 300, May 2007 17 How to choose location of file  Finding a series of empty blocks to expand existing file or add a new file  Which series of blocks do we choose, if there are multiple possible series of blocks available  First fit: Choose ‘first’ group found (for example next in free list)  Best fit: Choose smallest group that will hold the file  Nearest fit: Choose the series of blocks ‘closest’ (on the physical disk) to the previous allocation.

19. © Janice Regan, CMPT 300, May 2007 18 Internal Fragmentation  Divide the disk storage area into blocks  A block will contain the records or bytes that form part or all of a file  Placing the start of a file 2 after the end of a file 1 in the same block makes increasing file size of the file 1 very difficult. It also means that increasing the size of file 1 requires manipulation of file 2  Internal fragmentation: space wasted at the end of each block  To simplify extension of file 1 and to assure that other files do not need to be manipulated in the process of extending file 1, internal fragmentation is usually allowed. File 2 File 1 File 3 Block 1Block 3Block 2Block 4 Block 6 Block 5

20. © Janice Regan, CMPT 300, May 2007 19 Enlarging a file  To enlarge a file stored contiguously  Find a space to hold the enlarged file  Move the existing file to the space for the enlarged file  If the file is stored in one contiguous unit add the new data at the end of the file.  If the file is stored in contiguous blocks, fill the final block of the copied existing file then add new blocks to the end of the existing file and fill them with the remaining new data  To enlarge a file stored in non contiguous blocks  Fill the final block of the existing file  Add new blocks to hold the remainder of the new data. These new blocks need not be physically adjacent to the existing file on the disk.

21. © Janice Regan, CMPT 300, May 2007 20 File system: contiguous  What happens if file2 grows by two blocks?  Cannot allocate next contiguous block (block 4) because it is already in use  To assure the file is contiguous  Find a large enough series of empty blocks to hold the extended file  must copy existing portion to this series of block than append the new blocks (copy block 3 to block 1, then append new blocks, block 3 is now available) File 2 File 1 File 3 Block 7 Block 3 Block 8 Block 4 Block 6 Block 5 Block 9 Block 10 Block 1 Block 2 Block 0

22. © Janice Regan, CMPT 300, May 2007 21 Contiguous access  Advantages  File can be located using a single pointer to its first block  Need only pointer and number of blocks to retrieve entire file  Disadvantages  Need to know maximum file size at creation and how to find a space to place the file  Difficult and expensive to grow a file Must leave space at the end of each file and/or move as shown in previous slide  Deletion of files may cause available blocks to be fragmented or distributed throughout the disk (external fragmentation) with few or no large blocks of blocks available

23. © Janice Regan, CMPT 300, May 2007 22 Animation  Deletion of files may cause available blocks (white blocks) to be fragmented or distributed throughout the disk (external fragmentation) with few or no large groups of blocks available

24. © Janice Regan, CMPT 300, May 2007 23 External fragmentation  Deletion of files may cause available blocks (white blocks) to be fragmented or distributed throughout the disk (external fragmentation) with few or no large groups of blocks available

25.  What happens if file 1 grows by two blocks?  First fill remainder of empty block, then allocate further blocks (but cannot allocate next contiguous block because it is already in use) © Janice Regan, CMPT 300, May 2007 24 File system: not contiguous File 2 File 1 File 3 Block 1Block 3Block 2Block 4 Block 6 Block 5 Block 7Block 8

26. © Janice Regan, CMPT 300, May 2007 25 File system: not contiguous  No external fragmentation  Files themselves can become fragmented  Not all blocks of a file are contiguous  Less efficient for disk access (more seek time)  Need a mechanism to keep track of which blocks belong to which file  Linked list  Indexed list (File Access Table)

27. © Janice Regan, CMPT 300, May 2007 26 File systems: linked list  Each file is a linked list of blocks  Makes expansion/contraction of file simple  Sequential access of file is efficient  File is still accessed by a single pointer to its first block  Random access of file not efficient Must scan entire list to randomly access last element  Added overhead in each block, data blocks no longer 2 N long  Allows fragmentation of files on the disk which can become significant over time

28. © Janice Regan, CMPT 300, May 2007 27 Linked list File block 1 File block 0 File block 2 File block 3 File block 4 Physical block 26 3392931

29. © Janice Regan, CMPT 300, May 2007 28 File systems: indexed list  Store one pointer for each block of storage in main memory.  Called a file allocation table (FAT)  Windows uses a FAT (FAT12, FAT16, FAT32, each entry in the FAT has 12, 16, or 32 bits)  Does not scale well to large disks FAT12 max 64MB 4080 = 2 12 -16 4KB clusters FAT 16 max 4GB (2GB)≤2 16 -16 4KB clusters FAT 32 max 2TB = 2 28 -16 32KB clusters FAT 32 max 32MB ≤2 28 -16 16KB clusters  A cluster is a group of contiguous sectors

30. © Janice Regan, CMPT 300, May 2007 29 Operation: FAT  Store one pointer for each block of storage in main memory.  The pointer points to the block that follows the block corresponding to the pointer  Entire table must be in memory to make this work, table can be large For FAT32 32MB disk 2 28 entries each 32 bits (4 bytes) 1 MB (a significant fraction of your available memory)  Does not scale well to large disks

31. © Janice Regan, CMPT 300, May 2007 30 Indexed list: file allocation table File block 1 File block 0 File block 2 File block 3 File block 4 Physical block 26 332931 32 FAT 1056 BAD 1024 992 EOF (-1) 928 0 1 24 25 26 27 28 29 30 32 31 34 33 35 Each location in the FAT holds the address of the next block in the file, or an end of file marker For example: The address in the FAT at location 29 holds the address of the next block in the file following block 29, (block 31 with address 992) Starting address 832 1056928992 1024

32. © Janice Regan, CMPT 300, May 2007 31 Indexed Nodes  Indexed Nodes or I-Nodes are used in Unix/Linux systems  Need to keep the I-node structure for open files in memory (not all I-node structures for all files stored on the disk)  Each I-node used 64 bytes, with possible extra storage needed for the address array blocks (for large files)  The memory needed does not increase as the size of the disk increases (as it does for a FAT)

33. © Janice Regan, CMPT 300, May 2007 32 I-Node structure File Information Address block 0 Address block 1 Address block 2 Address block 3 Address block 4 Address block 5 Address block 6 Single indirect Address block 9 Address block 8 Address block 7 Double indirect Triple indirect Address array 1 block of addresses of further blocks ]block 0 block 1 block 2 block 3 block 4 block 5 block 6 block 8 block 7 indirect block 0 indirect block N block 9 Address array 1 block of addresses of further blocks of addresses 0 th Address array 1 block of addresses of further blocks indirect block 0 indirect block N N th Address array 1 block of addresses of further blocks indirect block 0 indirect block N

34. © Janice Regan, CMPT 300, May 2007 33 I-Node structure File Information Address block 0 Address block 1 Address block 2 Address block 3 Address block 4 Address block 5 Address block 6 Single indirect Address block 9 Address block 8 Address block 7 Double indirect Triple indirect Address array 1 block of addresses 0 th Address array indirect block 0 indirect block N N th Address array indirect block 0 indirect block N 0 th Address array indirect block 0 indirect block N N th Address array indirect block 0 indirect block N 0 th Address array N th Address array