2. KT6213 Lecture 14: Operating Systems – File Management & Internal OS Computer Organization and Architecture

3. KT6213 Introduction to Files Collection of (usually related) data Block ◦ Typically 256, 512, or 1024 bytes ◦ Files usually require a 1-block minimum Cluster ◦ Groups of one or more blocks Blocks or clusters correspond to one or more sectors on a disk’s single track or cylinder File extension ◦ Name of file includes identification of what type of file it is File Association ◦ The file specifies which program with which it is to be used

4. KT6213 Terminology Logical view ◦ Contents and attributes of files viewed by the user Physical view ◦ The actual way a file is stored within the computer system Sequential access ◦ Data files whose records always have to be retrieved from the beginning Random or relative access ◦ Data files whose records can be retrieved from anywhere in the file in random sequence Contiguous ◦ The blocks that hold a particular file are stored together Noncontiguous ◦ The blocks that hold a particular file are scattered all over the device

5. KT6213 Database File – Table Image

6. KT6213 Database File – Form Image

7. KT6213 Database File – Stream Image  Closer to physical representation of file  Example: YouTube video

8. KT6213 Logical View vs. Physical View

9. KT6213 File Management System Provides a logical view for the user and hides the physical implementation Consistent set of commands that are translated to a form appropriate for the device Consistent view of files regardless of file type, file characteristics, or device Supports manipulation of data within the file Manages directory structures which are presented in a logical view Command shell takes user file commands and program file requests and translates them for the file manager Requests data transfers from I/O device drivers File security and protection of file integrity

10. KT6213 File Manager Request Handling

11. KT6213 File Operations [1/3] File as a whole ◦ Copy, Move ◦ List, Print ◦ Load and execute a program ◦ Load file into memory ◦ Store file from memory ◦ Append data from memory to file ◦ Compile, assemble a file

12. KT6213 File Operations [2/3] Within a file ◦ Open a file ◦ Read a number of bytes from file ◦ Write a number of bytes to a file ◦ Move the file pointer forward or backward ◦ Move file pointer to beginning of a file ◦ Close a file

13. KT6213 File Operations [3/3] Record Storage ◦ Retrieve a record (read) ◦ Store a record (write) ◦ Add a record to a file ◦ Delete a record ◦ Modify contents of a record

14. KT6213 File Directory Operations File Directory ◦ Create a new (empty) file ◦ Move a file from one directory to another ◦ Rename a file ◦ Append one file to another ◦ Delete a file

15. KT6213 File Management and I/O Functions Separation between the two allows 1. I/O devices can change while keeping the file system the same 2. Redirecting of data is simple

16. KT6213 File Access Methods Sequential Access ◦ File is read in sequence from beginning to end ◦ Majority of all files ◦ Program source and binary files Random Access ◦ Assumes file is made up of fixed length logical records ◦ Hashing is a common method used to calculate the location of an internal logical record Indexed Access ◦ Additional means for accessing and viewing records in a file ◦ Key indexes ◦ ISAM – indexed sequential access method

17. KT6213 Physical File Storage Contiguous Non-contiguous ◦ Linked ◦ Indexed Examples ◦ DOS/Windows FAT ◦ UNIX i-nodes ◦ Windows NTFS Free space management

18. KT6213 Contiguous Storage Allocation Assign blocks (all in a row) to hold the file Access is simple for both sequential and random methods Disadvantages ◦ Space must be large enough ◦ Have to take into account file growth ◦ May need to be moved if it outgrows its space ◦ Fragmentation of disk Allocation strategies to minimize fragmentation ◦ First-fit, best-fit Eventually disk becomes fragmented

19. KT6213 Linked Allocation Non-contiguous Each block contains a link to the next physical block Variant – links in both directions Advantages ◦ no fragmentation ◦ Adding to a file is easy Disadvantages ◦ Not usable for random access ◦ Additional disk head searching ◦ Overhead in storing the pointers ◦ Recovery of a defective block is difficult

20. KT6213 File Allocation Contiguous Storage Allocation Linked Allocation

21. KT6213 Windows FAT File Allocation Table (FAT) ◦ Linked allocation with links stored in a table ◦ Table contains the first block of each file on the disk or disk partition ◦ Successive blocks contain a link to the next block Disadvantages ◦ Requires a tremendous amount of space ◦ File integrity can be easily compromised

22. KT6213 File Allocation Table Linked Allocation and File Allocation Table

23. KT6213 Indexed Allocation Non-contiguous All link pointers for a file are stored together in a single block called the index block One index block per file Advantages ◦ No fragmentation ◦ Can be used for random access Disadvantage ◦ Slower due to additional access of the index block ◦ Additional disk head searching ◦ Recovery of a defective block is difficult

24. KT6213 Indexed Allocation Index blocks for indexed allocation of linked files

25. KT6213 Unix i-nodes Indexed file allocation Index block contains ◦ File attributes ◦ 10 direct blocks ◦ 1 single indirect ◦ 1 double indirect ◦ 1 triple indirect Advantages ◦ Fast for small blocks ◦ Can accommodate very large files – 100’s of gigabytes

26. KT6213 Unix i-nodes

27. KT6213 Windows NTFS Dynamically sized volumes Volumes may be a fraction of a disk or span many disks Master File Table (MFT) of 1kb records ◦ First 16 records are metadata files which describe the volume ◦ First record stores the MFT attributes ◦ Each file has an MFT entry File records made up of attributes, including file information and data

28. KT6213 NTFS Volume Layout

29. KT6213 Free Space Management Bit map method ◦ one bit for each block to indicate if it is used or free Linked list method ◦ Pointer to first free block ◦ Each free block has a pointer to the next ◦ Blocks are allocated from the beginning ◦ Deleted files are placed at the end

30. KT6213 Other Secondary Storage Allocation Tape Allocation ◦ Not practical to reallocate space in the middle of the tape ◦ Files that grow must be re-written ◦ Files are stored contiguously whenever possible Optical drives and flash drive file allocation ◦ Similar to that of hard disks ◦ UDF (Universal Data Format) supports up to 2 terabytes of data ◦ Hierarchical directory format ◦ Support for both HD-DVD and Blu-Ray DVD formats

31. KT6213 Partitions, File Systems, Volumes, and Pools

32. KT6213 Directory Structure Provides a means of organization so that files can be located easily and efficiently Hide the physical devices from the logical view of the files Partitions ◦ Independent subsections of a device Volume ◦ Directory structure for a particular partition ◦ Needs to be mounted to be incorporated into the overall file system structure Contain file attributes

33. KT6213 Tree-Structure Directory Hierarchical with a top-level root directory from which all other directories stem All directories and files have names Separator ◦ Used to indicate subdirectories and files located in a directory ◦ / UNIX ◦ \ DOS, Windows Pathname ◦ Absolute – full pathname starting from the root directory ◦ Relative – pathname is created starting from the current directory Search Paths ◦ Directory locations that the operating system uses to locate files

34. KT6213 Tree-Structure Directory Pathname Examples: Absolute:C:\FINANCE\QUICKEN\Q.EXE Relative from the FINANCE directory:QUICKEN\Q.EXE Search Path:PATH=C:\DOS;C:\FINANCE\QUICKEN ◦ Now the programs in the two directories can be run by specifying the name of the program without the absolute or relative pathnames For our examples, root is on disk device C:

35. KT6213 Acyclic Directory Structures Tree-structure that permits links between separate branches of the tree Advantage ◦ Easy user access Disadvantages ◦ Cycles and dangling links Examples ◦ Windows shortcuts ◦ Unix hard and symbolic links ◦ MacIntosh aliases

36. KT6213 An Acyclic-Graph Directory

37. KT6213 Graph with a Cycle

38. KT6213 Hard Links vs. Symbolic Links

39. KT6213 Network File Access FTP ◦ File Transfer Protocol ◦ Part of the TCP/IP protocol family Network file systems ◦ Windows  Drive letters aliased to remote file systems ◦ UNIX  Network File System (NFS)  Remote Procedure Call (RPC)

40. KT6213 Typical NFS Configuration

41. KT6213 File Protection Logins and passwords Read, write, and execute protections ACL – access control list, permissions UNIX/Linux – owner, group, everyone

42. KT6213 Data Storage Approaches Standard client server configuration Storage area network configuration

43. KT6213 File Protection Most systems provide three forms of protection on files ◦ Read protection ◦ Write protection ◦ Execution protection Access control list (ACL) ◦ List of users who may access the file for each of the forms of protection ◦ Tremendous overhead if there are a lot of users Owner/Group/Everyone protection method ◦ UNIX, Linux

44. KT6213 File Directory Showing Protection ls –lFlist files in directory using a long format and indicate file type 10-char code for file protection 1 st chard for directory, - for file, s for symbolic link 2 nd to 4 th charpermissions for the owner 5 th to 7 th charpermissions for the group 8 th to 10 th charpermissions for everyone r - read permission, w - write permission, x - execute permission

45. KT6213 Journaling File Systems Log file records every system transaction that requires a write access to the file system Two levels of capability 1.Protect the integrity of the file system structure only  Example: Windows NTFS file system 2.Also guarantees the integrity of data that has not yet been written to the disk  Examples: Linux ext3 and ext4, IBM JFS

46. KT6213 Primary Kernel Functions File manager  translates logical file requests into specific physical I/O requests I/O Control System (IOCS)  performs resource allocation and device management Memory management  determines if it is possible to load programs and data into memory and if so where in memory Scheduler  allocates time for the program to execute

47. KT6213 Miniature Operating System Block Diagram Memory Map Process Dispatch

48. KT6213 Bootstrapping Execution begins with bootstrap loader (mini-loader, IPL) stored in ROM Looks for OS program in a fixed location (possibly on the network) Loads OS into RAM Transfers control to starting location of OS Loader program in OS used to load and execute user programs

49. KT6213 Bootstrapping Cold vs. warm boot (does not retest the system)

50. KT6213 Process [1/2] Process: basic unit of work in the OS ◦ A program together with all the resources that are associated with it as it is executed ◦ Program: a file or listing ◦ Process: a program being executed Independent vs. cooperating processes PID (process ID): a unique identifier for each process

51. KT6213 Process [2/2] Process creation ◦ Forking, spawning, cloning a new process ◦ Parent and child processes Process context ◦ All relevant register data including the program counter ◦ Allows interruption and restart invisibly

52. KT6213 Two Processes Sharing a Single Program

53. KT6213 Process Control Block A block of data for each process in the system Contains all relevant information about the process Typical process control block on the right 

54. KT6213 Process States Three primary process operating states ◦ Ready state ◦ Running state ◦ Blocked state Dispatching - Move from ready state to running state Wake-up - Move from blocked state to ready state Time-out - Move from running state to ready state Process completion ◦ killed, terminated, destroyed Additional states – suspend, swap Resumption – Move from suspended state to ready state

55. KT6213 Process State Diagram

56. KT6213 Threads ‘Miniprocess’ that can be run independent of other parts of the process ◦ Event-driven programs ◦ Shares resources allocated to its parent process including primary storage, files and I/O devices ◦ Each thread has its own context Advantage of process/thread families over multiple independent processes: ◦ Reduced OS overhead for resource allocation and process management ◦ Substantially less information than a normal PCB

57. KT6213 Loading and Executing a Process

58. KT6213 CPU Scheduling

59. KT6213 Dispatching Objectives Ensure Fairness Maximize throughput Minimize turnaround time Maximize CPU utilization Maximize resource allocation Promote graceful degradation Provide minimal and consistent response time Prevent starvation

60. KT6213 Nonpreemptive Dispatching First in, first out (FIFO) ◦ Unfair to short processes and I/O based processes Shortest Job First (SJF) ◦ Longer jobs can be starved Priority Scheduling ◦ Job with the highest priority is selected ◦ If multiple jobs have the highest priority then dispatcher selects among them using FIFO

61. KT6213 Preemptive Dispatching Round robin ◦ Inherently fair and maximizes throughput Dynamic Priority ◦ Based on ratio of CPU time to total time process has been in the system ◦ Smallest ratio has highest priority ◦ Linux; Windows 2000, XP, Vista, 7

62. KT6213 Preemptive Dispatching Multilevel feedback queues ◦ Favors short jobs, I/O bound jobs ◦ Each level assigns more CPU time

63. KT6213 Virtual Memory – Basic Ideas Virtual memory increases the apparent amount of memory by using far less expensive hard disk space Provides for process separation Demand paging ◦ Pages reside on hard disk and are brought into memory as needed Page table ◦ Keeps track of what is in memory and what is still out on hard disk Inverted page table ◦ Representation of physical memory with the page that resides in each frame

64. KT6213 Pages and Frames [1/2] ProgramMemory UnitPageFrame AddressLogicalPhysical Size 2 to 4KB Amount # of bits in instruction word Installed memory

65. KT6213 Pages and Frames [2/2] Each program has its collection of pages. The total number of pages can exceed the number of frames (physical memory). A Simple Page Table TranslationPages and Frames

66. KT6213 Page Translation Process

67. KT6213 Disk Virtual Memory Pages Page Frame Pages not in main memory: page fault when accessed 1234 5678 91011 1 2 3 4 5 6 7 8 9 10 11 6 4 8 10 1 2 7 Page Table Swap space

68. KT6213 Mapping for Three Processes

69. KT6213 Inverted Page Table Inverted Page Table for the previous slide The table represents what is in physical memory

70. KT6213 Steps in Handling a Page Fault

71. KT6213 Concept of Locality Most memory references confined to small region Well-written program in small loop, procedure or function Data likely in array and variables stored together Working set ◦ Number of pages sufficient to run program normally, i.e., satisfy locality of a particular program

72. KT6213 Page Replacement Algorithms Page fault ◦ Page is not in memory and must be loaded from disk Algorithms to manage swapping ◦ FIFO – First-In, First-Out  Belady’s Anomaly – when increasing number of pages results in more page faults ◦ LRU – Least Recently Used ◦ LFU – Least Frequently Used ◦ NUR – Not Used Recently  Referenced bit  Modified (dirty) bit ◦ Second Chance Replacement algorithms Thrashing ◦ too many page faults affect system performance

73. KT6213 Frame Lookup Procedures

74. KT6213 Segmentation

75. Virtual Memory Tradeoffs Disadvantages SWAP file takes up space on disk Paging takes up resources of the CPU Advantages Programs share memory space More programs run at the same time Programs run even if they cannot fit into memory all at once Process separation

76. KT6213 Virtual Memory vs. Caching Cache speeds up memory access Virtual memory increases amount of perceived storage ◦ Independence from the configuration and capacity of the memory system ◦ Low cost per bit compared to main memory

77. KT6213 Secondary Storage Scheduling FCFS - First-Come, First-Served Shortest Distance First ◦ Indefinite postponement problem Scan ◦ Middle of disk gets serviced twice N-Step C-Scan ◦ Disk seek in only one direction ◦ Return after last request in queue served ◦ Two queues  Queue of requests being processed  Queue of new requests

78. KT6213 Scan Scheduling Algorithm

79. KT6213 Comparison of Different Disk Algorithms

80. KT6213 Network OS Services File transfer programs Access to data files on other computers on the network ◦ Computer naming scheme is required for some network systems Print services ◦ Print requests are redirected by the OS to the network station that manages the requested printer Other peripheral sharing services Web services Messaging services API network services Security and network management services Remote processing services

81. KT6213 Access for a Networked OS

82. KT6213 Other OS Issues Deadlock ◦ Two or more processes simultaneously have resources that are required by one another in order to proceed ◦ Prevention ◦ Avoidance ◦ Detection and recovery Process Synchronization ◦ Required by cooperating processes when one process is dependent on the other

83. KT6213 Virtual Machines Virtualization ◦ Using a powerful computer to simulate a number of computers Virtual machines ◦ A simulated computer Hypervisor ◦ Layer of software and/or hardware that separates one or more operating systems from the hardware ◦ Type 1 (native) – hypervisor software interfaces directly with the computer hardware ◦ Type 2 (hosted) – hypervisor software runs as a program on the operating system

84. KT6213 Virtual Machine Configuration