Management and Optimization of File System

Figure 4-20. Percentage of files smaller than a given size Disk Space Management Block Size (1) Figure 4-20. Percentage of files smaller than a given size (in bytes). Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

Disk Space Management Block Size (2) Figure 4-21. The dashed curve (left-hand scale) gives the data rate of a disk. The solid curve (right-hand scale) gives the disk space efficiency. All files are 4 KB. Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

Keeping Track of Free Blocks (a) Storing the free list on a linked list. (b) A bitmap.

Creating a filesystem Creating a filesystem is done with an mkfs variant mke2fs, mke2fs -j mkreiserfs mkjfs Typical options: - b blocksize sets blocksize - i bytes-per-inode sets number of inodes - c checks disk for bad blocks Example: # mke2fs -b 1024 -i 4096 -c /dev/sda6 ... Writing inode tables: done Writing superblocks and filesystem accounting info: done

Mount # mount /dev/lv00 /home/patsie mount is the glue that logically connects file systems to the directory hierarchy File systems are associated with devices represented by special files in /dev (the logical volume) When a file system is mounted, the logical volume and its contents are connected to a directory in the hierarchical tree structure # mount /dev/lv00 /home/patsie

Comparing filesystems Journaled Filesystems used by Linux: ext2 ext3 jfs reiser xfs yes yes yes Journal no (10 MB (auto (32 MB yes default) resized) default) yes, but yes, but only yes, but only resizeable only when when yes yes when mounted unmounted unmounted maximum File: 2 TB File: 2 TB File: 4 PB File: 16 TB File: 2 TB size FS: 16 TB FS: 16 TB FS: 32 PB FS: 1 EB FS: 8 EB inodes inodes inodes inodes (completely (completely type (allocated b-Tree (allocated in a block block in a b-tree) b-tree) oriented) oriented)

SHMFS-specific information SHMFS: POSIX compliant Shared Memory Filesystem Filesystem stored in memory, expands when used to required size Not persistent across reboot Typically mounted on /dev/shm Required by certain applications

Each user may consume only 20 MB permanently and 25 MB temporarily Quota concepts Quotas limit the amount of data a user/group is allowed to store Defined on a per-filesystem basis Based on block and/or inode usage per user or group Two limits per quota: Soft and hard User exceeds soft limit warning only User exceeds hard limit error Grace period identifies how long the soft limit may be exceeded After that period, a user gets errors instead of warnings 20 MB Filesystem: 300 MB 5 Each user may consume only 20 MB permanently and 25 MB temporarily

Disk Quotas Figure 4-24. Quotas are kept track of on a per-user basis in a quota table. Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

Quota implementation on Linux Quota support compiled into the kernel No daemon necessary Implemented on a per-filesystem basis A user can have different quota on different filesystems Stored in aquota.user and aquota.groups in the root of the filesystem Quota checking should be enabled when mounting the filesystem Mount options: usrquota, grpquota Can be specified in /etc/fstab Quota checking should be turned on after mounting with the quotaon command Automatically executed from bootscript after mount -a

Enabling quota Modify /etc/fstab Create aquota.user and aquota.group in the filesystem's root directory Remount the partition Calculate current usage and turn on quota checking /dev/sda2 / ext3 defaults 1 1 /dev/sda4 /home ext3 defaults,usrquota,grpquota 1 2 /dev/sdb /mnt/cdrom iso9660 noauto,owner,ro 0 0 /dev/sda3 swap swap defaults 0 0 /dev/fd0 /mnt/floppy msdos noauto,owner 0 0 none /proc proc defaults 0 0 none /dev/pts devpts gid=5,mode=620 0 0 # touch /home/aquota.user /home/aquota.group # mount -o remount /home # quotacheck /home # quotaon /home

Configuring quota Group quota: edquota -g groupname Done with the edquota command Starts $EDITOR (default: vi) in a subshell Only edit the block/inode soft/hard quota numbers User quota: edquota -u username Group quota: edquota -g groupname Grace period: edquota -t Copy quota: edquota -p tux1 -u tux2 tux3 tux4 Disk quotas for user tux1 (uid 501): Filesystem blocks soft hard inodes soft hard /dev/sda4 10700 20000 25000 407 0 0 /dev/sda9 320 300 350 23 30 50 ~ "/tmp/Edp.a9fSEQK" 3L, 213C

Quota information quota command Reports on the quota of one user Can be executed by anyone A regular user can only view his own quota repquota command Reports on the quota of all users and groups Can only be executed by root tux1$ quota Disk quotas for user tux1 (uid 501): Filesystem blocks quota limit grace files quota limit grace /dev/sda4 10700 20000 25000 407 0 0 root# repquota /dev/hda4 Block limits File limits User used soft hard grace used soft hard grace root -- 848804 0 0 56892 0 0 . tux1 ++ 1500 1000 1500 7days 112 112 115 none tux2 -- 176 1000 1500 44 0 0

Physical Backup VS Logical Backup File System Backups (1) Backups to tape are generally made to handle one of two potential problems: Recover from disaster. Recover from stupidity. Physical Backup VS Logical Backup Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

Backup schemes Full backup Preserves the whole system System backup Preserves system directories and files Must include backup/restore tools Usually on bootable media (floppy, optical) Data backup Preserves user data Incremental or differential backup Only backup files that changed Very fast, but takes more time to restore Must be used carefully Needs more media

File System Backups (2) Figure 4-25. A file system to be dumped. Squares are directories, circles are files. Shaded items have been modified since last dump. Each directory and file is labeled by its i-node number. Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

Figure 4-26. Bitmaps used by the logical dumping algorithm. File System Backups (3) Figure 4-26. Bitmaps used by the logical dumping algorithm. Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

Incremental versus differential backup Full L 1 L 2 L 3 Incremental backup Day 1 Day 2 Day 3 Day 4 Full L 1 L 1 L 1 Differential backup

Sample monthly backup scheme Su Mo Tue We Thu Fr Sa 1 Level 0 2 3 4 Level 2 5 Level 3 6 Level 4 7 Level 5 8 Level 1 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29 30 31

Backup devices Tape drive Large capacity, fast Requires new tapes regularly CD-R, CD-RW, DVD Cheap but relatively slow (Removable) Hard disk Fast but expensive Diskette drive Often available but cumbersome for large backups Network Useful in large installations; usually requires commercial software (for instance, Tivoli Storage Manager)

Default backup tools tar Backs up individual files Widely available Excellent for transferring data between platforms cpio Difficulties with many symbolic links dump Backs up whole filesystems Can handle incremental backups (nine levels) dd Useful for making bit-for-bit dumps of disks and filesystems Tar这个命令，是unix里面常用的压缩指令。是最终要的打包指令。打包是真的打包，而不压缩。跟window中的rar不一样。只打包，不压缩，所以有时打包完更大。 Tar命令加入一个参数z，则压缩打包。*.gz, *.bz2是最长见的压缩扩展名。 Dd这个命令非常强大。Disk duplicate, 即磁盘复制，其功能不亚于ghost。 即把整个硬盘做成一个文件。 If：input file Of: output file Zero:比格式化硬盘还要强大，是毁掉硬盘。 美国国防部，拿0写一遍，拿1写一遍，这样重复三次，这样才算毁掉一块硬盘。硬盘你把他砸了，扔了都没用。总之，搞计算机的人应该都明白，删除是不靠谱的，格式化更是骗人的。0和1才是王道！

File System Consistency 影响文件系统可靠性的另一个重要问题是文件系统的一致性。很多文件系统读取磁盘块，进行修改后再写回磁盘。如果在修改过的磁盘块全部写回硬盘之前系统崩溃，则文件系统有可能处于不一致的状态。为了解决文件系统的不一致问题，很多计算机都带有一个实用程序以检验文件系统的一致性。 为了解决文件系统的不一致问题，很多计算机都带有一个实用程序以检验文件系统的一致性。例如windows有scandisk，系统启动后，特别是崩溃之后的重新启动，可以运行该实用程序。 一致性检查分为两种：块的一致性检查和文件的一致性检查。 在检查块的一致性时，程序构造两张表，每张表中为每个块设立一个计数器，都初始化为0.第一个表中的计数器跟踪该块在文件中的出现次数，第二个表中的计数器跟踪该块在空闲表中出现的次数。 块丢失：不会造成实际的损害，但的确浪费了磁盘空间，很简单，把他添加到空闲表中即可。 块在空闲表中重复：也很简单，只要重新建立空闲表，删掉一次即可； 块在数据块中重复出现：最糟糕，如果其中一个被删除，则会又添加到空闲表中，导致一个块处于两种状态。若删除这两个，那么在空闲表中这个磁盘块会出现两次。操作系统通常的处理方法是，先分配一空闲块，把块5中的内容复制到空闲块中，然后把它插刀其中一个文件之中。 Figure 4-27. File system states. (a) Consistent. (b) Missing block. (c) Duplicate block in free list. (d) Duplicate data block. Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

Figure 4-28. The buffer cache data structures. Caching (1) 访问磁盘比访问内存慢的多。如果只需要访问1个字，内存访问比硬盘访问快百万数量级。考虑到访问时间的这个差异，许多文件系统都采用了各种优化措施以改善性能。 高速缓存：是一系列的块，他们在逻辑上属于磁盘，实际上被保存在内存中。常用算法是：检查全部的读请求，查看在高速缓存中是否有所需要的块。如果存在，可执行读操作而无需访问磁盘。如果该块不在高速缓存中，首先要把它读到高速缓存，再复制到需要的地方，之后，对于同一个块的请求，都通过高速缓存完成。 块提前读：在需要用到块之前，试图提前将其写入高速缓存，从而提高命中率。特别的，许多文件都是顺序读的，如果正请求读块k，则系统会偷偷检查k+1是否已经在缓存中。如果还不在，系统会安排预读，因为文件系统希望在需要用到该块时，它已经在高速缓存或者马上就要在高速缓存了。 这种方法只适用于顺序读取的文件。对于随机读取的文件，丝毫不起作用，还会帮倒忙。因为读取无用的块以及从高速缓存删除潜在有用的块都会占用固定的磁盘带宽。 3. 减少磁盘臂运动 Figure 4-28. The buffer cache data structures. Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

Caching (2) Some blocks, such as i-node blocks, are rarely referenced two times within a short interval. Consider a modified LRU scheme, taking two factors into account: Is the block likely to be needed again soon? Is the block essential to the consistency of the file system? Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

Reducing Disk Arm Motion Figure 4-29. (a) I-nodes placed at the start of the disk. (b) Disk divided into cylinder groups, each with its own blocks and i-nodes. Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

Figure 4-30. The ISO 9660 directory entry. The ISO 9660 File System Figure 4-30. The ISO 9660 directory entry. Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

Rock Ridge Extensions Rock Ridge extension fields: PX - POSIX attributes. PN - Major and minor device numbers. SL - Symbolic link. NM - Alternative name. CL - Child location. PL - Parent location. RE - Relocation. TF - Time stamps. Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

Joliet Extensions Joliet extension fields: Long file names. Unicode character set. Directory nesting deeper than eight levels. Directory names with extensions Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

The MS-DOS File System (1) Figure 4-31. The MS-DOS directory entry. Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

The MS-DOS File System (2) Figure 4-32. Maximum partition size for different block sizes. The empty boxes represent forbidden combinations. Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

The UNIX V7 File System (1) Figure 4-33. A UNIX V7 directory entry. Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

The UNIX V7 File System (2) 。 Figure 4-34. A UNIX i-node. Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

The UNIX V7 File System (3) 当打开某个文件时，文件系统必须要获得文件名并且定位它所在的磁盘块。 首先文件系统要定位根目录，在unix系统中根目录的i节点存放于磁盘上固定的位置。接下来，系统读根目录，并且在根目录中查找路径的第一个usr，以获取其i节点号。由i节点号来定位i节点是很直接的，因为每个i节点在磁盘上都有固定的位置。根据这个i节点，系统定位该目录，并查找下一个字目录……. Figure 4-35. The steps in looking up /usr/ast/mbox. Tanenbaum, Modern Operating Systems 3 e, (c) 2008 Prentice-Hall, Inc. All rights reserved. 0-13-6006639

作业 P 183 27， 28， 29， 32