1. 计算机系 信息处理实验室 Lecture 15 File Systems xlanchen@06/03/2005

2. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 2 Contents Windows 2000 File System Formats NTFS Design Goals and Features File System Driver Architecture NTFS File System Driver NTFS On-Disk Structure

3. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 3 Windows 2000 File System Formats CDFS (CD-ROM File System) 1988, read-only formatting standard for CD-ROM media UDF FAT12, FAT16, and FAT32 NTFS the native file system format of Windows 2000

4. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 4 FAT series (12, 16, 32) FAT format organization Example: file allocation table

5. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 5 NTFS Design Goals Recoverability Security Data Redundancy Fault Tolerance

6. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 6 NTFS Features Multiple data streams Unicode-based names General indexing facility Dynamic bad-cluster remapping Hard links and junctions Compression and sparse files Change logging Per-user volume quotas Link tracking Encryption POSIX support Defragmentation

7. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 7 NTFS files NTFS file = set of ($attribute-name, data) pairs Std attributes include: FileName, flags, data, MSDOS-name, ACL, other… Filename up to 255 characters Values (data) held in MFT entry if possible Otherwise: Data attribute = set of (start-VCN, start-LCN, #clusters) Allows sequence of VCNs to be discovered Provides VCN->LCN->cluster mapping “attribute list” attribute added if MFT entry too small Points to (first) overflow MFT rec for mappings

8. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 8 NTFS Directories An index of filenames Index blocks organized as balanced (b+) tree Tree pointer gives (VCN,LCN) of next block Index entries contain: File Reference Number, plus Size, timestamp, etc (for directory browsing) (saves reading MFT rec to find file attributes) NT4 indexes on filename only NT5 indexes on other file attributes also

9. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 9 File System Driver (FSD) Architecture FSDs manage file system formats Kernel mode Two different types of FSD (2K) Local FSDs Remote FSDs

10. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 10 Local FSDs A local FSD must register with the I/O manager Local FSDs include: Ntfs.sys, Fastfat.sys, Udfs.sys, Cdfs.sys, and the Raw FSD (integrated in Ntoskrnl.exe).

11. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 11 Local FSD & other concept of the OS Boot sector I/O manager volume parameter block (VPB) Storage device  file system device Cache manager

12. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 12 Remote FSDs Remote FSDs consist of two components: a client and a server Client-side remote FSD (2K: LANMan Redirector) allows applications to access remote files and directories Accepts I/O request & translates into network commands Server-side FSD (2K: LANMan Server) Listens and fulfills the command

13. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 13 Remote FSD operation

14. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 14 EXPERIMENT Viewing the List of Registered File Systems

15. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 15 File system operation Two ways Directly, file I/O functions Indirectly, file mapping An FSD can be invoked through several paths Explicit file I/O From the memory manager's modified page writer Indirectly from the cache manager's lazy writer Indirectly from the cache manager's read-ahead thread From the memory manager's page fault handler

16. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 16 Components involved in file system I/O

17. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 17 NTFS FSD Components of the Windows 2000 I/O system Layered drivers

18. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 18 NTFS and related components

19. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 19 Log File Service (LFS) NTFS provides file system recoverability by means of a transaction-processing technique called logging LFS is a series of kernel-mode routines inside the NTFS driver

20. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 20 NTFS data structures

21. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 21 NTFS On-Disk Structure Volumes Clusters Directories The storage of actual file data and attribute information NTFS data compression

22. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 22 Volumes A logical partition on a disk A disk can have one volume or several stores all file system data as ordinary files, such as Bitmaps 、 directories 、 system bootstrap

23. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 23 Clusters Cluster size established when formatting Also called cluster factor 2 n sectors Cluster vs. sector NTFS is independent from physical sector sizes LCN, logical cluster numbers the numbering of all clusters of the volume VCN, virtual cluster numbers Number the clusters belonging to a particular file

24. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 24 Master File Table (MFT) All data is contained in files, including metadata Metadata the data structures used to locate and retrieve files, the bootstrap data, the bitmap that records the allocation state of the entire volume Easy to locate and maintain Each can be protected by a security descriptor

25. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 25 Master File Table (MFT) MFT, the heart Implemented as an array of file records File records, fixed size, 1KB Logically, contains one record for each file including the MFT itself. Metadata files (with name prefixed with “$”) $Mft

26. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 26 Metadata files in MFT

27. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 27 Mount a volume with MFT find the physical disk address of the MFT from the boot sector Find information inside the file record of MFT Open more metadata file Perform the file system recovery operation Open other metadata file

28. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 28 Other metadata files log file ($LogFile) root directory ("\") bitmap file ($Bitmap) security file ($Secure) boot file ($Boot) bad-cluster file ($BadClus) extensions ($Extend), a metadata directory object identifier file ($ObjId), the quota file ($Quota), the change journal file ($UsnJrnl), and the reparse point file ($Reparse).

29. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 29 File record vs. File Normally, 1:1 May n:1 If a file has a large number of attributes or becomes highly fragmented First one called base file record stores the locations of the others Others extended file record

30. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 30 File Reference Numbers A file on an NTFS volume is identified a file reference 64-bit File number, index to the file's file record position in the MFT Sequence number, the reused times of an MFT file record position

31. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 31 File Records File, a collection of attribute/value pairs Filename time stamp information unnamed data attribute additional named data attributes

32. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 32 Attribute Name Description Volume information$VOLUME_INFORMATION, $VOLUME_NAME These attributes are present only in the $Volume metadata file. They store volume version sand label information. Standard information$STANDARD_INFORMATIONFile attributes such as read-only, archive, and so on; time stamps, including when the file was created or last modified; and how many directories point to the file (its hard link count). Filename$FILE_NAMEThe file's name in Unicode characters. A file can have multiple filename attributes, as it does when a hard link to a file exists or when a file with a long name has an automatically generated "short name" for access by MS- DOS and 16-bit Microsoft Windows applications. Security descriptor$SECURITY_DESCRIPTORThis attribute is present for backward compatibility with previous versions of NTFS. The Windows 2000 version of NTFS stores all security descriptors in the $Secure metadata file, sharing descriptors among files and directories that have the same settings. Previous versions of NTFS stored private security descriptor information with each file and directory. Data$DATAThe contents of the file. In NTFS, a file has one default unnamed data attribute and can have additional named data attributes; that is, a file can have multiple data streams. A directory has no default data attribute but can have optional named data attributes. Index root, index allocation, and index bitmap $INDEX_ROOT, $INDEX_ALLOCATION, $BITMAP Three attributes used to implement filename allocation and bitmap indexes for large directories (directories only).

33. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 33 Attribute Name Description Attribute list$ATTRIBUTE_LISTA list of the attributes that make up the file and the file reference of the MFT file record in which each attribute is located. This seldom-used attribute is present when a file requires more than one MFT file record. Object ID$OBJECT_IDA 64-byte identifier for a file or directory, with the lowest 16 bytes (128 bits) unique to the volume. The link-tracking service assigns object IDs to shell shortcut and OLE link source files. NTFS provides APIs so that files and directories can be opened with their object ID rather than their filename. Reparse information$REPARSE_POINTThis attribute stores a file's reparse point data. NTFS junctions and mount points include this attribute. Extended attributes$EA, $EA_INFORMATIONExtended attributes aren't actively used but are provided for backward compatibility with OS/2 applications. EFS information$LOGGED_UTILITY_STREAMEFS stores data in this attribute that's used to manage a file's encryption, such as the encrypted version of the key needed to decrypt the file and a list of users that are authorized to access the file. The word logged is in the attribute's name because changes to this attribute are recorded in the volume log file (described later in this chapter) for recoverability.

34. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 34 Attribute streams Each file attribute is stored as a separate stream of bytes within a file Streams is the unit of file operation create, delete, read and write Attribute type code The file attributes in an MFT record are ordered by type codes Attribute Type code: value: optional name

35. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 35 Filenames NTFS, <=255 characters

36. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 36 Resident and Nonresident Attributes resident attribute the value of an attribute is stored directly in the MFT Example:

37. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 37 Several attributes are defined as always being resident The standard information Index root attributes

38. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 38 Resident attribute header and value Example: filename attribute Only need once disk accessing

39. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 39 MFT file record for a small directory If a particular attribute is too large Clusters outside MFT is allocated, called run If the value size grows, more runs is allocated This is nonresident attributes

40. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 40 Resident or nonresident Determined by the file system Location transparent to the process Example: MFT file record for a large file with two data runs

41. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 41 A large directory can also have nonresident attributes Example: MFT file record for a large directory with a nonresident filename index

42. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 42 Keeping track of the runs VCN-to-LCN mapping pairs VCN LCN Example: VCNs & LCNs for a nonresident data attribute

43. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 43 VCN-to-LCN mappings

44. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 44 Indexing A file directory is simply an index of filenames Example: Filename index for a volume's root directory

45. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 45 Index root attribute For large directories: Index buffer + B+ tree With index root attribute contains the first level of B+ tree File index entry: The file reference in the MFT Time stamp File size information

46. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 46 The index allocation attribute For index buffer runs: VCN  LCN

47. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 47 Data Compression and Sparse Files NTFS supports compression per-file, per-directory, or per-volume Related func GetVolumeInformation GetCompressedFileSize DeviceIoControl

48. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 48 Compressing Sparse Data Sparse data is often large but contains only a small amount of nonzero data relative to its size Runs of a noncompressed file and the related MFT record

49. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 49 Compression technique to remove long strings of zeros from the file NTFS allocates space only for runs that contain nonzero data So certain ranges of the file's VCNs have no disk allocations

50. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 50 example Runs of a compressed file containing sparse data 16~31 64~127

51. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 51 The related MFT record Read & write operation For hole, return zero/ allocate & write No mapping for 16~31; 64~127

52. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 52 Compressing Nonsparse Data Nonsparse data can also be compressed Individual files or a whole directory Compressing technology compression units = 16 clusters long For true compressing at least save 1 cluster of storage

53. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 53 example Data runs of a compressed file Actual storage

54. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 54 For compressed file Runs must started at a virtual 16-cluster boundary Read/write unit: compressed unit (why 16 clusters) Example: the related MFT record

55. 计算机系 信息处理实验室 xlanchen@06/03/2005Understanding the Inside of Windows2000 55 Reparse Points $REPARSE_POINT attribute A block of up to 16 KB of application-defined reparse data and a 32-bit reparse tag \$Extend\$Reparse metadata file