1 UNIX Internals – the New Frontiers Distributed File Systems

2 Difference between DOS and DFS u Distributed OS looks like a centralized OS, but runs simultaneously on multiple machines. It may provide a FS shared by all its host machines. u Distributed FS is a software layer that manages communication between conventional operating systems and file systems

3 General Characteristics of DFS u Network transparency u Location transparency & Location independence u User Mobility u Fault tolerance u Scalability u File mobility

4 Design Considerations u Name Space u Stateful or stateless u Semantics of sharing u UNIX semantics u Session semantics u Remote access method

5 Network File System(NFS) u Based on Client-server model u Communicate via remote procedure call

6 User Perspective u An NFS server exports one or more file systems u Hard mount: must get a reply u Soft mount: returns an error u Spongy mount: hard for mount, soft for I/O u Commands: u mount –t nfs nfssrv:/usr /usr u mount –t nfs nfssrv:/usr/u1 /u1 u mount –t nfs nfssrv:/usr /users u mount –t nfs nfssrv:/usr/local /usr/local

7

8 Design goals u Not restricted to UNIX u Not be dependent on any hardware u Simple recovery mechanisms u To access remote files transparently u UNIX semantics u NFS performance must be comparable to that of a local disk u Transport-independent

9 NFS components u NFS protocol u RPC protocol u XDR(Extended Data Representation) u NFS server code u NFS client code u Mount protocol u Daemon processes (nfsd, mountd,biod) u NLM( Network Lock Manager )& NSM( Network Status Monitor )

10 Statelessness u Each request is independent u It makes crash recovery simple u Client crash u Server crash u Problem: u It must commit all modifications to stable storage before replying to a request.

The protocol suite u Why XDR? u Differences among internal representation of data elements: u Order, sizes of types. u Opaque (byte stream) u Typed u Little-endian u Big-endian

12 XDR u Integers u 32 bits, (0 byte leftmost - most significant), (signed integers - 2’s compliment) u Variable-length opaque data u Length(4B),data is NULL padded u Strings u Length(4B), ASCII string, NULL padded u Arrays u size(4B),same type of data u Structures u Natural order

13

14 RPC u Specify the format of communications between the client and the server. u SUN RPC: synchronous requests only. u Implemented on UDP/IP. u Authentication to identify callers u AUTH _NULL, AUTH _UNIX, AUTH_SHORT, AUTH _DES, and AUTH _KERB u RPC language compiler: rpcgen

15

NFS Implementation u Control Flow u Vnode u Rnode

17 File Handle u Assign a file handle for lookup, create or mkdir. u Subsequent I/O operations will use it. u A file handle =Opaque 32B object = u Generation number is used to check if the file is not obsolete (its inode is allocated to another file)

18 The mount operation u nfs_mount(): u send RPC request with argument of pathname u Mountd daemon translate u Checks u Reply success with a file handle u Initialize vfs, records name, address u Allocate rnode & vnode u Server must check access rights on each request

19 Pathname Lookup u Client: u Initiate lookup during open, create & stat u From current or root directory, proceeds one component at a time u Send request if it is a NFS directory u Server u From file handle ->FS ID->vfs->VGET-> vnode ->VOP_LOOKUP->vnode & pointer u VOP_GETATTR->VOP_FID-> file handle u Reply message= status+file handle+file attributes u Client: u Gets the reply, allocates rnode+vnode, copy info and proceeds to search for the next component

UNIX Semantics u NFS leads to a few incompatibilities with UNIX because of stateless. u Open file permission u UNIX checks for open u NFS checks for each read and write u In NFS, the server always allows the owner of the file to read or write the file. u Write to the write-protected? u Save attributes containing the file permission when open

21 Deletion of open files u The server has no ideas about the open file. u The clients renames the file to be deleted. u Delete it when closing it u Delete on different machines?

22 Reads and Writes u UNIX locks the vnode at the start of I/O u NFS clients can lock the vnode on the same machine. u NFS offers no protection against overlapping I/O requests. u Using NLM(Network Lock Manager) protocol is only advisory.

NFS Performance u Bottlenecks u Writes must be committed to stable storage u Fetching of file attributes requires one RPC call per file u Processing retransmitted requests adds to the load on the server

24 Client-side caching u Caching both blocks and file attributes u To avoid invalid data u Keep an expiry time in the kernel u 60 seconds for rechecking the modified time u Reduces but not eliminates the problem

25 Deferral of writes u Asynchronous writes for full blocks u Delayed writes for partial blocks u Flush delayed writes when closing or 30 seconds by biod daemon u Server uses NVRAM buffer, flushes the buffer to disk u Write-gathering: u Wait, process >1 writes to one file and reply for each u The server process gathered write requests

26 The retransmissions cache u Idempotent u Nonidempotent u Problem: u Retransmissions (xid) cache (server): u Check xid, procedure number, & client ID u Check cache only when failure u Remove request u Remove, sends reply success, but lost u Client restransmit remove u Server processes remove request u Remove error, sends remove failure u Client receives the error message

27 New implementation u Caches all requests u Check xid, procedure number, client ID, state field & timestamp u If request in progress, discard; if done, discards if timestamp shows the request is in the throwaway window(3-6s) u Otherwise processes request if idempotent; u For nonidempotent, checks the file if modified, if not - send success; otherwise, retry it.

NFS Security u NFS Access Control u On mount and request u By an exports list u Mount: checks the list, denies the ineligible u Request: authentication information, AUTH_UNIX form(UID,GID) u Loophole: a imposter can use to access the files of others

29 UID Remapping u A translation map for each client. u Same UID may map to different UID on the server u Nobody if does not match in the map u Implemented at RPC level u Implemented at NFS level u Merging the map and /etc/exports file

30 Root Remapping u Map the super user to nobody u Limit the super user of the client to access files on the server u The UNIX framework is designed for an isolated, multi-user environment. The users trust each other.

NFS Version 3 u Commit request u Client writes, the kernel sends asynchronous write u Server saves to local cache, replies immediately u Client holds the data copy until the process closes the file and sends commit request u Server flushes data to disk u file length: u From 32 bits(4GB) to 64 bits(2 34 GB) u READDIRPLUS =(LOOKUP+GETATTR) u Returns names, file handles, file attributes

32 Other DFS u The Andrew File System (10.15 – 10.17) u The DCE Distributed File System (10.18 – )