1. CSS490 Distributed File Systems
Textbook Ch7 (p )
Instructor: Munehiro Fukuda
These slides were compiled from the course textbook and reference books.
Hello! Everyone,
My name is Shinya Kobayashi. Today, I am going to present our paper titled “Inter-Cluster Job Coordination Using Mobile Agents” on behalf of the first author, Munehiro Fukuda. Munehiro was hoping to show up and present the paper at AMS2001, however he got to wait in Japan until he will get an H1B visa. Since I received the presentation materials from him quite recently, please allow me to present this paper using this script.
I can respond to your questions as far as I know, however you can also ask Munehiro by . His address is on the title page of our paper.
(time 1:05)
Winter, 2004
CSS490 DFS

2. DFS Services Storage service
Disk service: giving a transparent view of distributed disks.
Block service giving the same logical view of disk-accessing units.
True file service
File-accessing mechanism: deciding a place to manage remote files and unit to transfer data (at server or client? file, block or byte?)
File-sharing semantics: providing similar to Unix but weaker file update semantics
File-caching mechanism: improving performance/scalability
File-replication mechanism: improving performance/availability
Name service
Mapping between text file names and reference to files, (i.e. file IDs)
Directory service
Winter, 2004
CSS490 DFS

3. DFS Desirable Features
Transparency: should include structure, access, naming, and replication transparency.
User mobility: should not force a user to work on a specific node.
Performance: should be comparable to that of a centralized file system.
Simplicity: should give the same semantics as a centralized file system.
Scalability: should cope with the growth of nodes.
Fault tolerance: should not face a failure stop and maintain backup copies.
Synchronization: should complete concurrent access requests consistently.
Security: should protect files from network intruders.
Heterogeneity: should allow a variety of nodes to share files in different storage media
Winter, 2004
CSS490 DFS

4. File Models Unstructured and Structured Files
An un-interpreted sequence of bytes: UNIX and MSDOS:
Non-indexed records: IBM mainframe
Indexed records such as B-tree: Research Storage System(RSS) and Oracle
Mutable and Immutable Files
Mutable: a single stored sequence altered by each update (ex. Unix and MSDOS)
Immutable: a history of immutable versions, each created every update (ex. Cedar File System)
Winter, 2004
CSS490 DFS

5. File-Accessing Models
Accessing Remote Files
File access
Merits
Demerits
Remote service model
At a server
A simple implementation
Communication overhead
Data caching model
At a client that cached a file copy
Reducing network traffic
Cache consistency problem
Unit of Data Transfer
Transfer level
Merits
Demerits
File
Simple, less communication overhead, and immune to server
A client required to have large storage space
Block
A client not required to have large storage space
More network traffic/overhead
Byte
Flexibility maximized
Difficult cache management to handle the variable-length data
Record
Handling structured and indexed files
More network traffic
More overhead to re-construct a file.
Winter, 2004
CSS490 DFS

6. File-Sharing Semantics
Define when modifications of the file data made by a user are observable by other users
Unix semantics
Session Semantics
Immutable shared-files semantics
Transaction-like semantics
Winter, 2004
CSS490 DFS

7. File-Sharing Semantics Unix Semantics
Absolute Ordering
Client A
read
Append(d)
delayed a b c a b a b c a b c a b c d a b c d e a b c d e t1 t2 t3 t4 t5 t6
Append(c)
Append(d)
read
Client B
Network Delays
Winter, 2004
CSS490 DFS

8. File-Sharing Semantics Session Semantics
Client A
Client B
Client C
Server a b
Open(file) a b
Append(c) a b c
Open(file) a b
Append(d) a b c d
Append(x) a b x
Append(e) a b c d e
Append(y) a b c y
Close(file) a b c d e
Append(z) a b c d z
Open(file) a b c d e
Close(file) a b c d z
Append(m) a b c d e m
Close(file) a b c d e m
Winter, 2004
CSS490 DFS
Close(file)

9. File-Sharing Semantics Transaction-Like Semantics (Concurrency Control)
Backward validation
Forward validation
Client A
Client B
Client C
Client D
Client A
Client B
Client C
Client D
Trans_start
Trans_start
Compare reads with
former writes
Compare write with
later reads R1 R2 W3 R4 W5 R1 R2 W3 R4 W5
Trans_start
Trans_start R1 R2 W6 R4 W7 R1 R2 W6 R4 W7
Trans_start
Trans_start
validation R1 R2 W9 R4 W8
validation R1 R2 W9 R4 W8
Commitment
Commitment
Trans_start
Trans_start
Trans_end
Trans_end R1 R2 R6 R8 W8 R1 R2 R6 R8 W8
Trans_abort
Trans_end
Trans_restart
Trans_end
Trans_end
Abort itself or conflicting active transactions
Trans_abort
Trans_restart
Trans_end
Which validation is better?
Winter, 2004
CSS490 DFS

10. File-Sharing Semantics Immutable Shared-Files Semantics
Server
Client A
Client B
Version
1.0
Tentative
based on
1.0
Tentative
based on
1.0
Version
1.1
Version conflict
Abort
Depend on each file system.
Abortion is simple (later, the client A can
Decide to overwrite it with its tentative 1.0
by changing the corresponding directory)
Version
1.2
Version
1.2
Ignore conflict
Merge
Winter, 2004
CSS490 DFS

11. File-Caching Schemes Cache Location
Node boundary
Location
Merits
Demerits
No caching
No modifications
Frequent disk access,
Busy network traffic
In server’s main memory
One-time disk access,
Easy implementation,
Unix-like file-sharing semantics
In client’s disk
One-time network access,
No size restriction
Cache consistency problem,
File access semantics,
No Diskless workstation
In client’s main memory
Maximum performance,
Diskless workstation,
Scalability
Size restriction,
File access semantics
Client
Server
Main
memory
Main
memory
copy
copy
Disk
Disk
copy
file
Winter, 2004
CSS490 DFS

12. File-Caching Schemes Modification Propagation
Client 1
Write-through scheme
Pros: Unix-like semantics and high reliability
Cons: Poor write performance
Delayed-write scheme
Write on cache displacement
Periodic write
Write on close
Pros:
Write accesses complete quickly
Some writes may be omitted by the following writes.
Gathering all writes mitigates network overhead.
Cons:
Delaying of write propagation results in fuzzier file-sharing semantics.
Client 2
Main
memory
Main
memory
copy W
new
copy W
Immediate write
Disk
file
Client 1
Client 2
Main
memory
Main
memory
new
copy W
copy W
Disk
delayed write
file
Winter, 2004
CSS490 DFS

13. File-Caching Schemes Cache Validation Schemes – Client-Initiated Approach
Main
memory
Main
memory
Checking before every access (Unix-like semantics but too slow)
Checking periodically (better performance but fuzzy file-sharing semantics)
Checking on file open (simple, suitable for session-semantics)
Problem: High network traffic
copy
copy W
Disk
Write through
Check before
every access
Delayed write?
file W
Client 1
Client 2
Main
memory
Main
memory
copy
new W
copy W
Disk W
Write-on-close
Check-on-open
file
Check-on-close? W
Winter, 2004
CSS490 DFS

14. File-Caching Schemes Cache Validation Schemes – Server-Initiated Approach
Client 1
Client 2
Client 3
Client 4
Main
memory
Main
memory
Main
memory
Main
memory
copy
copy
copy W W
Deny for a new open W
Write through Or
Delayed write?
Disk
Notify (invalidate)
file W
Keeping track of clients having a copy
Denying a new request, queuing it, and disabling caching
Notifying all clients of any update on the original file
Problem:
violating client-server model
Stateful servers
Check-on-open still needed for the 2nd file opening.
Winter, 2004
CSS490 DFS

15. Sun NFS Structure Client A Server Client B export export Winter, 2004/ / /
bin
usr
bin
usr
bin
opt
shared
org
shared
export
export
User
process
User
process
VFS
VFS
VFS
Local FS
NFS client
Local FS
NFS server
Local FS
NFS client
RPC stub
RPC stub
RPC stub
Winter, 2004
CSS490 DFS

16. Sun NFS Installation Client: Underlying Connections: Server:
Check if NFS is running: rpcinfo –p
Start NSF: /etc/rc.d/init.d/nfs start
Edit /etc/exports file: /dir/to/export client1(permissions), client2(…
Export dirs in /etc/exports: exportfs –a
Check exported directories: showmount –e
Client:
Import a server’s directory: mount –o options server_name:/dir /my_dir
bg: continue working on importing upon a failure,
intr: a process will be interupted if its I/O request to the server dir is pending.
soft: allowing a client to time out the connection after a number of retries
rw/ro: normal r/w or read only
Underlying Connections:
client
portmapper
NFS mount service port
mountd
permission
portmapper
2049
Winter, 2004
CSS490 DFS
rpc
nfs

17. Sun NFS Overviews Communication RPC: a compound procedure
Lookup, Open, and Read
Server status
Stateless: simple implementation in ver 3.
Statefull: allowing clients to cache files in ver 4.
RPC call back from a server to invalidate a client’s cache
Synchronization
Session semantics
File Locking in ver 4: lock, lockt, locku, and renew
Ex. Emacs: Tests with lockt when modifying buffer, locks a file with lockt, and unlock with locku after writing buffer contents to the file.
Share reservation: specify how to share a file (with ro, wo, or r/w)
Winter, 2004
CSS490 DFS

18. SUN NFS Overviews (Cont’d)
Caching
In client’s memory
Session semantics
Revalidation of client’s cache upon re-opening the same file
Open delegation:
A server delegates a open decision to a writing client which can handle an open request from other clients on the same machine.
A server calls back the client when receiving an open request from another machine.
Fault Tolerance
RPC failure: use a duplicate-request cache
File locking failure: provide a grace period during which a client reclaim locks previously granted and the server builds up its previous state.
Winter, 2004
CSS490 DFS

19. Sun NFS Duplicate Request Cache
client
server
client
server
client
server
XID = 1234
XID = 1234
XID = 1234
XID = 1234
Too soon, ignore
Too soon, ignore
Transaction
completed
Transaction
completed
Transaction
completed
XID = 1234
reply
reply
reply
Just replied, ignore
XID = 1234
reply
Then, when does the server delete this cached result?
Winter, 2004
CSS490 DFS

20. DFS Example Andrew File System
Client A
Client A / /
tmp
usr
tmp
usr
bin
bin
Symbolic links
Symbolic links
Venus
process
Venus
process
User
process
Unix Kernel
(Unix FS)
cache
Unix Kernel
(Unix FS)
Winter, 2004
CSS490 DFS

21. DFS Example XFS Metadata Manager Storage Server Metadata Manager
3: Fragment a segment
and sent them to a strip group of servers
2: Log them
in a segment
3: Collaborative caching
(Read data from another client if possible)
2: Query a manager
Client
Storage
Server
Client
Storage
Server
1: Write requests
1: Read request
LAN
Winter, 2004
CSS490 DFS

22. DFS Example Plan 9 Winter, 2004 CSS490 DFS Client / Union directory in
import a b c
net a d x y
import
import
export
File server 1
File server 2
Computation server
Network Interface d1 d2 d3 N
Network access a b c a d x y
net
Internet
Remote execution
Winter, 2004
CSS490 DFS

23. Paper Review by Students
Sun NFS
Andrew File System
XFS
Plan 9
LFS
Winter, 2004
CSS490 DFS