1. Federated DAFS: Scalable Cluster-based Direct Access File Servers Murali Rangarajan, Suresh Gopalakrishnan Ashok Arumugam, Rabita Sarker Rutgers University Liviu Iftode University of Maryland

2. SAN-2 Disco Lab 2 Network File Servers OS involvement increases latency & overhead TCP/UDP protocol processing Memory-to-memory copying NFS CLIENTS FILE SERVER TCP/IP

3. SAN-2 Disco Lab 3 Application NIC OS Send Receive NIC Application has direct access to the network interface OS involved only in connection setup to ensure protection Performance benefits: zero-copy, low-overhead User-level Memory Mapped Communication OS

4. SAN-2 Disco Lab 4 SEND QUEUE Kernel Agent VI Provider Library VI NIC Data transfer from user- space Setup & Memory registration through kernel Communication models Send/Receive: a pair of descriptor queues Remote DMA: receive operation not required Setup & Memory registration Application RECV QUEUE COMP QUEUE Virtual Interface Architecture

5. SAN-2 Disco Lab 5 Application Buffers DAFS Client VIPL User Kernel NIC VI NIC Driver DAFS Server NIC File access API DAFS File Server Buffers Driver VI NIC Driver KVIPL Direct Access File System Model DAFS File Server Buffers VI NIC Driver VIPL

6. SAN-2 Disco Lab 6 Goal: High-performance DAFS Server Cluster-based DAFS Server Direct access to network-attached storage distributed across server cluster Clusters of commodity computers - Good performance at low cost User-level communication for server clustering Low-overhead mechanism Lightweight protocol for file access across cluster

7. SAN-2 Disco Lab 7 Outline Portable DAFS client and server implementation Clustering DAFS servers – Federated DAFS Performance Evaluation

8. SAN-2 Disco Lab 8 User-space DAFS Implementation DAFS client and server in user-space DAFS API primitives translate to RPCs on server Staged Event Driven Architecture Portable across Linux, FreeBSD and Solaris DAFS Server VI Network Application VI Local FSVI DAFS Client DAFS Server Application VI DAFS Client VI Network Local FS DAFS API Request DAFS Server Application VI DAFS Client VI Network Local FS DAFS API Response

9. SAN-2 Disco Lab 9 DAFS Server CLIENT Connection Manager Protocol Threads SERVER Connection Request DAFS API Request Response

10. SAN-2 Disco Lab 10 Client-Server Communication VI channel established at client initialization VIA Send/Receive used except for dafs_read Zero-copy data transfers Emulation of RDMA Read used for dafs_read Scatter/gather I/O used in dafs_write DAFS Server VI Network Application VI DAFS Client Local FS dafs_read(file, buf) buf DAFS Server req VI Network VI DAFS Client Request Local FS dafs_read(file, buf) buf DAFS Server VI Network VI DAFS Client Response Local FS dafs_write(file, buf) DAFS Server buf req VI Network DAFS Client VI Local FS

11. SAN-2 Disco Lab 11 Asynchronous I/O Implementation Applications use I/O descriptors to submit asynchronous read/write requests Read/write call returns immediately to application Result stored in I/O descriptor on completion Applications need to use I/O descriptors to wait/poll for completion

12. SAN-2 Disco Lab 12 Benefits of Clustering Local FS VI DAFS Server Application VI DAFS Client Application VI DAFS Client Application VI DAFS Client Single DAFS Server Local FS VI DAFS Server Application VI DAFS Client Local FS VI DAFS Server Local FS VI DAFS Server Application VI DAFS Client Application VI DAFS Client Standalone DAFS Servers on a Cluster Local FS VI DAFS Server Application VI DAFS Client Local FS VI DAFS Server Local FS VI DAFS Server Application VI DAFS Client Application VI DAFS Client Standalone DAFS Servers on a Cluster Local FS VI DAFS Server Application VI DAFS Client Application VI DAFS Client Application VI DAFS Client Clustered DAFS Servers Clustering Layer Local FS VI DAFS Server Clustering Layer Local FS VI DAFS Server Clustering Layer

13. SAN-2 Disco Lab 13 Clustering DAFS Servers Using FedFS Federated File System (FedFS) Federation of local file systems on cluster nodes Extend the benefits of DAFS to cluster-based servers Low overhead protocol over SAN

14. SAN-2 Disco Lab 14 FedFS Goals Global name space across the cluster Created dynamically for each distributed application Load balancing Dynamic Reconfiguration

15. SAN-2 Disco Lab 15 Each VD is mapped to a manager node Determined using hash function on pathname Manager constructs and maintains the VD Virtual Directory (VD) Union of all local directories with same pathname / usr file1 / usr file2 file1 file2 / usr Virtual Directory (/usr) / usr file1 / usr file2

16. SAN-2 Disco Lab 16 Constructing a VD Constructed on first access to directory Manager performs dirmerge to merge real directory info on cluster nodes into a VD Summary of real directory info is generated and exchanged at initialization Cached in memory and updated on directory modifying operations

17. SAN-2 Disco Lab 17 File Access in FedFS Each file mapped to a manager Determined using hash on pathname Maintains information about the file Request manager for location (home) of file Access file from home Local FS VI FedFS VI Network DAFS Server Local FS VI FedFS DAFS Server Local FS VI FedFS DAFS Server f1 manager(f1) home(f1)

18. SAN-2 Disco Lab 18 Optimizing File Access Directory Table (DT) to cache file information File information cached after first lookup Cache of name space distributed across cluster Block level in-memory data cache Data blocks cached on first access LRU Replacement

19. SAN-2 Disco Lab 19 Communication in FedFS Two VI channels between any pair of server nodes Send/Receive for request/response RDMA exclusively for data transfer Descriptors and buffers registered at initialization Local FS DAFS Server FedFS VI VI Network Local FS DAFS Server FedFS VI Send/Receive for Request/Response Local FS DAFS Server FedFS VI VI Network Local FS DAFS Server FedFS VI Buffer RDMA for Response with data

20. SAN-2 Disco Lab 20 Performance Evaluation DAFS Server VI Network Application VI Local FSVI DAFS Client FedFS Application VI DAFS Client Application VI DAFS Client DAFS Server Local FSVI FedFS

21. SAN-2 Disco Lab 21 Experimental Platform Eight node server cluster 800 MHz PIII, 512 MB SDRAM, 9 GB 10K RPM SCSI Clients Dual processor (300 MHz PII), 512 MB SDRAM Linux-2.4 Servers and Clients equipped with Emulex cLAN adapter 32 port Emulex switch in full-bandwith configuration

22. SAN-2 Disco Lab 22 SAN Performance Characteristics VIA Latency and Bandwidth poll/wait for latency/bandwidth measurement respectively Packet Size (Bytes) Roundtrip Latency (  s) Bandwidth (MB/s) 25623.356 51227.385 102436.9108 204856.0109 409691.2110

23. SAN-2 Disco Lab 23 Workloads Postmark – Synthetic benchmark Short-lived small files Mix of metadata-intensive operations Benchmark outline Create a pool of files Perform transactions – READ/WRITE paired with CREATE/DELETE Delete created files

24. SAN-2 Disco Lab 24 Workload Details Each client performs 30,000 transactions Each transaction – READ paired with CREATE/DELETE READ = open, read, close CREATE = open, write, close DELETE = unlink Multiple clients used for maximum throughput Clients distribute requests to servers using a hash function on pathnames

25. SAN-2 Disco Lab 25 Base Case (Single Server) Maximum throughput 5075 transactions/second Average time per transaction For client ~ 200  s On server ~ 100  s

26. SAN-2 Disco Lab 26 Postmark Throughput # Servers248 Speedup1.7535

27. SAN-2 Disco Lab 27 FedFS Overheads Files are physically placed on the node which receives client requests Only metadata operations may involve communication first open(file) delete(file) Observed communication overhead Average of one roundtrip message among servers per transaction

28. SAN-2 Disco Lab 28 Other Workloads No client request sent to file’s correct location All files created outside Federated DAFS Only READ operations ( open, read, close ) Potential increase in communication overhead Optimized coherence protocol minimizes communication Avoid communication at open and close in the common case Data Caching helps reduce the frequency of communication for remote data access

29. SAN-2 Disco Lab 29 Postmark Read Throughput Each transaction = READ

30. SAN-2 Disco Lab 30 Communication Overhead Without Caching Without caching, each read results in remote fetch Each remote fetch costs ~65  s request message (< 256 B) + response message (4096 B) # Servers# Clients for Max. Throughput # Transactions# Remote Reads on each server 210300,000150,000 420600,000150,000

31. SAN-2 Disco Lab 31 Work in Progress Study other application workloads Optimized coherence protocols to minimize communication in Federated DAFS File migration Alleviate performance degradation from communication overheads Balance load Dynamic reconfiguration of cluster Study DAFS over a Wide Area Network

32. SAN-2 Disco Lab 32 Conclusions Efficient user-level DAFS implementation Low overhead user-level communication used to provide lightweight clustering protocol (FedFS) Federated DAFS minimizes overheads by reducing communication among server nodes in the cluster Speedups of 3 on 4-node and 5 on 8-node clusters demonstrated using Federated DAFS

33. Thanks Distributed Computing Laboratory http://discolab.rutgers.edu

34. SAN-2 Disco Lab 34 DAFS Performance