Almaden Rice University Nache: Design and Implementation of a Caching Proxy for NFSv4 Ajay Gulati, Rice University Manoj Naik, IBM Almaden Renu Tewari, IBM Almaden

Almaden © 2003 IBM Corporat2006n Rice Talk Outline  Federated File System  NFSv4: Overview and New Features  Nache Architecture  Nache Implementation  Experimental Results  Conclusions and Discussion

Almaden © 2003 IBM Corporat2006n Rice Federated File System  File system federation with unified namespace  Flexible data management –Replication, migration and read/write caching across distributed file servers  What are the options? –AFS, DCE/DFS provide distributed FS over WAN, but… difficult to deploy, not widely used –GPFS, Lustre are cluster filesystems, but… good for controlled environments, no federation –NFS is a standard, but… v2/v3 have been designed mainly for LAN, are chatty and not suited for WAN, firewall issues –NFSv4?

Almaden © 2003 IBM Corporat2006n Rice Why NFSv4?  Open industry standard  Optimizations for WAN –Compound operations A single RPC request contains multiple ops, reduces round-trips – Read-Write delegations Efficient cache consistency, no need to contact server once a delegation is obtained Delegations can be recalled on conflicting access  Federation support –Client redirection Server returns NFS4ERR_MOVED to redirect clients Client requests fs_locations attribute Client connects to new location

Almaden © 2003 IBM Corporat2006n Rice AlmadenHouston New York Admin Clients Filesystem Namespace Unification Separate namespace across servers Unified namespace for clients

Almaden © 2003 IBM Corporat2006n Rice Clients (San Jose) Server (San Jose) Server (New York) /project 1. OPEN /project/p1 2. NFS4ERR_MOVED 3. GETATTR 4. fs_locations Data Access using NFSv4 Redirection Limitations  High network latency when clients access remote servers  Delegation are unsuited for data sharing, fewer awarded and frequently broken  No unified caching, every client has to access remote server 6. /project/p1 7. DELEGRETURN 5.Client connects to New York server 8. Server responds

Almaden © 2003 IBM Corporat2006n Rice What is a Proxy Cache (Nache)?  An NFSv4 server acts as a proxy for another remote NFS server –caches data for reads and writes using delegations –becomes the temporary “owner” of the data servicing opens, lock requests and reads and writes  Multiple clients can share a proxy cache using a single server-to-proxy delegation  Extends beyond web-proxy caching

Almaden © 2003 IBM Corporat2006n Rice Data Access using a Proxy Cache Clients (San Jose) Proxy Cache (San Jose) Server (New York) /project 3./project/p1 1.OPEN /project/p1 Benefits with NFSv4 Proxy  Reduced network traffic and delay – Clients access files through local proxy cache – Clients can share delegations – fewer conflicts/recalls  Improved performance for cached files – Clients can collectively take advantage of aggressive readahead, write back and local locking at the proxy – Both data and delegations are shared 2. Delegate /project/p1 to proxy

Almaden © 2003 IBM Corporat2006n Rice Nache Architecture Nache ServerNache Client VFS Layer Buffer Cache cacheFS Local Client (export to) Remote Server (mount from) (Linux nfsd module)(Linux nfs module) Nache

Almaden © 2003 IBM Corporat2006n Rice Nache Implementation  Cascaded mounts  NFS operation forwarding  Sub-operation RPC call handling

Almaden © 2003 IBM Corporat2006n Rice Cascaded Mounts  Proxy exports a NFS mounted file system  Added export functionality for NFS – export_ops added, not available in vanilla NFS Nache Server /watson/almaden mount server:/ /almaden ls /net at client shows contents of /watson at server Client mount nache:/ /net /net /export

Almaden © 2003 IBM Corporat2006n Rice NFS Operation Forwarding  NFS ops are redirected from nfsd to nfs using vfs layer  Issues: certain operations are stateful –OPEN, CREATE, LOCK, ULOCK, CLOSE ClientNache Server VFS Nache Client NFSD VFS NFS ClientNache Server Application

Almaden © 2003 IBM Corporat2006n Rice Sub-Operation RPC calls  Each FS operation at Nache server translates to a separate RPC call from Nache client  Solution: –Remove such redundant checks from code path Nache ServerNache Client VFS Layer Local Client Remote Server (Linux nfsd module)(Linux nfs module) 1.OPEN 2.LOOKUP 3.ACCESS 4.OPEN

Almaden © 2003 IBM Corporat2006n Rice Experimental Methodology  Setup –2-6 local NFS clients (San Jose) –One local proxy with modified kernel modules (San Jose) –One local NFS server (San Jose) –One remote NFS server (New York)  Benchmarks –Filebench (developed by Sun) –Compilation of various packages – Linux kernel, emacs, gdb –Our micro-benchmarks

Almaden © 2003 IBM Corporat2006n Rice Experiment Categories  Delegation Benefits  Nache Benefits  Nache Overhead

Almaden © 2003 IBM Corporat2006n Rice Read Delegation Benefits Client repeats open-read-close on 100 files Server ops reduced by 50% due to delegations

Almaden © 2003 IBM Corporat2006n Rice Write Delegation Benefits A client repeats open-read/write-close on 100 files Server ops reduced by 4 times

Almaden © 2003 IBM Corporat2006n Rice Delegation Benefits (Server load) Server ops reduced due to delegations

Almaden © 2003 IBM Corporat2006n Rice Delegation Benefits (Client performance) Time taken is lower and ops/sec is higher due to delegations

Almaden © 2003 IBM Corporat2006n Rice Experiment Categories  Delegation Benefits  Nache Benefits  Nache Overhead

Almaden © 2003 IBM Corporat2006n Rice Nache Benefits - Filebench (webserver) Ops at server reduces by 38% with 4 clients

Almaden © 2003 IBM Corporat2006n Rice Nache Benefits – Kernel compile Ops at server unaffected by increase in number of clients

Almaden © 2003 IBM Corporat2006n Rice Nache Benefits – Compile Time Compile time unaffected by increase in number of clients

Almaden © 2003 IBM Corporat2006n Rice Experiment Categories  Delegation Benefits  Nache Benefits  Nache Overhead

Almaden © 2003 IBM Corporat2006n Rice Nache Overhead over WAN (Throughput) Micro benchmark Ops/s (Nache) Ops/s (no Nache) Overhead(%) Create-Files91643 Random-Reads Random-Writes Random-Append Nache induces an overhead of 8-43% Code can be further optimized to reduce overheads

Almaden © 2003 IBM Corporat2006n Rice Nache Overhead over WAN (Latency) Micro benchmark Latency (Nache) Latency (no Nache) Overhead (%) Create-Files Random-Reads Random-Writes Random-Append Proxy induces an overhead of 8-42% Code can be further optimized to reduce overheads

Almaden © 2003 IBM Corporat2006n Rice Result Summary  Delegations provide better throughput, lower latency and reduced load at server  Nache improves performance in presence of sharing among clients  Overhead of Nache in absence of sharing is not too high except for few cases and can be further reduced

Almaden © 2003 IBM Corporat2006n Rice Conclusions  NFSv4 is a good alternative for building federated file systems  Delegations help in reducing the number of server operations and provide better caching  Nache can be integrated with a federated file system architecture and improve performance in presence of sharing