1. Efficient Access to Many Small Files in a Grid Filesystem Douglas Thain and Christopher Moretti University of Notre Dame

2. Efficient Access to Many Small (and Big) Files in a Grid Filesystem Douglas Thain and Christopher Moretti University of Notre Dame

3. Abstract Many grid data tools focus on transferring, storing, and managing large (GB-TB) files. But, many users need to manage, transfer, and process lots (1000s) of small (KB-MB) files. We describe protocols and interfaces for manipulating many small files over wide area networks. (Doesn’t hurt large files, either.) Implemented in the Chirp file system. Performance: –Best case: order of magnitude improvement. –Worst case: no slower than before.

4. The Small File Problem

5. Who has lots of small files? Anyone using a batch system. –One file for submit, input, output, error, log... Anyone using a large software package. –Executables, libraries, config files... Anyone using a filesystem like a database. –Genomics, astronomy, physics... Anyone who likes to write shell scripts. –foreach host in list ssh $host > $host.output

6. Why is this a problem? Users do the “sensible” thing: –foreach file in (list) do transfer done The “sensible” thing performs miserably: –New TCP Connection –SSL Authentication –Configuration Operations –Slow Start Again Result is KB/s on a GB/s link.

7. Why not just use tar? If you can, you should! Sometimes you cannot: –The system semantics demand multiple files. –Packing and unpacking can be very slow. –Not enough disk space to unpack. –Different apps select different data subsets. –Using an existing script or program. Users don’t know or care that it’s a dist system, why should they change?

8. The Challenge: How to design interfaces so that users get the expected performance and behavior?

9. Chirp and Parrot: A Grid Filesystem

10. Requirements for a Grid Filesystem Transparent access to files in the same manner as a local Unix filesystem. Non privileged deployment at both client and server. (root not possible on the grid.) User control over policies for naming, caching, consistency, and fault tolerance. Flexible access controls for sharing. Good performance on both small and large files.

11. Chirp/Parrot – A Grid Filesystem Chirp Ordinary Unix Filesystem Ordinary Unix Program Parrot unix system calls Authorization: kerberos:joe@nd.edu RWLDA globus:/O=ND/CN=Joe RWLDA hostname:*.nd.edu RL group:server.nd.edu/team RWL Protocol: open / pread / pwrite / close stat / mkdir / rmdir / unlink getfile / putfile / movefile Authentication: Kerberos / Globus / Hostname / Unix Single TCP Stream No Privs Needed! No Privs Needed! Automatic Recovery ptrace trap

12. Ordinary Unix Commands > parrot tcsh > ls /chirp alpha.nd.edubeta.nd.edu... > cd /chirp/alpha.nd.edu/mydir > cp /tmp/bigdata. > emacs mydata.txt

13. Parrot Specific Commands > parrot tcsh > parrot_whoami globus:/O=ND/CN=Joe > parrot_getacl /chirp/alpha.nd.edu/ kerberos:joe@nd.edu RWLDA globus:/O=ND/CN=Joe RWL hostname:*.nd.edu RL

14. Chirp as Remote Filesystem Grid Site AGrid Site B App Parrot App Parrot App Parrot App Parrot App Parrot App Parrot App Parrot Chirp Server Unix Filesystem Grid Middleware App Parrot Cert Secured by GSI

15. Chirp as Cluster Filesystem Grid Site AGrid Site B App Parrot App Parrot App Parrot App Parrot App Parrot App Parrot App Parrot Chirp Server Unix Filesystem Chirp Server Unix Filesystem Chirp Server Unix Filesystem Chirp Server Unix Filesystem dir server aux db

16. http://www.cse.nd.edu/~ccl/viz

17. Sample Applications Image Processing for Biometrics –Moretti et al, PCGRID 2007 Bioinformatics on EGEE –Blanchet et al, Grid 2006 High Energy Physics on LCG –Sfiligoi et al, CHEP 2005, Molecular Dynamics Repository –Wozniak et al, HPDC 2005 Remote DB Access on EDG –Klous et al, CCPE 2005

18. Protocols for Small Files

19. What About FTP? FTP is a great data transfer system, but it was never designed to be a file system: –New TCP stream per data transfer. –New TCP stream for each directory list. –Lots of connections can overwhelm net devices. –Coarse errors: 550 for all file system errors. –Semantic problems: e.g. empty directory. –Unix access controls, (But, see SecPAL) –Wildly varying implementations and support.

20. FTP Protocol Reminder AUTH GSSAPI MIC Data Transfer AUTH GSSAPI MIC PORT RETR Control Connection Data Connection FTP Client FTP Server Minimum of four round trips (plus auth overhead) to fetch a file + loss of TCP window. Common practice is new control connection for every data transfer!

21. What About NFS? NFS was designed for a local area network among (relatively) trusted hosts. –Fine-grained file access very slow on WAN. –Kernel support and root assistance needed to start server, mount client, change target. –Unix UID for ownership, access control. –Need to bind to privileged port, often filtered. –Use of “file handles” to refer to files makes it very difficult to build a user-level server. + lots of lookup operations over the WAN.

22. NFS Protocol Reminder NFS Client NFS Server On a WAN, throughput limited to 4KB/latency. 10ms = 400 KB/s 100ms = 40 KB/s lookup(00,a) lookup(10,b) lookup(20,c)... read 4KB...

23. Chirp Hybrid Protocol Overview Chirp Client Chirp Server auth globus (8 RTT) open read write close... getfile(“mydata”) putfile(“otherdata”,size) size and data data

24. Protocol Comparison FTP - Stream per File –Latency = 4+ RTT for each file –Throughput = TCP limit after slow start NFS – Remote Procedure Call –Latency = 1 RTT for each file –Throughput = block size / latency Chirp - Hybrid –Latency = 1 RTT for each file –Throughput = TCP limit in steady state

25. Local Area Performance

26. Wide Area Performance

27. Real WAN Performance

28. Interfaces for Small Files

29. Standard Unix Copy Parrot cp LocalChirp Local Disk Chirp Server open(source) read open write open(source) open(target) loop: read/write cp /tmp/source /chirp/B/target

30. Problem: The system does not know the context of the operation! Solution: Introduce a higher-level operation copyfile that exploits the context.

31. Improved Copy with Copyfile Parrot new cp LocalChirp Local Disk Chirp Server copyfile(source,target) open(source) putfile(target) cp /tmp/source /chirp/B/target

32. Is it reasonable to modify cp? Installation: –Cannot modify /bin/cp. –Install new parrot_cp –Alias cp or link named “cp” in PATH. Backwards compatibility: –parrot_cp without Parrot falls back to normal. –Ordinary cp on Parrot behaves as before. –Parrot_cp on a different filesystem falls back.

33. Improved Copy with Copyfile Parrot new cp Chirp Server B copyfile(source,target) thirdput(source,B,target) Chirp Server A cp /chirp/A/source /chirp/B/target putfile(target) thirdput(source,B,target)

34. Directory Copy Chirp Server B Chirp Server A ACL XYZ mydir thirdput(/mydir/X,B,/mydir/X) X setacl(mydir) ACL mydir thirdput(/mydir/X,B,/mydir/Y) Y thirdput(/mydir/X,B,/mydir/Z) Z cp Parrot mkdir(mydir) cp –r /chirp/A/mydir /chirp/B/mydir

35. Improved Directory Copy Chirp Server B Chirp Server A ACL XYZ mydir ACL XYZ mydir mkdir putfile*3 setacl cp Parrot thirdput(/mydir,B,/mydir) cp –r /chirp/A/mydir /chirp/B/mydir

36. Third Party Performance

37. You get the idea... ls –la D ls –la D –Original: getdir D + N*stat –Improved: getlongdir D rm –rf D –Original: getdir D + N*unlink (recursive) –Improved: rmall D md5sum F –Original: open F + N*read + close –Improved: md5 F

38. Final Example ls –la /chirp/alpha/data md5sum /chirp/alpha/data/* cp -r /chirp/alpha/data /chirp/beta/data /chirp/beta/data md5sum /chirp/beta/data/* rm –rf /chirp/alpha/data

39. Original Implementation ls -lamd5cprmcpmd5 chirp server A chirp server B parrot app

40. Improved Implementation rm chirp server A chirp server B parrot app ls -lamd5cpmd5

41. Performance on Script

42. The Challenge: How to design interfaces so that users get the expected performance and behavior?

43. Summary Good small file performance requires attention to low level network protocols. –getfile, putfile, thirdput, rmall, checksum Exploiting protocols requires minor changes to the Unix I/O interface. –copyfile, rmall, checksum, others? Easy to apply those changes in a user transparent way. – cp, rm, md5sum all operate as normal Usable performance in a wide-area FS.

44. For more information... Douglas Thain –dthain@nd.edu dthain@nd.edu Chris Moretti –cmoretti@nd.edu cmoretti@nd.edu Parrot and Chirp –http://www.cctools.org http://www.cctools.org