1. 1 A Study of Applications for Optical Circuit-Switched Networks Xiuduan Fang May 1, 2006 Supported by NSF ITR-0312376, NSF EIN-0335190, and DOE DE-FG02-04ER25640 grants

2. 2 Outline Introduction CHEETAH Background ― CHEETAH concept and network ― CHEETAH end-host software Analytical Models of GMPLS Networks Application (App) I: Web Transfer App App II: Parallel File Transfers Summary and Conclusions

3. 3 Introduction Many optical connection-oriented (CO) testbeds ― E.g., CANARIE's CA*net 4, UKLight, and CHEETAH ― Primarily designed for e-Science apps Use Generalized Multiprotocol Label Switching (GMPLS) ― Immediate request, call blocking Motivation: extend these GMPLS networks to million of users Problem Statement ― What apps are well served by GMPLS networks? ― Design apps to use GMPLS networks efficiently

4. 4 Circuit-switched High-speed End-to-End Transport ArcHitecture (CHEETAH) Designed as an “ add-on ” service to the Internet and leverages the services of the Internet Optical circuit- switched CHEETAH network Packet-switched Internet End host NIC I NIC II End host NIC I NIC II IP router Ethernet-SONET gateway Ethernet-SONET gateway CHEETAH concept

5. 5 CHEETAH Network zelda4 Sycamore SN16000 1G ORNL, TN Atlanta, GA NC Direct fibers VLANs MPLS tunnels mvstu6 UVa CUNY zelda5 Sycamore SN16000 zelda3 zelda1 zelda2 OC-192 lambda MCNC Catalyst 7600 wukong SN16000 UVa Catalyst 4948 NCSU M20 Centuar FastIron FESX448 WASH Abilene T640 NYC HOPI Force10 WASH HOPI Force10 CUNY Foundry CUNY Host

6. 6 CHEETAH End-host Software OCS: Optical Connectivity Service RD: routing decision RSVP-TE: ReSerVation Protocol-Traffic Engineering C-TCP: Circuit-TCP

7. 7 Outline Introduction CHEETAH Background ― CHEETAH concept and network ― CHEETAH end-host software Analytical Models of GMPLS Networks Application (App) I: Web Transfer App App II: Parallel File Transfers Summary and Conclusions

8. 8 Assumptions: ― Call arrival rate, (Poisson process) ― Single link ― Single class: all apps are of the same type A link of capacity C; m circuits; per-circuit BW=C/m m is a measure of high-throughput vs. moderate-throughput For high-throughput (e.g., e-Science apps), m is small Problem: what apps are suitable for GMPLS networks? Analytical Models of GMPLS Networks ― Measure of suitability: Call-blocking probability, P b Link utilization, U ― App properties: Per-circuit BW Call-holding time,

9. 9 BW sharing models is independent of … 1 N Link L, capacity C …, … 1 N RD is dependent on File size distribution: :shape, k :scale Two kinds of apps: whether is dependent on The Erlang-B formula:crossover file size

10. 10 Numerical Results: is independent of Two equations, four variables Fix U and m, compute P b and

11. 11 Numerical Results: is independent of m=10 P b =23.62% Conclusions: to get high U Small m (~10): high P b, thus book-ahead or call queuing Large m (~1000): high, thus large N Intermediate m (~100): large is preferred

12. 12 Conclusions: to get high U Small m (~10): high P b, thus book-ahead or call queuing As m increases, N does not increase m=100, to get U>80%, P b <5%: 6MB< <29MB, thus Numerical Results: is dependent on, when

13. 13 Conclusions for Analysis Ideal apps require BW on the order of one- hundredth the link capacity as per-circuit rate Apps where is independent of ― long call-holding time is preferred Apps where is dependent on ― need short call-holding time

14. 14 Outline Introduction CHEETAH Background ― CHEETAH concept and network ― CHEETAH end-host software Analytical Models of GMPLS Networks Application (App) I: Web Transfer App App II: Parallel File Transfers Summary and Conclusions

15. 15 APP I: Web Transfer App on CHEETAH Why web transfer? ― Web-based apps are ubiquitous ― Based on the previous analysis, m=100 is suitable for CHEETAH Consists of a software package WebFT ― Leverages CGI for deployment without modifying web client and web server software ― Integrated with CHEETAH end-host software APIs to allow use of the CHEETAH network in a mode transparent to users

16. 16 Control messages via Internet WebFT Architecture Web server Web client Web Server (e.g. Apache) CGI scripts (download.cgi & redirection.cgi URL Response WebFT sender OCS APIRD API RSVP-TE API C-TCP API Web Browser (e.g. Mozilla) WebFT receiver RSVP-TE API C-TCP API Data transfers via a circuit OCS daemon RD daemon RSVP-TE daemon RSVP-TE daemon Cheetah end-host software APIs and daemons Cheetah end-host software APIs and daemons

17. 17 Experimental Testbed for WebFT zelda3 and wukong: Dell machines, running Linux FC3 and ext2/3, with RAID-0 SCCI disks RTT between them: 24.7ms on the Internet path, and 8.6ms for the CHEETAH circuit. load Apache HTTP server 2.0 on zelda3 CHEETAH Network Internet zelda3 NIC I NIC II wukong NIC I NIC II IP routers NCSUAtlanta, GA Sycamore SN16000 Atlanta, GA Sycamore SN16000 MCNC, NC

18. 18 Experimental Results for WebFT The web page to test WebFT Test parameters: ― Test.rm: 1.6 GB, circuit rate: 1 Gbps Test results ― throughput: 680 Mbps, delay: 19 s

19. 19 Outline Introduction CHEETAH Background ― CHEETAH concept and network ― CHEETAH end-host software Analytical Models of GMPLS Networks Application (App) I: Web Transfer App App II: Parallel File Transfers Summary and Conclusions

20. 20 APP II: Parallel File Transfers on CHEETAH Motivation: E-Science projects need to share large volumes of data (TB or PB) Goal: achieve multi-Gb/s throughput Two factors limit throughput ― TCP ’ s congestion-control algorithm ― End-host limitations Solutions to relieve end-host limitations ― Single-host solution ― Cluster solution, which has two variations General case: non-split source file Special case: split source file

21. 21 General-Case Cluster Solution Original Source Host 1 Host i Host n split Host 1 ’ Host i ’ Host n ’ Original Sink transfer assemble … … … … … …

22. 22 Software Tools: GridFTP and PVFS2 GridFTP: a data-transfer protocol on the Grid ― Extends FTP by adding features for partial file transfer, multi-streaming and striping ― We mainly use the GridFTP striped transfer feature. PVFS: Parallel Virtual File System ― An open source implementation of a parallel file system ― Stripes a file across multiple I/O servers like RAID0 ― A second version: PVFS2

23. 23 SPOR response to SPOR GridFTP server globus-url-copy GridFTP striped transfer Block 1 Block n+1 … Block 1 Block n+1 … data node R1 data node Rn Parallel File System GridFTP server … Block 1 Block n+1 … Block 1 Block n+1 … data node S1 data node Sn Parallel File System … receiving front end sending front end SPAS a list of host-port pairs Sending data nodes initiate data connections to receiving nodes

24. 24 General-Case Cluster Solution: Design StepsApproachPros.Cons. Splitting & Assembling GridFTP partial file transfer Wastes disk space, Performance overhead Socket program Avoids wasting disk space Performance overhead pvfs2-cp Avoids wasting disk space Transferring GridFTP partial file transfer Many independent transfers incurring much overhead to set up and release connections GridFTP striped transfer A single file transfer

25. 25 General-Case Cluster Solution: Implementation To get a high throughput, we need to make data nodes responsible for data blocks in their local disks Block 1 Block n+1 … Block 1 Block n+1 … data node R1 data node Rn PVFS2 Block 1 Block n+1 … Block 1 Block n+1 … data node S1 data node Sn PVFS2 … … ― Make PVFS2 and GridFTP have the same stripe pattern Problems: ― PVFS2 1.0.1 does not provide a utility to inspect data distribution ― Data connections between sending and receiving nodes are random

26. 26 Random data connections Block 1 Block n+1 … Block 1 Block n+1 … data node R1 data node Rn PVFS2 Block 1 Block n+1 … Block 1 Block n+1 … data node S1 data node Sn PVFS2 … …

27. 27 Random data connections Block 1 Block n+1 … Block 1 Block n+1 … data node R1 data node Rn PVFS2 Block 1 Block n+1 … Block 1 Block n+1 … data node S1 data node Sn PVFS2 … …

28. 28 Implementation - Modifications to PVFS2 Goal: know a priori how a file is striped in PVFS2 Use strace command to trace systems calls called by pvfs2-cp ― Pvfs2-fs-dump gives the (non-deterministic) I/O server order of file distribution ― Pvfs2-cp ignores the – s option for configuring stripe size Modify PVFS2 code ― For load balance, PVFS2 stripes files starting with a random server: jitter = (rand() % num_io_servers); ― Set jitter = -1 to get a fixed order of data distribution ― Change the default stripe size (original: 64KBytes)

29. 29 Implementation - Modifications to GridFTP Goal: use a deterministic matching sequence between sending and receiving data nodes Method: modify the implementation of SPAS and SPOR commands ― SPAS: sort the list of host-port pairs based on the IP-address order for receiving data nodes ― SPOR: request sending data nodes to initiate data connections sequentially to receiving data nodes

30. 30 Experimental Results Conducted on a 22-node cluster, sunfire Reduced network-and-disk contention Performance of PVFS2 implementation was poor

31. 31 Summary and Conclusions Analytical Models of GMPLS Networks ― Ideal apps require BW on the order of one- hundredth the link capacity as per-circuit rate Application I: Web Transfer Application ― provided deterministic data services to CHEETAH clients on dedicated end-to-end circuits ― No modifications to the web client and web server software by leveraging CGI Application II: Parallel File Transfers ― Implemented a general-case cluster solution by using PVFS2 and GridFTP striped transfer ― Modified PVFS2 and GridFTP code to reduce network-and-disk contention

32. 32 Publication Lists M. Veeraraghavan, X. Fang, and X. Zheng, On the suitability of applications for GMPLS networks, submitted to IEEE Globecom2006 X. Fang, X. Zheng, and M. Veeraraghavan, Improving web performance through new networking technologies, IEEE ICIW'06, February 23-25, 2006 Guadeloupe, French Caribbean Improving web performance through new networking technologies

33. 33 Future Work Analytical Models of GMPLS Networks ― Multi-class ― Multiple links and network models Application I: Web Transfer Application ― Design a Web partial CO transfer to enable non- CHEETAH hosts to use CHEETAH ― Connect multiple CO networks to further reduce RTT Application II: Parallel File Transfers ― Test the general-case cluster solution on CHEETAH ― Work on PVFS2 or try GPFS to get a high I/O throughput

34. 34 A Classification of Networks that Reflects Sharing Modes

35. 35 The flow chart for the WebFT sender

36. 36 The WebFT Receiver Integrates with the CHEETAH end-host software modules similar to the WebFT sender. Runs as a daemon in the background on the client host to avoid manual intervention. Also provides the WebFT sender a desired circuit rate.

37. 37 Experimental Results for WebFT

38. 38 PVFS2 Architecture

39. 39 Experimental Configuration Configuration of PVFS2 I/O servers ― The 1 st PVFS2: sunfire1 through sunfire5 ― The 2 nd PVFS2: sunfire10, and sunfire6 through 9 Configuration of GridFTP servers ― Sending front end: sunfire1 with data nodes sunfire1 through sunfire5 ― Receiving front end: sunfire10 with data nodes sunfire10, sunfire6 through sunfire9 GridFTP striped transfer globus-url-copy -vb – dbg -stripe ftp://sunfire1:50001/pvfs2/test_1G ftp://sunfire10:50002/pvfs2/test_1G1 2>dbg1.txt

40. 40 Four Conditions to Avoid Unnecessary Network-and-disk Contention Know a priori how data are striped in PVFS2 PVFS2 I/O servers and GridFTP servers run on the same hosts GridFTP stripes data across data nodes in the same sequence as PVFS2 does across PVFS2 I/O servers GridFTP and PVFS2 have the same stripe size

41. 41

42. 42 The Specific Cluster Solution for TSI

43. 43 Numerical Results for is dependent on Conclusions: Large m (~1000): does not increase N