1. A WAN-in-LAB for Protocol Development Netlab, Caltech Lachlan Andrew, George Lee, Steven Low(PI), John Doyle, Harvey Newman

2. Outline What and why is WAN-in-Lab? What and why is WAN-in-Lab? What can I do with WiL? What can I do with WiL? Why would I use WiL? Why would I use WiL? How do I use WiL? How do I use WiL? Future plans Future plans

3. What is WAN-in-Lab? “Wide Area” Network in a laboratory “Wide Area” Network in a laboratory Real fibre delaysReal fibre delays Carrier-class routers, switches, …Carrier-class routers, switches, …

4. Why -- Spectrum of tools cost abstraction mathssimulationemulationlive netwkWANinLab NS2 SSFNet QualNet JavaSim Mathis formula Optimization Control theory Nonlinear model Stocahstic model DummyNet EmuLab ModelNet WAIL UltraLight PlanetLab Abilene NLR LHCNet CENIC etc ? All scales are important– WAN-in-Lab fills a gap

5. What can I do with WAN-in-Lab?

6. Other groups’ interests Protocol development Protocol development FAST, delay-basedFAST, delay-based MaxNet, explicit signallingMaxNet, explicit signalling ADPM, single-bit explicit signallingADPM, single-bit explicit signalling Impact of small buffers (U. Pittsburgh) Impact of small buffers (U. Pittsburgh) Test automatic configuration of routers (MonALISA, Ultralight) Test automatic configuration of routers (MonALISA, Ultralight) Test distributed file-system (MojaveFS) Test distributed file-system (MojaveFS)

7. TCP Benchmarking Our current main direction Our current main direction Evaluating others’ protocols, not ours Evaluating others’ protocols, not ours Web interface Web interface Submit kernel patchSubmit kernel patch Standard tests automatically performedStandard tests automatically performed Results mailed backResults mailed back Explicit or implicit signalling protocols Explicit or implicit signalling protocols

9. Capabilities: Topology Four Cisco 7609 OC-48 routers Four Cisco 7609 OC-48 routers Line cards for four physical linksLine cards for four physical links More “virtual” links using IP routingMore “virtual” links using IP routing 18 GbE servers 18 GbE servers Two wired as software routers for AQMTwo wired as software routers for AQM 2 standalone 10GbE servers 2 standalone 10GbE servers Library of standard topologies Library of standard topologies Users can also create their ownUsers can also create their own

10. Physical topology

11. Capabilities: Delay 24 spools of 100km fibre, many loopbacks 24 spools of 100km fibre, many loopbacks Set delay by MEMS switching loops in/outSet delay by MEMS switching loops in/out 130ms physical delay 130ms physical delay more with IP loopbackmore with IP loopback 2 Dummynets: long delay for cross-traffic 2 Dummynets: long delay for cross-traffic 125 ms, 1.8ms steps

12. External connections Linked to Ultralight, 10Gbps Physics WAN Linked to Ultralight, 10Gbps Physics WAN Smooth migration testing -> deployment Smooth migration testing -> deployment Delay Delay longerlonger jitterjitter Cross traffic Cross traffic Monitor data routed through WiL Monitor data routed through WiL

13. Why use WAN-in-Lab?

14. Why use WiL? Complement other levels of abstraction, not replace them Complement other levels of abstraction, not replace them Different ways to use it: reasons for each Different ways to use it: reasons for each Standard platform for TCP benchmarking Standard platform for TCP benchmarking Easier to compare with others’ resultsEasier to compare with others’ results No need to write your own test suiteNo need to write your own test suite

15. Artifacts of software delays Packets sent on 1ms “ticks” Packets sent on 1ms “ticks” 1Gbps = 83,333 pk/s 1Gbps = 83,333 pk/s 1ms 83 packets

16. How can I use WAN-in-Lab?

17. Management structure

19. Time sharing Coarse switching between projects Coarse switching between projects Servers rebooted, routers reconfiguredServers rebooted, routers reconfigured Switchover takes ~5 minutes Switchover takes ~5 minutes Book in advance Book in advance For longer bookings, book further in advanceFor longer bookings, book further in advance Also “ad hoc” bookings for individual hostsAlso “ad hoc” bookings for individual hosts Can log in while others have booked Can log in while others have booked

20. Future plans

21. Benchmarking infrastructure Benchmarking infrastructure Standardise testsStandardise tests Use it ourselvesUse it ourselves Develop “indices” of TCP performanceDevelop “indices” of TCP performance Better control over capacities and buffers Better control over capacities and buffers Better cross-traffic generation Better cross-traffic generation Currently HarpoonCurrently Harpoon Investigate differences from DummyNet Investigate differences from DummyNet Integrate DAG cards Integrate DAG cards

22. Conclusion WAN-in-Lab fills the gap between emulation and live network experiments WAN-in-Lab fills the gap between emulation and live network experiments Seeks to be as realistic as possible Seeks to be as realistic as possible Long links, simple topologyLong links, simple topology Focus will be on TCP benchmarking Focus will be on TCP benchmarking We welcome people to use it We welcome people to use it<http://wil.cs.caltech.edu>

23. Spare Slides

24. Case Study: MaxNet

25. WAN in Lab WAN in Lab Capacity: 2.5 – 10 GbpsCapacity: 2.5 – 10 Gbps Delay: 0 – 120 ms round tripDelay: 0 – 120 ms round trip Breakable Breakable Won’t take down live networkWon’t take down live network Flexible, active debugging Flexible, active debugging Passive monitoring, AQMPassive monitoring, AQM Configurable & evolvable Configurable & evolvable Topology, rate, delays, routeTopology, rate, delays, route Modular design stays up to dateModular design stays up to date Integral part of R&A networks Integral part of R&A networks Transition from theory, implementation, demonstration, deploymentTransition from theory, implementation, demonstration, deployment Transition from lab to marketplaceTransition from lab to marketplace Global resource Global resource Part of global infrastructure UltraLight led by Harvey NewmanPart of global infrastructure UltraLight led by Harvey Newman Aim: Wind Tunnel of Networking

26. Equipment 4 Cisco 7609 routers with OC48 line cards 4 Cisco 7609 routers with OC48 line cards 6 Cisco ONS 15454 switches 6 Cisco ONS 15454 switches A few dozen high speed servers A few dozen high speed servers 1G switch to routers/servers 1G switch to routers/servers Calient switch for OC48 Calient switch for OC48 2,400 kilometres of fibre, optical amplifiers, dispersion compensation modules 2,400 kilometres of fibre, optical amplifiers, dispersion compensation modules 63ms aggregate RTT delay, in two hops 63ms aggregate RTT delay, in two hops 120ms using IP loopbacks120ms using IP loopbacks

27. Accounts Mail wil at cs.caltech.edu Mail wil at cs.caltech.edu Sudo access to “network” commands Sudo access to “network” commands Ifconfig/…/Ifconfig/…/ Custom commands to set topologiesCustom commands to set topologies Login to routers if required Login to routers if required Separate accounts for “benchmark only” Separate accounts for “benchmark only”

28. Configuration -- Delays Want maximum delay from limited fibre Want maximum delay from limited fibre Signals traverse fibre 16 timesSignals traverse fibre 16 times 4 WDM wavelengths 4 WDM wavelengths 4 OC48 (2.5G) MUXed onto OC192 (10G) 4 OC48 (2.5G) MUXed onto OC192 (10G) Lots of transponders Lots of transponders WDM amplifier joins 100km spools  200kmWDM amplifier joins 100km spools  200km

29. Configuration – delays OC48 slot Amp -------WDM Wavelength-------- Bidirectional 100km 16x200km

30. Configuration – delays Delay varied by adjusting the number of OC48 hops traversed Delay varied by adjusting the number of OC48 hops traversed Calient optical switch selects required hops Calient optical switch selects required hops Hop lengths 200km up to 1600km Hop lengths 200km up to 1600km Maximise granularity given limited switch portsMaximise granularity given limited switch ports Switch

31. Projects TCP benchmarking TCP benchmarking FAST FAST Delay-based congestion controlDelay-based congestion control MaxNet MaxNet Explicit signalling congestion controlExplicit signalling congestion control MojaveFS MojaveFS New distributed file systemNew distributed file system University of Pittsburgh University of Pittsburgh TCP with small buffersTCP with small buffers University of Melbourne University of Melbourne Single-bit congestion markingSingle-bit congestion marking

32. WAN-in-Lab testbed Dummynet and simulation introduce artifacts Dummynet and simulation introduce artifacts Also need to test on real equipment Also need to test on real equipment WAN with real delays, located in a single room WAN with real delays, located in a single room Connected to an external WAN (Ultralight)Connected to an external WAN (Ultralight) Open for the community to use for benchmarking Open for the community to use for benchmarking OC-48

33. WAN-in-Lab capabilities CurrentPlanned Two 2.5G bottlenecks Multiple 1G bottlenecks Six 2.5G bottlenecks Two “real” delays (Emulate cross traffic delay) Up to six “real” delays End-to-end RTT, drop Per-router delay, drop (movable DAG cards)

34. Configuration -- delays OC48 slot Amp -------WDM Wavelength-------- Bidirectional 100km

35. Using WAN-in-Lab Contact me – lachlan at caltech. Edu Contact me – lachlan at caltech. Edu Coarse timesharing Coarse timesharing Some users set up experiments while others run experimentsSome users set up experiments while others run experiments Software setup still being developed Software setup still being developed Your chance to influence our directions to tailor it to your needsYour chance to influence our directions to tailor it to your needs

36. Sample MaxNet results Achieves realistic delay at 1Gbit/s Achieves realistic delay at 1Gbit/s