1. emulab.net: An Emulation Testbed for Networks and Distributed Systems Jay Lepreau and many others University of Utah Intel IXA University Workshop June 21, 2001

2. The Main Players Undergrads –Chris Alfeld, Chad Barb, Rob Ricci Grads –Dave Andersen, Shashi Guruprasad, Abhijeet Joglekar, Indrajeet Kumar, Mac Newbold Staff –Mike Hibler, Leigh Stoller Alumni –Various (Red: here at Intel today)

3. What? A configurable Internet emulator in a room –Today: 200 nodes, 500 wires, 2x BFS (switch) –virtualizable topology, links, software Bare hardware with lots of tools An instrument for experimental CS research Universally available to any remote experimenter Simple to use

4. What’s a Node? Physical hardware: PCs, StrongARMs Virtual node: –Router (network emulation) –Host, middlebox (distributed system) Future physical hardware: IXP1200 +

5. Why? “We evaluated our system on five nodes.” -job talk from university with 300-node cluster “We evaluated our Web proxy design with 10 clients on 100Mbit ethernet.” “Simulation results indicate...” “Memory and CPU demands on the individual nodes were not measured, but we believe will be modest.” “The authors ignore interrupt handling overhead in their evaluation, which likely dominates all other costs.” “Resource control remains an open problem.”

6. Why 2 “You have to know the right people to get access to the cluster.” “The cluster is hard to use.” “ runs FreeBSD 2.2.x.” “October’s schedule for is…” “ is tunneled through the Internet”

7. Complementary to Other Experimental Environments Simulation Small static testbeds Live networks Maybe someday, a large scale set of distributed small testbeds (“Access”)

8. Switched “Backplane” PC Web/DB/SNMP Switch Mgmt Users Internet Control Switch/Router Serial Sharks 160 40 PowerCntl

9. Zoom In: One Node

10. Fundamental Leverage: Extremely Configurable Easy to Use

11. Key Design Aspects Allow experimenter complete control … but provide fast tools for common cases –OS’s, disk loading, state mgmt tools, IP, traffic generation, batch,... Virtualization –of all experimenter-visible resources –node names, network interface names, network addresses –Allows swapin/swapout

12. Design Aspects (cont’d) Flexible, extensible, powerful allocation algorithm Persistent state maintenance: –none on nodes –all in database –leverage node boot time: only known state! Separate control network Familiar, powerful, extensible configuration language: ns

13. Some Unique Characteristics User-configurable control of “physical” characteristics: shaping of link latency/bandwidth/drops/errors (via invisibly interposed “shaping nodes”), router processing power, buffer space, … Node breakdown today: –40 core, 160 edge

14. More Unique Characteristics Capture of low-level node behavior such as interrupt load and memory bandwidth User-replaceable node OS software User-configurable physical link topology Completely configurable and usable by external researchers, including node power cycling

15. Obligatory Pictures

16. Then

17. Now

18. A Few Research Issues and Challenges Network management of unknown entities Security Scheduling of experiments Calibration, validation, and scaling Artifact detection and control NP-hard virtual --> physical mapping problem Providing a reasonable user interface ….

19. An “Experiment” emulab’s central operational entity Directly generated by an ns script, … then represented entirely by database state Steps: Web, compile ns script, map, allocate, provide access, assign IP addrs, host names, configure VLANs, load disks, reboot, configure OS’s, run, report

20. Mapping Example

21. Automatic mapping of desired topologies and characteristics to physical resources Algorithm goals: –minimize likelihood of experimental artifacts (bottlenecks) –“optimal” packing of multiple simultaneous experiments – Extensible for heterogenous hardware, software, new features Randomized heuristic algorithm: simulated annealing May move to genetic algorithm

22. Virtual Topology

23. Mapping into Physical Topology

24. Mapping Results < 1 second for first solution, 40 nodes “Good” solution within 5 seconds Apparently insensitive to number of node “features”

25. Disk Loading 13 GB generic IDE 7200 rpm drives Was 20 minutes for 6 GB image Now 88 seconds

26. How? Do obvious compression … and a little less obvious: zero the fs Disk writes become the bottleneck Hack the disk driver Carefully overlap I/O and decompression ==> 6 minutes

27. Last Step Domain-specific compression Type the filesystem blocks: –allocated –free Never write the free ones

28. Experiment Creation Time

29. Experiment Termination Time

30. Ongoing and Future Work Multicast disk images IXP1200 nodes, tools, code fragments –Routers, high-capacity shapers Event system Scheduling system Topology generation tools and GUI Simulation/enulation transparency Linked testbeds Wireless nodes, Mobile nodes Logging. Visualization tools Microsoft OSs, high speed links, more nodes!

31. Final Remarks 18 projects have used it (14 external) –Plus several class projects Two OSDI’00 and three SOSP’01 papers –20% SOSP general acceptance rate –60% SOSP acceptance rate for emulab users! More emulab’s under construction: –Yes: Univ. of Kentucky, Stuttgart –Maybe: WUSTL, Duke Sponsors (red: major ones, current or expected) –NSF, DARPA, University of Utah –Cisco, Intel, Compaq, Microsoft, Novell, Nortel

32. Available for universities, labs, and companies at: www.emulab.net