1. 1 Cluster or Network? An Emulation Facility for Research Jay Lepreau Chris Alfeld David Andersen (MIT) Mac Newbold Rob Place Kristin Wright Dept. of Computer Science University of Utah http://www.cs.utah.edu/flux/testbed/ February 3, 2000

2. 2 Research We Do Operating systems, local and distributed Distributed systems ÜWeb caching schemes, distributed objects,... Active Networks Ücode in every packet: route me! ÜConfigurable router Router operating systems

3. 3 What? A configurable Internet (cluster) in a room Ü230 nodes, 1000 links, BFS (switch) Üvirtualizable topology, links, software An instrument for experimental CS research Universally available to any remote experimenter Simple to use!

4. 4 Why? “We evaluated our system on five nodes.” -job talk from university with 300-node cluster “We evaluated our Web proxy design with 10 clients on 100Mbit ethernet.” “Simulation results indicate...” “Memory and CPU demands on the individual nodes were not measured, but we believe will be modest.” “The authors ignore interrupt handling overhead in their evaluation, which likely dominates all other costs.” “Resource control remains an open problem.”

5. 5 Why 2 “You have to know the right people to get access to the cluster.” “The cluster is hard to use.” “ runs FreeBSD 2.2.x.” “October’s schedule for is…” “ is tunneled through the Internet”

6. 6 Complementary to Other Experimental Environments Simulation Small static testbeds Live networks Maybe someday, a large scale set of distributed small testbeds (“Access”)

7. 7 Some Unique Characteristics Significant scale: initially 225 nodes, degree four 100Mb links between 42 core routers. User-configurable control of “physical” characteristics: shaping of link latency/bandwidth/drops/errors (via invisibly interposed “shaping nodes”), router processing power, buffer space, … Node breakdown: 42 core, 160 edge, 26 shaping, 2 management

8. 8 More Unique Characteristics Capture of low-level node behavior such as interrupt load and memory bandwidth User-replaceable node OS software User-configurable physical link topology (VLAN via BFS; “P-LAN” via BFPP) Completely configurable and usable by external researchers, including node power cycling

9. 9 Fundamental Research Leverage: Extremely Configurable

10. 10 Obligatory Pictures

11. 11 Prototype Pieces: edge nodes

12. 12 Big Iron

13. 13 A View from the Dark Side

14. 14 And the Light Side

15. 15 Artist’s Conception

16. 16 Zoom in: “Delay” Node

17. 17 Feature: Automatic mapping of desired topologies and characteristics to physical resources Algorithm goals: Üminimize likelihood of experimental artifacts (bottlenecks) Ü“optimal” packing of multiple simultaneous experiments Ü Complete in finite time! Constraint-based heuristic algorithm (version 2!) Feature: accepts ns-compatible specification

18. 18 Current Algorithm Simulated annealing ÜMake random change (move node from one switch to another), compute score, accept/reject based on current temp. Heuristic algorithm ~ 4 seconds for 30 nodes; polynomial Improve: ÜHardwired node connections will slow it down x100 ÜEdge nodes ÜSpeed - incremental score recomputation

19. Virtual Topology

20. Mapping into Physical Topology

21. 21 Roatan: Remote Console for a Node

22. 22 Early Network Configuration GUI

23. 23 Research Applications Simulation validation Active networks Resource demands of services inside routers Denial-of-service resistance Interaction of adaptive applications and protocols All sorts of distributed system experiments...

24. 24 Research Applications (continued) Detailed performance monitoring and analysis Relationships between {node, link, topology} characteristics and ÜApplication performance ÜTask scheduling and assignment ÜCommunication software ÜApplication algorihms Ü….

25. 25 Study: Interconnection Techniques Point-to-point vs.always through a switch ÜSalmon et al (Caltech) Cost vs. performance Of most interest on large clusters Locality of communication patterns Interference with local processing Ad hoc mobile networking

26. 26 Research Issues and Other Challenges Calibration, validation, and scaling: how to emulate different speed networks? Scaling behavior of emulating faster links by slowing nodes? Can we sufficiently capture real router internal behavior in a PC? Assuring validity: detecting switch bottlenecks, measuring and controlling physical characteristics without introducing artifacts. Algorithms and software to map requirements to resources while minimizing artifacts. Integrate with ns? Providing a reasonable user interface to all this.

27. 27 Final Remarks Should be limping next month Looking for feedback on your potential use Looking for early users Collaborators/clients: UU Physics, CMU CS, MIT CS, Georgia Tech, IBM research Sponsors: University of Utah, Novell, DARPA, Compaq, Nortel,