Presentation on theme: "UW EE Colloquium Feb. 20091 Network Simulation and Emulation in a Testbed Era Tom Henderson University of Washington The Boeing Company UW EE Colloquium."— Presentation transcript:
UW EE Colloquium Feb. 20091 Network Simulation and Emulation in a Testbed Era Tom Henderson University of Washington The Boeing Company UW EE Colloquium February 24, 2009
UW EE Colloquium Feb. 2009 What is a network simulator? A software tool and set of models used to evaluate computer network behavior A discrete-event network simulator describes networks in terms of a sequence of discrete events –The simulation kernel temporally orders and dispatches events to models –Models generate output or more events –Time advances in discrete steps
UW EE Colloquium Feb. 20094 Application The basic model Application Protocol stack Node NetDevice Application Protocol stack Node NetDevice Channel Packet Simul
UW EE Colloquium Feb. 2009 The basic model // Define your topology Node n0 = new Node; Node n1 = new Node; AddInternetStack (n0, n1); Channel c0 = new Channel;. Connect (n0, n1, c0); Application a0 = new TrafficGenerator; // Configure things a0.SetDataRate (1Mb/s); a0.Start (10.0 seconds); // Define outputs WriteTraceFile (“outfile”); // Run the simulator Simulator::Run();
UW EE Colloquium Feb. 20096 Network simulation remains popular Many commercial products –e.g. OPNET, QualNet, Shunra VE, Matlab Simulink, OMNet++, NCTUns, SSFNet, Extend A few free, open source projects –ns-2 is the most widely used such tool –roughly 10,000 downloads/month Widely used in research Source: Search of ACM Digital Library papers citing simulation, 2001-04
UW EE Colloquium Feb. 20097 Skepticism in the research community “For years, the community had to rely on simulators, which now seem a little dated, and it’s not clear who was convinced to adopt anything new based on ns2 simulations;” Nick McKeown, VINI public review, ACM Sigcomm 2006
UW EE Colloquium Feb. 2009 What has changed? Simulators have historically been used for experiments in place of real machines –A tradeoff of realism for scale But now... Cost of test hardware has plummeted Virtualization technologies proliferate
UW EE Colloquium Feb. 2009 Emulab (University of Utah) Source: http://www.emulab.net
UW EE Colloquium Feb. 2009 Emulab Original Emulab allowed experiments of roughly ~100 nodes –FreeBSD Dummynet and VLANs to interconnect the nodes Emulab now offers virtualization –Experiments of up to ~2000 (virtual) nodes –Requires new approaches for experimental configuration Also, some 802.11 wireless nodes Source: M. Hibler et al., “Large-scale Virtualization in the Emulab Network Testbed,” Proc. 2008 Usenix Technical Conference.
UW EE Colloquium Feb. 2009 PlanetLab: Global-scale Internet testbed Source: http://www.planet-lab.org PlanetLab “slices”
UW EE Colloquium Feb. 2009 PL-VINI and Trellis PL-VINI adds virtual networking capabilities (via network overlay) to PlanetLab –A. Bavier et al, “In VINI Veritas: Realistic and Controlled Network Experimentation,” Proc. ACM Sigcomm, 2006 Trellis is a kernel-based implementation of the same concept –S. Bhatia et al, “Trellis: A Platform for Building Flexible, Fast Virtual Networks on Commodity Hardware,” ROADS Workshop, Dec. 2008
UW EE Colloquium Feb. 2009 ORBIT (Rutgers WINLAB) Source: http://www.winlab.rutgers.edu/docs/focus/ORBIT.html 802.11a/b/g radios Bluetooth GNU/USRP radios Noise generators
UW EE Colloquium Feb. 200914 Simulation “out of the loop” Source: NSF GENI project: http://www.geni.net/docs/GENIOvrvw092908.pdf
UW EE Colloquium Feb. 200915 Skepticism in the research community “For years, the community had to rely on simulators, which now seem a little dated, and it’s not clear who was convinced to adopt anything new based on ns2 simulations;” Nick McKeown, VINI public review, ACM Sigcomm 2006
UW EE Colloquium Feb. 200916 Overheard* on e2e-interest mailing list “...Tragedy of the Commons...” “...around 50% of the papers appeared to be... bogus...” “Who has ever validated NS2 code?” “To be honest, I'm still not sure whether I will use a simulation in a paper.” “...I will have a hard time accepting or advocating the use of NS-2 or any other simulation tool”... * September 2005 archives of the e2e-interest mailing list
UW EE Colloquium Feb. 2009 Questions Do we need to simulate networks anymore? What kind of network simulator is needed? Why should you believe network simulation results?
UW EE Colloquium Feb. 2009 Why simulate? Wireless –Reproducibility –Fidelity (especially, real-time constraints) –Radios may not exist or be available –Field tests in realistic conditions cost $$ Scalability –10,000+ nodes? –For smaller configurations, execution time Simulation is often an appropriate tool for the job
UW EE Colloquium Feb. 200919 Challenges for ns (and simulators) Align with how research is now conducted Improve credibility Can a new simulator help with these problems?
UW EE Colloquium Feb. 200920 What is ns-3? An open source project building a new network simulator to replace ns-2
UW EE Colloquium Feb. 200921 ns-3 project goal Develop a preferred, open simulation environment for networking research 1) a tool aligned with the simulation needs of modern networking research 2) an open-source project that encourages community contribution, peer review, and validation of the software
UW EE Colloquium Feb. 200922 Relationship to ns-2 ns-3 is a new simulator, without backward compatibility Similarities to ns-2: C++ software core GNU GPLv2 licensing ported ns-2 models: random variables, error models, OLSR, Calendar Queue scheduler, (more planned) Differences: Python scripting (or C++ programs) replaces OTcl most of the core rewritten also based on the yans and GTNetS simulators new animators, configuration tools, etc. are in work
UW EE Colloquium Feb. 200923 Funding support ns-3 project is supported by: NSF CISE CRI awards (2006-10) –PIs: Tom Henderson and Sumit Roy (University of Washington), George Riley (Georgia Institute of Technology) –NSF CNS-0551686 (University of Washington) INRIA Planete and the French government With additional support from: 2008 Google Summer of Code University of Washington Georgia Institute of Technology ‘
UW EE Colloquium Feb. 200924 Challenges for ns (and simulators) Align with how research is now conducted Improve credibility
UW EE Colloquium Feb. 2009 Test and Evaluation Options When mathematical analysis alone won’t do Increasing realism Increasing complexity Pure simulation Simulation cradles Virtual/Physical testbeds Field experiments Live networks Test and evaluation options Can we develop tools to span this space?
UW EE Colloquium Feb. 2009 Some ns-3 design goals 1.Realism 2.Software reuse 3.Emulation 4.Composition
UW EE Colloquium Feb. 200927 1) Attention to realism Research should often involve a mix of simulations and testbed or live experiments If the simulator cannot be made to closely model a real system: –hard to compare results or validate the model –hard to reuse software between the two domains When in doubt, do what implementations do
UW EE Colloquium Feb. 200928 Attention to realism (example) An ns-3 Node is a husk of a computer to which applications, stacks, and NICs are added Application “DTN”
UW EE Colloquium Feb. 200929 Realism (cont.) Align ns-3 to be more faithful representation of real computers –sockets API –packets are “packed” binary representations ready for direct serialization to/from network devices –alignment with Linux architecture –multiple interfaces handled properly
UW EE Colloquium Feb. 200930 ns-3 follows a Linux architecture class NetDevice class Ipv4Protocol Linux corollaries –each NetDevice is modelled like struct net_device –each layer-3 protocol has a struct in_device component for each NetDevice –this interface (transmit side) is aligned with Linux dev_queue_xmit() –on receive side, demux is modelled after Linux protocol handlers
UW EE Colloquium Feb. 200931 2) software integration Do not reimplement models and tools for which open-source implementations abound ns-3 conforms to standard input/output formats so that other tools can be reused. –e.g., pcap trace output, ns-2 mobility scripts ns-3 is adding support for running implementation code –Network Simulation Cradle integration has met with success: Linux TCP code –ns-3 “process” API
UW EE Colloquium Feb. 2009 ns-3 simulation cradle Port by Florian Westphal of Sam Jansen’s Ph.D. work Figure reference: S. Jansen, Performance, validation and testing with the Network Simulation Cradle. MASCOTS 2006.
UW EE Colloquium Feb. 2009 ns-3 simulation cradle For ns-3: Linux 2.6.18 Linux 2.6.26 Linux 2.6.28 Others: FreeBSD 5 lwip 1.3 OpenBSD 3 Other simulators: ns-2 OmNET++ Figure reference: S. Jansen, Performance, validation and testing with the Network Simulation Cradle. MASCOTS 2006.
UW EE Colloquium Feb. 200934 ns-3 “processes” and Posix API Support for a synchronous Posix socket API each Posix type and function is redefined in the simulator processes get their own private stack –somewhat like a lightweight virtual machine Benefits: –makes porting real world application code much easier –makes writing applications easier because the BSD socket API is faithfully followed see the “mathieu/ns-3-simu” code repository
UW EE Colloquium Feb. 200935 3) emulation support Support moving between simulation and testbeds or live systems A real-time scheduler, and support for two modes of emulation
UW EE Colloquium Feb. 200936 ns-3 emulation modes virtual machine ns-3 virtual machine 1) ns-3 interconnects real or virtual machines real machine ns-3 Testbed real machine ns-3 2) testbeds interconnect ns-3 stacks real machine Various hybrids of the above are possible
UW EE Colloquium Feb. 200937 Example: ORBIT and ns-3 Support for use of Rutgers WINLAB ORBIT radio grid
UW EE Colloquium Feb. 2009 4) Composition ns-3 is a component of an overall workflow Define ns-3 with interfaces to support integration with other tools Topology generators Traffic generators Data analysis frameworks Configuration tools Animators
UW EE Colloquium Feb. 200943 ns-3 and research alignment In summary, make it easier to move from simulation to emulation to experiments Realism and alignment with popular interfaces Support use of real code Emulation capabilities Combine ns-3 with other tools
UW EE Colloquium Feb. 200944 Challenges for ns (and simulators) Align with how research is now conducted Improve credibility
UW EE Colloquium Feb. 200945 Background on simulation credibility  “Why We STILL Don’t Know How to Simulate Networks” –Mostafa Ammar, Georgia Institute of Technology, Annual Simulation Symposium 2005  “Maintaining a Critical Attitude Towards Simulation Results” –Sally Floyd, WNS2 Workshop Keynote, October 2006  “MANET Simulation Studies: The Incredibles” –Kurkowski, Camp, and Colagrosso, ACM Sigmobile, MC2R, Volume 9, Issue 4, October 2005  “An Integrated Approach to Evaluating Simulation Credibility” –Muessig, Laack, and Wrobleski, U.S. Naval Air Warfare Center, August 2001
UW EE Colloquium Feb. 200946 Criteria for Credibility Repeatable Unbiased Realistic Scenarios Statistically Sound Model Accuracy Results Accuracy (Validation) Data Accuracy Usability from  from 
UW EE Colloquium Feb. 200947 Repeatability A hallmark of the scientific method... Papers should identify simulator, version, operating system, parameters, etc. Better yet, make code and configuration scripts available to the community –Yet, 0 out of 84 ACM Mobihoc MANET simulation papers (2000-2004) referenced publicly available code (from )
UW EE Colloquium Feb. 2009 ns-3 approach We are adding support to systematically document how simulations were conducted –A configuration subsystem that dumps simulation parameters to a configuration file (input and output) We will host code, publications, validation results, etc.
UW EE Colloquium Feb. 200949 Model accuracy “error-free-ness” of software and models ns-3 goals here: –Support real code where possible –Open source models –Maintainers
UW EE Colloquium Feb. 200950 Open source models “Given enough eyeballs, all bugs are shallow” –Eric Raymond, “The Cathedral and the Bazaar” ns-3 needs ways to certify models, too –capture level of community acceptance –publication lists, cross-reference –need to identify maintainers, or state the absence of a maintainer –validation techniques and results
UW EE Colloquium Feb. 2009 Example: ns-3 Wifi development Several research groups are maturing the original INRIA model: Karlsruhe Institute of Technology: 802.11 PHY, 802.11e –Equalizing PHY models including capture effects, user-definable coding rates (e.g. 5.9 GHz from 802.11p), EDCA QoS extensions of 802.11e, Nakagami/Rayleigh propagation loss model University of Florence: 802.11n features –Frame Aggregation, Block ACK, HCF (EDCA and support for HCCA),TXOP, HT terminal (also with protection modes), MIMO Russian Academy of Sciences: 802.11s –a complete model of IEEE802.11s D2.0 Draft Standard Deutsche Telekom Laboratories in Berlin: 802.11 PHY Boeing: 802.11b channel models, validation (and others...)
UW EE Colloquium Feb. 2009 Pledge break ns-3 needs participation from the research community 1) improving simulation credibility 2) contributed and supported models 3) maintainers Please support your open source projects!
UW EE Colloquium Feb. 200953 Summary of simulation credibility Learn from good and bad examples of simulation research, to produce credible simulations Consider open source (or publishing of models and scripts) to be integral part of your research Please give back to the simulators that you use
UW EE Colloquium Feb. 200954 Acknowledgments Thanks to: the core development team and research project leads –Raj Bhattacharjea, Gustavo Carneiro, Walid Dabbous, Craig Dowell, Joe Kopena, Mathieu Lacage (software lead), George Riley, Sumit Roy 2008 Google Summer of Code mentors and students many code authors and testers the ns-2 PIs and developers for creating ns-2 and for supporting ns-3 activities USC ISI for hosting project mailing lists Contact: email@example.com (office M354)