1. Sponsored by the National Science Foundation Systematic Experimentation in GENI Sarah Edwards GENI Project Office

2. Sponsored by the National Science Foundation 2 Systematic Experimentation – GEC 20 – June 23, 2014 General Principle #1: One thing at a time Experimentation 101 and Debugging 101 Only change one thing at a time Examples: Software/image Configuration Number of nodes/links Geographic distribution of nodes/links

3. Sponsored by the National Science Foundation 3 Systematic Experimentation – GEC 20 – June 23, 2014 General Principle #2: Start Small Start by building smallest possible topology by hand. Automate as needed. Test and measure as you go. clientserver host OVS switch host router node

4. Sponsored by the National Science Foundation 4 Systematic Experimentation – GEC 20 – June 23, 2014 General Principle #3: Save what you do Log all of your experimental artifacts for every experiment that works –RSpec –image –install script –custom software –measurements –etc Use version control to store your artifacts Always know the last configuration that worked

5. Sponsored by the National Science Foundation 5 Systematic Experimentation – GEC 20 – June 23, 2014 Process Recommendation A. Build (smallest possible) topology by hand at a single aggregate B. Automate topology creation at a single aggregate C. Orchestrate and Instrument D. Increase scale E. More nodes F. More aggregates Automate

6. Sponsored by the National Science Foundation 6 Systematic Experimentation – GEC 20 – June 23, 2014 Design Choices Custom images vs install scripts vs both Single aggregate vs multi-aggregate Which aggregates? Bare metal vs virtual machine What tools are you comfortable with? There are no right answers. Pick what’s needed for your experiment.

7. Sponsored by the National Science Foundation 7 Systematic Experimentation – GEC 20 – June 23, 2014 Avoiding Common Pitfalls Don’t start with a large or inter-aggregate topology In general, if you are keeping your experiment up because you can’t recreate your setup, then something is wrong. GENI is not a simulator Common Issue: OpenFlow controllers used on an multi-aggregate topology with loop MUST be tested in a single aggregate topology with loop

8. Sponsored by the National Science Foundation 8 Systematic Experimentation – GEC 20 – June 23, 2014 “How To” pages Listed under the “Experimenters” section Each “How To” is a short descriptions of how to do various tasks New entries being added all the time

9. Sponsored by the National Science Foundation 9 Systematic Experimentation – GEC 20 – June 23, 2014 Finding other resources GENI wiki –Pages for Instructors and Experimenters http://groups.geni.net/geni/wiki

10. Sponsored by the National Science Foundation 10 Systematic Experimentation – GEC 20 – June 23, 2014

11. Sponsored by the National Science Foundation 11 Systematic Experimentation – GEC 20 – June 23, 2014 Systematic Experimentation Running an experiment on a testbed requires best practices and methodology from a combination of: –Science (Scientific method) –Programming –System Administration Today, talking about strategies and techniques for bringing up experiments in a systematic way.

12. Sponsored by the National Science Foundation 12 Systematic Experimentation – GEC 20 – June 23, 2014 Why experiment systematically? Debugging Easier to debug experiments. Easier to get help debugging. Especially for complex experiments. Scalability and Repeatability A well designed experiment will be repeatable and systematically scalable Valid Develop a solid understanding of and validate your experimental setup Rigorous Perform a scientifically rigorous (and publishable) study

13. Sponsored by the National Science Foundation 13 Systematic Experimentation – GEC 20 – June 23, 2014 GENI-FIRE 150 node topology Courtesy of Thierry Rakotoarivelo, NICTA, GEC 20 Demo Clear plan Automation Scale and Visualization

14. Sponsored by the National Science Foundation 14 Systematic Experimentation – GEC 20 – June 23, 2014 Virtual Computer Networking Lab Courtesy of Cong Wang and Divya Bhat, UMass Amherst, GEC 20 Demo Start small

15. Sponsored by the National Science Foundation 15 Systematic Experimentation – GEC 20 – June 23, 2014 Intelligent Data Movement System Courtesy of Ezra Kissel, Indiana University, GEC 20 Demo Long-lived slice for stitched, shared VLAN Long-lived slice for stitched, shared VLAN Dynamically add/remove nodes as needed

16. Sponsored by the National Science Foundation 16 Systematic Experimentation – GEC 20 – June 23, 2014 Why doesn’t this work? (Anonymous) Past Bug Reports repeatable incremental test controller in one AM? test controller in one AM? GRE works Stitch test controller in one AM test controller in one AM Multi-site Hadoop Used control plane to send high volume data between racks Interfere with rack ! OpenFlow multi-site OpenFlow multi-site OpenFlow multi-site OpenFlow multi-site

17. Sponsored by the National Science Foundation 17 Systematic Experimentation – GEC 20 – June 23, 2014 Process: Start Small A. Build smallest possible topology by hand at a single aggregate B. Automate topology creation at a single aggregate –Store RSpecs, software, etc in version control –TOOLS Use install scripts and custom images –Determine how to update software on your nodes Repeat A/B as needed. C. Orchestrate and Instrument your experiment procedure TOOLS GENI Desktop and LabWiki Automate as needed. worker master worker node

18. Sponsored by the National Science Foundation 18 Systematic Experimentation – GEC 20 – June 23, 2014 Process: Scale D. Increase scale (number of nodes, volume of traffic, etc) TOOLS Flack for small scale. geni-lib for larger scale client server client server

19. Sponsored by the National Science Foundation 19 Systematic Experimentation – GEC 20 – June 23, 2014 Process: Split across aggregates E. Split an existing topology across second aggregate –How connect across aggregates matters (stitching vs OF etc) TOOLS stitcher for VLAN stitching and GRE tunnels. ExoSM for Intra-ExoGENI stitching. Flack for GRE tunnels. F. Split across multiple aggregates client server client Colors represent different aggregates Inter-aggregate link