1. Sponsored by the National Science Foundation Scaling Up: Growing the Topology of an Existing Experiment in GENI Sarah Edwards GENI Project Office with input from Xuan Liu, UMKC

2. Sponsored by the National Science Foundation 2 Scaling Up – GREE-SC 15 – May 28, 2015 Agenda Background and Motivation Hands-On: Medium sized, multi-site topology –A multi-slice, multi-site technique for building larger stitched topologies –Building large RSpecs geni-lib based scaleup tool

3. Sponsored by the National Science Foundation 3 Scaling Up – GREE-SC 15 – May 28, 2015 Why are we here? At GEC20, we talked about Systematic Experimentation A quick review… Build smallest reasonable topology by… –… changing one thing at a time … –… automating as you go … –… saving what you do. THEN scale up. Today clientserver router

4. Sponsored by the National Science Foundation 4 Scaling Up – GREE-SC 15 – May 28, 2015 Scaling Up Assumption #1 You have a small scale working experiment Assumption #2 Install and orchestration scripts are auto-configuring Today we will discuss how to make your topology BIGGER OR use a configuration management tool

5. Sponsored by the National Science Foundation 5 Scaling Up – GREE-SC 15 – May 28, 2015 General Principle #1: One thing at a time Experimentation 101 and Debugging 101 Only change one thing at a time Examples: Software/image Configuration Number of nodes/links Geographic distribution of nodes/links

6. Sponsored by the National Science Foundation 6 Scaling Up – GREE-SC 15 – May 28, 2015 General Principle #2: Start Small … and then scale up Start by building smallest possible topology by hand. Automate as needed. Test and measure as you go. clientserver host OVS switch host router worker master worker node

7. Sponsored by the National Science Foundation 7 Scaling Up – GREE-SC 15 – May 28, 2015 General Principle #3: Save what you do Log all of your experimental artifacts for every experiment that works –RSpec –image –install script –custom software –measurements –etc Use version control to store your artifacts Always know the last configuration that worked

8. Sponsored by the National Science Foundation 8 Scaling Up – GREE-SC 15 – May 28, 2015 Process Recommendation A. Build (smallest possible) topology by hand at a single aggregate B. Automate topology creation at a single aggregate C. Orchestrate and Instrument D. Increase scale E. More nodes F. More aggregates Automate

9. Sponsored by the National Science Foundation 9 Scaling Up – GREE-SC 15 – May 28, 2015 Process Recommendation A. Build (smallest possible) topology by hand at a single aggregate B. Automate topology creation at a single aggregate C. Orchestrate and Instrument D. Increase scale E. More nodes F. More aggregates Automate Getting Started Tutorials

10. Sponsored by the National Science Foundation 10 Scaling Up – GREE-SC 15 – May 28, 2015 Process Recommendation A. Build (smallest possible) topology by hand at a single aggregate B. Automate topology creation at a single aggregate C. Orchestrate and Instrument D. Increase scale E. More nodes F. More aggregates Automate LabWiki, OEDL, and GENI Desktop tutorials

11. Sponsored by the National Science Foundation 11 Scaling Up – GREE-SC 15 – May 28, 2015 Process Recommendation A. Build (smallest possible) topology by hand at a single aggregate B. Automate topology creation at a single aggregate C. Orchestrate and Instrument D. Increase scale E. More nodes F. More aggregates Automate Scaling Up & geni-lib tutorials

12. Sponsored by the National Science Foundation 12 Scaling Up – GREE-SC 15 – May 28, 2015 Writing scalable install scripts Example from xorp hostname | sudo tee vm_info.txt > /dev/null /sbin/ifconfig | egrep 'eth|inet addr' | sudo tee -a vm_info.txt > /dev/null router-1.xuan-xorp2.ch-geni-net.geni.it.cornell.edu eth0 Link encap:Ethernet HWaddr 02:07:47:72:19:fb inet addr:172.17.1.1 Bcast:172.31.255.255 Mask:255.240.0.0 eth1 Link encap:Ethernet HWaddr 02:7e:40:75:3f:b2 inet addr:192.168.3.6 Bcast:192.168.3.255 Mask:255.255.255.0 eth2 Link encap:Ethernet HWaddr 02:c0:ca:1e:f4:97 inet addr:192.168.1.30 Bcast:192.168.1.255 Mask:255.255.255.0 inet addr:127.0.0.1 Mask:255.0.0.0

13. Sponsored by the National Science Foundation 13 Scaling Up – GREE-SC 15 – May 28, 2015 Experiment Workflow Part I: Design/Setup Part II: Execute Part III: Finish

14. Sponsored by the National Science Foundation 14 Scaling Up – GREE-SC 15 – May 28, 2015 Medium-scale GENI Experiments 150 node international topology –Brecht Vermeulen, iMinds & Thierry Rakotoarivelo, NICTA Domain Science Applications –Paul Ruth, RENCI Courtesy of Ezra Kissel, Indiana University, GEC 20 Demo Long-lived slice for stitched, shared VLAN Long-lived slice for stitched, shared VLAN Dynamically add/remove nodes as needed Intelligent Data Movement System

15. Sponsored by the National Science Foundation 15 Scaling Up – GREE-SC 15 – May 28, 2015 Why use this topology? Large topologies at a single aggregate can take a long time to come up Stitched links fail with some frequency (this is less and less true over time) Updating topologies is hard

16. Sponsored by the National Science Foundation 16 Scaling Up – GREE-SC 15 – May 28, 2015 Single Aggregate Single Aggregate / 1 Slice Single Aggregate / 2 Slices with Shared VLAN Proto-Backbone Share VLAN: scalingup Start small

17. Sponsored by the National Science Foundation 17 Scaling Up – GREE-SC 15 – May 28, 2015 backbone router.1 10.100.1.0/24.1 router scalingup.2 Illinois Stanford Wisc scalingup 10.100.2.0/24.2 site-10 2 server 3 host 1 router.110 10.10.2.0/24 10.10.1.0/24 site-5site-15 Topology: Multi-site, multi-slice, stitched … then scale up

18. Sponsored by the National Science Foundation 18 Scaling Up – GREE-SC 15 – May 28, 2015 General Procedure 1.Backbone Slice –Reserve stitched backbone –Share VLAN at each geographic location 2.At each geographic location –Reserve a topology on the appropriate shared VLAN 3.Add, remove, update topologies at each geographic location as needed http://groups.geni.net/geni/wiki/HowTo/ShareALan

19. Sponsored by the National Science Foundation 19 Scaling Up – GREE-SC 15 – May 28, 2015 Tools to generate a scaled RSpec (from painful to easy) Copy paste existing RSpec in a text editor CON Tedious AND very error prone Manually create each node using an Rspec Editor (Jacks, jFed) CON Tedious AND somewhat error prone Write a shell script to generate the RSpec –Brecht and Thierry’s 150 node topology PRO Large topologies CON Single purpose Duplicate and Auto-IP buttons in Jacks PRO General purpose CON Medium topologies scaleup tool PRO Large topologies AND General purpose We’ll use today!!!

20. Sponsored by the National Science Foundation 20 Scaling Up – GREE-SC 15 – May 28, 2015 geni-lib library and scaleup tool geni-lib is a python library to interact with the GENI Federation. –Written by Nick Bastin, Barnstormer Softworks geni-lib supports the creation of tools. An example of such a tool is … scaleup is a script to create arbitrary topologies based on user-defined node types –Written by Xuan Liu, University of Missouri, Kansas City –Distributed with geni-lib in tools directory http://geni-lib.readthedocs.org/en/latest/intro/install.html

21. Sponsored by the National Science Foundation 21 Scaling Up – GREE-SC 15 – May 28, 2015 Site-X scaleup config [general] topo_type=star subnet=10. X node_type=host, router single_am=yes output_rspec=site- X.xml # Star Topology # n+1 nodes, node 1 is center [star] num_nodes=2 [host] # Plain Ubuntu 12 image... node_list=2,3 [router] # XORP software router # OSPF auto-configuring install script node_list=1 [add-shared-vlan] shared_vlans=scalingup [scalingup] lans=[(1,)] [am_nodes] # unbound RSpec any=ALL

22. Sponsored by the National Science Foundation 22 Scaling Up – GREE-SC 15 – May 28, 2015 Procedure: Create a small topology 1.Modify site.txt config by replacing X with the value from your worksheet 2.Run scaleup on site.txt config 3.In Jacks, modify IP of shared VLAN interface with the value from your worksheet 4.Save file 5.Reserve Resources

23. Sponsored by the National Science Foundation 23 Scaling Up – GREE-SC 15 – May 28, 2015 Experiment Workflow Part I: Design/Setup Part II: Execute Part III: Finish

24. Sponsored by the National Science Foundation 24 Scaling Up – GREE-SC 15 – May 28, 2015 Procedure: Test Connectivity 1.Login to a host node 2.Ping the backbone nodes ping 10.100.1.1 ping 10.100.2.2 3.Ordinarily, this is when you would run your procedure (e.g. using LabWiki) 4.Configure nodes using Ansible (see Appendix of instructions)

25. Sponsored by the National Science Foundation 25 Scaling Up – GREE-SC 15 – May 28, 2015 Experiment Workflow Part I: Design/Setup Part II: Execute Part III: Finish

26. Sponsored by the National Science Foundation 26 Scaling Up – GREE-SC 15 – May 28, 2015 Procedure: Make a bigger topology 1.Delete your existing slice 2.Make a copy of site.txt cp site.txt bigsite.txt 3.Add nodes to the star topology num_nodes = 5 node_list=2,3,4,5,6 output_rspec=bigsite-X.xml 4.Repeat everything from Step 3.1.d onward including reservation

27. Sponsored by the National Science Foundation 27 Scaling Up – GREE-SC 15 – May 28, 2015 Procedure: Try other topologies Create and view (but don’t reserve) other topologies: –How make a topology besides a star? –How install different software on one of the nodes?

28. Sponsored by the National Science Foundation 28 Scaling Up – GREE-SC 15 – May 28, 2015 Scaling Up Assumption #1 You have a small scale working experiment Assumption #2 Install and orchestration scripts are auto-configuring Today we have discussed how to make your topology BIGGER OR use a configuration management tool

29. Sponsored by the National Science Foundation 29 Scaling Up – GREE-SC 15 – May 28, 2015 Thank you! Sarah Edwards GENI Project Office help@geni.net Xuan Liu University of Missouri, Kansas City xuan.liu@mail.umkc.edu

30. Sponsored by the National Science Foundation 30 Scaling Up – GREE-SC 15 – May 28, 2015 site-X 2 server 3 host 10.___.___.___ 1 router X 10.___. ___.___ X X X 10.100.___.___ 100+X backbone router 10.100.1.1 10.100.1.2 10.100.2.1 router scalingup 10.100.2.2 Illinois Stanford Wisconsin scalingup

31. Sponsored by the National Science Foundation 31 Scaling Up – GREE-SC 15 – May 28, 2015 backbone site-X 2 server 3 host router 10.100.1.1 10.100.1.2 10.100.2.1 10.___.___.___ router 1 scalingup 10.100.2.2 X 10.___. ___.___ X X X 10.100.1.___ Illinois Stanford Wisconsin 100+X

32. Sponsored by the National Science Foundation 32 Scaling Up – GREE-SC 15 – May 28, 2015 backbone site-X 2 server 3 host router 10.100.1.1 10.100.1.2 10.100.2.1 10.___.___.___ router 1 scalingup 10.100.2.2 X 10.___. ___.___ X X X 10.100.2.___ Illinois Stanford Wisconsin 100+X

33. Sponsored by the National Science Foundation 33 Scaling Up – GREE-SC 15 – May 28, 2015 2 server 3 host router 10.100.1.1 10.100.1.2 10.100.2.1 10.___.___.___ router 1 scalingup 10.100.2.2 X 10.___. ___.___ X X X 10.100.2.___ Illinois Stanford Wisconsin backbone site-X 100+X

34. Sponsored by the National Science Foundation 34 Scaling Up – GREE-SC 15 – May 28, 2015 site-X 2 server 3 host 10.___.___.___ 1 router X 10.___. ___.___ X X X 10.100.___.___ 100+X backbone router 10.100.1.1 10.100.1.2 10.100.2.1 router scalingup 10.100.2.2 UKYPKS2 CENIC GeorgiaTech scalingup

35. Sponsored by the National Science Foundation 35 Scaling Up – GREE-SC 15 – May 28, 2015 backbone site-X 2 server 3 host router 10.100.1.1 10.100.1.2 10.100.2.1 10.___.___.___ router 1 scalingup 10.100.2.2 X 10.___. ___.___ X X X 10.100.1.___ UKYPKS2 CENIC GeorgiaTech 100+X

36. Sponsored by the National Science Foundation 36 Scaling Up – GREE-SC 15 – May 28, 2015 backbone site-X 2 server 3 host router 10.100.1.1 10.100.1.2 10.100.2.1 10.___.___.___ router 1 scalingup 10.100.2.2 X 10.___. ___.___ X X X 10.100.2.___ CENIC GeorgiaTech 100+X UKYPKS2

37. Sponsored by the National Science Foundation 37 Scaling Up – GREE-SC 15 – May 28, 2015 2 server 3 host router 10.100.1.1 10.100.1.2 10.100.2.1 10.___.___.___ router 1 scalingup 10.100.2.2 X 10.___. ___.___ X X X 10.100.2.___ CENIC GeorgiaTech backbone site-X 100+X UKYPKS2

38. Sponsored by the National Science Foundation 38 Scaling Up – GREE-SC 15 – May 28, 2015 site-X 2 host 3 10.___.___.___ 1 router X 10.___. ___.___ X X X 10.100.___.___ 100+X backbone router 10.100.1.1 10.100.1.2 10.100.2.1 router scalingup 10.100.2.2 Illinois Stanford Wisconsin scalingup

39. Sponsored by the National Science Foundation 39 Scaling Up – GREE-SC 15 – May 28, 2015 backbone site-X 2 host 3 router 10.100.1.1 10.100.1.2 10.100.2.1 10.___.___.___ router 1 scalingup 10.100.2.2 X 10.___. ___.___ X X X 10.100.1.___ Illinois Stanford Wisconsin 100+X

40. Sponsored by the National Science Foundation 40 Scaling Up – GREE-SC 15 – May 28, 2015 backbone site-X 2 host 3 router 10.100.1.1 10.100.1.2 10.100.2.1 10.___.___.___ router 1 scalingup 10.100.2.2 X 10.___. ___.___ X X X 10.100.2.___ Illinois Stanford Wisconsin 100+X

41. Sponsored by the National Science Foundation 41 Scaling Up – GREE-SC 15 – May 28, 2015 2 host 3 router 10.100.1.1 10.100.1.2 10.100.2.1 10.___.___.___ router 1 scalingup 10.100.2.2 X 10.___. ___.___ X X X 10.100.2.___ Illinois Stanford Wisconsin backbone site-X 100+X