Sponsored by the National Science Foundation Using Omni to Build Tools Sarah Edwards, GENI Project Office 23 July 2013 $ omni.py createsliver aliceslice myRSpec.xml INFO:omni:Loading config file omni_config INFO:omni:Using control framework pgeni INFO:omni:Slice urn:publicid:IDN+pgeni.gpolab. expires within 1 day on INFO:omni:Creating sliver(s) from rspec file INFO:omni:Writing result of createsliver for INFO:omni:Writing to ‘aliceslice-manifest-rspe INFO:omni: INFO:omni: Completed createsliver: Options as run: aggregate: framework: pgeni native: True Args: createsliver aliceslice myRSpec.xml Result Summary: Slice urn:publicid:IDN+pgeni Reserved resources on Saved createsliver results to aliceslice-man INFO:omni: ===================================

Sponsored by the National Science Foundation 2 GEC17: July 23, 2013 Outline What is omni? Using omni as a script to build other tools Example WIP: Awesome Omni What’s next?

Sponsored by the National Science Foundation 3 GEC17: July 23, 2013 What is omni? Omni is a command line client tool for use with the GENI Aggregate Manager (AM) API Supports reserving resources from any aggregate which supports the AM API Supports complete resource reservation workflow Distributed with gcf which is a set of developer tools, including an AM, which allow for testing the AM API clientaggregate Flack Omni(-based): omni portal.geni.net gcf-am.py Flavors: InstaGENI ExoGENI FOAM (OpenFlow) AMSoil-based

Sponsored by the National Science Foundation 4 GEC17: July 23, 2013 Strengths and Weaknesses Strength Reliably at the forefront of implementing changes to the AM API Easy and powerful to script with –e.g. readyToLogin.py –Returns easy to use python objects (no parsing CLI output) Weakness Changes fairly frequently User must be aware of differences in v1/v2 vs v3 Lots of command line options (-h is very long) Packaging System exit codes In addition, to the usual strengths and weakness of CLIs…

Sponsored by the National Science Foundation 5 GEC17: July 23, 2013 (Not So) Recent Additions Multiple aggregates specified in one command (use –a multiple times) omni-configure.py configure omni for ProtoGENI, Portal and PlanetLab user account readyToLogin.py --no-keys gets login info even if don’t have ssh keys – works well with multi-user slices stitcher.py dynamically reserve cross-domain VLANs Aggregate nicknames In an upcoming release RSpec nicknames –Specify in omni_config vm = –Look in default directory or url default_rspec_location = default_rspec_extension = rspec omni createsliver myslice myspecialtopology -a ig-utah RSpec nickname agg nickname

Sponsored by the National Science Foundation 6 GEC17: July 23, 2013 omni-configure.py Distributed with omni Not built on omni (ie not an omni script) Automatically generates your omni_config Supports ProtoGENI, PlanetLab, and GENI Portal accounts Reasonable defaults for the common case –Just run: omni-configure.py All defaults can be overridden with command line options

Sponsored by the National Science Foundation 7 GEC17: July 23, 2013 readyToLogin.py Distributed with and built on omni Automatically determines your ssh login info Supports ProtoGENI, ExoGENI resources $ readyToLogin.py egtest10 -a pg-ky my-node's geni_status is: ready (am_status:ready) User sedwards logs in to my-node using: ssh -p i …/geni_key_portal ssh -p i …/geni_cert_portal_key $ readyToLogin.py egtest10 -a pg-ky --no-keys my-node's geni_status is: ready (am_status:ready) User sedwards logs in to my-node using: ssh -p

Sponsored by the National Science Foundation 8 GEC17: July 23, 2013 Omni scripting #!/usr/bin/env python import omni # Get a parser from omni that understands omni options parser = omni.getParser() parser.set_usage("%s [options] username"%sys.argv[0]) options, args = parser.parse_args(sys.argv[1:]) # (1) Run equivalent of 'omni.py listmyslices username’ text, sliceList = omni.call( ['listmyslices', username], options ) # (2) For each returned slicename run equivalent of: # 'omni.py print_slice_expiration slicename’ for slicename in sliceList: text, expiration = omni.call( ['print_slice_expiration', slicename], options ) printStr += "%s\n"%(str(expiration)) printStr += "="*80 return printStr ================================================================================ User sedwards has 2 slice(s): Slice urn:publicid:IDN+ch.geni.net:tutorial+slice+portal99 expires on :54:28 UTC Slice urn:publicid:IDN+ch.geni.net:tutorial+slice+portal01 expires on :03:25 UTC ================================================================================ gcf/examples/expirationofmyslices.py Easily execute and parse the output of omni calls from within another script Returns python objects

Sponsored by the National Science Foundation 9 GEC17: July 23, 2013 Omni scripting #!/usr/bin/env python import omni # Get a parser from omni that understands omni options parser = omni.getParser() # Add additional optparse.OptionParser style options for your script as needed. # Be sure not to re-use options already in use by omni parser.add_option("--myScriptPrivateOption", help="A non-omni option added by %s"%sys.argv[0], action="store_true", default=False) # options is an optparse.Values object, and args is a list options, args = parser.parse_args(sys.argv[1:]) if options.myScriptPrivateOption: # do something special for your private script's options print "Got myScriptOption” gcf/examples/myscript.py Inherit command line options from omni while adding custom options Easily inherit omni’s command line arguments… …while adding your own Get -a, -c, --api-version for free

Sponsored by the National Science Foundation 10 GEC17: July 23, 2013 Tools built using omni scripting For experimenters gcf/src/ stitcher.py Tool which supports stitching (as seen in yesterday’s plenary) gcf/examples/ readyToLogin.py determines ssh command to login to any node gcf/examples/ remote-execute.py remotely runs commands on the nodes in your slices For developers gcf/acceptance_tests/AM_API/ am_api_accept.py –AM API acceptance tests verifies compliance with the AM API specification supports AM API v1, v2, and v3 Looking for an example code snippet? Look here For operators Aggregate health checks run by GPO Lab’s nagios are simple omni scripts

Sponsored by the National Science Foundation 11 GEC17: July 23, 2013 Tools using omni under the covers GENI Experimenter Portal ( runs omni under the covers to talk to aggregates GENI Desktop and GIMI Each use omni for parts of their GENI Portal integration HyperNets a.k.a. Shufeng Huang’s PhD Thesis

Sponsored by the National Science Foundation 12 GEC17: July 23, 2013 Best Practices Using Omni to Build a Tool Cache slice credential omni.py getslicecred mySlice -o # save slice cred to a file omni.py createsliver --slicecredfile myslice-cred.xml Cache user credential omni.py getusercred mySlice -o # save slice cred to a file omni.py listmyslices --usercredfile jdoe-usercred.xml GetVersion cache by default reduces calls to GetVersion but may or may not be appropriate to your situation To not use the cache use --NoGetVersionCache To use the cache use --ForceUseGetVersionCache To age out the cache --GetVersionCacheAge To set the location of the cache file --GetVersionCacheName Cache a list of aggregates used in each slice –So only have to query used aggregates

Sponsored by the National Science Foundation 13 GEC17: July 23, 2013 What next for omni? Possible next steps? What would be most useful? Packaging and installing omni -> easier to build tools using omni –Completed initial installation on Windows –Unix package Cache user & slice creds -> fewer/quicker calls Ability to run multiple commands in one invocation Exit codes returned when run from the command line make it hard to shell script with omni Smooth over difference in v1/v2 vs v3  easier for users Common CH API support –Multi-user slice support: dynamically query CH for users and keys for each slice instead of using a statically configured list Others?

Sponsored by the National Science Foundation 14 GEC17: July 23, 2013 Where to get it? Where to get omni/gcf? Can be downloaded from the gcf wiki: Have an idea for an improvement? Send a suggestions to Or, make a ticket (requires a wiki account)

Sponsored by the National Science Foundation Awesome Omni: Or how I learned to stop worrying and love Omni inspired by Niky Riga Work In Progress

Sponsored by the National Science Foundation 16 GEC17: July 23, 2013 Motivation Omni originated as a developer test tool and has been adapted to support advanced experimenters Omni’s user interface reflects this history –Many (~30) commands and many (~40) options –The output is thorough (aka long and verbose) omni-configure.py, readyToLogin.py, and stitcher.py are separate tools

Sponsored by the National Science Foundation 17 GEC17: July 23, 2013 Imagine that omni were more awesome. Imagine it worked like this….

Sponsored by the National Science Foundation 18 GEC17: July 23, 2013 Configure and setup awemni init * awemni configured! * * Try: * awemni help * awemni help *.....

Sponsored by the National Science Foundation 19 GEC17: July 23, 2013 A complete workflow awemni create vm as topo1 at utah_rack * Created one virtual machine called ”topo1" at aggregate "utah_rack” * To login/ssh, try: * awemni connect node1 * To release resources try: * awemni delete node1 awemni connect topo1 at utah_rack * ”topo1" is booting...wait a minute and try again awemni connect topo1 at utah_rack * ”topo1" is ready...logging in * vm> hostname vm.utah.geniracks.com vm> exit awemni delete topo1 * Deleted virtual machines

Sponsored by the National Science Foundation 20 GEC17: July 23, 2013 Or even more simply… awemni create vm * Created one virtual machine named "novelSlice” * at each of: utah_rack, mass_rack * To login, try: * awemni connect novelSlice at utah_rack * To release resources try: * awemni delete novelSlice awemni connect novelSlice at utah_rack * "novelSlice" is ready...logging in vm> hostname vm.utah.geniracks.com vm> exit awemni delete vm * Deleted virtual machines

Sponsored by the National Science Foundation 21 GEC17: July 23, 2013 Reserving a custom stitched topology With stitching this becomes even easier… awemni create nikysawesometopology.....

Sponsored by the National Science Foundation 22 GEC17: July 23, 2013 What’s needed to do this?  Have omni support nicknames for RSpecs  A way to store a small amount of current state A wrapper to:  parse the natural language-ish syntax into the underlying omni  parse the omni outputs into something natural language-ish  pull in the functionality of omni-configure.py to support init  pull in the functionality of readyToLogin.py to support connect  parse RSpec into something human readable  Testing

Sponsored by the National Science Foundation 23 GEC17: July 23, 2013 What’s needed to do this?  Have omni support nicknames for RSpecs  A way to store a small amount of current state A wrapper to:  parse the natural language-ish syntax into the underlying omni  parse the omni outputs into something natural language-ish  pull in the functionality of omni-configure.py to support init  pull in the functionality of readyToLogin.py to support connect  parse RSpec into something human readable  Testing

Sponsored by the National Science Foundation 24 GEC17: July 23, 2013 Demo