1. Sponsored by the National Science Foundation GENI Instrumentation and Measurement WG: 2 nd I&M Workshop and Priority Architecture Topics GENI Engineering Conference 8 UC San Diego GPO System Engineer: Harry Mussman July 22, 2010 www.geni.net

2. Sponsored by the National Science Foundation 2 July 22, 2010 2 nd I&M Workshop Objective: –Gather contributors from key I&M prototyping projects that have already implemented pieces of I&M functionality in a manner consistent with GENI goals. –Define priority pieces of the I&M architecture by consensus –Assemble teams to complete the discussion and documentation –Draft a roadmap for implementations during Spirals 2 and 3. –See http://groups.geni.net/geni/wiki/2ndInstMeasWork#References http://groups.geni.net/geni/wiki/2ndInstMeasWork#References Initial set of priority topics: –GENI I&M use cases –GENI I&M services –GENI measurement plane –Interfaces, protocols and schema for measurement data in GENI

3. Sponsored by the National Science Foundation 3 July 22, 2010 (continued) Approach: –Each priority topic was discussed in a structured manner –The organizers outlined an approach or solution, and representatives from the key projects discussed it – The group identified where they had a consensus, and where there were issues that needed further discussion –One new topic was identified: how to discover, specify, authorize and assign all of the various types of resources required for I&M capabilities –The interfaces topic was split into four topics, divided upon function –Finally, teams were assembled to discuss each priority topic, come to a consensus, and write a revised section for the architecture document Initial set of priority topics: –GENI I&M use cases –GENI I&M services –GENI measurement plane –Interfaces, protocols and schema for measurement data in GENI

4. Sponsored by the National Science Foundation 4 July 22, 2010 (continued) Organizing Committee: –Paul Barford - University of Wisconsin – Madison –Bruce Maggs – Duke University and Akamai –Harry Mussman – BBN/GPO –Vic Thomas - BBN/GPO –Evan Zhang – BBN/GPO Key projects who sent representatives: –OML (ORBIT Measurement Library) OMF (ORBIT Management Framework) –Instrumentation Tools –perfSONAR –Scalable Sensing Service (S3) –OnTimeMeasure for network measurements –GENI Meta-Operations Center and NetKArma –Virtual Machine Introspection (VMI) –Experiment Management Service – Digital Object Registry

5. Sponsored by the National Science Foundation 5 July 22, 2010 (continued) Final set of priority topics: –Topic 1: GENI I&M Use Cases –Topic 2: GENI I&M Services –Topic 3: GENI I&M Resources –Topic 4: GENI I&M Measurement Plane and Interfaces –Topic 5: GENI I&M Interfaces and Messages/Flows/APIs : Manage Services –Topic 6: GENI I&M Interfaces and Messages/Flows/APIs : Data Flows and Data File Transfers –Topic 7: GENI I&M Interfaces and Messages/Flows/APIs : Service Registration and Discovery –Topic 8: GENI I&M Interfaces and Messages/Flows/APIs : GUIs –Topic 9: GENI Measurement Data Schema

6. Sponsored by the National Science Foundation 6 July 22, 2010 Topic 1: GENI I&M Use Cases Team members: –Paul Barford - University of Wisconsin – Madison –Jim Griffioen - Univ Kentucky –Prasad Calyam* - Ohio Supercomputing Ctr –Camilo Viecco - Indiana Univ –Brian Hay – Univ Alaska *agreed to organize first writing and discussion Identify all user groups, and provide basic use cases for each user group: –Experiment researchers –Experiment (opt-in) users –Central (i.e., GMOC) operators –Cluster and aggregate providers and operators –Archive service providers and operators –Researchers that use archived measurement data

7. Sponsored by the National Science Foundation 7 July 22, 2010 Topic 2: GENI I&M Services Team members: –Harry Mussman* and Evan Zhang – BBN/GPO –Giridhar Manepalli – CNRI –Chris Small and Beth Plale - Indiana Univ *agreed to organize first writing and discussion Summarize current view of services –Measurement Orchestration (MO) Service –Measurement Point (MP) Service –Measurement Information (MI) Service –Measurement Collection (MC) Service –Measurement Analysis and Presentation (MAP) Service –Measurement Data Archive (MDA) Service Need basic definition of a Measurement Data Archive (MDA) service

8. Sponsored by the National Science Foundation 8 July 22, 2010 (continued) Have identified three different types of services: –Type 1: Dedicated service platform for customized Information. (Completely dedicated to an experiment) –Type 2: Common service platform with dedicated slivers for customized Information. (Common portion, plus parts associated with different experiments) –Type 3: Common service for common or customized information. (Common service, with data provided to multiple experiments)

9. Sponsored by the National Science Foundation 9 July 22, 2010 GENI I&M Architecture: Services and Interfaces

10. Sponsored by the National Science Foundation 10 July 22, 2010 Topic 3: GENI I&M Resources Team members: –Vic Thomas - BBN/GPO –Jim Griffioen* - Univ Kentucky –Martin Swany - Univ Delaware –Camilo Viecco - Indiana Univ –Brian Hay – Univ Alaska –Giridhar Manepalli - CNRI *agreed to organize first writing and discussion How are each of these resources discovered, specified, authorized and assigned: –Hosts, VMs, etc. –Network connectivity –Software, e.g., I&M software that can be included in an experiment –I&M services –I&M data flows and file transfers –I&M data files stored in archives

11. Sponsored by the National Science Foundation 11 July 22, 2010 (continued) Options: –Always by mechanisms provided by the CF? –With CF plus additional mechanisms? –Consider example of LS in perfSONAR –Consider example of data file stored in archive, owned by an experimenter –Need to understand interop with CF for each option –Does CF setup secondary authorization mechanisms in some cases? If so, how? For each item, consider –Unique name? –Unique and persistent identifier? –Ownership –What sort of policies the owner may want to apply –How to create –How to name –How to register and discover –How to authorize and assign

12. Sponsored by the National Science Foundation 12 July 22, 2010 (continued) Chart:

13. Sponsored by the National Science Foundation 13 July 22, 2010 Topic 4: GENI I&M Measurement Plane and Interfaces Team members: –Harry Mussman* – BBN/GPO –Ezra Kissel – Univ Delaware –Chris Small - Indiana Univ *agreed to organize first writing and discussion Assume: Use IP backbone network (with Internet access) for most measurement traffic (like control traffic) Consider: Use Layer 2 (VLAN) connections only when necessary Issues with measurement traffic: –Which protocols are active? minimize number to simplify? –Need access to resources in aggregates, even when resources are in private address space, via proxies (i.e., gateways) –Need to provide authentication and authorization, without and with proxies –Need to be able to reserve bandwidth (provide QoS) to protect measurement traffic from other traffic, and vice-versa –Which of these requires the use of Layer 2 (VLAN) connections?

14. Sponsored by the National Science Foundation 14 July 22, 2010 GENI Measurement Traffic Flows

15. Sponsored by the National Science Foundation 15 July 22, 2010 GENI I&M Interfaces and Messages/Flows/APIs 1) Discover Resources and Assign Slivers –EC srvc uses CF to discover resources, and then assign slivers to slice/researcher for I&M srvcs 2) Configure and Program Slivers –EC srvc uses CF and/or ssh to load std or customized software images for I&M srvcs –Note: 1) and 2) are not specific to I&M services 3) Manage Services –EC srvc and MO srvc use CF and/or https to check status of I&M srvcs, receive event notifications, and execute functions such as start, stop, reset, reboot, and checkpoint 4) Measurement Data Flows and Data File Transfers –Measurement data flows and data file transfers between I&M srvcs. Options: Pull, Push, Pub/Sub. 5) Register I&M Service –Operator configures I&M srvc to register with Lookup Srvc, advertising name, location, and available metadata 6) Discover I&M Service and Establish Meas Data Flow –ECS or I&M srvc discovers I&M srvc advertisement, and establishes data flow 7) Conduct and Observe Experiment –Researcher uses browser to interact with and observe services via web portals (GUIs)

16. Sponsored by the National Science Foundation 16 July 22, 2010 GENI I&M Architecture: Services and Interfaces

17. Sponsored by the National Science Foundation 17 July 22, 2010 Topic 5: GENI I&M Interfaces and Messages/Flows/APIs: Manage Services Team members: –Vic Thomas - BBN/GPO –Ivan Seskar – Rutgers WINLAB –Max Ott – NICTA –Sonia Fahmy* – Purdue –Giridhar Manepalli – CNRI *agreed to organize first discussion and writing Define an approach based on OMF/OML and S3 Consider: –HTTP(S) –REST vs SOAP –Authorization by credentials or ? If credentials, how to revoke? –Pass XML fragments –Define basic API

18. Sponsored by the National Science Foundation 18 July 22, 2010 Topic 6: GENI I&M Interfaces and Messages/Flows/APIs: Data Flows and Data File Transfers Team members: –Harry Mussman* – BBN/GPO –Ivan Seskar – Rutgers WINLAB –Max Ott – NICTA –Ezra Kissel – Univ Delaware –Prasad Calyam - Ohio Supercomputing Ctr –Michael Zink - UMass Amherst *agreed to organize first writing and discussion Consider data flows and data file transfers between all services –Fundamental to I&M, and not yet defined in GENI –Linked to measurement data schema, which is primary

19. Sponsored by the National Science Foundation 19 July 22, 2010 (continued) Consensus to allow a wide range of options: –Both data flows and data files transfers –Types: Pull, Push, Pub/Sub –Protocols: SNMP, SCP, FTP, gridFTP, HTTP, XMPP, TCP, SCTP –Consider: Naming, Discovery, Connectivity, Authentication and authorization mechanisms Goal is to define an initial set: –Minimum set required for GENI –Mapped to current projects –Permit later additions

20. Sponsored by the National Science Foundation 20 July 22, 2010 Topic 7: GENI I&M Interfaces and Messages/Flows/APIs: Registration and Discovery of Services with Available Measurement Data Team members: –Jason Zurawski* – Internet2 (yes) –Prasad Calyam - Ohio Supercomputing Ctr (yes) *agreed to organize first writing and discussion Consider approach used in perfSONAR Summarize for: –Services with available data flows –Also sources of file transfers? –Also GUIs?

21. Sponsored by the National Science Foundation 21 July 22, 2010 Topic 8: GENI I&M Interfaces and Messages/Flows/APIs: GUIs Team members: –Jeremy Reed and Zongming Fei - Univ Kentucky (yes) –Guilherme Fernandes* – Univ Delaware (yes) –Puneet Sharma - HP Labs (yes) –*agreed to organize team Consider all types of GUIs: –Control experiments –Display I&M results –Report status –View archive service Define overall goals for GENI GUIs Consider portal, for access to multiple GUIs Consider need for authentication and authorization

22. Sponsored by the National Science Foundation 22 July 22, 2010 Topic 9 GENI Measurement Data Schema Team members: –Bruce Maggs – Duke University and Akamai –Max Ott – NICTA –Ivan Seskar – Rutgers WINLAB –Martin Swany* - Univ Delaware –Camilo Viecco - Indiana Univ –Michael Zink - UMass Amherst –Jim French – CNRI *agreed to organize first discussion and writing Consider: –Measurement data schema –Metadata schema –Metadata contents

23. Sponsored by the National Science Foundation 23 July 22, 2010 (continued) Goal is to define an initial set: –Minimum set required for GENI –Mapped to current projects –Permit later additions Consider measurement data schema and/or metadata schema from: –perfSONAR –GMOC-provided –Current OML approach –Proposed OML approach using IPFIX –NetCDF (as used by DI Cloud) Provide overall template for GENI metadata: –Which items in GENI metadata template are required? –Invariant?

24. Sponsored by the National Science Foundation 24 July 22, 2010 GENI I&M Architecture Priority Topics: Current Status TopicGroup discussion First writingWG review Topic 1: GENI I&M Use Cases Topic 2: GENI I&M Services Topic 3: GENI I&M Resources Topic 4: GENI I&M Measurement Plane and Interfaces Topic 5: GENI I&M Interfaces and Messages/Flows/APIs : Manage Services Topic 6: GENI I&M Interfaces and Messages/Flows/APIs : Data Flows and Data File Transfers Topic 7: GENI I&M Interfaces and Messages/Flows/APIs : Service Registration and Discovery Topic 8: GENI I&M Interfaces and Messages/Flows/APIs : GUIs Topic 9: GENI Measurement Data Schema

25. Sponsored by the National Science Foundation 25 July 22, 2010 GENI I&M Architecture Document: Current Status Status: –By GEC7 : v0.1 DRAFT completed, by GPO; see http://groups.geni.net/geni/wiki/GeniInstrumentationandMeasurem entsArchitecture http://groups.geni.net/geni/wiki/GeniInstrumentationandMeasurem entsArchitecture –By GEC8: v0.5 DRAFT completed, by GPO, with contributions from WG; see Plan: –By GEC9: v1.0 draft, reviewed by WG

26. Sponsored by the National Science Foundation 26 July 22, 2010 GENI I&M Architecture Priority Topics and Document: Next Steps Complete group discussion and first writing for each priority topic: –When? –Add members to each group? Review each priority topic with WG, and rewrite: –How? –When? Assemble and review revised Architecture Document as a whole –How? –Before GEC9 Draft a roadmap for implementations during Spiral 3.