1. GEC3 Oct. 28-30 2008www.geni.net1 Experiment Workflow and Services Working Group System Engineering Report Vicraj (Vic) Thomas, Ph.D. Working Group System Engineer groups.geni.net GENI working group wiki

2. GEC3 Oct. 28-30 2008 We Are Here… www.geni.net2 The Experiment Services and Workflow WG focuses on experimenter- users needs for planning, scheduling, running, debugging, analyzing and archiving experiments.

3. GEC3 Oct. 28-30 2008 Lifecycle of a GENI Experiment User view of an experiment Tools and services supporting this lifecycle map to different subareas identified by Jeff Chase –Slice control Construct, deploy, monitor and adapt slices –Information plane Instrumentation: Specification, collection, processing –Experiment building blocks Code artifacts: Specification, qualification, sharing/reuse, deployment, archival, etc. Objective: Think about how your ideas and projects relate to workflow presented here –How they support this workflow –Identify areas of disagreement www.geni.net3

4. GEC3 Oct. 28-30 2008 Lifecycle of a GENI Experiment: Top- Level View www.geni.net4 Experiment Design S/w and H/w Development (Lab) Experiment Planning Experiment Execution Experiment Sunsetting Experiment Deployment GENI Tools and Services support these phases of the lifecycle Outside the scope of GENI User Registration

5. GEC3 Oct. 28-30 2008 GENI Relevant Lifecycle Phases www.geni.net5 Establish Credentials Experiment Control Archive Experiment Establish Credentials Resource Discovery Tool Discovery Develop GENI Version Lease Resources Install and Configure Software Data Collection/Anal ysis Transition to Product Experiment Planning Experiment Deployment Experiment Execution Experiment Sunsetting Verify Deployment

6. GEC3 Oct. 28-30 2008 User Registration Needed: Admin tools to register users and grant them appropriate authorizations –Administrator uses real- world documents to authenticate user –User is registered with a Principal Registry/Identity Provider –User is granted appropriate authorizations Browse resources Lease resources and run experiments www.geni.net6 Admin and Account Tools

7. GEC3 Oct. 28-30 2008 Experiment Planning www.geni.net7 Establish Credentials Experiment Control Archive Experiment Establish Credentials Resource Discovery Tool Discovery Develop GENI Version Lease Resources Install and Configure Software Data Collection/Anal ysis Transition to Product Experiment Planning Experiment Deployment Experiment Execution Experiment Sunsetting Verify Deployment

8. GEC3 Oct. 28-30 2008 Establishing Credentials User establishes identity with Clearinghouse –Assumption: Already registered with an Identity Provider/Principal Registry www.geni.net8

9. GEC3 Oct. 28-30 2008 Resource Discovery Multiple ways of finding resources –Automatically from an experiment specification Needed: A tool that can interpret an experiment specification and query appropriate clearinghouses for resources –Programmatically Needed: An API on Clearinghouses to query for resources –Browse and select Needed: A graphical tool for browsing resources –May lease resources during discovery (for immediate or future use) Researcher credentials must allow leasing Assumption: All components and aggregates are registered with one or more Clearinghouses www.geni.net9

10. GEC3 Oct. 28-30 2008 Tool Discovery Determine tools needed to program GENI components discovered by researcher –Need: Component Rspec to provide links to information on using/programming component –Need: Component Rspec to provide links to information on resource and other constraints imposed by component www.geni.net10

11. GEC3 Oct. 28-30 2008 Develop GENI Version Version of experiment that will run on GENI components –Developed and tested in the lab Based on component information obtained during resource and tool discovery Develop instrumentation plan and implement GENI Tools/Services: None needed www.geni.net11

12. GEC3 Oct. 28-30 2008 GENI Relevant Lifecycle Phases www.geni.net12 Establish Credentials Experiment Control Archive Experiment Establish Credentials Resource Discovery Tool Discovery Develop GENI Version Lease Resources Install and Configure Software Data Collection/Anal ysis Transition to Product Experiment Planning Experiment Deployment Experiment Execution Experiment Sunsetting Verify Deployment

13. GEC3 Oct. 28-30 2008 Establish Deployment Credentials Assumption: Researcher has rights to lease resources –Credentials to lease resources and use them may be different from those for browsing resources Needed: Researcher’s Principal Registry/Identity Provider to be trusted by Clearinghouses www.geni.net13

14. GEC3 Oct. 28-30 2008 Lease Resources Similar to browsing for resources –Automatically from an experiment specification Needed: Tool to interpret experiment specification and lease resources at appropriate Clearinghouses –Programmatically Needed: API on Clearinghouses to lease resources –Browse and lease Needed: Graphical tool to lease resources Researcher specifies duration of lease and start time www.geni.net14 Leasing Interface Details of Control Plane operations (slice management, tickets, etc) are hidden from the casual user

15. GEC3 Oct. 28-30 2008 Install and Configure Software Download software into components and install them –Experimenter must have valid lease for resource Configure components such as firewalls and Internet gateways Needed: Tool to manage downloads, track configurations, etc. Needed: Mechanisms for composing experiments Experimental hardware: Specific to type of hardware and where it goes –Post Spiral 1 www.geni.net15

16. GEC3 Oct. 28-30 2008 Validate Deployment Needed: Tools to ensure slice is properly configured –Required resources have been obtained –Components connected and communicating www.geni.net16

17. GEC3 Oct. 28-30 2008 GENI Relevant Lifecycle Phases www.geni.net17 Establish Credentials Experiment Control Archive Experiment Establish Credentials Resource Discovery Tool Discovery Develop GENI Version Lease Resources Install Hardware/Soft ware Data Collection/Anal ysis Transition to Product Experiment Planning Experiment Deployment Experiment Execution Experiment Sunsetting Verify Deployment

18. GEC3 Oct. 28-30 2008 Experiment Control Needed: Services and tools to support –Start/pause/resume/stop experiments Granularity: Slice? Sliver? Experiment relevant groupings? All of the above? –Grow/shrink slice: Add/remove resources –Control flow of traffic from/to Internet –Connect to other experiments (experiment composition) –Distributed synchronization for multi-phased experiments –Debugging and checkpointing Resources being used by experiment Experiment state –Failure management Handling failures of components and links: www.geni.net18

19. GEC3 Oct. 28-30 2008 Data Collection and Analysis Needed: Non-intrusive instrumentation –Data collection and storage Needed: Tools for monitoring and visualizing resources, flows, etc. in real-time while experiment is running www.geni.net19

20. GEC3 Oct. 28-30 2008 GENI Relevant Lifecycle Phases www.geni.net20 Establish Credentials Experiment Control Archive Experiment Establish Credentials Resource Discovery Tool Discovery Develop GENI Version Lease Resources Install Hardware/Soft ware Data Collection/Anal ysis Transition to Product Experiment Planning Experiment Deployment Experiment Execution Experiment Sunsetting Verify Deployment

21. GEC3 Oct. 28-30 2008 Experiment Archival Make experiment available to the research community to replicate, extend or incorporate into another experiment Needed: Checklist of tasks to be completed before experiment is archived –Documentation of experiment objectives, code, resources used; keywords; experiment description in a formal specification language; published reports; etc. Needed: Tools to search archives Needed: Embedding experiment specifications www.geni.net21

22. GEC3 Oct. 28-30 2008 Transition Technology to Product Outside scope of GENI tools and services? www.geni.net22

23. GEC3 Oct. 28-30 2008 Working Group Deliverables and GPO Support www.geni.net23

24. GEC3 Oct. 28-30 2008 WG Deliverables GENI Experiment Lifecycle Spiral 1 Capabilities-based Use Case –Control plane oriented Experiment Support Subsystem Technical Description Experiment Support Tools and Services Technology Survey www.geni.net24

25. GEC3 Oct. 28-30 2008 Goals Understand the GPO system engineering role and contributions to this working group Track hard problems and open issues relevant to this working group – And identify researchers and projects Identify how each of us can contribute to addressing these problems – Including developing cogent descriptions of these problems www.geni.net25

26. GEC3 Oct. 28-30 2008 Who am I? GENI Systems Engineer for Distributed Systems and Security Background in highly dependable distributed systems and networks –Distributed control system infrastructures for process control systems –Distributed collaboration system for operators of process control plants –PI on DARPA NEST and Ultra*Log projects –Lead on DARPA proposal that led to the development of a very successful intrusion report aggregator –Ground-breaking work on wireless networks for on- aircraft flight-essential applications –Industrial Technical Area Lead for Sensor Information Processing and Delivery on US-UK program (ITA program) on Network Sciences www.geni.net26

27. GEC3 Oct. 28-30 2008 My Role in this Working Group Frame issues from top-down through document development –Collect & frame issues –Collect input from group & revise Synthesize input from bottom-up –Test for & document consensus (or lack of it) –Take & distribute notes –Maintain wiki Track hard problems and open issues – Solicit help with developing cogent descriptions of problems – Identify researchers and projects that can address these problems www.geni.net27

28. GEC3 Oct. 28-30 2008 GPO System Engineering and Working Groups SEs draft documents Internal GPO review Post publically –On wiki initially, repository up shortly Discuss doc on working group list –Possible one-on-one followups –Please send text Revision is posted –Repeat process www.geni.net28

29. GEC3 Oct. 28-30 2008 GEC3 www.geni.net29 GENI Spiral 1 Integration: 5 Control Framework Clusters 1578 Overlay Hosting Nodes 1579 ProtoGENI 1595 GpENI 1595 GpENI 1646 CMU Testbeds 1609 DETER Trial Integration 1609 DETER Trial Integration 1642 Instrumentation Tools 1613 Enterprise GENI 1600 PlanetLab 1658 Mid-Atlantic Crossroads 1658 Mid-Atlantic Crossroads 1602 Sensor/Actuat or Network 1582 ORCA/BEN 1599 Vehicular Mobile Network 1660 ORBIT Framework 1601 Virtual Tunnels 1657 WIMAX 1657 WIMAX 1663 Digital Object Registry 1663 Digital Object Registry 1633 Kansei Sensor Network 1628 Measurement System 1604 GENI Meta Operations 1643 Programmable Edge Node 1621 GUSH Tools 1622 Provisioning Service 1650 Regional Opt-In 1650 Regional Opt-In 1632 Security Architecture 1645 Million- Node GENI 1610 GENI at 4-Year Colleges 1610 GENI at 4-Year Colleges 1653 Data Plane Measurements Cluster A Cluster ECluster D Cluster CCluster B Key: Projects with active Spiral 1 clearinghouse interfaces STUDY ALL PICK ONE 1631 Embedded Real-time Measurements 1631 Embedded Real-time Measurements 1619 Optical Access Networks 1619 Optical Access Networks Column labels show common control framework Highlighted projects are relevant to this wg. Is this the right set?

30. GEC3 Oct. 28-30 2008 Follow-up Notes, slides, actions, etc will be sent to the working group mail list and posted on the wiki page: http://www.geni.net/wg/services-wg.htmlhttp://www.geni.net/wg/services-wg.html Please give Jeff/me copies of your slides immediately after this session Please sign Non-Exclusive Copyright License www.geni.net30