1. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. GENI and InstaGENI: An Architecture for the Intercloud Rick McGeer HP Labs

2. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice.

3. 3 Outline The Need for an Intercloud Architecture The GENI Initiative: from Testbed to Architecture The ProtoGENI MetaCloud The InstaGENI Distributed Cloud The InstaGENI Rack: The Apple-II of Clouds 3

4. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 4 The World is Closing In On Us Big Data dominates both scientific and social computing Astronomical Imagery Collider events Genomic data Environmental Sensing (e.g., CASA networked weather radar) Large social graphs (10 12 nodes and beyond) Getting rid of the clusters-in-the-basement Consolidation into the “condo-of-condos” New demands on the network… Disaster Response, social media,…. 4

5. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 5 Mixing the Operational and Advanced Networks New network requirements mean new network demands Big data, high QoS on operational network SLAs for “condo of condos” Movement of Big Data Across campus networks Between campuses Between campus and sensor sites Programmability at network collection site 5

6. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 6 Solution Compute where the data is Ubiquitous computation Cloud that can be instantiated anywhere Advanced Networking to move the data to where the compute is Customizable virtual networks Layer-2 end-to-end QoS guarantees Programmable networking Custom, app-specific protocols 6

7. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 7 But it can’t compete with Production Traffic Right Solution: customizable virtual networks Isolation at the level of a flow Guaranteed network properties: virtual network per flow Offers the prospect of Routing Bandwidth Limits On a per-flow basis 7

8. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 8 HP, Internet2, GENI, US-Ignite – Working to Advance Internet Technology for Research & Local Communities Internet2 GENI US-Ignite HP Advanced network infrastructure HP InstaGENI rack – GENI cluster deployment in-a- box, ready out of the box Next-gen network apps for public benefit Collaborative platform to build the next-gen Internet

9. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 9 GENI Ubiquitous cloud with deeply-programmable networking Ubiquitous Cloud Abstracted API that can be implemented by any popular cluster manager (Slice Federation Architecture) Designed for federation Certificate-based access control (No need for single sign-on, common AUP) Implementations with fine and deep control of resources (ProtoGENI) Deeply Programmable Network Open Flow native Layer 2 backbone 9

10. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 10 GENI Each facility implements Slice-Based Federation Interface Standard, unified means of allocating Virtual machines at each layer of the stack (“slivers”) Networks/sets of virtual machines (“slices”) Already supported by PlanetLab, ORCA, ProtoGENI Now supported by Eucalyptus and OpenStack (our contribution) 10

11. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 11 GENI Mesoscale 11

12. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 12 GENI Mesoscale Deployment Putting the backbone in GENI 47 sites with GENI racks 33 InstaGENI racks 14 ExoGENI racks Fully Interoperable Interoperable with existing GENI aggregates Extensible to new campuses OpenFlow-enabled Layer 2 connectivity between campuses 12

13. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 13 GENI And NSFNet: Back to the Future GENI today is NSFNet circa 1985 GENI and the SFA: Set of standards (e.g., TCP/IP) Mesoscale: Equivalent to NSF Backbone InstaGENI: Hardware/software instantiation of standards that sites can deploy instantly Equivalent to VAX 11 running Berkeley Unix InstaGENI cluster running ProtoGENI and OpenFlow

14. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 14 What About IGNITE? Application suite Analog here is to the Web, and web applications In other words, no one can predict this Did you predict Google?

15. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 15 Foundational Technologies Software-defined networking OpenFlow Highly-Flexible Clouds MetaCloud (ProtoGENI)

16. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 16 Traditional Switch Data Plane Control Plane Management Apps Traditional switches integrate control plane (policy) and data plane (forwarding) Only way to introduce new apps is to integrate with switch Slow, expensive Sole knob is configuration

17. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 17 OpenFlow approach: Factor control plane to off- switch controller SW Off-switch controller offers programmability, visitbility Off-switch controller offers programmability of whole network simultaneously can directly manage global properties without multiple layers Enforces isolation by manipulating FIB on control plane stubs Control Plane Management Apps Management SW Data Plane Simplified switch Control Plane Stub Data Plane Simplified switch Control Plane Stub Data Plane Simplified switch Control Plane Stub

18. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 18 OpenFlow Basics Flow Table Entries Switch Port MAC src MAC dst Eth type VLAN ID IP Src IP Dst IP Prot TCP sport TCP dport RuleActionStats 1.Forward packet to port(s) 2.Encapsulate and forward to controller 3.Drop packet 4.Send to normal processing pipeline 5.Modify Fields + mask what fields to match Packet + byte counters

19. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 19 OpenFlowSwitch.org Controller OpenFlow Switch PC OpenFlow Usage Dedicated OpenFlow Network OpenFlow Switch OpenFlow Switch OpenFlow Protocol Aaron’s code RuleActionStatistics RuleActionStatisticsRuleActionStatistics

20. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 20 20 HP is The OpenFlow Technology Leader 2007 2008 2009 2010 2011 2012 2013 HP & Stanford collaborate on Ethane HP is founding member of ONF HP Labs forms OpenFlow research team HP early-release OpenFlow software to researchers HP demos OpenFlow-enabled switch HP makes OpenFlow software generally available HP helps establish InCNTRE HP will extend OpenFlow across the FlexNetwork architecture 10 20 40 60 Growth in Customer Deployments

21. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 21 HP Labs Contributions to OpenFlow OpenFlow QoS controller ToS support, priority queuing, rate limiters Improved Support for research multi-pathing solutions: – SPAIN (Smart Path Assignment In Networks / NSDI 2010) – HBR (Hash-Based Routing / ICC 2010, ANCS 2010) Improved Deployment Scalability – DevoFlow (SIGCOMM 2011) – TCAM Synthesis (INFOCOMM 2009) Verification of OpenFlow Networks (SDN 2012) OpenFlow™ deployment – Defined GENI deployment strategy with Stanford Univ – OpenFlow™ deployment testing with Stanford Univ – Contributions to OpenFlow™ test suite – OpenvSwitch contributions

22. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 22 HP Labs OpenFlow Research (sample) “DevoFlow: Cost-Effective Flow Management for High Performance Enterprise Networks,”, HotNets, October 2010, SIGCOMM 2011 “Network Integrated Transparent TCP Accelerator,” AINA, April 2010 "ElasticTree: Saving Energy in Datacenter Networks,” NSDI, April 2010 "Automated and Scalable QoS Control for Network Convergence," WREN/INM 2010 Workshop, held in conjunction with NSDI, April 2010. “API Design Challenges for Open Router Platforms on Proprietary Hardware,” HotNets, October 2008 “Verification of Switching Network Properties Using Satisfiability”, SDN, 2012 “A Safe Update Protocol for OpenFlow Networks”, HOT SDN, 2012

23. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 23 Enables a scalable control plane to deliver virtual service networks Rapid, service-oriented provisioning Fine-grain policy control, integrated security Service-focused monitoring Use models across service provider, data center, campus, and branch networks OpenFlow/SDN: A New Way to Build & Manage Networks 23

24. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 24 The MetaCloud (ProtoGENI) Multiple Cloud Infrastructures offering different services and interfaces Hardware as a service VM’s as a Service Containers as a service Threads as a service MapReduce as a service Platform as a service Multiple tools and interfaces Nova, euca2ools, Keystone,…

25. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 25 Problem Different users want different clouds! Varying behavior of various platforms Different toolsets to create, manipulate, manage virtual resources Much of code base replicated All systems involve allocation of physical resources All systems require image store, image load, persistent store… All systems allocate x86 nodes… Problems Currently, cloud systems persistent across hardware base (can’t reuse hardware base between different cloud managers) Have to rebuild common functionality to do new managers

26. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 26 Solution: MetaCloud Underlying API that supports cluster/cloud managers Hardware as a Service Resource Allocation and Specification API Underlying key/authentication solution

27. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 27 The MetaCloud ProtoGENI

28. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 28 ©2010 HP Created on xx/xx/xxxxof 222 The Instageni rack Designed for GENI Meso-scale deployment Eight 2012 deployments, 24 2013 deployments ProtoGENI and FOAM as native Aggregate Managers and Control Frameworks Boots to ProtoGENI instance with OpenFlow switch Designed for wide-area PlanetLab federation PlanetLab image provided with boot InstaGENI PlanetLab Central stood up Designed for expandability Approx 30U free in rack

29. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 29 29 ©2010 HP Created on xx/xx/xxxxof 222 Understanding the instageni rack Two big things: IT’S JUST ProtoGENI It’s this thing

30. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 30 30 ©2010 HP Created on xx/xx/xxxxof 222 It’s just protogeni Key Design criterion behind the InstaGENI rack Reliable, proven control framework Familiar UI to GENI experimenters and administrators Well-understood support and administrative model We’re not inventing new Control Frameworks, we’re deploying Control Frameworks and Aggregate Managers you understand and know how to use Network of baby ProtoGENI’s, with SDN native to the racks Allocation of resources with familiar tools Flack... Easy distribution and proven ability to run many images Support model well-understood If something goes wrong, we know how to fix it... PlanetLab and OpenFlow integration out-of-the-box

31. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 31 31 ©2010 HP Created on xx/xx/xxxxof 222 The “Apple-II of Clouds” Key insight: the Apple II wasn’t the first mass market computer because it was innovative, but because it was packaged Pre Apple-II, computers were all hobbyist kit “Much Assembly, Configuration, Software Writing, Installation required” But the Apple-II worked out of the box Plug it in and turn it on And that’s what made a revolution Same Idea Plug in the InstaGENI Rack Put in the wide-area network connection Rob will install the software and bring it up over the net You’re on the Mesoscale!

32. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 32 32 ©2010 HP Created on xx/xx/xxxxof 222 The InstaGENI rack Designed for easy deployability Power: 220V L6-20 receptacle (or two 110V) Network: 10/100/1000 Base-T Pre-wired from the factory On the Mesoscale Network connections pre-allocated VLANs and connectivity pre-wired before the rack arrives Designed for Remote Management HP iLO on each node Designed for flexible networking 4 1G NICs/node, 20 1G NICs, v2 linecards OpenFlow switch

33. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 33 33 ©2010 HP Created on xx/xx/xxxxof 222 Instageni rack hardware Control Node for ProtoGENI Boss, ProtoGENI users, FOAM Controller, Image storage… HP ProLiant DL 360G7, quad-core, single-socket, dual NIC (1 Gb/sec), 12GB RAM, 4TB Disk (RAID), iLO Five Experiment Nodes HP ProLiant DL 360G7, six-core, dual-socket, quad NIC (1 Gb/sec), 48GB RAM, 1TB Disk, iLO OpenFlow Switch HP E 5406, 20 1 Gb/s, v2 linecards Hybrid mode

34. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 34 34 ©2010 HP Created on xx/xx/xxxxof 222 Instageni planned deployment GENI funding 8 sites in Year 1 24 sites in Year 2 All in USA Other Racks US Public Sector except Federal Government: Special HP program Contact Michaela Mezo, HP SLED

35. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 35 35 ©2010 HP Created on xx/xx/xxxxof 222 Instageni year 1 sites

36. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 36 36 ©2010 HP Created on xx/xx/xxxxof 222 Instageni rack diagram

37. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 37 37 ©2010 HP Created on xx/xx/xxxxof 222 Instageni rack topology

38. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 38 38 ©2010 HP Created on xx/xx/xxxxof 222 instageni photo

39. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 39 39 ©2010 HP Created on xx/xx/xxxxof 222 Instageni software architecture ProtoGENI (Hardware as a Service, Infrastructure as a Service) FOAM (Networks as a Service) ProtoGENI Image PlanetLabImagePlanetLabImage InstaGENI PLC Layer 2 and 3 connectivity GENI L2/L3 Slice

40. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 40 40 ©2010 HP Created on xx/xx/xxxxof 222 Control Infrastructure Control / External switch Data Plane Switch Control Node: Xen Hypervisor ProtoGENI “boss” ProtoGENI “ops” FOAM FlowVisor

41. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 41 41 ©2010 HP Created on xx/xx/xxxxof 222 (rE)Provisioning Nodes ProtoGENI Shared ProtoGENI Exclusive ProtoGENI Exclusive ProtoGENI Exclusive PlanetLab Shared

42. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 42 42 ©2010 HP Created on xx/xx/xxxxof 222 Selected Other Interconnections

43. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. Thanks!

44. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 44 44 The “Grand Challenge” Phase of Research Transition from individual experimenter to institution or multi-institution team Typically necessitated because problems go beyond the scale of an individual research group Investigation of new phenomena required dramatic resources Ex: particle physics 1928-1932

45. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 45 45 Experimental Physics Before 1928 Dominated by tabletop apparatus Ex: Rutherford’s discovery of the nucleus, 1910 Done with tabletop apparatus, shown here Major complication: had to observe in darkened room

46. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 46 46 Example: Chadwick and the Neutron Chadwick used high-energy particles from polonium to bombard nucleus Neutron only method to account for high-energy radiation from bombardment Key apparatus “leftover plumbing” – pipe used to focus radiation beam Date: February, 1932

47. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 47 47 Entry of Institutional Physics Nuclear Fission, Cockcroft and Walton, April, 1932 Key: needed high voltages (est 250,000+ volts) to split nucleus Room(!) to hold apparatus major constraint Needed major industrial help (Metropolitan-Vickers)

48. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 48 48 What a difference two months makes.. Chadwick, 2/32 Cockcroft/Walton, 4/32

49. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 49 Since Then…

50. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 50 50 Key Differences Apparatus now takes many years to construct, costs billions Requires multi-national consortia Discoveries made by large teams of scientists Hundreds on the Top Quark team\ Thousands on the Higgs Team Experiments last for 30+ years Ex: ALICE at LHC, Babar at SLAC Experimental devices measured by energies of collisions produced Driven by cost and complexity of apparatus Cockcroft and Walton heralded era of institutional Grand Challenge physics

51. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 51 The era of institutional systems research Computer Systems Research, 1980-2010 Dominated by desktop-scale systems 1980-~1995: The desktop was the experimental system Ex: Original URL of Yahoo! was akebono.cs.stanford.edu/yahoo.html Akebono was Jerry Yang’s Sun workstation! Named for a prominent American Sumo wrestler – Jerry had spent a term in Kyoto in 1992 Sometimes “servers” used to offload desktops But rarely: “Server” ca. 1990 was a VAX 11, less powerful than a SUN or DEC workstation ~1995-~2005: Used servers primarily because desktop OS unsuitable for serious work ~2005-: Need clusters (and more) for any reasonable experiment The Era of Institutional Systems Research has begun 51

52. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 52 Why? Activity in 21 st Century Systems Research focused on massively parallel, loosely-coupled, distributed computing Content Distribution Networks Key-Value Stores Cloud Resource Allocation and Management Wide-Area Redundant Stores Fault Recovery and Robust Protocols End-system multicast Multicast messaging Key Problem: Emergent Behavior at Scale Can’t anticipate phenomena at scale from small-scale behavior Hence: Moderate-to-large scale testbeds: G-Lab, PlanetLab, OneLab,… 52

53. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 53 What we need, what we don’t What we need Method of creating slices on clouds and distributed infrastructures Method of communicating between clouds and distributed infrastructures Method of interslice communication between clouds What we don’t Single sign-on! Single AUP Single resource allocation policy or procedure Unified security policy Principle of Minimal Agreement What is the minimum set of standards we can agree on to make this happen? 53

54. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 54 What do we need from the clouds Building Blocks Eucalyptus: Open-source clone of EC-2 OpenStack: Open-source Widespread developer mindshare (easy to use, familiar) What we want: Slice-Based Federation Architecture Means of creating/allocating slices Authorization by Attribute-Based Access Control (ABAC) Delegation primitive Explicit costs/resource allocation primitives Need to be able to control costs for the developer 54

55. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 55 Why GENICloud? Minimal set of facilities to permit seamless interconnection without trust Motivation: the Web Web sites mutually untrusting Key facilities: DNS, HTTP. HTML What are the equivalents for Clouds? Our cut: Slices, ABAC, DNS conventions....transcloud.net 55

56. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 56 Introduction – TransCloud TransCloud = A Cloud Where Services Migrate, Anytime, Anywhere In a World Where Distance Is Eliminated Joint Project Between GENICloud, iGENI, G-Lab GENICloud Provides Seamless Interoperation of Cloud Resources Across N-Sites, N- Administrative Domains iGENI Optimizes Private Networks of Intelligent Devices G-Lab contributes networking and advanced cloud resources

57. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 57 Seamless Computation Services Available Anytime, Anywhere “The Cloud” offers the prospect of ubiquitous information and services…BUT… Performance of Cloud services Highly Dependent On Location Of End-User, Applications, Middle Processes, Network Topology Of Cloud Data, Compute Processes, Storage, etc Why? Performance of Legacy Protocols Highly Dependent on Latency Therefore: Want to compute anywhere convenient Want to be able to compute everywhere

58. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 58 What do we need to make this work? Ability to instantiate and run a program anywhere Common API at each level of the stack IaaS/NaaS (VM/VN Creation) PaaS (guaranteed OS/Progamming environment) OaaS (Standard Query/Data Management API) Easy, Standard Naming Scheme I need to know the name of my VM’s, logins, store etc without asking

59. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 59 Solution – TransCloud Introducing TransCloud Prototype An Early Instantiation of the Architecture A Distributed Environment That Enables Component and Interoperability Evaluation A Testbed On Which Early Experimental Research Can Be Conducted An Environment That Can Be Used To Explain/Showcase New Innovative Architecture/Concepts Through Demonstrations

60. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 60 DEMO What is the World’s Greenest City? Answering this question through analysis of landsat data Perfect job for distributed cloud Currently running on HP Labs GENICloud But we can distribute it anywhere… 60

61. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 61

62. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 62

63. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 63

64. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 64

65. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 65

66. © Copyright 2012 Hewlett-Packard Development Company, L.P. The information contained herein is subject to change without notice. 66 TransCloud Today Approx 40 nodes at 4 sites, 10 Gb/s connectivity