Presentation is loading. Please wait.

Presentation is loading. Please wait.

GRNETDeIC konference 20121 20121112 Building a real-world public cloud from the ground up Vangelis Koukis Technical.

Similar presentations


Presentation on theme: "GRNETDeIC konference 20121 20121112 Building a real-world public cloud from the ground up Vangelis Koukis Technical."— Presentation transcript:

1 GRNETDeIC konference Building a real-world public cloud from the ground up Vangelis Koukis Technical Coordinator,  okeanos Project

2 Greek Research and Technology NetworkDeIC konference Outline   okeanos ?  Rationale  Design – Platform - Features  Unity - Automation  Opensource – Upcoming

3 Greek Research and Technology NetworkDeIC konference What is  okeanos? ‘okeanos’ is Greek for ‘ocean’. Oceans capture, store and deliver energy, oxygen and life around the planet.

4 GRNETDeIC konference Simplicity

5 GRNETDeIC konference

6 GRNETDeIC konference Compute Network Storage Security Virtual Machines Virtual Ethernets Virtual Disks Virtual Firewalls

7 GRNETDeIC konference Flexibility

8 GRNETDeIC konference

9 GRNETDeIC konference x 2x5x 8x

10 Greek Research and Technology NetworkDeIC konference  okeanos service  Goal: Production-quality IaaS  Beta in Dec, current Alpha: >1600 VMs / >1000 users  Target group: GRNET’s customers  direct: IT depts of connected institutions  indirect: university students, researchers in academia  Users manage resources over  a simple, elegant UI, or  a REST API, for full programmatic control

11 Greek Research and Technology NetworkDeIC konference  okeanos features  Compute/Network Service: Cyclades  File Storage Service: Pithos+  Image Service: Plankton  Identity Service: Astakos  Volume Service: Archipelago

12 GRNETDeIC konference Rationale

13 Greek Research and Technology NetworkDeIC konference How it all started  Need for easy, secure access to GRNET’s datacenters  User friendliness, simplicity  Scalable to the thousands  #VMs, TBs, users (Pithos:  10k)  running within GRNET’s AAI Federation  Resell or build your own?  IaaS cloud provider, vendor, or own infrastructure?  It all depends on your needs

14 Greek Research and Technology NetworkDeIC konference Build on commercial IaaS?  Commercial IaaS  Amazon EC2 not an end-user service  Need to develop custom UI, AAI layers  Vendor lock-in  Unsuitable for IT depts persistent, long-term servers custom networking requirements  GRNET has invested heavily in its core network  > 8000km of dark fiber

15 Greek Research and Technology NetworkDeIC konference Bring vendor into datacenter?  Hypervisor lock-in  Is a turn-key solution suitable for a public cloud?  Building public clouds is an ongoing process  Manageable by GRNET’s operation  Integrated into the rest of the infrastructure  Scaling to thousands of users  Build on existing know-how  Gain know-how, build own IaaS  reuse for own services

16 Greek Research and Technology NetworkDeIC konference What about opensource?  OpenStack, Eucalyptus, OpenNebula  Need a mature opensource core to build around  Maturity, production-readiness?  proven in production environments, predictable  Extensibility?  Flexibility?  Upgradeability, maintainability?

17 GRNETDeIC konference Design

18 Greek Research and Technology NetworkDeIC konference  okeanos design decisions  Reuse existing components  Build on Google Ganeti  target commodity hardware  release to the community as opensource

19 Greek Research and Technology NetworkDeIC konference  okeanos design principles  No need to make the world  No need to support everything  Service developed and maintained by  people  Start from the architecture…  …then discover, combine, reuse the right components  And for everything that’s not already available  Do it yourself!

20 GRNETDeIC konference

21 Greek Research and Technology NetworkDeIC konference Jigsaw puzzle  Synnefo  custom cloud management software to power  okeanos  Google Ganeti backend  VM cluster management: physical nodes, VMs, migrations  OpenStack APIs: Compute API v1.1, Object Storage API  with custom extensions whenever necessary  Then everything comes together  UI, Networking, Images, Storage, Monitoring, Identity management, Accounting, Billing, Clients, Helpdesk

22 Greek Research and Technology NetworkDeIC konference Why Ganeti?  No need to reinvent the wheel  Scalable, proven software infrastructure  Built with reliability and redundancy in mind  Combines open components (KVM, LVM, DRBD)  Well-maintained, readable code  VM cluster management in production is serious business  reliable VM control, VM migrations, resource allocation  handling node downtime, software upgrades

23 Greek Research and Technology NetworkDeIC konference Why Ganeti?  GRNET already had long experience with Ganeti  provides  280 VMs to NOCs through the ViMa service  involved in development, contributing patches upstream  Build on existing know-how for  okeanos  Common backend, common fixes  reuse of experience and operational procedures  simplified, less error-prone deployment

24 GRNETDeIC konference Platform

25 Greek Research and Technology NetworkDeIC konference Software Stack Multiple users, multiple resources Multiple VMs on cluster Single VM Synnefo Ganeti KVM REST API

26 Greek Research and Technology NetworkDeIC konference Platform Design Web ClientCLI ClientWeb Client 2 GRNET datacenter Debian Virtual Hardware OpenStack Compute API v1.1 Direct Outof Band Access Synnefo cloud management software Google Ganeti KVM GRNET Proprietary

27 GRNETDeIC konference Features

28 Greek Research and Technology NetworkDeIC konference Virtual Machine Actions My_Windows_desktop Shutdown Reboot Start Console Destroy

29 Greek Research and Technology NetworkDeIC konference IaaS – Compute (1)  Virtual Machines  powered by KVM Linux and Windows guests, on Debian hosts  Google Ganeti for VM cluster management  accessible by the end-user over the Web or programmatically (OpenStack Compute v1.1)

30 Greek Research and Technology NetworkDeIC konference IaaS – Compute (2)  User has full control over own VMs  Create Select # CPUs, RAM, System Disk OS selection from pre-defined or custom Images popular Linux distros (Fedora, Debian, Ubuntu) Windows Server 2008 R2  Start, Shutdown, Reboot, Destroy  Out-of-Band console over VNC for troubleshooting

31 Greek Research and Technology NetworkDeIC konference IaaS – Compute (3)  REST API for VM management  OpenStack Compute v1.1 compatible  3rd party tools and client libraries  custom extensions for yet-unsupported functionality  Python & Django implementation  Full-featured UI in JS/jQuery  UI is just another API client  All UI operations happen over the API

32 Greek Research and Technology NetworkDeIC konference IaaS – Network (Virtual Ethernets) Internet Private Network 1 Private Network 2Private Network 3

33 Greek Research and Technology NetworkDeIC konference IaaS – Network - Functionality  Dual IPv4/IPv6 connectivity for each VM  Easy, platform-provided firewalling  Array of pre-configured firewall profiles  Or roll-your-own firewall inside VM  Multiple private, virtual L2 networks  Construct arbitrary network topologies  e.g., deploy VMs in multi-tier configurations  Exported all the way to the API and the UI

34 GRNETDeIC konference Unity

35 Greek Research and Technology NetworkDeIC konference Spawn Freeze Images Ubuntu my own Ubuntu

36 Greek Research and Technology NetworkDeIC konference Custom Images: snf-image  Untrusted images  Host cannot touch user-provided data  Resize fs, change hostname, change passwords, inject files  Split design  snf-image-host  snf-image-helper  All customization in helper VM

37 Greek Research and Technology NetworkDeIC konference  OpenStack Object Storage API  Block storage  Content-based addressing for blocks  Every file is a collection of blocks  Web-based, command-line, and native clients  Synchronization, deduplication  An integral part of  okeanos  User files, Image registry for VM Images  Goal: use common backend with Archipelago

38 Greek Research and Technology NetworkDeIC konference Spawn Freeze Images Ubuntu my own Ubuntu

39 Greek Research and Technology NetworkDeIC konference Clone Snapshot Images  Storage Ubuntu root Ubuntu + user data

40 Greek Research and Technology NetworkDeIC konference Images – Golden Image golden Debian

41 Greek Research and Technology NetworkDeIC konference IaaS – Storage

42 Greek Research and Technology NetworkDeIC konference Archipelago RADOS Object Storage nodes IaaS – Storage Maps Volume Composer object I/O Monitor nodes Storage

43 Greek Research and Technology NetworkDeIC konference RADOS Storage nodes IaaS – Storage Maps Volume Composer object I/O Monitor nodes

44 Greek Research and Technology NetworkDeIC konference IaaS – Storage (1)  First-phase deployment  System-provided and custom user Images  Redundant storage based on DRBD  VMs survive physical node downtime or failure  Currently under testing  Reliable distributed storage over RADOS  Combined with custom software for snapshotting, cloning  Dynamic virtual storage volumes

45 Greek Research and Technology NetworkDeIC konference IaaS – Storage (2)  Multi-tier storage architecture  Dedicated Storage Nodes (SSD, SAS, and SATA storage)  OSDs, e.g., for RADOS  Custom storage layer: Archipelago  manages snapshots, creates clones over block pools  OS Images held as snapshots  VMs created as clones of snapshots

46 GRNETDeIC konference Integration

47 Greek Research and Technology NetworkDeIC konference

48 Greek Research and Technology NetworkDeIC konference

49 Greek Research and Technology NetworkDeIC konference

50 Greek Research and Technology NetworkDeIC konference

51 Greek Research and Technology NetworkDeIC konference Support services  Identity: Astakos  Provides the user base for  okeanos  Once authenticated, the user retrieves a common auth token for programmatic access

52 GRNETDeIC konference Automation

53 Greek Research and Technology NetworkDeIC konference /kamaki $./kamaki Usage: kamaki [options] … --api=API API can be either openstack or synnefo --url=URL API URL --token=TOKEN use token TOKEN … Commands: flavor info get flavor details flavor list list flavors … image create create image image delete delete image $./kamaki server shutdown url=http://localhost:8000/api/v1.1 --token= db2…

54 Greek Research and Technology NetworkDeIC konference /kamaki $ ipython In [1]: from kamaki.client import Client In [2]: c = Client('http://localhost:8000/api/v1.1', " db2…") In [3]: c.list_flavors() … In [4]: i = c.list_images() In [5]: i[5] {u'created': u' T00:00:00+00:00', u'id': 7, u'metadata': {u'values': {u'OS': u'windows', u'size': u'11000'}}, u'name': u'Windows', u'progress': 100, u'status': u'ACTIVE', u'updated': u' T14:47:12+00:00'} In [6]: c.create_server('mywin1', 3, 5)

55 GRNETDeIC konference Sights

56 Greek Research and Technology NetworkDeIC konference Live Demo  Prepare and upload Image from local template VM  Spawn compute cluster to run MPI app  Make local modifications and repeat  … What if it was over a 3G connection?  Time needed to upload 1GB Image file?  Time needed to prepare and spawn virtual nodes?

57 GRNETDeIC konference Internals

58 Greek Research and Technology NetworkDeIC konference Ganeti Master Queue Deployment DB Ganeti node KVM … Web Server REST API Logic RAPI SQL snf-gnt-eventd snf-gnt-hook snf-dispatcher API Server apiaai SQL ui

59 Greek Research and Technology NetworkDeIC konference Upcoming goals  Credit-based resource allocation  Abstract away the Ganeti backend, replace with backend connector behind the MQ  Release to community as reference implementation of OpenStack Compute v1.1  Support live modification of VMs in Ganeti  Snapshots, clones in storage layer  Dramatic decrease in VM initialization time  Support workloads with 100s of ephemeral VMs e.g. for scientific computation, MPI jobs

60 GRNETDeIC konference

61 GRNETDeIC konference

62 GRNETDeIC konference

63 GRNETDeIC konference

64 GRNETDeIC konference

65 GRNETDeIC konference

66 GRNETDeIC konference

67 GRNETDeIC konference

68 GRNETDeIC konference

69 GRNETDeIC konference

70 GRNETDeIC konference

71 GRNETDeIC konference

72 GRNETDeIC konference

73 GRNETDeIC konference

74 GRNETDeIC konference

75 GRNETDeIC konference

76 GRNETDeIC konference

77 GRNETDeIC konference

78 GRNETDeIC konference

79 GRNETDeIC konference

80 GRNETDeIC konference

81 GRNETDeIC konference

82 GRNETDeIC konference

83 GRNETDeIC konference

84 GRNETDeIC konference

85 GRNETDeIC konference

86 GRNETDeIC konference

87 GRNETDeIC konference

88 GRNETDeIC konference

89 GRNETDeIC konference

90 GRNETDeIC konference

91 GRNETDeIC konference

92 GRNETDeIC konference

93 GRNETDeIC konference

94 GRNETDeIC konference

95 GRNETDeIC konference

96 GRNETDeIC konference

97 GRNETDeIC konference

98 GRNETDeIC konference

99 GRNETDeIC konference

100 GRNETDeIC konference

101 GRNETDeIC konference

102 GRNETDeIC konference

103 GRNETDeIC konference

104 GRNETDeIC konference

105 GRNETDeIC konference

106 GRNETDeIC konference

107 GRNETDeIC konference

108 GRNETDeIC konference

109 GRNETDeIC konference

110 GRNETDeIC konference

111 GRNETDeIC konference

112 GRNETDeIC konference

113 GRNETDeIC konference

114 GRNETDeIC konference

115 GRNETDeIC konference

116 GRNETDeIC konference Upcoming

117 Greek Research and Technology NetworkDeIC konference Current and Upcoming features  Now: Alpha2  Common user base, custom user images on Pithos+  short-term: Synnefo v0.12, Beta  Ultra-lightweight VMs on Archipelago with RADOS backend  medium-term  Volumes: clonable / snapshottable / attachable disks  Network and storage hotplugging  Upcoming beta in fully populated datacenter

118 Greek Research and Technology NetworkDeIC konference Opensource  Synnefo: Cyclades / Pithos+ / Astakos  https://code.grnet.gr/projects/synnefo  https://code.grnet.gr/projects/pithos  https://code.grnet.gr/projects/astakos  kamaki  https://code.grnet.gr/projects/kamaki pip install or apt-get install everything!

119

120 Greek Research and Technology NetworkDeIC konference Thank You! Questions?

121 Greek Research and Technology NetworkDeIC konference Asynchronous design  DB contains All state needed to handle API queries  no need to reach the backend  Ganeti GetInstanceInfo() is a proper job, too slow  Two distinct paths, effect and update  Effect changes to VMs  when servicing API requests to modify VM state  issue commands to Ganeti backend, over RAPI  ACK reception of request to user  Update DB, when interesting things happen  user or admin initiated  Queue notifications to Message Queue, over AMQP

122 Greek Research and Technology NetworkDeIC konference Ganeti Master Queue Synnefo deployment DB Ganeti node KVM … Web Server REST API Logic RAPI SQL snf-gnt-eventd snf-gnt-hook snf-dispatcher API Server apiaai SQL ui

123 Greek Research and Technology NetworkDeIC konference The “effect” Path  Reception of API request to modify VM state (e.g., PUT /servers over HTTP)  API enforces access rights and policy  Ganeti knows no cloud users or access rights  Need to translate from Openstack Compute to backend ops (e.g., CreateInstance())  Asynchronous request processing  Return HTTP 202 Accepted  it’s up to the API client to poll for completion

124 Greek Research and Technology NetworkDeIC konference Ganeti Master Queue Synnefo deployment DB Ganeti node KVM … Web Server REST API Logic RAPI SQL snf-gnt-eventd snf-gnt-hook snf-dispatcher API Server apiaai SQL ui

125 Greek Research and Technology NetworkDeIC konference The “update” path  May run at any time  Completely decoupled from “effect” path  Design goal:  Ganeti admins free to bypass frontend  Synnefo adapts  Synnefo logic triggered on backend events  Ganeti operation progressing in the queue  Synnefo hook running inside Ganeti Hooks run at various phases in a VM’s lifecycle

126 Greek Research and Technology NetworkDeIC konference Ganeti Master Queue Synnefo deployment DB Ganeti node KVM … Web Server REST API Logic RAPI SQL snf-gnt-eventd snf-dispatcher ui API Server apiaai SQL snf-gnt-hook

127 Greek Research and Technology NetworkDeIC konference The Ganeti event daemon  Ganeti master manages job queue  Jobs pass Queued, Waiting, Running, end up in Canceled, Success, Error.  Need a way for Synnefo to monitor job progress  Synnefo-specific solution: Ganeti event daemon  Passively monitor the Ganeti job queue  Notifications over AMQP on job progress  Synnefo logic listens to Message Queue, updates DB  inotify()- based mechanism, no code changes to Ganeti

128 Greek Research and Technology NetworkDeIC konference The Synnefo hook in Ganeti  Different phases in a VM’s lifecycle  { pre, post } – { add, start, stop, reboot, modify }  Run Synnefo-specific hook in post-*  Pushes VM configuration notifications to MQ  e.g., NIC setup

129 Greek Research and Technology NetworkDeIC konference IaaS – Network - Implementation  Custom modifications to Ganeti  IP pool management for the public network  Custom-written DHCP server over NFQUEUE  Custom interface handling scripts  Enforce VM networking configuration  Private Networks  Alpha: pre-provisioned bridges to 802.1Q VLANs  Later on: MAC-prefix based filtering

130 Greek Research and Technology NetworkDeIC konference Ganeti Master Queue Synnefo deployment DB Ganeti node KVM … Web Server REST API Logic RAPI SQL snf-gnt-eventd snf-dispatcher ui API Server apiaai SQL snf-gnt-hook

131 Greek Research and Technology NetworkDeIC konference Reconciliation with Ganeti  What if the MQ is down, and messages are lost?  Ganeti is the Single Source of Truth for VM state  Reconcile DB state asynchronously  On success notification for a Ganeti GetInstanceInfo() op  Triggered periodically, e.g., using cron  or even by the administrator, running gnt-instance info manually


Download ppt "GRNETDeIC konference 20121 20121112 Building a real-world public cloud from the ground up Vangelis Koukis Technical."

Similar presentations


Ads by Google