Presentation is loading. Please wait.

Presentation is loading. Please wait.

Mastering Cloud Computing

Similar presentations


Presentation on theme: "Mastering Cloud Computing"— Presentation transcript:

1 Mastering Cloud Computing
Rajkumar Buyya, Christian Vecchiola, Thamarai Selvi

2 Chapters Part I: Foundations Chapter 1 — Introduction
Chapter 2 — Principles of Parallel and Distributed Computing Chapter 3 — Virtualization Chapter 4 — Cloud Computing Architecture Part II: Cloud Application Programming and the Aneka Platform Chapter 5 — Aneka: Cloud Application Platform Chapter 6 — Concurrent Computing: Thread Programming Chapter 7 — High-Throughput Computing: Task Programming Chapter 8 — Data Intensive Computing: Map-Reduce Programming Part III: Industrial Platforms and New Developments Chapter 9 — Cloud Platforms in Industry Chapter 10 — Cloud Applications Chapter 11 — Advanced Topics in Cloud Computing

3 Chapter 1 - Introduction

4 Global Cloud Marketplace
I need to grow my infrastructure, but I do not know for how long… I cannot invest in infrastructure, I just started my business…. I want to focus on application logic and not maintenance and scalability issues I want to access and edit my documents and photos from everywhere.. I have a surplus of infrastructure that I want to make use of I have a lot of infrastructure that I want to rent … I have infrastructure and middleware and I can host applications I have infrastructure and provide application services Global Cloud Marketplace

5

6 Private Cloud (Government)
Public Clouds Compute Storage Applications Development and Runtime Platform All users, on any device Private Resources Cloud Manager Private Cloud Private Cloud (Government) Organization Personnel Government Agencies

7 Cloud Deployment Models
Public/Internet Clouds * 3rd party, multi-tenant Cloud infrastructure & services: * available on subscription basis to all. Private/Enterprise Clouds * A public Cloud model within a company’s own Data Center / infrastructure for internal and/or partners use. Hybrid/Inter Clouds * Mixed usage of private and public Clouds: Leasing public cloud services when private cloud capacity is insufficient

8 Cloud Computing? Virtualization IT outsourcing Pay as you go
Provisioning on demand Elasticity Utility computing Scalability IaaS PaaS SaaS Virtual Datacenters Green computing Internet No capital investments Quality of Service Cloudbusrting Security Privacy & Trust Service Level Agreement Billing Cloud Computing?

9 Infrastructure as a Service
Runtime Environment for Applications Development and Data Processing Platforms Examples: Windows Azure, Hadoop, Google AppEngine, Aneka Platform as a Service Virtualized Servers Storage and Networking Examples: Amazon EC2, S3, Rightscale, vCloud Infrastructure as a Service End user applications Scientific applications Office automation, Photo editing, CRM, and Social Networking Examples: Google Documents, Facebook, Flickr, Salesforce Software as a Service Web 2.0 Interfaces

10 Clouds Grids Clusters Mainframes 1950 1960 1970 1980 1990 2000 2010
2010: Microsoft Azure 1970: DARPA’s TCP/IP 1999: Grid Computing 1984: IEEE 802.3 Ethernet & LAN 1997: IEEE (Wi-Fi) 2008: Google AppEngine 1966: Flynn’s Taxonomy SISD, SIMD, MISD, MIMD 1989: TCP/IP IETF RFC 1122 2007: Manjrasoft Aneka 1969: ARPANET 1984: DEC’s VMScluster 2005: Amazon AWS (EC2, S3) 1951: UNIVAC I, First Mainframe 1975: Xerox PARC Invented Ethernet 2004: Web 2.0 Clouds 1990: Lee-Calliau WWW, HTTP, HTML 1960: Cray’s First Supercomputer Grids Clusters Mainframes 1950 1960 1970 1980 1990 2000 2010

11 Chapter 2 – Parallel and Distributed Computing

12 High speed network connection
Compilers, PVM, MPI, …. Parallel Programming Environments Cluster Middleware: Single System Image and Availability Infrastructure Network Interface HW Operating System Common SW & Applications PC / Workstation High speed network connection Sequential Applications ParallelApplications

13 Computer and network hardware Hardware
Support for heterogeneous resource sharing, communication, and programming environments for application development Middleware User interface for presentation Applications Distributed System Stack Execution platform including network connectivity services Operative System

14 Java RMI, Remoting, CORBA, ORBIX.DCOM, COM+ RPC / Specific Tasks
Paradigms / Architectural Models Message Based Communication Client - Server Object Space Distributed Objects RPC Peer –to-Peer Java Spaces Java RMI, Remoting, CORBA, ORBIX.DCOM, COM+ RPC / Specific Tasks MPI Programming, JMS, MSMQ, MQS Video Conferencing Istant Messaging, DistributedAgents Mobile Agents

15 1940 1950 Parallel Era Sequential Era 1960 1970 1980 1990 2000 2010
Applications Parallel Era Sequential Era Compilers Architectures 1960 1970 1980 1990 2000 2010 2020 2030 Problem Solving Environments

16 Processor Data Input Data Output Instruction Stream

17 Single Instruction Stream
Processor N Data Output N Single Instruction Stream Processor 2 Processor 1 Data Input 1 Data Input 2 Data Input N Data Output 2 Data Output 1

18 Processor 1 Processor 2 Processor N Instruction Stream 1
Single Data Input Stream Single Data Output Stream Stream 2 Stream N

19 Processor 1 Processor 2 Processor N Instruction Stream 1 Stream 2
Stream N Data Input 1 Data Input 2 Data Input N Data Output 1 Data Output 2 Data Output 3

20 Global System Memory Cache L1 Cache L1 Core 2 IPC Channel Processor 1
Memory Bus Processor 2 Processor N Local Memory IPC Channel

21 + x load Large Level (Processes, Tasks) Task 1 Task 2 Task N
Function 1 Function 2 Function J Statements x load function f1() {…} function f2() function fJ() a[0] = … b[0] = … a[1] = … b[1] = … a[k] = … b[k] = … Shared Memory Messages IPC Medium Level (Threads, Functions) Fine Level (Processor, Instructions) Very Fine Level (Cores, Pipeline, Instructions)

22 Applications Middleware Operating System Hardware
Networking and Parallel Hardware IPC primitives for control and data. Frameworks for distributed programming

23 Applications (SaaS) Middleware (PaaS) Hardware and OS (IaaS)
Social Networks, Scientific Computing, Enterprise Applications Virtual hardware, networking, OS images, and storage. Frameworks for Cloud Application Development

24 Client-server Two Tier (Classic Model) Three Tier N Tier request
Server/client server request response Two Tier (Classic Model) Three Tier N Tier

25 P2P peer

26 RPC RPC Service RPC Library Network Node A Node B Main Procedure
Procedure A Procedure B Procedure C:Node B RPC Library Node A Program A (RPC Client) Procedure C Node B Program C (RPC Server) Procedure Registry Parameters Marshaling and Procedure Name Return Value Marshaling Parameters Unmarshaling and Procedure Name Unmarshaling Network

27 Remote Reference Module
DOF interaction Instance Object Proxy Remote Reference Module Node A Application A Node B Application B Network Object Skeleton Remote 2 3 7 8 12 13 18 19 1: Ask for Reference 4 6 9 5: Object Activation 10 11 14 15 16 17 20 21

28 Web WSDL(s) Web Service Server UDDI Registry WS Client WSDL
Application WS Client Invocation Query Response (WSDL) Web Server UDDI Registry WSDL(s) WSDL Web Service

29 HTTP, FTP,e-mail, MQ, IIOP, ….
WS technology Stack Web Service Flow Service Discovery Service Description XML-based Messaging Network Security Service Publication Quality of Service Management WSDL SOAP HTTP, FTP, , MQ, IIOP, …. Direct  UDDI Static  UDDI WSFL

30 Header: Metadata & Assertions
SOAP Messages POST /InStock HTTP/1.1 Host: Content-Type: application/soap+xml; charset=utf-8 Content-Length: <Size> <?xml version=“1.0”> <soap:Envelope xmlns:soap=“http//www.w3.org/2001/12/soap-envelope” soap:encondingStyle=“http//www.w3.org/2001/12/soap-encoding”> <soap:Header></soap:Header> <soap:Body xmlns:m=http://www.stocks.org/stock> <m:GetStockPrice> <m:StockName>IBM<m:StockName> </m:GetStockPrice> </soap:Body> </soap:Envelope> Envelope Header: Metadata & Assertions Body: Method Call

31 SOAP Messages Envelope Header: Metadata & Assertions
POST /InStock HTTP/1.1 Host: Content-Type: application/soap+xml; charset=utf-8 Content-Length: <Size> <?xml version=“1.0”> <soap:Envelope xmlns:soap=“http//www.w3.org/2001/12/soap-envelope” soap:encondingStyle=“http//www.w3.org/2001/12/soap-encoding”> <soap:Header></soap:Header> <soap:Body xmlns:m=http://www.stocks.org/stock> <m:GetStockPriceResponse> <m:Price>34.5<m:Price> </m:GetStockPriceResponse> </soap:Body> </soap:Envelope> Envelope Header: Metadata & Assertions Body: Execution Result

32 Chapter 3 – Virtualisation

33 Guest Virtualization Layer Software Emulation Host Virtual Hardware
Virtual Networking Virtual Storage Software Emulation Host Physical Hardware Physical Storage Physical Networking Guest Applications Virtual Image

34 Programming Languages Applications
Operative Systems Execution Stack Hardware - level Virtualization Hardware Programming Languages Applications OS- level Virtualization Programming Language level Virtualization Application - level Virtualization

35 Aggregation Sharing Emulation Isolation Virtual Resources
Virtualization Physical Resources Virtual Resources

36 …. Virtualization Execution Environment Storage Network Emulation
High-Level VM Multiprogramming Hardware-assisted Process Level System Level Paravirtualization Full Virtualization How it is done? Technique Virtualization Model Application Programming Language Operating System Hardware Partial Virtualization

37 API ABI ISA Applications API calls Libraries System calls User
Hardware Operative System ISA Applications API calls System calls User

38 Ring 3 Ring 2 Ring 1 Ring 0 Least privileged mode (user mode) Privileged modes Most privileged mode (supervisor mode)

39 Guest Virtual Image Storage VMM Virtual Machine Host In memory
binary translation instruction mapping interpretation …… Guest In memory representation Storage Virtual Image Host emulation

40 Virtual Machine Manager
ABI Hardware Operative System ISA Virtual Machine Manager VM

41 Virtual Machine Manager
ISA Virtual Machine Instance Instructions (ISA) Interpreter Routines Allocator Dispatcher

42 User Instructions Sensitive Instructions Privileged Instructions

43 Virtual Machine Manager
Server A (running) VM Server B Virtual Machine Manager (inactive) Before Migration After Migration

44 Privileged instructions
Xen Hypervisor (VMM) Memory management CPU state registers Devices I/O User Domains (Domain U) Guest OS Modified codebase Hypercalls into Xen VMM User Applications (unmodified ABI) Management Domain (Domain 0) VM Management HTTP interface Access to the Xen Hypervisor Ring 3 Ring 2 Ring 1 Ring 0 Hardware (x86) Privileged instructions Hardware trap

45 Guest Operating System
Hypervisor Binary translation Instruction caching Guest Operating System Unmodified codebase VMM unaware User Applications (unmodified ABI) Ring 3 Ring 2 Ring 1 Ring 0 Hardware (x86) Hardware trap (sensitive instructions) Dynamic / cached translation (sensitive instructions)

46 Virtual Machine Instance
Hardware (x86) Host Operating System VMware Hypervisor (VMM) Direct access to hardware I/O, memory, networking for guests Save/Restore CPU state for host OS VMware Driver Virtual Machine Instance User Applications VMware Workstation Guest Operating System I/O

47 VMware Hypervisor (VMM)
Hardware (x86) Host Operating System VMware Hypervisor (VMM) Direct access to hardware I/O, memory, networking for guests Save/Restore CPU state for host OS VMware Driver VM Instance serverd (daemon) VMware Web Server

48 Hardware VMkernel hostd CIM broker User world API Resource scheduling
VMX CIM broker VM User world API Resource scheduling Device drivers Storage stack Network stack Distributed VM file system Virtual Ethernet adapter and switch VMM DCUI syslog vxpa SNMP Third-party CIM plug-ins

49 vCenter vCloud vFabric Zimbra
Server ESXi ESX vSphere Data Center vCenter vCloud Cloud Infrastructure Virtualization vFabric Platform Zimbra Application

50 Latest Pictures. Hardware (x86) Processor Memory Hypervisor (Ring -1)
Hypercalls MSRs APIC Scheduler Address Management Partition Management Root / Parent Partition VMWPs VMMS WMI Hypervisor-aware Kernel (Ring 0) VSPs VID WinHv I/O Stack Drivers VMBus Enlightened Child Partition User Applications (Ring 3) Hypervisor-aware Wndows Kernel (Ring 0) VSCs / ICs Linux Kernel (Ring 0) LinuxHv Unenlightened Child Partition Hypervisor-unaware Kernel (Ring 0) Processor Memory Latest Pictures.

51 Chapter 4 – Cloud Computing Architecture

52 SaaS PaaS IaaS Cloud Applications
Virtual Machine (VM), VM Management and Deployment QoS Negotiation, Admission Control, Pricing, SLA Management, Monitoring, Execution Management, Metering, Accounting Cloud Hosting Platforms Web 2.0, Mashups, Concurrent and Distributed Programming, Workflows , Libraries, Scripting Cloud Programming Environment and Tools User Applications Social Computing, Enterprise ISV, Scientific Computing, CDNs Cloud Applications Cloud Resources Autonomic Cloud Economy Adaptive Management User-level Middleware Core System Infrastructure SaaS PaaS IaaS IaaS (M) Pure PaaS Pure SaaS

53 Physical Infrastructure
Datacenter Cluster Desktop / Heterogeneous Resources Third Party IaaS Cloud Infrastructure Management Software Scheduling Pricing / Billing Monitoring QoS SLA Management Reservation VM Pool Management VM Image Repository Provisioning Web Services, Portals, REST API Web based Management Interface

54 Programming API / Libraries
Physical Infrastructure IaaS Providers PaaS Core Middleware Resources Management Application Management User QoS / SLA Management & Billing Elasticity & Scaling Runtime Web Services, Portals, REST API Web based Interface

55 PaaS IaaS Platform as a Service Solutions DataSynapse, Zimory Pools,
Elastra CloudServer, Aneka, …. Platform as a Service Solutions PaaS IaaS IaaS (M) Datacenters, Clusters, Desktop Grids Physical Infrastructure KVM, Xen, VMWare, … Virtual Machine Technology Eucalyptus, OpenNebula, VMWare vCloud, OpenPEX, InterGrid, … Infrastructure Management Software

56 Hybrid / Heterogeneous Cloud
Datacenter Public Cloud Cluster Desktop Grids / NOWs Dynamic Provisioning IaaS (M) Paas Private Cloud Hybrid / Heterogeneous Cloud

57 Community Cloud 3rd Party Cloud Private Cloud Public Cloud
Users Public Services Industries Enterprises 3rd Party Cloud Public Cloud Private Cloud Community Cloud Federal and Government Bodies Development Platforms Infrastructure Application Services

58 Chapter 5 – Aneka: Cloud Application Platform

59 Multiple Applications Multiple Infrastructures
Thread Task ... MapReduce 1. SDK Aneka 2. Runtime Multi-core Cluster Grid Cloud 2100 Multiple Infrastructures

60 PAL – Platform Abstraction Layer
ECMA 334: .NET or Mono / Windows, Linux, & Mac Infrastruscture Enterprise Desktop Grid Data Centers Clusters Public Cloud Persistence and Security Fabric Services Hardware Profiling Dynamic Resource Provisioning Foundation Services Billing & Reporting Resource Reservation Storage Licensing & Accounting Execution Services Independent Bag of Tasks Distributed Threads MapReduce other models… Middleware - Container Application Development & Management SDK: APIs & Tools Management: Tools, Interfaces and APIs PAL – Platform Abstraction Layer

61 update HTTP File Share …… Aneka Repository Management Console Node
Containers Node Manager Management Console update HTTP File Share ……

62

63

64

65 PAL – Platform Abstraction Layer
ECMA 334: .NET or Mono / Windows, Linux, & Mac Infrastruscture Enterprise Desktop Grid Data Centers Clusters Public Cloud Persistence & Security Fabric Services Hardware Profiling Resource Provisioning High-Availability Membership Foundation Services Billing & Reporting Resource Reservation Storage Licensing & Accounting Application Services Bag of Tasks Distributed Threads MapReduce Other models… PSM Middleware - Container Application Development & Management Software Development Kit: APIs & Tools Management: Tools, Interfaces and APIs PAL – Platform Abstraction Layer

66 Failover Index (master) Index (slave) Scheduling Storage Accounting
Reservation Provisioning Reporting Accounting Mandatory Index (master) Scheduling Failover Index (slave) Storage Storage Node Allocation Execution Worker Node Master Node

67 Task Scheduling Service Reporting, Billing, Accouting Master Node
Dynamic Resources Amazon EC2 Storage Service Provisioning Service Task Scheduling Service Reporting, Billing, Accouting Master Node Slave Node Task Execution Service Static Resources AWS

68 QoS Aneka Hybrid Cloud Reporting, Billing, Accouting Public Clouds
Provisioning Service Application Management & Scheduling Reporting, Billing, Accouting Desktops & Workstations Low priority workloads High Performance and QoS Public Clouds Cluster Resources Aneka Hybrid Cloud QoS

69 Resource Provisioning
Application Management & Scheduling Resource Reservation Master Node Desktop Machines Virtual Clusters Workstations Clusters

70 Resource Provisioning Application Management & Scheduling Master Node
Reporting, Billing, Accouting Amazon EC2 Instance Amazon EC2 Instances Slave Nodes EC2 Boundaries

71 Management & Scheduling Reporting, Billing, Accouting Public Clouds
Provisioning Service Application Management & Scheduling Reporting, Billing, Accouting Public Clouds Desktops & Workstations Clusters Virtual Cluster Resources Resource Reservation Master Node Local Infrastructure

72 IService.Start() IService.Pause() <Error> IService.Continue()
Unknown Initialized Running Paused Stopped Pausing Resuming Starting Stopping IService.Start() IService.Stop() IService.Continue() IService.Pause() <Error>

73 Chapter 6 – Concurrent Computing: Thread Programming

74 CPU: single die Core 1 Core 2 Core N Cache L1 Cache L2 To Memory

75 Process Shared Memory Execution Timeline Main Thread
Thread Local Storage Instructions (program counter) Thread Main Thread Process Shared Memory

76 a. Embarrassingly parallel
Process Units of work Result a. Embarrassingly parallel b. Inherently sequential

77 A 4 x 6 B 6 x 3 C 4 x 3 2 1 2,1

78 Process Units of work Result Unit 1 Unit 2 Unit 5 Unit 4 Unit 3

79 Wait()/Sleep()/Join()
Unstarted Running Stopped Wait SleepJoin Start() Interrupt() Abort() Suspended Suspend Requested Suspend() Resume () Abort Wait()/Sleep()/Join() Normal Completion Started Completed StagingIn Queued Staging Out Aborted Rejected Failed a. System.Threading.Thread life cycle. b. Aneka.Threading.AnekaThread life cycle.

80 Chapter 7 – High-Throughput Computing: Task Programming

81 Middleware Access Node(s) Tasks Remote Environment Local Node
Computing Nodes Local Node Tasks Remote Environment

82 2 8 1 10 9 7 6 4 3 11 5 MPI_COMM_WORLD Group A Group B Communicator A
11 10 MPI_COMM_WORLD Group A Group B Communicator A Communicator B

83 MPI Environment Initialization
MPI Include File MPI Environment Initialization MPI Environment Shutdown Prototypes declaration Do Work and Message Passing Do Work Serial Code Parallel Code Code Section Header Section

84 Workflow Application Modeling & Definition Tools
Specification Task + Data Scheduler Data Movement Data Provenance Dependency Manager Runtime Manager Language Interpreter Fault Management Workflow Enactment Engine Middleware Supporting Clusters, Grids, and Clouds Distributed Computing Resources Interaction with Distributed Resources Interaction with Information Services Users Workflow Design & Definition Workflow Execution & Control Resource Info Service Application Info Service Information Services Run Time Build Time

85 Aneka Cloud StorageService TaskScheduler AnekaApplication TaskManager
Local Node (… more tasks) AnekaApplication Task1 : ITask Task2 : ITask AnekaTask Index Node Membership Catalogue StorageService TaskScheduler TaskExecution Service Aneka Cloud TaskManager

86 Aneka Cloud File dependencies Result Files Input/Output Files
Client Machine File dependencies Index Node Membership Catalogue StorageService TaskScheduler TaskExecution Service Aneka Cloud Web Service TaskManager AnekaTask Remote FTP Result Files Input/Output Files

87 AnekaApplication<AnekaTask, TaskManager>
IJobManager TaskManager AnekaApplication<AnekaTask, TaskManager> PSM Model PSMJobInfo PSMCommandInfo PSMParameterInfo PSMFileInfo PSMContext AnekaTask ITask Aneka ApplicationModel

88 Aneka Cloud Offspring Environment IDistributionEngine AppInstance
IOffspringPlugin DefaultPlugin LogPlugin AutoPlugin Strategy Controller ITaskStrategy Workflow Logic 1 Workflow Logic N TaskStrategyBase Local Engine AnekaEngine DistributionEngineBase TaskManager Aneka Cloud

89 (Strategy Management) Monitoring Thread
StrategyController Control Thread (Strategy Management) Monitoring Thread (Feedback from Distribution Engine) Init Application Init Strategy Submit Tasks Wait Release Strategy Shutdown Application Yes No Complete or Stop? Task Feedback Task Failed? Invoke OnSuccess Invoke OnFailed New Task? Signal Iteration Complete? Submit

90 Chapter 8 – Data Intensive Computing: Map-Reduce Programming

91 Heterogeneous Formats Distributed Petabytes
Data/compute intensive problems Compute Current Problems Data Heterogeneous Formats Distributed Petabytes Data Complexity (volume/format/distribution) Homogeneous Formats Non-distributed Megabytes or Gigabytes Computational Complexity Model Solvers Simulations Simple search Statistical Models Decision support Knowledge generation

92 Information & Discovery
Compute Site Instrument Storage Facility Scientist Information & Discovery

93 Key Ring Request Coordinator Pluggable Storage Engine (BDB, MySQL)
Storage Peer Key Ring . Pluggable Storage Engine (BDB, MySQL) Failure and Membership Detection Request Coordinator

94 Tablet Server Master Server Chubby GFS Logs Tablet Files

95 MapReduce Infrastructure / Runtime
Task Reduce Aggregation by Key Input Output Input Data File Output Data File Distributed File System

96 Worker Master User Program (1) fork (2) assign map (2) assign reduce
Split 0 Split 1 Split 2 Split 3 Split 4 Output File 0 Output File 1 User Program (1) fork (2) assign map (2) assign reduce (3) read (4) local write (5) remote read (6) write input files map phase reduce phase output files intermediate files

97 R X M Stage Input File Output File Vertices (processes) Channels

98 MapReduceApplication
Aneka Cloud MapReduce Scheduling Service MapReduceApplication Reducer Mapper Input Data File Client Machine Execution Service Map Task Reduce DFS Implementation Split 0 Split 1 Split 2 Split 3 Split 4 intermediate files Output File 0 Output File 1

99 MapReduceSchedulerService
upload queue initialization queue dispatch queue failure queue resubmission queue reporting queue upload thread initialization thread dispatch thread report thread DFS Implementation Message translation and Event Triggering Aneka Middleware MapReduceApplication MapReduce Execution Services

100 MapReduceExecutorService
ExecutorManager job and task information command thread report thread completion thread DFS Implementation Message translation and Event Triggering Aneka Middleware MapReduce Scheduling Service MapReduceExecutor Map Task Reduce Task Sandbox command queue

101 ……. Header: 3 bytes magic number, 1 byte version Record Block:
Integer (4 bytes) Length of the block Length of the key Key byte array Value byte array

102

103

104 SaaS PaaS IaaS User (detailed view) Large Number 1 2 3 6 4 5
User Requests Wireless / Mobile 3G Network ECG Sensor ECG Sensor Module Embedded Bluetooth Enabled Data Communication & Processor Module Bluetooth connectivity IaaS PaaS Dynamically Scalable Runtime (QoS-based scaling of computing resources) Security Runtime SaaS ECG Data Analysis as a Service User (detailed view) Large Number 1 2 3 6 4 5

105 A B C D E F G H I Jeeva Portal Aneka Task Graph A : BLAST
B : Create Data Vector C : HH Classifier D : SS Classifier E : TT Classifier F : HS Classifier G : ST Classifier H : TH Classifier I : Predict Final Secondary Structure Initial Phase Classification Phase Final Phase Task Graph Aneka

106 Chapter 9 – Cloud Platforms in Industry

107 Amazon AWS Platform Communication Services
Amazon Simple Queue Service (SQS) Amazon Simple Notification Service (EBS) Amazon Simple Service (SES) Amazon Route 53 Amazon Virtual Private Cloud (VPC) Amazon Direct Connect Amazon Elastic Load Balancing Storage Services Amazon Simple Storage Service (S3) Amazon Elastic Block Store (EBS) Amazon ElastiCache Amazon SimpleDB Amazon Relational Database Service (RDS) Amazon CloudFront Amazon Import/Export Compute Services Amazon Elastic Compute Cloud (EC2) Amazon Elastic MapReduce AWS Elastic Beanstalk AWS Cloudformation Autoscaling Amazon AWS Platform Additional Services Amazon GovCloud Amazon Flexible Payment Service (FPS) Amazon DevPay Amazon Fullfillment Web Service (FWS) Amazon Mechanical Turk Alexa Web Information Service Amazon CloudWatch Alexa Top Sites

108 Google AppEngine Infrastructure
Development Runtime Local Machine Python SDK Google AppEngine Infrastructure Sandboxed Runtime Environment Data Store Url Fetch Image Manipulation Task Queue Cron Jobs Web App

109 Enterprise Level Infrastructure
Compute Enterprise Level Infrastructure Storage Networking Identity Marketplace Management portal Windows Azure Platform Local Development Environment Performance Development Tools Windows Azure Compute Windows Azure Storage Windows Azure Connect Content Delivery Network (CDN) AppFabric Caching Service Bus Integration Access Control Tools Windows Azure SDK SQL Azure DataMarket Applications Windows Azure AppFabric

110 TDS + Secure Socket Layer
Infrastructure Layer Platform Layer Tabular Data Stream (TDS) TDS + Secure Socket Layer SQL Server SQL Azure Fabric Management Services Services Layer Provisioning Billing and Metering Connection Routing ODBC ADO.NET PHP SQL Server Applications and Tools WCF Data Services HTTP / REST Client Layer Client Premises or Windows Azure Platform

111 Chapter 10 – Cloud Applications

112 Datasets CoXCS Tasks Aneka Elastic Sizing (EC2,…, ) Cloud-CoXCS
(Task orchestrator) Elastic Sizing (EC2,…, ) CoXCS Tasks Datasets

113 Aneka Portal (SaaS) Distribution Archiving Local Storage Public Cloud
Private Cloud Public Cloud Archiving Distribution Portal (SaaS)

114 Bulk Processing Engine
Force.com Platform Metadata Cache Bulk Processing Engine Multitenant-aware Query Optimiser Runtime Application Generator Full-text Search Engine Shared Database Data Tables Pivot Tables Metadata Tables (UDD) Virtual Application Components COMMON APPLICATION SCREENS TENANT-SPECIFIC SCREENS OBJECTS (TABLES) Indexes User-base 3: Application 3 User-base 1: Application 1 User-base 2: Application 2

115 eyeos apps extern extras system users workgroups Client-side scripts
EyeOS Server eyeos apps extern extras system users workgroups Client-side scripts

116 EC2 instances: video rendering
Scalability engine EC2 instances: video rendering EC2 instances: web front end Amazon SQS Amazon S3: Video storage

117 Aneka Networked LAN Aneka MayaRender Maya: Train Design Render Files Scenarios Render Tasks

118 Gamer Titan Gaming Portal Game Server Aneka Log Processing Task

119 Dropbox Folder Automatic synchronization

120 Chapter 11 – Advanced Topics in Cloud Computing

121 Revenue Power Consumption Green Cloud Solution User Applications
Reducing Power Cost and Carbon Emission Reducing Power Cost and Maximizing Revenue Revenue Power Consumption Cloud Data Center User Applications Green Cloud Solution

122 Cloud Exchange Bank Directory Auctioneer Enterprise IT Consumer
Users Brokers Compute Cloud Pool Node Cluster (VM Pool) Storage Cloud Cloud Exchange Directory Bank Auctioneer VMM Enterprise IT Consumer Enterprise Resource Server (Proxy) Request Capacity Negotiate Bid = Cloud Coordinator

123 Parallel Applications Others eg. Storage User Centric Resource Centric
Local Global Allocation Decision Profit-based System-based Hybrid Objective Market Model Application Model Participant Focus Bag of Tasks Workflow Parallel Applications Others eg. Storage User Centric Resource Centric System Centric Game Theory Proportional Share Bargaining Commodity Market Posted Price Auction Contract Net Single Sided Double Auction Combinatorial Auction Vickery Auction Dutch Auction English Auction First Price Sealed Auction Market-Oriented Models

124 Service Request Examiner and Admission Control
- Customer-driven Service Management - Computational Risk Management - Autonomic Resource Management Pricing Accounting Dispatcher VM Monitor Service Request Monitor Users Brokers Physical Machines Virtual Machines SLA Allocator Cloud Data Center

125 PaaS Vendor 1 PaaS Vendor 3 PaaS Vendor 2
PEX OSC IBM Blue Cloud multimedia education science engineering business …. bag of tasks parameter sweep workflow message passing map-reduce Aneka Task Model Thread Model PSM MapReduce Accounting Metering Pricing Billing QoS and SLA Res. Provisioning & Reservation Workflow Engine Broker Cloudbus Middleware Other Middleware Market Maker Meta-broker Web 2.0 Interfaces: Ajax, Web Services, Service Portals, REST Desktop Computers Applications Models Technology InterCloud Public Cloud Infrastructure Amazon EC2 / S3 Microsoft Azure Google AppEngine Core Technologies Infrastructure PaaS Vendor 1 PaaS Vendor 3 PaaS Vendor 2 physics biology medicine optimization modeling CAD teaching risk analysis banking finance 3D rendering imaging Scalable Execution Application Manager Cluster Datacenter social networking blogging photos

126 Cloud Federation Stack
Motivations Advantages Opportunities Obligations Federation Model Cloud Service, Provider, Agreements Market and Pricing Models Service Level Agreements Protocol, Interfaces, and Standards Programmatic Interoperation Federation Platforms (RESERVOIR, InterCloud) Infrastructural Level Logical and Operational Level Conceptual Level

127 HTTP Provider’s Infrastructure Resource Resources Management Framework
OCCI Interface Provider’s APIs Resources Resource Management Framework OCCI Clients Other Clients HTTP Provider’s Infrastructure any channel, any protocol Most interoperable Least interoperable Communication is internal and proprietary Communication is related to IMF interface requests

128 Container Information Services (Future) Data Services Storage Services Cloud Data Management Storage Cloud Data Management Interface (CDMI) Data Storage Cloud Draw Resources on Demand Multiple, Proprietary Interfaces iSCSI, FC, FCoE, LUNs, Targets POSIX (NFS, CIFS, WebDAV) CDMI Export Data to the Cloud Block Storage Client File System Client Object Storage Client XAM Client Database / Table Client Clients Acting in the Role of Using Data Storage Interface Clients can be in the Cloud or Enterprise and provide additional services (data, computing, etc…) Data Storage Management Client Clients Acting in the Role of Managing Data / Storage Management of the Cloud Storage can be standalone or part of the overall management of your Cloud Computing

129 Computational Resource
VEE Host Computational Resource Virtualizer RESERVOIR Site 1 RESERVOIR Site 2 Service Application 1 Service Application 3 Service Application 2

130 QoS Service Provider Service Manager VEE Manager (VEEM)
VEE Hosts (e.g. hypervisor, VJSC Host, …) QoS VEE Hosts Site A Site B Site C SLA Service Provider manifest

131 Cloud Exchange Broker (e.g. Eucalyptus) (e.g. EC2) Cloud Coordinator
VM Manager (e.g. Open Nebula) (e.g. EC2) (e.g. Eucalyptus) User Interface Broker Resource Negotiation Publish / Query Offers / Request

132 Coral CDN Windows Azure Nirvanix Amazon S3 XYZ Web Service Access
Web Portal Connector Coral Azure S3 MetaCDN Manager Database Allocator Load Redirector SLA Users Applications

133 XYZ Nirvanix Coral CDN Windows Azure Amazon S3 Web Service Access
Web Portal Connector Coral Azure S3 MetaCDN Manager Database Allocator Load Redirector Users Applications SLA

134 SpotCloud Buyers Capacity Provider (Storage)
Comparative Listing of Offerings Buyers SpotCloud Capacity Provider (Compute) Connector Capacity Provider (Storage) VM Package Repository Transactions Bookkeeping and Billing Engine QoS and SLA Monitoring User Profiles Database SLA Deployment

135 Other Stuff

136 Index / Map Management Nodes Management Infrastructure Management Failover master Storage management Local caching File lookup forwarding Slave

137 Client N Client 1 Client 2 DFS Raw File System
File Map (GUID  Location) Bucket List Failover master Slave – Bucket 1 Slave – Bucket 2 Slave – Bucket N DFS Client 1 Client 2 Client N Hierarchical File System (Files and Directories) Flat File System (Files) Raw File System (Byte stream)

138 DFS Client DFS Infrastructure DFS Service Aneka Container
Aneka DFS Wrapper Service : IService DFS Service DFS API DFS Client DFS Infrastructure Application-based Storage Interface


Download ppt "Mastering Cloud Computing"

Similar presentations


Ads by Google