Presentation is loading. Please wait.

Presentation is loading. Please wait.

Cloud Computing Term Project Cloud Monitoring and Scaling Merged CD Group Pankaj Kumar Qinglan Zhang Sagar Davasam Sowjanya Puligadda Wei Liu.

Similar presentations


Presentation on theme: "Cloud Computing Term Project Cloud Monitoring and Scaling Merged CD Group Pankaj Kumar Qinglan Zhang Sagar Davasam Sowjanya Puligadda Wei Liu."— Presentation transcript:

1 Cloud Computing Term Project Cloud Monitoring and Scaling Merged CD Group Pankaj Kumar Qinglan Zhang Sagar Davasam Sowjanya Puligadda Wei Liu

2 Overview What we have done – Openstack installation – Create and manage VM instances – Monitoring tools: Ganglia & Zenoss Future work – Compare monitoring tools and present union of data metrics from Ganglia and Zenoss – Performance analysis and VM scaling – Controlling and computing availability

3 Architecture Diagram of the setup Cloud for Monitoring, Scaling and performance measurement Cloud for Monitoring, Scaling and performance measurement UBUNTU OPENSTACK UBUNTU cloud OS VM-1 UBUNTU cloud OS VM-1 Host 1 – Host 2 – Host 3 – Similar VM-2 and VM- 3 Ganglia and Zenoss Tools UBUNTU OPENSTACK UBUNTU cloud OS VM-1 UBUNTU cloud OS VM-1 Similar VM-2 and VM-3 Ganglia and Zenoss Tools UBUNTU OPENSTACK UBUNTU cloud OS VM-1 UBUNTU cloud OS VM-1 Similar VM-2 and VM-3 Ganglia and Zenoss Tools

4 Openstack installation Use devstack multi-node installation – Network configuration & NTP Add User & Set Up SSH Configure Controller & Slaves

5 Issues and best practices Devstack script does not work well. The problems we have solved are as follow: – Mysql conflicts – Invalid download link – Permission denial Su; chown –R stack:stack devstack/; – No reboot post devstack installation – VM installation using openstack dashboard

6 Create and manage VM instance Verify the Identity Service installation Verify the Image Service installation Enable Networking Generate a keypair and choose a flavor Launch the instance

7 Scaling Analyze the performance result recorded by Ganglia and Zenoss. Set threshold to scale out or shrink back Try dynamic strategies to scale automatically

8 Availability Monitor the availability Test availability in different situations: – Shut down server – Shut down single vm – Shut down multiple vms – Turn on vms

9 Ganglia Vs Zenoss

10 Zenoss Performance Metrics Aggregate Reports – CPU usage – Free memory – Free swap – Network Input and Output Availability - Percentage of time that a device or component is considered available CPU Utilization – Shows monitored interfaces, devices, load average and % utility

11 Zenoss Performance Metrics Filesystem Utilization - Shows total bytes, used bytes, free bytes, and percentage utilization for each device. Interface Utilization - Shows the traffic through all network interfaces monitored by the system. – Interface’s rated bandwidth – Average input traffic – Average output traffic – Total average traffic across interface – % utilization of bandwidth

12 Zenoss Performance Metrics Memory Utilization - Provides system-wide information about the memory usage for devices in the system. – Total memory – Available memory – Cache memory – Buffered memory – Percentage of memory utilized

13 Zenoss Extended Monitoring Apache Web Server DNS File Transfer Protocol Internet Relay Chat Jabber Instant Messaging LDAP Response Time MySQL Database Network News Transport Protocol Network Time Protocol ONC – System Remote Procedure Call Webpage Response Time (HTTP) VMware esxtop Xen Virtual Hosts

14 Ganglia Vs Zenoss Zenoss can automatically discover hosts and start monitoring them automatically. Ganglia requires an agent to run on every host to gather information Ganglia does not monitor events, Zenoss reports about event count, event queue length Zenoss can monitor services but Ganglia can not Status of a running task can be received by Zenoss tool but Ganglia can not be used for

15 Zenoss Monitoring System Architecture

16 Zenoss features Zenoss offers visibility over the entire IT stack, from network devices to applications. Automatic discovery, Inventory via CMDB, Availability monitoring, Easy-to-read performance graphs, Sophisticated alerting, An easy-to-use web portal

17 Why Both Ganglia and Zenoss More data and parameters received from both tools will make – Decision making easier – Decision making robust and efficient – Decision and action will be more reliable

18 All VMs All VMs UBUNTU cloud OS VM-1 UBUNTU cloud OS VM-1 Ganglia and Zenoss Tools Union of data read by Ganglia and Zenoss Tool read_availability < demanded_avail ability VM provisioning Module 0 Scale out to meet availability – add VMs Scaling algorithm Decision making Module Yes Project Architecture Diagram

19 How to Scale Scale up(horizontal) or Scale out (vertically )? – scale up: add resources to a single node – scale out: add more nodes to a system We decide – Scale out – Add VMs on nodes in system So when we scale our system? – System availability is under our expectation. – Scaling !!!

20 Our goal Add minimum VMs to get “three nines” availability (Assuming adding VMs will enhance the availability) Availability %Downtime per year Downtime per month* Downtime per week 90% ("one nine")36.5 days72 hours16.8 hours 95%18.25 days36 hours8.4 hours 97%10.96 days21.6 hours5.04 hours 98%7.30 days14.4 hours3.36 hours 99% ("two nines")3.65 days7.20 hours1.68 hours 99.5%1.83 days3.60 hours50.4 minutes 99.8%17.52 hours86.23 minutes20.16 minutes 99.9% ("three nines") 8.76 hours43.8 minutes10.1 minutes 99.95%4.38 hours21.56 minutes5.04 minutes 99.99% ("four nines") minutes4.32 minutes1.01 minutes % ("five nines") 5.26 minutes25.9 seconds6.05 seconds % ("six nines") 31.5 seconds2.59 seconds0.605 seconds % ("seven nines") 3.15 seconds0.259 seconds seconds

21 Our Algorithm Initialize several VMs, get the monitoring data and availability. – CPU, Network, I/O, Memory usage percentage – Availability Run our algorithm to make decision – 1. Calculate the initialized weight of each factor – 2. Calculate the number of VM we needed (W c, W n, W i, W m ) = f (U c, U n, U i, U m, N i,N i+1, A i,A i+1 ) W c, W n, W i, W m are weights of CPU, Network, I/O and Memory U c, U n, U i, U m are usage of CPU, Network, I/O and Memory N i,N i+1, A i,A i+1 are number of VMs and Availability in training i and i+1. – 3. Lance algorithm every a period of time to make decision and revise the algorithm itself based on data collected by Ganglia and Zenoss Machine learning: semi-supervised approach

22 Problem Now How can we get the function f? Two possible ways: – 1. Generate the function theoretically, and use the collected data to verify and revised the function. Hard to accomplish for current stage. – 2. Collected the data first, and guess the function and use some tool (MATLAB) to generate the function. After training the data, improve the accuracy and revise the algorithm. This is what we will use. Example next

23 Guess and Example Number of VMs CPU usageNetwork usage I/O usageMemory usage Availability Training 1180%50%90%60%90% Training 2275%43%88%52%92% Training 3369%38%70%48%96% Training 4460%35%65%43%97.8% Expected ? // // 99.9% Data from ZenossData from Ganglia Goal

24 Example and Guess

25 OpenStack installation Controller Node

26 OpenStack installation on Compute Nodes

27

28 Instance on Compute node (CirrOS)

29 Instance on Compute node (Ubuntu)

30 Zenoss Installation on Ubuntu VM

31 Zenoss Login

32 Ganglia Installation


Download ppt "Cloud Computing Term Project Cloud Monitoring and Scaling Merged CD Group Pankaj Kumar Qinglan Zhang Sagar Davasam Sowjanya Puligadda Wei Liu."

Similar presentations


Ads by Google