1. Cloud Benchmarking, Tools, and Challenges
Advisor: Dr. Yong Chen
Student: Elham Hojati
This research is supported by the Cloud and Autonomic Computing site at Texas Tech University via the StackVelocity membership contribution and the Aerospace Corporation technical partnerships.

2. Outline INTRODUCTION TOOLS AND FRAMEWORKS CHALLENGES OUR PROJECT PLAN
Benchmarking
Cloud benchmarking
TOOLS AND FRAMEWORKS
CHALLENGES
OUR PROJECT PLAN
QUESTION AND FEEDBACKS

3. Team and Leverage
This research is supported by the Cloud and Autonomic Computing site at Texas Tech University via the StackVelocity membership contribution and the Aerospace Corporation technical partnerships.

4. PART 1: Introduction 4

5. BENCHMARKING (Generic Concept)
What is a benchmark?
How does a benchmark do its tasks?
What are benchmark`s requirements? 5

6. Project Overview
Benchmarking of a system is a process of assessing its performance and other characteristics, to be able to compare them with other systems.
Benchmark tools try to answer the question “which is the best system in a given domain?”
Cloud is one of the systems need benchmarking process.

7. Project Overview
On one hand, considering the number of cloud computing providers and the different services each provider offers, cloud users need benchmark information to select the best service and provider for their needs.
On the other hand, cloud providers and cloud architects need benchmarking results to create optimized architectures.
The problem is that cloud is a very complex and dynamic environment. Therefore there are some important differences between benchmarking in dynamic cloud environment with traditional benchmarking methods in static systems.

8. Project Overview
In this research we have studied and compared several tools and frameworks of cloud benchmarking.
We have performed a survey research about benchmarking in cloud and various challenges about creating and deploying cloud benchmarking;
The most important problem for cloud users and providers are:
data isolation,
security
unreliable performance
inference challenging and performance isolation.
It is possible to improve performance isolation by considering fairness property especially in current implementations, mainly in I/O intensive scenarios.

9. WHAT ARE BENCHMARK`S REQUIREMENTS?
1. General Requirements
Strong Target Audience
Relevant
Economical
Simple
Implementation Requirements
Fair
Repeatable
Configurable
Workload Requirements
Scalable
Metric 9

10. PART 2 Tools and Frameworks10

11. 11 CloudCmp CloudStone HiBench YCSB CloudSuite Perfkit
price-performance benchmarking
Target Application
Legacy application
Web 2.0
MapReduce
Database
Media streaming
Servers functionality-performance
Server price performance functionality
Test
environment
Multiple instance types
Amazon EC2
instances
Hadoop cluster
Data serving system
Characterize scale-out
workloads
Public and private cloud
Amazon, Google, Microsoft, Rackspace, IBM, HP and Linode
Service
IaaS
PaaS
Target
Estimate performance
and costs of running a
legacy application on a cloud
Social web applications
Hadoop
(MapReduce)
programs including
real-world
applications
Performance
comparisons of the new
generation
of cloud data serving
systems
Server 11

12. PART 3 Challenges 12

13. The Art of Building and Running a Good Cloud Benchmark
Main steps for building and running a good benchmark in the Cloud:
Meaningful Metric
Workload Design
Workload Implementation
Creating Trust 13

14. Challenges Step 1: Meaningful Metric Step 2: Workload Design
Challenge 1: defining meaningful metric with considering elasticity and fairness
Step 2: Workload Design
Challenge 2: Resources allocation/ Scalability and performance isolation
Step 3: Workload Implementation
Challenge 3: Workload Generation.
Challenge 4: Fairness
Step 4: Creating Trust
Challenge 5: Location.
Challenge 6: Ownership.
Challenge 7: security 14

15. Fairness Property A system is fair if it has the below conditions:
1. Customers working within their assigned quota should not suffer from customers exceeding their quotas.
2. Customers exceeding their quotas should suffer performance degradation if they have a negative impact on others.
3. Customers with higher quotas should be provided with better performance than customers with lower quotas.” 15

16. System Types based on performance isolation [7]16

17. performance isolation
Having different customers and hosting their application, and isolating their tasks from each other as much as possible and reducing the effect of sharing resources on the performance of each user is one of the major goals of cloud service providers.
It is important to have a metric for measuring performance isolation in cloud systems.
Architects and developers need to have a mechanism to evaluate performance isolation between users and the effectiveness of their methodology to guarantee the quality of the artefact and avoid distrust.
Performance isolation is providing a constant Quality of Service for one user which is independent from the load used by the other users.
Limitations of this method
The current method and metrics does not consider all three aspects of fairness property. As a feature work we are going to improve this method and develop an advanced isolation approach to consider all the aspects of fairness property.

18. using game theory
Resource allocation using game theory by considering fairness property.
Limitations of this method
Do not consider all the 3 part definition of fairness property.
the problem modeling is static here. 
In this project, our goal is improving this method for considering complex dynamic environment, and using that in cloud benchmarking process. It should consider elasticity. And also, our new method should consider all three aspects of fairness property, which are not covered by the existing approaches.

19. Conclusion / future Project Plan
We have performed a survey research about benchmarking in cloud and various challenges about creating and deploying cloud benchmarking.
Existing cloud benchmarks and metrics mostly focus on traditional metrics like throughput in a virtualized environment, or just consider single aspects like databases, or just focus on some cloud features. For example there are some methods for benchmarking resource allocation which consider fairness or elasticity. But there is not a method which consider both of them. Our goal is considering various aspects such as performance isolation along with fairness and elasticity features for dynamic shared resources environment.

20. Experimental plan
Study and compare several tools and frameworks of Cloud benchmarking.
Evaluate and understand challenges about creating and deploying cloud benchmarking
Develop new methods for Cloud benchmarking for better performance isolation
Develop new methods to improve benchmarking resource allocation process and develop an advanced isolation approach to consider all the aspects of fairness property.

21. References
[1] Alexandru Iosup, Radu Prodan, Dick Epema ,"IaaS Cloud Benchmarking: Approaches, Challenges, and Experience", Cloud Computing for Data-Intensive Applications 2014, pp
[2] Enno Folkerts, Alexander Alexandrov, Kai Sachs, Alexandru Iosup, Volker Mark, Cafer Tosun, "Benchmarking in the Cloud: What it Should, Can, and Cannot Be", Lecture Notes in Computer Science Volume 7755, 2013, pp
[3] C.Vazquez, R. Krishnan, and E. John, "Cloud Computing Benchmarking: A Survey", 2014.
[4] Edward Wustenhoff, CTO, Burstorm, "Cloud Computing Benchmark RB-A, the 1st step to continuous price-performance benchmarking of the cloud", Rice Burstorm Price Performance Benchmark Report (RB-A) June 2015.
[5] Bin Sun, Brian Hall, Hu Wang, Da Wei Zhang, Kai Ding, "Benchmarking Private Cloud Performance with User- Centric Metrics", 2014 IEEE International Conference on Cloud Engineering.
[6] GuiyiWei · Athanasios V. Vasilakos · Yao Zheng ·, Naixue Xiong, "A game-theoretic method of fair resource allocation for cloud computing services", Springer Science+Business Media, LLC 2009.
[7] RouvenKrebs, ChristofMomm, SamuelKounev,"Metrics and techniques for quantifying performance isolation in cloud environments", Elsevier 2013. 21