C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing,

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. How to test the readiness of cloud services? Chaowei Yang, Min Sun, Jizhe Xia, Jing Li, Kai Liu, Qunying Huang, and Zhipeng Gui 1

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. Learning Objectives Developing the fundamental technology for testing the readiness cloud services in support of geosciences 1.Geoscience computing characteristics 2.Cloud test environment 3.Concurrent intensity test with Clearinghouse 4.Data and computing intensity test with Climate@Home 5.Comprehensive test using dust storm forecasting 2

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. Learning Materials Online Videos: o Chapter_12_Video_1.mp4 Chapter_12_Video_1.mp4 o Chapter_12-Video_2-dust-storm-test.mp4 Chapter_12-Video_2-dust-storm-test.mp4 o Chapter_12-Video_3-modelEtest.mp4 Scripts, Files and others: o Chapter_12_Section12.3.zip Chapter_12_Section12.3.zip o modelEtest_materials.zip modelEtest_materials.zip o scripts.zip scripts.zip 3

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. Geoscience computing characteristics 1. Concurrent intensity: for popular geospatial applications, such as data.gov or clearinghouse. 2. Data intensity: geoscience observation and research will produce significant amount of data. 3. Computing intensity: geoscience research needs significant computer power to conduct simulations. 4. Spatiotemporal intensity: geosciences involves the understanding of past, present, and future of the Earth system. 5.These characteristics are represented by the three applications of GEOSS clearinghouse, climate@home, and dust storm forecasting. 4

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. Cloud test environment Cloud service runs at computing centers and are accessed through the Internet The geographic region and computing facility will matter in the test. This test will look at the performance of different applications running on different cloud services The cloud services are located at different regions across the U.S. and connected through the national network backbone. Different service instance configure will produce different performance results. 6

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. Cloud test environment 7 The test will be conducted from each cloud and a local server to other computing service. Comparison will be made cross the cloud service and the local server. Computing ServiceGeographic LocationCloud ServicesHost Organization EC2Reston, VAIaaSAmazon AzureChicago, ILPaaSMicrosoft NebulaAmes, CAIaaSNASA Local ClusterFairfax, VATraditional ClusterGMU

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. Cloud test environment 8 Service namesCPU CoreCPU Speed (GHz)Memory (GB) EC2 112.30.5 EC2 222.37.5 EC2 342.37.5 EC2 482.87.5 Azure 111.61.75 Azure 221.63.5 Azure 341.67 Azure 481.614 Nebula 112.92 Nebula 222.94 Nebula 342.98 Nebula 482.916 Local Server82.423

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. Concurrent intensity test with Clearinghouse 1.Clearinghouse computing requirement 2.Test Design 3.Test Workflow 4.Test Results Analyses 10

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. Concurrent intensity test with Clearinghouse: Computing Requirements CLH is a FGDC, GEO, and NASA collaborative project to support the sharing of Earth Observation (EO) data in the global context. Global user access to the clearinghouse concurrently, and the access has different spatiotemporal patterns for different regions and different metadata. For example, clearinghouse receives more accesses at local day time than evening hours, and the access can be massive when a significant event (e.g., earthquake) happens. 11

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. Cloud test environment 12 The test will be conducted from each cloud and a local server to other computing service. Comparison will be made cross the cloud service and the local server.

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. Concurrent intensity test with Clearinghouse: Test Design 1.To compare the capabilities of different cloud services in supporting concurrent requests, two experiments designed: 2.Matrix of concurrent search requests test: 1)With CLH running on the four computing services: three cloud services (AWS, Azure and Nebula) and a local server (figure 12.1), 2)Concurrent requests are issued from one service to all other services. Apache JMeter is used for testing the performance of concurrent requests between the services. 3)The test result is a 4*4 matrix which can represent the concurrent performance of each computing against other computing services. 3.Load balance and scalability tests: 1)The load balancing experiment compares concurrent performance when different numbers of instances are used behind the Load Balancer; 2)The auto-scaling experiment tracks the change of performance when cloud services scale new instances dynamically with the increasing number of concurrent requests. 4.http://jmeter.apache.org/ (please check relevant video and cloud images)http://jmeter.apache.org/ 13

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. 14 Matrix of concurrent search requests test: Workflow Video: Chapter_12_ Chapter_12_ Video_1.mp4

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. Steps of Concurrent Test: 1.Installing CLH on different cloud services by launching Instance using virtual image. 2.Setting up CSW GetRecords request. CSW GetRecords request can be issued to the CLH (refer to Chapter 8). 3.Setting up JMeter test plan. Testers can set a test plan to simulate the concurrent requests to CLH. The concurrent number can be increased, such as 1, 50, 100, 150, 200, 250, 300, 350, 400, 450, 500, and 600 by setting up thread group and controller. 4.Running Jmeter through Graphic User Interface or Command line. 5.Analyze JMeter Results (2.3.4) 15

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. 16 Load balancing and auto-scaling tests : Workflow

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. Steps of Load balancing and auto-scaling tests: 1.These two tests can be used to test the improvements of the performance when using load balancing or auto-scaling for CLH. Testers can step-wise to use 10 to 400 numbers of concurrent requests to do the tests. 2.Steps 1, 3, 4, 5, 6 can be referred to as Matrix test. 3.Step 2 sets up the load balancing or auto-scaling as detailed in Chapter 8. CPU utilization is recommended to be used as resource trigger. 17

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. Concurrent intensity test with Clearinghouse: Test Results Analyses The JMeter records the results information in the result file (set up in step 3). Testers can compare the response times using the results file. Figure 12.6 shows the average response time of GEOSS using Amazon EC2 to scale different levels of concurrent requests. Three groups of experiments are conducted: no scale up and scale down with just one instance, automatically scale up with up to 2 instances, and automatically scale up to 5 instances (response/latency time longer than 4s is trigger). Using 2 and 5 instances to scale the concurrent access can decrease the average response time. After 50 to 60 of concurrent requests, average response time is longer than 4 seconds, group “2 instances” and “5 instances” started a new instance. The amazon AWS log file shows group “5 instances” start the third instance 10 minutes later after the second instance starts, the fourth instance 8 minutes later after the third instance starts and the fifth instance 9 minutes later after the fourth instance starts. Hence there are some drops when the concurrent number is 170, 210 and 270 for the group “5 instances”. 18

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. Data and computing intensity test with Climate@Home 1.Computing Requirements 2.Test Design 3.Test workflow 4.Test Results Analyses 20

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. Data and computing intensity test with Climate@Home: Computing Requirements The Climate@Home project needs multiple virtual machines to run many model runs. Climate model: ModelE Minimum computing requirements: A Linux machine equipped with one CPU core of 1.5+ Ghz speed and 1G RAM The initial capacity of data storage: >10G The cloud instance: sufficient hard disk space for the basic installation, which includes the model, input data and model outputs as well. 21

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. Data and computing intensity test with Climate@Home: Test Design Experiment how different cloud services support climate modeling. To be specific in three tasks: Identifying a cost-effective hardware configuration of the instance. Evaluating the stability and reliability of virtual machine instances by recording the total failure times and average time spent on executing multiple runs. Comparing the performance of cloud-based virtual machines and physical machines. 22

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. Data and computing intensity test with Climate@Home: Test workflow 23 Video: Chapter_12-Video_3-modelEtest.mp4Chapter_12-Video_3-modelEtest.mp4

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. Data and computing intensity test with Climate@Home: Test Results Analyses The configurations of virtual machines include 1-core, 2-core, 4-core and 8-core instances with comparable main memory. After executing the shared script on every instance, the time cost of each model-run is recorded in a log file tagged with the end time of each simulation period. When analyzing the log file, we need to clearly label the association of the log files to their machine configurations which are not captured in the file name. 24 An example of the log file (“logModel19491208.txt”, 1-core instance from Nebula).

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. Comprehensive test using dust storm forecasting 1.Computing Requirements 2.Test Design 3.Test Workflow 4.Test Results Analyses 26

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. Comprehensive test using dust storm forecasting: Computing Requirements Dust storm forecasting involves big geoscience data in model input and output, intensive data processing and computing (Chapter 10). The periodic phenomena simulation of dust storm requires multiple computing resources to run the simulation in parallel for running the same set of intensive computation for many times. Advanced computing methodologies and techniques are normally used to address the computing challenges (Yang et al. 2011a). Previous study of dust storm simulation (Huang et al. 2012) found that CPU and networking speed have significant impact on the simulation performance because of massive data exchange and synchronization (as described in the Chapter 10). Therefore, the cloud virtual machines for dust storm forecasting should have fast CPU speed and be connected with high bandwidth network. 27

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. Comprehensive test using dust storm forecasting: Test Design 28 Examining the performance of local HPC cluster and the two cloud services: Different numbers of computing nodes from different services for supporting the same dust storm simulations. The same amount of computing resources from different services to support different simulation tasks. The impact of hyperthreading. 1)a single physical processor appear to be multiple logical processors, and significantly improves performance on computing workloads (Koufaty and Marr 2003). 2)Improve processor resource utilization (Bulpin and Pratt 2004). 3)Experiment is designed to test the performance of one EC2 instance before and after shutting down the hyperthreading capability.

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. Comprehensive test using dust storm forecasting: Test Workflow 29 Video: Chapter_12-Video_2-dust-storm-test.mp4Chapter_12-Video_2-dust-storm-test.mp4

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. Comprehensive test using dust storm forecasting: Test Results Analyses 1.Figure 12.8 shows a typical test result with different process numbers. 1)The first line “****** Test type EC2.2VMs**********” indicates that it is the test result by EC2 cloud service with two instances. 2)The third line indicates the model running directory. 3)The fourth line means the starting time to run the model with 128 process numbers. 4)The fifth line tells the model completing time with 128 processes. 5)The following lines indicate the starting and ending time with other process numbers. 6)The last line with “Finish” shows the test has been finished successfully. 2.The execution time of a 3-hour and 2.3 * 3.5 degree model forecasting by EC2 cloud service using two instances and different process numbers can be calculated and analyzed (Figure 12.9 a). 3.Combining different test output files, more complex analysis can be performed (Figure 12.9 b). 30

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. Discussion Questions 1.What’s the different between using and without using preconfigured images for the experiments? 2.Do you know any other geoscience application that fit into the intensities of data, computing, concurrent, and spatiotemporal? 3.How to analyze the testing results? 4.How to compare the performance of different cloud services? 31

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing, edited by C.Yang, Q. Huang, Z. Li, C. Xu, K. Liu, CRC Press: pp. 201-221. References 1.Huang, Q., Yang, C., Nebert, D., Liu, K., & Wu, H. (2010, November). Cloud computing for geosciences: deployment of GEOSS clearinghouse on Amazon's EC2. In Proceedings of the ACM SIGSPATIAL International Workshop on High Performance and Distributed Geographic Information Systems (pp. 35-38). ACM. 2.Huang, Q., Yang, C., Liu, K., Xia, J., Xu, C., Li, J.,... & Li, Z. (2013). Evaluating Open-source Cloud Computing Solutions for Geosciences. Computers & Geosciences. 3.Sun, M., Li, J., Yang, C., Schmidt, G. A., Bambacus, M., Cahalan, R.,... & Li, Z. (2012). A Web-Based Geovisual Analytical System for Climate Studies. Future Internet, 4(4), 1069-1085. 32

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing,

Similar presentations

Presentation on theme: "C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing,"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing,

Similar presentations

Presentation on theme: "C. Yang, M. Sun, J. Xia, J. Li, K. Liu, Q. Huang and Z. Gui, 2013. Chapter 12 How to test the readiness of cloud services, In Spatial Cloud Computing,"— Presentation transcript:

Similar presentations

About project

Feedback