Performance Assessment for Web-based Systems Min Song, Ph.D. IS 465

2 Topology of a Web Site Web Server Web Server Security Server Security Server Database Server Database Server Application Server Application Server ISPISP ISPISP Browser Internet FirewallFirewall FirewallFirewall Browser

3 General Considerations Application conception What’s to be done? What’s the whole purpose of this task? Design How is it going to be done? Flexibility, Dynamic Development What tools to use? What technologies to implement? TESTING What? it’s working well, isn’t it? Deployment How soon?, Pronto!, Is fast isn’t it?

4 The NEED to Test (Are we wondering…) Is the site content meaningful? Is this application easy to use? How about browser compatible? Are we using the right tools? How reliable is our technology? Do the Servers have enough power? How many visitors are we expecting? Are the machines fast enough? How much activity can the site handle?

5 What and How to test? (Fundamental Considerations) Test Planning Browser Compatibility Design compliance Usability Efficiency Scalability Best suited OS platform Network bandwidth Hardware capacity Site availability Monitoring Security

6 Testing a Web Site Understanding what types of testing are needed for each application, is essential for several reasons: Ensures you that the testing effort, will support the business and technical requirements. The activities required for each type of test and its associated phases, are included within the master test plan (and schedule). H elps identifying and plan for the environments and resources that are needed to prepare and execute each test type. Assures the achievement of any Testing Goals.

7 Testing a Web Site (continued) There are mainly two essential types of testing that should be accomplished. Functionality Performance

8 Functionality Testing What is it? Why is so important? What is: – Unit Test – Integrated Test – Functional Test – System Test – Regression Test

9 What is Functionality Testing? Is a set of skilled approaches and procedures which systematically verify that the application’s components and process modules, are available and working as expected. Compliance with Business rules, Technical requirements, System Documentation, and User manuals is of course… Mandatory!

10 Why is Functionality Testing important? Missing Image

11 Functionality Testing -Unit Test Unit Test Unit tests perform basic tests at component level and test a specific business process, application, and/or system configuration. Unit tests ensure that each unique path of a business process performs accurately to the documented specifications and contains clearly defined inputs and expected results

12 Functionality Testing -Integration Test Integration Test Integration test is testing a specific feature together with other newly developed features Testing the interface of two components explores how components interact with each other Integration testing inspects the variables passed not only between two components, but also the global variables. Integration testing is specifically aimed at exposing the problems that arise from the combination of components Previously captured unit test scripts can be combined to create a variety of integration test cases This test phase assumes the components and the objects they manipulate have all passed their local unit tests

13 Functionality Testing - Functional Test Functional Test Functional testing is centered around the following items: 1.Valid Input - identified classes of valid input must be accepted. 2.Invalid Input - identified classes of invalid input must be rejected 3.Functions - identified functions must be exercised 4.Output - identified classes of application outputs must be exercised. 5.Systems/Procedures - interfacing systems or procedures must be invoked.

14 Functionality Testing – System & Regression Test Regression Test Regression tests are a series of tests to ensure that no adverse changes are introduced to the application during maintenance changes, upgrades, or other modifications. Regression testing is composed of a suite of components that are rerun when any one- application component has been modified This provides confidence that the system will function with newly implemented requirements. Regression testing is often an iterative process during a testing cycle System Test System testing studies issues and behaviors that can only be exposed by testing the entire integrated system Verifying these characteristics is very important for products intended for users on diverse OS and Hardware platforms. Identification of faulty Hardware and Software components, is feasible this way.

15 Performance Testing What is it? Why is so important? What are the types of Performance Tests? What is the difference between Load, Performance, and Stress Testing ?

16 What is Performance Testing ? Performance Tests determine the runtime “behavior” of the application and its supporting infrastructure, under certain conditions. Performance Testing is used to measure several system characteristics, such as processing speed, response time, resource consumption, throughput, and efficiency.

17 Why is Performance Testing important? The week of Feb 6, 2000: Hackers delivered over 1-Billion transactions concurrently to each one of these sites. Yahoo eBay Buy.com Amazon eTrade How did you think they performed ?

18 Performance Testing - Load Test Load Test Load Tests determine the applications behavior under load, up to and including its limits (not just at its limits). Load tests specifically refer to the load size (number of concurrent users) and related values.

19 Performance Testing - Stress Test Stress Test Stress testing determines the applications ability to handle large amounts of data. Stress Testing can be much more successful with a full load applied to the server. Running addition applications on the server in conjunction with the client side tests is an additional form of stress testing.

20 Performance, Load, and Stress Testing ? They all have different meanings, although they seem to be used almost interchangeably sometimes. To better understand these differences; let’s apply these terms while testing a car: Performance Testing - a car does 0-60mph in a certain number of seconds under normal conditions Load Testing – The car can pull a trailer weighing x - tons before it can no longer move Stress Testing - The car continues to work fine driving 60 mph for two hours, in 120 degree temperatures, with only 1 gallon of coolant in the radiator

21 Why Automate Testing? Is it necessary to repeat the sequence of actions many times? Do you need to run the same tests on multiple hardware configurations? Need to test with several concurrent users? Exercise multiple options, components, and configurations Save time and money

22 What are we testing for? Where should I start? How do we proceed? What tool do I select?

23 Simulating the Real World The MAIN purpose of running automated tests is to emulate the REAL world as close as possible while saving time and effort.

24 Tips for selecting an Automated Testing Tool Ease to use (and learn) Standards compliance Robust and Reliable (accurate) Flexible Affordable Maintenance and Support Important: Is it the right tool for the job?

25 What are we Testing for? Everyone involved should be clear on what exactly we are testing for and the goals for the outcome of the test Testing For: How many Total Concurrent Users ? At what point does the performance degrade ? Which server will max out first ? (Web, DB, App, etc.) How about Data integrity ? What is the Total Throughput ? (Bandwidth, pipeline) Goals: Round time =< 15 sec Total Concurrent Users => 300 / Min

26 Automation Web Testing Methodology 5-Phases 5.Analyze 4.Execute 3.Modify 2.Record 1.Define Modeling Paradigm

27 The Application 1.Describe the application to test using a application model 2.Describe the Hardware environment 3.Create a Benchmark (Agenda) to be recorded in Phase 2. A.Define what tasks each user will perform B.Define (or estimate) the percentage of users per task. Benchmark - A benchmark procedure uses a predefined set of data and measures the results returned by the Application Being Tested (ABT). Benchmarks are intended to make comparisons easy! 1.Define

28 Testing Types and Metrics needed 1.Load Increasing Workload 1.Start / End with how many users? 2.Increment by how many users? 3.How often to increment (Seconds)? Steady-State 1.How many users? 2.How long (Days/Hours/Minutes/Seconds)? 2.Performance A.What are the reasonable response times in seconds? (each task/page) 3.Stress A.What are the expected limits? B.How many Peak users ? 1.Define

29 3.Modify 2.Record Record 1.Record the defined testing activities that will be used as a foundation for your load test scripts. 2.One activity per task or multiple activities depending on user task definition 1.Define Modify Modify load test scripts defined by recorder to reflect more realistic Load test simulations. 1.Defining the project, users 2.Randomize parameters (Data, times, environment) 3.Randomize user activities that occur during the load test

30 Performance Test scenario Test Date: 07/17/2001 Virtual Users (VUs): Test Goals Start: 5 Max Cycle Time <= 40 Sec Incremented by: 5 Maximum: 200 Initial Cycle Time = 35 Sec Sleep = 32 Seconds Test Script(s): One typical user from logon through completion (continuously) This would be an average user 4. Execute

31 Performance Test scenario Cycle Time starts to oscillate, indicating a bottleneck 5. Analyze

32 Performance Test scenario (Continued) The CPU Started making large spikes right from the start

33 Performance Test scenario (Continued) The TPS & Throughput peaked out indicating another Bottleneck

34 Performance Test scenario Both Load Generator CPU’s are spiking to 100%

35 Conclusions Define the purpose of taking on a test automation effort Develop a test automation strategy (map out what's to be automated) Treat the automated process as any other software development task Do your homework (Investigate!) Test early and test often! Remember: The effort of test automation is an investment Those performing the work should have the correct skill sets YOU SHOULD TEST UNDER LOAD!!!