Presentation on theme: "Going for low hanging fruits first… NONINVASIVE PERFORMANCE TUNING"— Presentation transcript:
1 Going for low hanging fruits first… NONINVASIVE PERFORMANCE TUNING Why single user performance testing is important-overview-February 2012Gihan B madawalaSenior systems engineer
2 Why PerformanceNo matter how well applications are designed, no matter how well they meet business requirements, they are virtually useless to end-users if the performance is sluggish or unreliable
3 Why this is doable and its benefits But the good news is very majority of these performance issues can be found without sophisticated, expensive testing tools.You can find majority of performance issues with baseline testingYou can find a sub set of these performance issues with single user testing (SUT)Benefit two fold. Verify feature as well as single user performance testing (when the duration for the full scenario is measured)This could be a health check of the product (when run regularly)Test high ROI feature scenarios
4 Have you observed any of these behaviors in your PC or in the Test Server? Computer suddenly freezes and unusable often start working on MondayAny operations that you try to execute is extremely slowRunning out of disk space due to huge log filesOut of Memory Exception from applicationsStack Overflow errors from applicationsRendering of some pages of applications is extremely slow.
5 why single user performance testing is so important? Gives an early indication of system design feasibilityCan be measured with existing set of tools availableCost of single user testing is low and can be done prior to multiple user testingCan be done with minimal experience in Performance TestingCan be done repetitively with minimum efforts
6 Single User, Baseline and Benchmark testing Baseline Performance testing (Done for each scenario say with 1, 2, 10, 20 and 50 users to determine baselines for mainly response times) But the performance objective could be to run 500 usersSuppose you determine that 5 seconds for a webpage load is acceptable performance. If you execute your performance test cases and see that the webpage is loading at 8 seconds, that 8 second load time is your baseline results. But this is not acceptable performance. From that point, the development team need to make the webpage load faster. Each time you run the same test, these will be your regression tests. If the webpage makes it to a 5 second load time, at that point your test case passesContd…
11 Benchmark testing (IT) Tests that use representative sets of programs and data designed to evaluate the performance of computer hardware and software in a given configuration.A benchmark is the act of running a computer program, a set of programs, or other operations, in order to assess the relative performance of an object, normally by running a number of standard tests and trials against it.Compare with competition for availability and stability etcUsually done by certified labs. VeriTest, Lions Bridge and Microsoft etc
12 SUT Scenarios are…SUT can be considered as a sub set of baseline and benchmark performance testingIf your performance requirement is 5 seconds for a SUT this should be in sub second rangeDetermine the ideal performance of the systemSUT scenarios should cover most used scenarios of the application or systemSUT scenarios can include think timeSUT can be done manually(stop watch) or automated(tool)Should consider measuring performance of multiple scenarios (Probably 10 to 20 scenarios but no more)
13 When SUT can be executed Each weekEach iterationEach releaseEach customer deploymentEach product upgrade (Software and hardware)When network mode or infrastructure changesWith data centre relocations
14 Be organized and methodical… Follow a processDo benchmarks regularlyDedicate approved test bench for this testingPublish results where it is visible to everyone in development teamShare results and raise concerns when needed with appropriate peopleWork closely with key people in development teamForm a performance SWAT team if necessary
15 Benchmarking Single User Performance manually (numbers are superficial)
16 Example SUT Scenario1. Create a visitor 2. Create an equipment manufacturer, model and unit 3. Assign the equipment to a mobile device 4. Assign a visitor for the equipment created 5. Delete an equipment 6. Modify an equipment 7. Create a a prerequisite for the equipment 8. Create a designated area policy 9. Assign the policy to equipment 10. Create speed limits for roads 11. Simulate movement of the equipment 12. Verify the feature MEASURE – Total duration from Strep 1 to 12
21 Why it is important to define performance targets in the beginning of the project This will result in technology selectionImpact on application architecture and designStage optimizations can be done until meeting the targetResponse Time TargetDeviation AllowedResponse time with deviationsSmall1 Sec0%1 |SecMedium1.5 Sec50%0.75 – 2.25 SecLarge2 Sec100%1 – 4 SecDurationResponse TimeReport AUp to 6 months< 4 secondsUp to 2 years< 8 secondsUp to 4 years< 12 seconds
22 PERFORMANCE OBJECTIVES Following four areas we need to look into when meeting performance objectives. Values for these need to be defined under requirements in the beginning of the projectResponse TimeThis is the time it takes for a system to complete a particular operation, such as a user operationThroughputThis is the amount of work a system can support. Throughput can be measured in terms of bytes per second in our application (This could be requests per second or transactions per second in a web application)Resource UtilizationThis is the % of system resources that are used by particular operations and how long they are used. This is the cost of the server and network resources, including CPU, memory, Disk I/O and network I/OWorkload or PayloadThis is usually derived from marketing data and includes total number of users, concurrently active users, data volumes, and byte rates etc.
23 Probable performance bottlenecks - A bottleneck is the resource that constrains throughput- Identifying bottlenecksMeasure response timeMeasure throughputMeasure resource utilization- System resource issues: (Need System tuning)CPUMemoryDisk I/ONetwork I/O- External resource issues: (Need .NET Framework tuning)Application Server - Session and state management, thread contention, interfacesDatabase Server - Database design, inefficient indexes, stored procedures and queries
28 SOME SPECIFIC GUIDELINES FOR SINGLE USER PERFORMANCE TESTING Consider raising issues first with more frequently used operations. They are the ones which are going to reflect most as bad user experience for your customers.Mobile client performance (or response time) is more important than office client performance.If you want to run performance counters of a different machine in your network from your PC then run \\machinename\C$ to open them.When measuring response time with a stopwatch take reading of more than one run. .NET applications may reflect a bad reading first time when you run a certain operation. It is always better if you record the second reading or even better average out 2nd and 3rd readings.
29 Contd…Do not try to use too many performance counters first time when you are trying to identify performance issues. Start with a set 20 to 25 counters the most and then when you sense a problem then dig in more with increasing the number of counters in that area of the bottleneck.Don’t do SQL profiling and use of performance counters to monitor performance issues at the same time. There is too much overhead to the system that you are measuring performance.If you are using the SQL Profiler to look at how the system performs any query taking more than 50ms with a single user may lead to potential performance and scalability issues when running with multiple users.
30 APPROACH FOR MEETING PERFORMANCE OBJECTIVES PlanUnderstand the needIdentify requirements (Objectives)PrepareProvide a plan with an approachSet aside a test benchExecuteOrganize and prioritizeExecute regularlyMeasure and analyzePublishShare and publish resultsRetuning and PostmortemOptimize/tune as necessaryTake appropriate action
31 SOME TIPS FOR USING WINDOWS PERFORMANCE MONITOR Use counter logs to monitor activity of certain operation or a simulation for a long time. Once you add a set of performance counters to the log you can start them manually or start this automatically at a certain time or triggered by another counter exceeding a limit.Work with small set of counters if you are measuring any response time of an application. Otherwise the overhead from this could affect your reading.You can set the interval of monitoring points longer if you are running these counters for a longtime. Say from default every 15 seconds to every 1 minute.Change the scale of the performance counters appropriately for it to be properly visible in the system monitorIf you are creating a counter or trace log, store it in the local disk not in a shared location in the network
32 You can agree or disagree with these… Successful test is not a test case that passes execution. It is a one that fails and which can find a defectSoftware engineers and test engineers need to have different mindset to be successful in their positions. SEs’ mostly build systems and in contrary TEs’ break systems.Intelligent short duration ad hoc testing exercise can be more effective and successful than a well planned well thought of regression test exercise in certain times. Success will vastly depends on the test engineer’s knowledge of the product and his/her testing skills.
33 References… Q&A Improving .NET Application Performance and Scalability Performance tuning and Optimizing ASP.NET ApplicationsSilk Performer User manualsThe Art of Software TestingQ&A