1. 1 Presented by: Val Pennell, Test Tool Manager Date: March 9, 2004 Software Testing Tools – Load Testing

2. 2 AGENDA Load Test Phases Assessment Analysis and Specification Strategy and Test Plans Setup Environment and Develop Scripts Test Preparation and Execution Lessons Learned Cost Environment Understanding the Application Future Benefits Successes Q&A

3. 3 Load Test Phases Assessment Client fills out a Load Assessment form. (what to test) Client relays load test desires - educate Client on different types of load tests and workload types. (how to test) Client provides network topology diagram to assess test and production environments. (where and when to test) Determine load test monitoring team. (who to test)

4. 4 Load Test Phases Defining Measurement Points

5. 5 Load Test Phases Analysis & Specification Infrastructure Meeting – determine monitoring requirements. Determine resources and licenses required. If external resources required arrange work place. Application demonstration for load test engineer. Determine user ids, data requirements and refresh method. Determine number of scripts and size of scripts.

6. 6 Load Test Phases Test Strategy and Plan Create test strategy plan document with clearly defined objectives for client signoff Test Strategy / Environment Test Objective / Description / Assumptions Test Plan / Team List Test Models / Scripts Team review of Load Test Detail Management Summary Report

7. 7 Load Test Phases Environment and Script Development Acquire and setup agent test machines Complete test lab requests or Change Management Request Obtain access to application / platform for script coding Code scripts Obtain test ids, test data and validate refresh method Run script and validate data refresh method Provide sample report for Client sign off

8. 8 Load Test Phases Performance Testing Production test - validate have change approvals and management notified Run mock test day/evening before Team meeting to level set – review objectives and plan Validate configuration still working run mini test day of test Very Important – evening testing requires food! Data Correlation – Run reports – Review with monitoring team Generate summary report, review with Client, and determine if additional tests are required HANDOUT – Sample Report

9. 9 Lessons Learned Cost  Internal vs. External Resources  Licensing vs. Spike Licenses – finding the right mix  Vendor provided engagement  Due diligence – perform POC when selecting vendor

10. 10 Lessons Learned Environment  The closer to production load equipment is, the better production results can be predicted.  The infrastructure used for load testing should be as similar as possible to production to truly understand end to end throughput.  An application should be verified to work in a new environment prior to load testing.

11. 11 Lessons Learned Understanding the Application  Detailed flowchart for monitoring components.  Having people knowledgeable about application and components is important.  Having system components monitored by the expert for that component during a load test greatly increases the success of analyzing and understanding the results.  Real world load.

12. 12 Lessons Learned Future Benefits  Load testing should be performed periodically (with new releases, major operating system or hardware upgrades or increased user base) in order to validate continued acceptable performance.  Understanding how the system performs under load is helpful with future application designs and support.

13. 13 Successes Significant Finds from Testing Under Load  Firewall - capacity issue, first layer bottleneck  MQ - memory issue, servers spawning  Web Access - application user limit  Cold Fusion - application code leaving sessions open  WIR - session thread settings  Time and Attendance - database queries

14. 14 Q&A