1. Automated Software Testing: Test Execution and Review Amritha Muralidharan (axm16u)

2. Contents What is test execution? Phases of Testing –Unit Testing –Integration Testing –System Testing Test Results Analysis of Regression Tests User Acceptance Testing Defect Tracking and Automated Tracking tool Test Program Status Tracking and EVMS Test Metrics Collection and Analysis –White Box –Black Box Test Execution Review

3. Test Execution Flow

4. What Is Test Execution? Stage 5 of the ATLM Creation of testing environment Tests are carried out on the application –Input = test procedures –Output = satisfied/altered acceptance criteria Software Problem Reporting E.g. CourseMarker

5. Phases of Testing Unit, Integration, System and User Acceptance testing. Tests executed according to the Test Plan for each phase.

6. Phases of Testing: Unit Testing Consistent with Development Schedule Automated Results Checkout Process White Box Level –Involves detailed look at code Best to execute unit tests as soon after the code has been created. Static Analysis Code Profiling Development Test Tools Test Documentation –Metric Collection –Process Improvement –Software development folders (SDF) and Unit Test Evaluation Criteria

7. Phases of Testing: Integration Testing “Units that are incrementally integrated are tested together based on control flow” Some integration testing takes place during unit testing Test procedures based on integration test design addressed earlier in the ATLM Results are analyzed and documented Refining Test Procedures Some test scripts can be automated for re-use in system testing End-user approval = end of unit and integration testing

8. Phases of Testing: System Testing Higher Level Performed by a separate test team May require many individual test procedures Test Execution –False Negatives –False Positives Software Problem Report

9. Test Results/Analysis of Regression Tests New application baseline –Build release notes should accompany the new build Smoke Tests Regression Testing (after defects have been fixed/modified) –Investigate high risk areas of code –Indicator of developer behaviour Test team investigates areas with larger number of problem reports Analysis is a clear indicator of whether test execution procedures are useful in locating errors. –Where more test effort is required System testing is complete when criteria have been met

10. User Acceptance Testing Involves End-user participation Consists of the subset of the suite of tests performed at system level Defects at UAT level will be documented in SPRs Satisfactory resolutions of level 1 and 2 SPRs = conclusion to testing

11. Defect Tracking and New Build Process Test team performs problem reporting operations in compliance with a defined process –SPR creation –E-mail Notification to configuration management(CM) and developers New software is checked via CM tools New software build issuance Defect Priority –Fatal, high, medium, low Engineering review Board (ERB) Automated Defect Tracking Tool –Ensures reported defects receive proper attention –Central defect tracking repository –Has several basic characteristics Automated Test Tools –Maintain test results/permit automatic generation of test results Recommended that SPRs are generated per life-cycle stage

12. Test Program Status Tracking and EVMS Helps to efficiently handle problems that arise during test effort Test engineer must produce meaningful reports based on test plan measures and metrics –Test log –Test coverage reports

13. Test Program Status Tracking and EVMS Cont… To effectively monitor testing progress and report to senior management, an earned value approach is used. EVMS = one of the best ways to track progress Tracks value of completed works and compares it to planned costs and actual –Provides true measure of schedule and cost status –Enables creation of effective corrective actions 4 steps –Identify short tasks –Schedule each task –Assign a budget to each task –Measure progress of each task Earned value for work completed – planned budget = Schedule Variance Earned value for work completed – actual cost = Cost Variance

14. Test Metrics Collection and Analysis Metrics = key indicators of test coverage progress and quality of test effort Separated into white box and black box testing efforts HOWEVER, gathering too many metrics can be time consuming

15. Test Metrics Collection and Analysis: White Box Testing Targets application’s internal working Measures the DEPTH of testing –Coverage Analysis Code Analysis and Code Profiling e.g. Rational Quantify –Objective is to locate complex areas of source code Fault Density (Fd) = Nd/KSLOC –Predicts the remaining number of faults Design Complexity Metrics measures the number of ways a module can cal other modules

16. Test Metrics Collection and Analysis: Black Box Testing Focus on the BREADTH of testing Based on application’s external considerations Each metric is assigned to one of three categories –Coverage –Progress –Quality

17. Test Metrics Collection and Analysis: Black Box Testing Coverage –Test Coverage –System Coverage –Functional Coverage Progress –Test Procedure Execution Status –Error Discovery Rate –Defect Aging –Defect Fix Rate –Defect Trend Analysis Quality –Test Success Index –Defect Density –Test Effectiveness –Problem Report Acceptance Criteria Metric –Test Automation Metric

18. Test Execution Review Stage 6 of ATLM Review test performance e.g. metrics –Helps determine enhancements to be made Document well performed activities Lessons Learned should be acknowledged/documented –Aim to ensure best practices are used Integrate a continual iterative learning process –Encouraged as improvement opportunities “check pulse” Documentation should be easily accessible to all team members Corrective actions suggested must be evaluated Represents he need for process improvement

19. Questions?