1. Testing Metrics Software Reliability
CSSE 376, Software Quality Assurance
Rose-Hulman Institute of Technology
April 5, 2007

2. Outline
Testing Metrics
An Important Metric: Reliability

3. Common Metrics Product Process KLOC Function Points #Bugs Staff hours
thousands of lines of code
need to remove comment lines?
Function Points
#Bugs
Process
Staff hours
Tests planned
Tests passed

4. Bug Density Measure #Bugs/KLOC
Expect different densities at different stages of a project
May categorize bugs by severity

5. Example Bug Density

6. Cartoon of the Day (1/3)

7. Using Bug Metrics Count bugs discovered during each phase of a project
Compare to previous projects
provides estimates of expected values at each phase -- could use to set milestone
deviation of more than 20% from expected indicates need for investigation

8. Analysis of Bug Data Root cause analysis Trend analysis
Search for explanations
Might look at other process data (effort, experience of team, etc.)
Trend analysis
Make predictions from current data

9. Reliability

10. Failures vs. Faults Fault: Failures: developer-oriented
6 faults/1000 source lines
Failures:
customer-oriented
3 failures/1000 CPU hours

11. Calculating Reliability
probability of failure-free operation for a specified time interval
0.82 for 8 CPU hours
Failure Intensity
number of observed failures within a specified time interval
3 failures/1000 CPU hours

12. Factors Influencing Reliability
Fault removal
by error correction (debugging)
Fault introduction
by error correction (unintended)
by new feature development
Operational profile

13. Operational Profile
Probability of Use
Functions

14. Example Function Usage Probability Distribution Interval Change 32%
0-31
Delete
14%
32-45
Insert
46%
46-91
Print 8%
92-99

15. Test Generation Test Random Numbers Test Cases 1
29, 11, 47, 52, 26, 94
C, C, I, I, C, P 2
62, 98, 39, 78, 82, 65
I, P, D, I, I, I 3
83, 32, 58, 41, 36, 17
I, D, I, D, D, C 4
36, 49, 96, 82, 20, 77
D, I, P, I, C, I

16. Test Compression Real use of a product involves repetitive operations
different users invoke the same operations
same user invokes the same operations on different days
Redundant tests waste computer and personnel time
Compression: when generating random tests, do not duplicate previous tests

17. Cartoon of the Day (2/3)

18. Cartoon of the Day (3/3)

19. Curve Fitting Reliability models focus on failure removal
Use a random process to model the failure removal process

20. Execution Time Model
Failure Intensity
Execution Time
Goal

21. Resource Constraints Early phase of a project Middle phase Late phase
constrained by availability of developers (debuggers)
Middle phase
constrained by availability of testers
Late phase
constrained by availability of machines
may run tests in parallel to increase number of tests per CPU hour

22. Adjusting for Calendar Time
Estimate resource usage during each phase of the project
Model calculates failure intensity in terms of execution time
Model adjusts fault removal rate according to resource constraints

23. Calendar Time Component
Constrained by debuggers
Failure Intensity
Constrained by testers
Constrained by machines
Goal
Execution Time

24. Calculating CalendarTime/ExecutionTime Ratio
10 staff-hours to fix each failure
2 failures/CPU-hr
That means it will take 10 * 2 = 20 staff-hrs per CPU-hr
Suppose you have 5 developers
Then you have 20 / 5 = 4 hrs per CPU-hr
Each CPU-hr will take 4 calendar hours

25. Estimating Completion
Establish a failure intensity objective
Record execution times of failures
Run model to estimate reliability
Model reports estimated completion date
Values are not absolute---within confidence bounds

26. Estimating Completion
Failure Intensity
Calendar Time
Goal
Ship Date

27. Acceptance Charts
Bugs
Reject
Continue Testing
Accept
Time