1. 1841f06detprob3 MM Stroustrup Ch26 u Comments? u Agree or disagree with his testing approach?

2. 2841f06detprob3 MM Stroustrup ch26 u Additional topics?

3. 3841f06detprob3 L2asg C++ u Good job – very interesting programs u Common mistakes –No failure set or not well labelled –Easy fault = large failure set

4. 4841f06detprob3 Failure Set

5. 5841f06detprob3 Failure Sets What was the shape of your failure sets? How could test case selection criterion improve the probability of finding these faults?

6. 6841f06detprob3 From last year’s 1-minute paper u How do we measure test adequacy? Or what is test adequacy?

7. 7841f06detprob3 Test terms u Fault u Failure u Error

8. 8841f06detprob3 Test terms u Test selection criterion – a methodology for selecting test cases u Test adequacy criterion – a methodology for deciding when to stop testing

9. 9841f06detprob3 Testing Basics Detection probability

10. 10841f06detprob3 Detection Probability The probability of detecting a specific fault using a specified testing strategy Or The probability that a randomly-generated test set that satisfies a specific criterion will detect a specified fault

11. 11841f06detprob3 Operational Profile u The set of inputs from which operational data is chosen is called the operational profile –It usually includes a description of the probability of which points will be used u Is 7.99 more likely to be encountered at a WalMart POS terminal than at an exclusive store that only sells fur coats?

12. 12841f06detprob3 Methodology to evaluate testing criterion u Pick sample program, sample faults, sample operational profile, two criteria. u For each criterion, randomly generate test suites that satisfy the criterion. u Calculate the percentage of the suites that detect each of the faults. u The criterion that has the higher percentage is better.

13. 13841f06detprob3 Triangle Example cin >> a >> b >> c ; type = “scalene”; if (a == b || a == c || b == c) type= “isosceles”; if (a == b && a == c) type = “equilateral”; if (a >= b+c || b >= a+c || c >= a+b) type=“not a triangle”; if (a <= 0 || b <= 0 || c <= 0) type=“bad input”; cout<< type;

14. 14841f06detprob3 Control Flow Graph Operational profile 3,3,3abcdegiequi 3,3,4abcegiisos 3,3,5abcegiisos 3,3,6abcefginot 3,4,3abcegiisos 3,4,4abcegiisos 3,4,5acegiscal 3,4,6acegiscal All inputs are equally likely

15. 15841f06detprob3 What are the failure probability for each color (separately)? cin >> a >> b >> c ; type = “scalene”; if (a == b || a == c && b == c) type= “isosceles”; if (a == b || a == c) type = “equilateral”; if (a >= b+c || b >= a+c || c > a+b) type=“not a triangle”; if (a <= 0 || b <= 0 || c <= 0) type=“bad input”; cout<< type; Blue GreenRed

16. 16841f06detprob3 TTYP – probability of detection u What is the probability of detection with one randomly chosen test case? u What is the probability of detection with two randomly chosen test cases?

17. 17841f06detprob3 TTYP – per path u What is the probability of detection with one randomly chosen test case per path? u What is the probability of detection with an equal number of randomly chosen test cases?

18. 18841f06detprob3 TTYP – smaller subdomains u What might be better smaller subdomains?

19. 19841f06detprob3 TTYP - subdomains u Are paths the best subdomains? u Would a functional decomposition be better? u Should we re-define the term subdomain?

20. 20841f06detprob3 fault 1 fault 2 fault 3 fault 4 criterion 10.080.030.040.85 criterion 20.070.10.050.91 criterion 30.060.120.080.88 criterion 40.050.40.060.89 Comparing Criteria

21. 21841f06detprob3 Solving this choice u What are the assumptions? u Analogy with dice?

22. 22841f06detprob3 TTYP – smaller subdomains u What might be better smaller subdomains? u Would MCC (multiple condition coverage) be better subdomains

23. 23841f06detprob3 TTYP2 – C0 and C1 coverage u How do we deal with C0 and C1 coverage since they are not subdomain testing methodologies?

24. 24841f06detprob3 TTYP3 u How could you estimate the det prob of C0 or C1 testing?

25. 25841f06detprob3 Marble Problem u Assume that there is a bag of marbles from which marbles are drawn with replacement. u What is the maximum likelihood estimate of p (the probability of drawing a purple) marble if you draw exactly n purple marbles in a row? u ? P such that (p) n (1-p) is max or u ? P such that (p) n =.5

26. 26841f06detprob3 Evaluating Testing Methods by Delivered Reliability Frankl, Hamlet, Littlewood, Strigini IEEE TOSE Aug98 For Thursday, Aug 30 – study through section 2.3