Presentation is loading. Please wait.

Presentation is loading. Please wait.

Adaptive Random Test Case Prioritization Speaker: Bo Jiang * Co-authors: Zhenyu Zhang *, W.K.Chan, T.H.Tse * * The University of Hong Kong City University.

Similar presentations


Presentation on theme: "Adaptive Random Test Case Prioritization Speaker: Bo Jiang * Co-authors: Zhenyu Zhang *, W.K.Chan, T.H.Tse * * The University of Hong Kong City University."— Presentation transcript:

1 Adaptive Random Test Case Prioritization Speaker: Bo Jiang * Co-authors: Zhenyu Zhang *, W.K.Chan, T.H.Tse * * The University of Hong Kong City University of Hong Kong 1

2 Contents 2 Background Motivation Adaptive Random Test Case Prioritization Experiments and Results Analysis Related Works Conclusion & Future work

3 Regression Testing Techniques 3 Program P Program P Program P Program P Test Suite T Test Suite T Test Suite T Test Suite T Test Suite T Test Suite T Test Suite T Test Suite T Test Suite T Test Suite T Test Suite T Test Suite T Obsolete Test Case Elimination Test Case Augmentation Test Case Prioritization Test Case Reduction Test Case Selection Accounts for 50% of the cost of software maintenance.

4 Test Case Prioritization 4 Definition Test case prioritization permutes a test suite T for execution to meet a chosen testing goal. Typical testing goals Rate of code coverage Rate of fault detection Rate of requirement coverage Merits No impact on the fault detection ability

5 Coverage-based Test Case Prioritization Technique 5 Total-statement/function/branch Highest code coverage first Resolve tie-case randomly Additional-statement/function/branch Additional highest code coverage first Reset when no more coverage can be achieved Resolve tie-case randomly Disadvantages Hard to scale to larger programs

6 Contents 6 Background Motivation Adaptive Random Test Case Prioritization Experiments and Results Analysis Related Works Conclusion & Future work

7 Problem With Total Techniques 7 GREPFLEX Elbaum et TSE 2002 APFD

8 Problem With Total(greedy) Techniques 8 GREPFLEX Total strategy may NOT be effective for real-life program Elbaum et TSE 2002 APFD

9 Problems with Additional Techniques 9 Random Unix Additional Siemens Additional Unix Total Siemens Total Unix

10 Problems with Additional Techniques 10 Random Unix Additional Siemens Additional Unix Total Siemens Total Unix Additional Techniques may NOT be efficient for real-life programs.

11 Problems with Additional Techniques 11 Random Unix Additional Siemens Additional Unix Total Siemens Total Unix Can we find a prioritization techniques that is both effective and efficient for real life program?

12 Adaptive Random Testing (ART) 12 Adaptive Random Testing (ART) A technique for test case generation Evenly spread randomly generated test cases across the input domain. In empirical study, ART can detect failures using up to 50% fewer test cases than random testing.

13 Random generate a test case and execute it. Fixed-Sized-Candidate-Set ART Algorithm 13

14 Randomly generate a set of candidate test cases. Fixed-Sized-Candidate-Set ART Algorithm 14

15 For each candidate test case, find its nearest neighbor within the executed test cases. Fixed-Sized-Candidate-Set ART Algorithm 15

16 Select the test case which has longest distance with its nearest neighbor and execute it. Fixed-Sized-Candidate-Set ART Algorithm 16

17 Randomly generate a set of candidate test cases. Fixed-Sized-Candidate-Set ART Algorithm 17

18 For each candidate test case, find its nearest neighbor within the executed test cases. Fixed-Sized-Candidate-Set ART Algorithm 18

19 For each candidate test case, find its nearest neighbor within the executed test cases. Fixed-Sized-Candidate-Set ART Algorithm 19

20 For each candidate test case, find its nearest neighbor within the executed test cases. Fixed-Sized-Candidate-Set ART Algorithm 20

21 For each candidate test case, find its nearest neighbor within the executed test cases. Fixed-Sized-Candidate-Set ART Algorithm 21

22 For each candidate test case, find its nearest neighbor within the executed test cases. Fixed-Sized-Candidate-Set ART Algorithm 22

23 Select the test case which has longest distance with its nearest neighbor and execute it. Fixed-Sized-Candidate-Set ART Algorithm 23

24 Repeat until a failure is encountered. Fixed-Sized-Candidate-Set ART Algorithm 24 X

25 Adaptive Random Testing (ART) 25 ART is based on the observation that failure turned to cluster across the input domain. Intuitively, evenly spread the test case may increase the probability of exposing the first fault faster. In test case prioritization, we also want to increase the rate of fault detection.

26 Use ART directly for test case prioritization? 26 The variety of black-box input information makes it hard to define a general distance metric. Video streams Images Xml … The white-box coverage information of the previously executed test cases are readily available Statement coverage Branch coverage Function coverage And…

27 Distribution of Failures in Profile Space on LilyPond 27 William Dickinson et FSE, 2001.

28 MDS Display of Distribution of Failures in Profile Space on LilyPond 28 William Dickinson et FSE, Failures tend to cluster together.

29 MDS Display of Distribution of Failures in Profile Space on GCC 29 William Dickinson et FSE, 2001.

30 Distribution of Failures in Profile Space on GCC 30 William Dickinson et FSE, Failures tend to cluster together.

31 Use ART directly for test case prioritization? 31 The variety of black-box input information makes it hard to define a uniform distance metric. Video streams Images Xml … The white-box coverage information of the previously executed test cases are readily available Statement coverage Branch coverage Function coverage … Why NOT use such low-cost white-box information to evenly spread test cases across the code coverage space?

32 Contents 32 Background Motivation Adaptive Random Test Case Prioritization Experiments and Results Analysis Related Works Conclusion & Future work

33 Adaptive Random Test Case Prioritization 33 Generate candidate set Random select a test case into the candidate set If code coverage improve, continue; Otherwise, stop. Merits: No magic number, non-parametric Select the farthest candidate from the prioritized set Distance between test cases Distance between a candidate test case and the already prioritized test cases Repeat until all test cases are prioritized

34 Adaptive Random Test Case Prioritization 34 How to measure the distance of test cases Jaccard Distance General distance metric for binary data Can also use other distance metric for substitution. How to select the test case from the candidate set that is farthest away from the already prioritized test cases? Maximize the minimum distance (maxmin for short) Chen et ASIAN '04, LNCS 2004 Maximize the average distance (maxavg for short) Ciupa et ICSE 2008 Maximize the maximum distance (maxmax for short)

35 Contents 35 Background Motivation Adaptive Random Test Case Prioritization Experiments and Results Analysis Related Works Conclusion & Future Work

36 Research Questions 36 Do different levels of coverage information have significant impact on ART techniques? Do different definitions of test set distances have significant impacts on ART techniques? Are ART techniques efficient?

37 Subject Programs 37 Subject No. of Faulty Versions LOCTest Pool Size tcas 41133– schedule 9291– schedule – tot_info 23272– print_tokens 7341– print_tokens – replace 32508– flex – grep – gzip – sed –

38 Techniques Studied in the Paper 38 GroupNameDescriptions RandomrandomRandom prioritization Level of Coverage Info. Total total-ststatement total-fnfunction total-brbranch Additional addtl-ststatement addtl-fnfunction addtl-brbranch ARTLevel of Coverage Info.Test Set Distance (f 2 ) ART ART-fn-maxmin Function Maximize minimum distance ART-fn-maxavgMaximize average distance ART-fn-maxmaxMaximize maximum distance ART-br-maxmin Branch Maximize minimum distance ART-br-maxavgMaximize average distance ART-br-maxmaxMaximize maximum distance ART-st-maxmin Statement Maximize minimum distance ART-st-maxavgMaximize average distance ART-st-maxmaxMaximize maximum distance

39 Experiment Setup 39 Dynamic coverage information collection gcov tool Effectiveness Metric APFD: weighted average of the percentage of faults detected over the life of the suite Process For each of the 11 subject programs, randomly select 20 test suite, and repeat 50 times for each ART techniques.

40 Research Questions 40 Do different levels of coverage information have significant impact on ART techniques? Do different definitions of test set distances have significant impacts on ART techniques? Are ART techniques efficient?

41 Do different levels of coverage information have significant impact on ART techniques? 41 Fix the other variable: definitions of test set distances. Perform multiple comparison between each pair of coverage information and gather the statistics.

42 Do different levels of coverage information have significant impact on ART techniques? 42 Fix the other variable: definitions of test set distances. Perform multiple comparison between each pair of coverage information and gather the statistics. As confirmed by previous research: Branch > Statement > Function As confirmed by previous research: Branch > Statement > Function

43 Research Questions 43 Do different levels of coverage information have significant impact on ART techniques? Branch > Statement > Function Do different definitions of test set distances have significant impacts on ART techniques? Is ART techniques efficient?

44 The Impact of Test Set Distance 44 Fix the other variable: definitions of coverage information Perform multiple comparison between each pair of test set distance and gather the statistics.

45 The Impact of Test Set Distance 45 Fix the other variable: definitions of coverage information Perform multiple comparison between each pair of test set distance and gather the statistics. Max-Min > Max-Avg Max-Max

46 Best ART Technique 46 ART-br-maxmin is the best ART prioritization Technique

47 Research Questions 47 Do different levels of coverage information have significant impact on ART techniques? Branch > Statement > Function Do different definitions of test set distances have significant impacts on ART techniques? Max-Min > Max-Avg > Max-Max How does ART-br-maxmin compare with greedy? Is ART techniques efficient?

48 Multiple Comparisons for ART-br-maxmin on Siemens 48

49 Multiple Comparisons for ART-br-maxmin on Siemens 49 Only maginal difference difference between ART-br-maxmin and traditional coverage- based techniques, and it is not statistical significant.

50 Multiple Comparisons for ART-br-maxmin on UNIX 50

51 Multiple Comparisons for ART-br-maxmin on UNIX 51 Only maginal difference difference between ART-br-maxmin and traditional coverage-based techniques, and it is not statistically significant.

52 Research Questions 52 Do different levels of coverage information have significant impact on ART techniques? Branch > Statement > Function Do different definitions of test set distances have significant impacts on ART techniques? Max-Min > Max-Avg > Max-Max How does ART-br-maxmin compare with greedy? ART-br-maxmin Additional > Total Is ART techniques efficient?

53 Time Cost Analysis across All Programs Time Random AdditionalTotalART

54 Time Cost Analysis across All Programs Time (s) Random AdditionalTotalART ART << Additional ART Total ART << Additional ART Total

55 Research Questions 55 Do different levels of coverage information have significant impact on ART techniques? Branch > Statement > Function Do different definitions of test set distances have significant impacts on ART techniques? Max-Min > Max-Avg > Max-Max Is there a best ART technique? ART-br-maxmin ART Additional > Total Is ART techniques efficient? YES (<

56 Contents 56 Background Motivating Example Adaptive Random Test Case Prioritization Experiments and Results Analysis Related Works Conclusion & Future work

57 Related Works 57 Greedy Techniques for Test Case Prioritization Rothermel et ICSM 1999, S. Elbaum et TSE02. Greedy Algorithms ART Seminal Paper Chen et ASIAN '04, LNCS 2004 ART techniques can improve the effectiveness of random test case selection by 40%-50% Theoretical Aspects of ART Techniques Chen et ACM TOSEM 17, 3, No technique can improve the effectiveness of random test case selection by more than 50%.

58 Related Works 58 ART for Object-Oriented Software Ciupa et ICSE 2008 Define the metric for measuring object distance ARTOO is faster to find fault Detect faults not found by directed random. Profile Guided Test Case Generation Dickinson et FSE, Study the how failure is distributed in profile space in real software Improve test case generation by perusing failure regions

59 Contents 59 Background Motivating Example Adaptive Random Test Case Prioritization Experiments and Results Analysis Related Works Conclusion & Future work

60 Conclusion 60 Adaptive Random Test Case Prioritization can be much more effective than random prioritization. There is marginal difference in effectiveness between ART-br-maxmin and additional greedy techniques (but not statistically significant), yet ART- br-maxmin is much more efficient. Compared to the total technique, ART-br-maxmin is more effective on real-life program but slightly less efficient.

61 Future Work 61 Are there any better metrics to measure test case distance? Improve greedy techniques by using ART to resolve tie cases. Extend the ART prioritization techniques to the testing of concurrent programs and other domain specific techniques.

62 Comments are welcome! 62


Download ppt "Adaptive Random Test Case Prioritization Speaker: Bo Jiang * Co-authors: Zhenyu Zhang *, W.K.Chan, T.H.Tse * * The University of Hong Kong City University."

Similar presentations


Ads by Google