Presentation is loading. Please wait.

Presentation is loading. Please wait.

Automation of Test Case Generation

Similar presentations

Presentation on theme: "Automation of Test Case Generation"— Presentation transcript:

1 Automation of Test Case Generation

2 Testing is expensive We have talked about 3 different kinds of coverage However, the three coverage are mainly used As measure for progress Not as target to be achieved Why? Money coverage Time coverage The most used targets… 2

3 Testing is expensive 3 Numbers How many lines of test code
About 40% in Microsoft code base How much is spent each year on software testing It is not easy to even estimate the time/effort spent on testing inside a IT company (Recall words of Bill Gates) The software testing market (out source testing) is about $17B in year 2012 How many new testers More than 30k each year in US recently Well more than 150K each year globally 3

4 Automation is the way to reduce cost
Good or bad news, no way to fully automate testing in predictable future Three phase Execution automation Mostly done for unit testing: xUnit Somewhat done for system testing: scripts Test case generation automation Quite hot topic in academia Great progress made recently Test oracle check automation Some good trials 4

5 Automating different steps
Automating test execution Unit Testing Framework Record and Replay for GUI testing Scripts for Command-line tools and configuration testing Automating test case generation (this lecture) 5

6 Automatic test case generation
Random testing Pure Random Rule-based Combinatory Feedback-guided Exploration testing Systematical testing Symbolic execution Chaining Search-based testing 6

7 Random Testing Randomly generate inputs to feed in a software Problems
Easiest way to do automatic test case generation Do not require any preparation and easy to implement Problems Semantically redundant inputs E.g., for a simple program 10/x, providing any input except 0 means the same 7

8 Rule-based Random Testing
Try to reduce redundancy Use rules Try boundary cases 0, 1, -1, and other known boundaries Use distributions Generate random values following certain distribution Very effective if the distribution of input is known e.g, scores of a final exam 8

9 Combinatory Testing 9 Recall input combination coverage
Try to achieve better coverage until reach 100% Add a new test case that covers uncovered value combinations Give priority to the test case that covers multiple new possible values For multiple features of a given input, use constraint solvers to generate a value that satisfies feature combinations (e.g., length = 2 && starts-with A) 9

10 Adaptive random testing
Intuition Input patterns for bug revealing Strips: bug is revealed when x + y = 4 Blocks: bug is revealed when x > 3 & y < 4 Trying two very similar test cases does not make much sense 10

11 Adaptive random testing
Approach When choosing the next test case, choose the test case having the largest minimum distance from existing test cases {t1, t2,…tn} are existing test cases, choose ch so that Example: one input, int16 x T3: T2:32767 T5: T1: -20 T6:... T4: 16394

12 Feedback-Guided Random Testing
Question: It is easy to generate random values for primitive types, how to generate a random object? You may generate a random object id = -1, content = “”, child = null May not make sense… For a class, id may be always positive, content may not be empty, and child cannot be null… 12

13 Feedback-Guided Random Testing
Consequence: Randomly generated objects may cause a lot of bugs But they are not real bugs… class BugClass{ int id; String content; Object child; public BugClass(String str){ = ++; this.content = "bug:" + str; this.child = Bug.createDefaultChild(); } public do(){ String type = Global.type[]; String content = this.content.substring(4);; id = -1; content = ""; child = null; Index array < 0!!! Index out of string length!!! Null pointer reference!!! 13

14 Feedback-Guided Random Testing
Solution: Try from the methods taking primitive types first When an object of a class is generated, use the object in following test cases Do not use duplicate and null objects Do not use objects generated with exception 14

15 Exploration Testing Very commonly used in testing of GUI & Web software Explore the user interface Click all clickable Feed in random value to input boxes May often achieve good coverage One problem is the login part Or any inputs that requires some semantics 15

16 Exploration Testing 16 Example Menu Window About Play Records Setting
Audio Setting Select Level Login Game Setting Play Confirm 16

17 Exploration Testing 17 Return to the last window
Return button, go back, … Not always available Record the state of the software at the previous window Sometimes high overhead Usually Require instrumentation of the code To check outputs (sent to the screen) with oracles To tell the explorer that the window is ready (the drawing of the window is done) 17

18 Exploration Testing Signal based exploration testing 18 Driver
Software under test Click event Event handler finishes Analyze new GUI Click event Event handler finishes 18

19 Tool Introduction-Randoop
Feedback-based random testing Joint work by MIT and Microsoft Research The most robust and effective random testing tool for Java Have both stand-alone tool and eclipse plug-in Tool website: 19

20 Tool Introduction - Randoop
Process Install with eclipse update site: Right click on the class to be tested Choose as Randoop Input Configuration How long should randoop run Size of test methods Timeout of certain thread #Test methods per file 20

21 Tool Introduction - Randoop
Results Add support Jar file Use Junit to re-execute the test suite Remove some invalid test cases Use EclEmma to check results 21

22 Search-based Testing Deem test case generation as a optimization problem Somewhat between random testing and systematical testing Based on random testing, and focus on the input domains Use code coverage as guidance 22

23 Optimization Try to find the maximal or minimal value of a certain function Numerous practical problems can be viewed as optimization problem Least cost to travel to a number of cities Least camera to cover an area Distribution of stores to attract most customers Design of pipe systems with least material Put items into a backpack (with limited volume) with highest value 23

24 Illustration of Optimization
global peak local peak 24

25 Solution of Optimization
Hill climbing Start from a random point Try all neighboring points, and go to the point with highest value, until all neighboring points has a value lower than the current point Easy to find a local peak Random-restart Hill climbing Restart hill climbing for many times To avoid local peaks 25

26 Solution of Optimization
Annealing simulation Adaptation of hill climbing Has a probability to move after reached local peak The probability drops as time goes by Genetic algorithm Simulate the process of evolution Start with random points Select a number of best points Combine and mutate these points Until no more improvements can be made 26

27 Transform Testing to Optimization
Using code coverage as criterion: Try to generate a test case that covers certain code element (method, statement, branch, …) Measure how well we have solve the problem A simple fitness function How far is the already covered elements from the target code elements Try to make the distance 0 27

28 Go from random testing A list of random test cases as the start point Each test case is a point in the input domain Using various optimization algorithms to find the best test case 28

29 An example with hill climbing
read x, y Target is st2 Start from 0, 0 f(0, 0) = 2 steps Try (0,1) (1,0), (0,-1), (-1,0) No improvement Go all directions equally Until reach (5,5) f(5,5) = 1 step Increase x and y Until reach (7,5) Done! if x+y >= 10 N y st1 st3 N if x >= 7 read x, y; if(x+y>=10){ st1; if(x>=7){ st2; // target } st3; y y st2 29

30 Problem 30 The fitness function is too simple
Requires a lot of random walk before getting to the correct place (5,5) Better algorithms may help, but not much, because all algorithms require to compare the value of fitness functions A better fitness function: Consider the value gap at all branches 30

31 Rerun example with hill climbing
read x, y Target is st2 Start from 0, 0 f(0, 0) = 10 value gap Try (0,1) (1,0), (0,-1), (-1,0) Go to (0,1) or (1,0) Until reach (0,10) f(0,10) = 7 value gap Increase x Until reach (7,10) Done! if x+y >= 10 N y st1 st3 N if x >= 7 read x, y; if(x+y>=10){ st1; if(x>=7){ st2; // target } st3; y y st2 31

32 EvoSuite: Demo 32 Search based software testing tool
Work in around 2010 by Gordon Fraser and colleagues at U Saarland, Germany Use genetic algorithm as the optimization algorithm Use combination of step + value gap as the fitness function 32

33 Process 33 Their eclipse plug-in is rather instable
Have a relative stable command-line tool for mac os X Very easy to setup Download the jar Package your own code to jar file Run the command with options Coverage criterion Timeout Random seeds 33

34 Results 34 Junit test cases
Even with comments Very high coverage on relatively simple examples 34

35 Systematical testing 35 Target at certain code coverage
Try to understand how the code works Analyze the code structure to find out a path to go to a certain statement Analyze the code structure to find out the constraint of the inputs to let the program follow the path 35

36 Symbolic execution 36 Target at better statement coverage Basic idea
If a statement is not covered yet, try to provide an input to go over that statement A statement is covered only when a path goes to it is covered Then, what input will cause the program to go through certain paths? Only when the input satisfy all if-conditions along the path! 36

37 Symbolic execution: example
void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == ) throw new Exception("bug"); } a==null F T a.Length>0 F T a[0]==123… F T

38 How to know what input to feed in
Static symbolic execution Construct a constraint for each statement, the statement will be executed when the constraint is satisfied The parameters (variables) in the constraint are inputs of the software Solve the constraint The variables used in conditions may not be inputs So they must be represented by expressions of inputs 38

39 Static symbolic execution
Here T is the condition for the statement to be executed, (y=s) is the relationship of all variables to the inputs after the statement is executed Basic Example y = read(); y = 2 * y; if (y <= 12) fails(); else success(); print ("OK"); T (y=s), s is a symbolic variable for input T (y=2*s) T (y=2*s) T^y<=12 (y=2*s) T^y<=12 (y=2*s) | T^!(y<=12) (y=2*s) T^!(y<=12) (y=2*s) T^y<=12 (y=2*s) | T^!(y<=12) (y=2*s) 39

40 Static symbolic execution
Generating test cases y = read(); T (y=s), s is the input y = 2 * y; T (y=2*s) if (y <= 12) T (y=2*s) fails(); T^y<=12 (y=2*s) s<=6 -> 8 else success(); T^!(y<=12) (y=2*s) s>6 ->3 print ("OK"); T^y<=12 (y=2*s) | T^!(y<=12) (y=2*s) 40

41 Constraint Solver Solve constraints -> provide a value set satisfying the constraint Float values Linear programming if the constraints are all linear Boolean values SAT problem Integer values Can be deduced to SAT String values SMT problem Mostly NPC problems Can be deduced from/to SAT problem 41

42 Problems of static symbolic execution
Path explosion Remember n branches will cause 2n paths Infinite paths for unbounded loops Calculate constraints on all paths is infeasible for real software Too complex constraint The constraint gets very complex for large programs Not to mention the resolving part is NPC 42

43 Chaining and Goal-oriented Approach
Not all branches are actually relevant If the outcome of a branch will not affect whether a statement will be executed We should not waste time on the branch A classification of branches Critical branches The branch must not be taken if target is executed Non-essential branches Other branches 43

44 Example of branch classification
Read a, b Read a, b; if (a > 0){ a = a + b; } if (b >= 10){ st2; //target if a > 10 N-E a = a+b; N-E if b >= 10 C st2 End 44

45 Chaining 45 Find a path to reach the target code
Start with the start node and the target node <read a,b>, <target> Avoid critical branches: b>=10 Choose b = 10 45

46 A different example 46 b=0 Read a; if a > 0 b = 0; if (a > 0){
b = a + b; } if (b >= 10){ st2; //target if a > 0 N-E b = a+b; N-E if b >= 10 C st2 End 46

47 Chaining 47 Find a path to reach the target code
Start with the start node and the target node <read a>, <target> Avoid critical branches: b>=10 , no solution Go to definition statement of b: b = 0; no solution b = a + b; -> a + b>=10 47

48 Chaining: next steps <read a>, <b=a+b, a+b>=10>, <target> Avoid critical branches: a>0, a+b>=10 go to definition of a, b b = 0; read a; -> solve the constraint: a = 10 48

49 Chaining 49 Compared to static symbolic execution: Problems
Reduce the branches to be considered Reduce the definitions to be considered More scalable than static symbolic execution Problems Still not scalable enough Spend much time to generating only 1 test case to cover a certain statement 49

50 Dynamic symbolic execution
Koushik Sen et al. 2005 One of the most important progress in software engineering in the 21st century Basic idea Actually run the software Generate constraints as the program runs Flip constraints to reach other statements 50

51 Dynamic symbolic execution
Choose next path Code to generate inputs for: Solve Execute&Monitor void CoverMe(int[] a) { if (a == null) return; if (a.Length > 0) if (a[0] == ) throw new Exception("bug"); } Data null {} {0} {123…} Constraints to solve a!=null a!=null && a.Length>0 a.Length>0 && a[0]== Observed constraints a==null a!=null && !(a.Length>0) a.Length>0 && a[0]!= a[0]== Negated condition a==null F T a.Length>0 F T a[0]==123… Done: There is no statement left. F T 51

52 Dynamic symbolic execution
Advantages No need to analysis the whole system before perform testing All constraints and expressions can be executed along one execution path Can handle library method calls Testing as the analysis taking place Disadvantages Overhead in testing 52

53 Review of automatic test case generation
Random testing Rule-based Adaptive Feedback-based Exploration Search-based testing Optimization algorithms Fitness function Systematical testing Symbolic execution Chaining and goal-oriented 53

54 Review of testing 54 General guidelines Levels of testing
Unit testing, higher level testing, GUI testing Test coverage Code, input, mutation Regression testing Non-function testing Security testing, performance testing Test automation 54

55 Review: General guidelines
Test is the practical choice: the best affordable approach Concepts: test case, test oracle, test suite, test driver, test script, test coverage Granularity: unit, integration, system, acceptance Type by design principle: black-box, white-box Black-box-testing: boundary, equivalence, decision table White-box-testing: branch coverage, complexity 55

56 Review of Unit Testing 56 Using unit test framework to do unit testing
It does all the common things, reduce test interference Inside test framework setUp -> test -> assert -> tearDown Writing informative assertions Writing complete tear downs Dependencies are evil! Dependency injection Remove dependencies Test doubles 56

57 Review: Higher level testing
Integration Testing Strategies: big bang, top-down, bottom-up, sandwich, path-based System Testing Environment issues: building, platforms, web services, environment failures, distribution issues Acceptance Testing GUI testing Event-based, Screen-based, CV-based Record and replay 57

58 Review of test coverage
Code coverage Target: code Adequacy: no -> 100% code coverage != no bugs Overhead: low (instrumentation cause some overhead) Input combination coverage Target: inputs Adequacy: yes -> 100% input coverage == no bugs Overhead: none Mutation coverage Target: bugs Adequacy: no -> 100% mutant coverage != no bugs Overhead: very high (execution on instrumented mutated versions) 58

59 Review of Regression Testing
Test Prioritization Try only the most important test cases Test Relevant Code Try the most relevant test cases Record and Replay Reuse the execution results of previous test cases 59

60 Review of Non-Functional Testing
Performance Testing Test whether the efficiency (time and space) of a software meets requirements Security Testing Test whether the software is vulnerable to attacks (special invalid inputs designed to control the software or reveal info from the software) 60

61 Review of automatic test case generation
Random testing Rule-based Adaptive Feedback-based Exploration Search-based testing Optimization algorithms Fitness function Systematical testing Symbolic execution Chaining and goal-oriented 61

Download ppt "Automation of Test Case Generation"

Similar presentations

Ads by Google