1. Wishnu Prasetya Software Testing Email: wishnu@cs.uu.nlwishnu@cs.uu.nl URL: www.cs.uu.nl/~wishnuwww.cs.uu.nl/~wishnu

2. Content Chapter 1 of LN. Here I will just give you a summary of the chapter: Coverage White box testng Black box testing 2

3. Testing vs Verification Verification : show that all possible executions satisfy the given specification. Expensive Testing : show that at least few executions satisfy the specification. not trivial to determine which executions to choose not trivial to figure out how to trigger those executions (undecidable) 3 tax(income) { if (income  10000) return 0 ; income := income – 10000 ; if (income  10000) return 0.1 * income ; return 1000 + 0.2 * (income – 10000) ; }

4. Coverage 4 E.g. CFG-based coverage: Try to cover all nodes, or all edges Try to cover all paths, or all prime paths, or all linearly independent paths

5. Automated testing Consider a simplified case of a program f(x) that returns a value y. A specification of f is a predicate S(x,y). A test-case for f is just an instance of x. Automated testing here means: the generation of test- cases for f. S is a so-called “test-oracles”. While inputs can be generated, no algorithm can guess what’s in your mind. So, there is no way to “generate” S. Coverage problem: come up with an instance of x, such that the resulting execution covers some target c (undecidable). 5

6. From Unit Tetsing Tool Competition 2015 Benchmarking of automated unit testing tools for Java; “unit testing” at the class level. 63 classes from various open source projects Base lines: randoop (random testing tool), human testers. Tools  all exploit reflection T3 (random + pair-wise) GRT (guided random + light static analysis) JTexpert (guided random + light static analysis) Mosa (evolutionary algorithm + light static analysis) Evosuite (evolutionary algorithm + light static analysis + memetic) Commercial (secret) 6

7. From Unit Tetsing Tool Competition 2015 Rando op HumanT3GRTJTexpe rt MosaEvosuit e CT cov b 3268576155565526 cov m 2050414531394112 Tgen1.77h23h0.85h4.6h1.55h1.48h6.5h1.01h 7

8. Cov. problem as a search problem 8 Consider again the example program; is the target to cover. The program is extended with “instrumentation” ; for simplicity, fit is a global-var. The problem can be re-expressed to searching an input for tax, such that the value of “fit” is minimized. tax(income) { fit := max(0, 10001 - income) if (income  10000) return 0 ; income := income – 10000 ; fit += min(0, 10001 - income) if (income  10000) return 0.1 * income ; return 1000 + 0.2 * (income – 10000) ; }

9. Hill climbing (actually hill descent here) start with some input vector i adjust i, if this improves fit, repeat the process. stop if no improvement is obtained. you can get stuck in a local-minimum other search-algorithgms, e.g. genetic 9 income fit 1000010001 2000120000 10000

10. Using “symbolic execution” Consider a path p in the program f(x) that pass a certain target node c. The linearized statement that corresponds to p, is the sequence of assignments and program calls along p. We convert all branching conditions along p to the corresponding assert, and in the “right orientation”. let’s ignore loop and exception for now. Suppose A is the wlp of this linearized statement over “true” as the post-condition. For any x satisfying A, executing f(x) will covers c  solving the coverage problem. 10

11. Example an LS covering assert income > 10000 ; tax := 0 ; income := income – 10000 ; assert income > 10000 ; wlp LS true = income > 10000 /\ income > 20000 Any solution of that wlp will cover 11 tax(income | tax) { if (income  10000) tax := 0 ; else{ income := income – 10000 ; if (income  10000) tax := 0.1* income ; else tax := 1000 + 0.1 * (income – 10000) ; }

12. Issues with the wlp-based approach What if there are multiple paths leading to the target c? choose one; catch: the chosen path turns out to be unfeasible. consider them all (the resulting wlp would be just a disjunction of each path’s wlp). Catch: can be wastful. What if the route to c has to pass a loop? you can do [loop] k unrolling, but again with the risk that the unrolled paths turn out to be unfeasible. 12

13. Combined concrete and symbolic execution We execute the program and collect conditions to satisfy. Suppose we start with f(0,0) goes the wrong way at the first guard. Generate the condition “y ≥ 100” suppose we solve the condition with y=100, then we try f(0,100). This goes the wrong way in the second guard. Generate the condition “y ≥ 200”. We now need to solve both “y ≥ 100” and “y ≥ 200”. The solution would now take us to 13 f(x,y) { while x<100 do x++ ; if (y<x) return ; while x<200 do x++ ; if (y<x) return ; }

14. T3 test against a Java class a test-case is a meta representation of a sequence of calls to the class’ methods and constructors a large number of these meta sequences can be generated, executed, and inspected can inspect/verify LTL, Hoare triples, algebraic equations has modules to “mine” specifications 14

15. Experimentation project and thesis projects within automated testing TP: will a combined random testing and wlp-based symbolic execution deliver better performance (coverage and time) than search-based variations? the whole Unit Testing Competition infrastructure is available for study EP: evolutionary and memetic improvements of T3’s meta sequences. 15