1. Learning based Software Testing --Marriage between “learning” and “testing” Dan Hao (haodan@pku.edu.cn) Peking University 2015.5.28

2. About Me Associate Professor, Peking University Education Background – 2002.9-2008.1, Peking University, Ph.D. – 1998.9-2002.7, Harbin Institute of Technology, B.S. Research Interest: Software Testing 2 Homepage: sei.pku.edu.cn/~haod

3. Software contains bugs 3

4. Software Testing Software Everywhere SOFTWARE 4

5. Simplified Software-Testing Process SUT Test Input Expected Output Execute Actual Output Compare revealed faultsno revealed faults Test Case Test Oracle 5

6. Important Problems in Software Testing Test Oracle Generation Test Input Generation Test Repair Test Selection/reducti on/prioritization …… 6

7. Important Problems in Software Testing Test Oracle Generation Test Input Generation Test Repair Test Selection/reducti on/prioritization …… How to solve these problems? Program Analysis, Machine Learning, Searching Algorithms, … 7

8. Learning --- Outsider’s perspective Machine Learning – learn rules from history data – apply rules to new data Search-based Algorithms – find the optimum quickly in the solution space …… Learning Algorithms 8

9. Marriage between Learning & Testing Learning Algorithms 9

10. Marriage between Learning & Testing Learning Algorithms search-based test generation search-based test prioritization/reduction/selection automated program repair machine learning based bug prediction …… 10

11. Marriage between Learning & Testing Learning Algorithms 11

12. Our Work in the Marriage between Learning & Testing Learning Algorithms Test oracle generation[ASE14] Obsolete test identification[ECOOP12] Test effectiveness measurement[ISSTA16] …… 12

13. In: Test Oracle Generation[ASE14] Metamorphic relation inference + PSO What is the test oracle problem? Test oracles are widely recognized as a difficult problem!!! 13 [ASE14] Jie Zhang, Junjie Chen, Dan Hao, Yingfei Xiong, Bing Xie, Lu Zhang, Hong Mei, Search- based Inference of Polynomial Metamorphic Relations, ASE 2014.

14. Metamorphic Relation --- A specific type of oracles MR: a particular change to the input “changes” the output 14 R I (I 1,I 2 )=> R O (O 1,O 2 ) I1I1 I2I2 O1O1 O2O2

15. Statistics on Metamorphic Relations 60% 50% 1-MR 2-MR 15 R I (I 1,I 2 )=> R O (O 1,O 2 )

16. 1-MR 2-MR 16 R I (I 1,I 2 )=> R O (O 1,O 2 ) R I : I 2 =αI 1 +β R O : c 1 O 1 +c 2 O 2 +c 3 =0 P(I 1 ) P(αI 1 +β) c 1 P(I 1 )+c 2 P(αI 1 +β)+c 3 =0 c 1 P 2 (I 1 )+c 2 P 2 (αI 1 +β)+c 3 P(I 1 )P(αI 1 +β)+c 4 P (I 1 ) +c 5 P(αI 1 +β)+c 6 =0

17. PSO  MR Inference PSO algorithm: – Each candidate solution is called a particle – Each particle has a velocity and a location which keep changing – A fitness function is used to evaluate how close the location of a particle is to an optimal location PSO algorithm (Particle Swarm Optimization) simulating the birds foraging behavior Why PSO : Very effective to search in continuous space Can lead a particle to escape local optimal locations Simple, no many parameters to adjust 17 1-MR: c 1 P(I 1 )+c 2 P(αI 1 +β)+c 3 =0 2-MR: c 1 P 2 (I 1 )+c 2 P 2 (αI 1 +β)+c 3 P(I 1 )P(αI 1 +β)+c 4 P (I 1 )+c 5 P(αI 1 +β)+c 6 =0

18. YN Y N a.In the beginning, the N particles are assigned with locations L (initial values of the parameters). b.The N particles keep updating their velocities and locations. c.When reaching the termination threshold, over. PSO based MR Inference 18

19. MR Filtering 19 1-MR: c 1 P(I 1 )+c 2 P(αI 1 +β)+c 3 =0 2-MR: c 1 P 2 (I 1 )+c 2 P 2 (αI 1 +β)+c 3 P(I 1 )P(αI 1 +β)+c 4 P (I 1 )+c 5 P(αI 1 +β)+c 6 =0

20. Evaluation Results(1/2) 20 Applied to 189 scientific functions of JDK, Apache, Matlab, and GSL

21. In: Obsolete Test Identification[ECOOP12] Obsolete Tests public class Testcases Account a; protected void setUp() a=new Account(100.0,"user1"); protected void tearDown() public void test1() a.transfer(50.0,"user2"); a.withdraw(40.0); assertEquals(9.5,a.getBalance()); public void test2() a.withdraw(40.0); assertEquals(56,a.getBalance();//should be 60... P T P’ 21 [ECOOP12] Dan Hao,Tian Lan, Hongyu Zhang, Chao Guo, Lu Zhang, Is This a Bug or an Obsolete Test? ECOOP 2012.

22. Problem Description Obsolete test identification Given a failing execution, is it caused by a bug in the source code or an obsolete test case? Importance – Without knowing the cause of a failure, how to decide whether repairing a test [Daniel:ASE09,Daniel:ISSTA10] or debugging in the source code [Jones: ASE06,Liblit: PLDI03,Weimer:ICSE09,Kim: ICSE13] ? Krishna Ratakonda (IBM Fellow @FOSE2014) “Determining this reason for failure is the critical first step before any corrective action can be taken…” 22

23. Best-First Decision Tree Algorithm  Obsolete Test Identification Binary classification problem (T v.s. P’) public class Testcases Account a; protected void setUp() a=new Account(100.0,"user1"); protected void tearDown() public void test1() a.transfer(50.0,"user2"); a.withdraw(40.0); assertEquals(9.5,a.getBalance()); public void test2() a.withdraw(40.0); assertEquals(56,a.getBalance();//should be 60... T P’ 23

24. Learning based Obsolete Test Identification failure- inducing test collection feature collection classifier building 24

25. Features Complexity Features Change Features Testing Features Maximum depth of the call graph Number of methods in the call graph File change Type of failure Count of plausible nodes in the graph Existence of highly fault-prone node Product innocence 25

26. Evaluation Results within the same version between versions across projects Effective when being applied within the same versions, or between versions 26

27. In: Test Effectiveness Measurement[ISSTA16] Mutation Testing: – Whether a mutant is killed by a test suite – Mutation score 1 begin 2int x,y; 3input(x,y); 4if(x<y) 5 output(x+y); 6else 7 output(x*y); 8end 1 begin 2int x,y; 3input(x,y); 4if(x<=y) 5 output(x+y); 6else 7 output(x*y); 8end Program Mutant 27 [ISSTA16] Jie Zhang, Yiling Lou, Lingming Zhang, Dan Hao, Lu Zhang, Hong Mei, Predictive Mutation Testing, ISSTA 2016.

28. Challenge in Mutation Testing Costly – E.g., 512LOC Program has 23848 mutants Mutant generation Mutant execution Literature – Do Fewer – Do Faster 28 Don’t run at all

29. Predictive Mutation Testing Mutation testing results – Whether a mutant is killed by a test suite – Mutation score: percentage of killed mutants Prediction: Whether a mutant is killed or survived 29

30. Random Forest  Mutation Testing Binary classification problem (killed v.s. survived) 1 begin 2int x,y; 3input(x,y); 4if(x<y) 5 output(x+y); 6else 7 output(x*y); 8end 1 begin 2int x,y; 3input(x,y); 4if(x<=y) 5 output(x+y); 6else 7 output(x*y); 8end Mutant Program Whether a test kills a mutant? PIE Theory ： Propagation Infection Execution 30

31. Features 31

32. Evaluation Results Besides, in the cross-version scenario, the precision is mostly about 90%. 32

33. Evaluation Results Besides, in the cross-version scenario, the precision is mostly about 90%. Prediction results with high precision Make prediction quickly 33

34. Commonality Analysis Test Oracle Generation Obsolete Test Identification Test Effectiveness Measurement Sampling Learning Algorithms PSO Decision-Tree Random Forest 34

35. Learning-based Software Testing Learning Algorithms 35 Problems: test generation test-execution optimization defect prediction bug fixing …… Algorithms: genetic algorithms PSO hill climbing random forest ……

36. Challenges in Learning-based Software Testing 36 Testing Perspective: Transform a testing problem into a typical learning problem Learning Perspective: Design of fitness function Influence of Imbalance data Choice of algorithms and parameter values ……

37. Summary 37

38. Thanks!