1. Random Test Generation of Unit Tests: Randoop Experience
Wujie Zheng

2. Outline Automated Test Generation of Unit Tests
Randoop: Feedback-directed Random Test Generation
Randoop Experience
Potential Improvements

3. Outline Automated Test Generation of Unit Tests
Randoop: Feedback-directed Random Test Generation
Randoop Experience
Potential Improvements

4. Automated Test Generation of Unit Tests
Problem
Given a method of a class, how to automatically generate test cases to execute most of its code, i.e., to achieve high code coverage.

5. Automated Test Generation of Unit Tests
A simple example
class under test: java.util.Stack
method under test:
public synchronized int search(Object o) {
int i = lastIndexOf(o);
if (i >= 0) {
return size() - i; }
return -1;

6. Automated Test Generation of Unit Tests
What need to be generated?
Test Inputs
not only primitive values but also objects, including the receiver object
Test Oracles
Could use some basic contracts
For not much test inputs, manually added by testers

7. Automated Test Generation of Unit Tests
An example test case
// a test case for Stack.search(Object)
// existing (previous generated) sequences that create parameters
Stack var0 = new Stack();
String var1 = "hi!";
var0.push((Object)var1);
// the method under test
int var2 = var0.search(var1);
// test oracles
assertTrue(var0.equals(var0));
assertTrue(var2==1); // Regression assertion

8. Automated Test Generation of Unit Tests
Main solutions
Random Approaches
Randomly create and mutate objects
For a method under test, select objects created in existing sequences as its parameters
Simple, scalable, but may not efficient
Systematic Approaches
Explore the possible paths systematically, for each path, collect the path condition and solve it to generate test inputs
Potential high coverage, but complex
Systematic approaches
Build a connection between inputs and program paths
Rely on constraint solvers: good for primitive values, but may not easy for objects

9. Outline Automated Test Generation of Unit Tests
Randoop: Feedback-directed Random Test Generation
Randoop Experience
Potential Improvements

10. Randoop: Feedback-directed Random Test Generation
The basic idea
Undirected random test generation is not efficient because it does not prune redundant or error objects (generated by duplicate or different method sequences)
Identify redundant or error objects early can prune large search space

11. Randoop: Feedback-directed Random Test Generation
The basic idea
Identify redundant objects
Use the equals() method
Identify error objects
Exceptions.
Exceptions frequently correspond to precondition violations for a method, and therefore there is little point extending them
Null
Null dereference exceptions caused by using null as an argument are often uninteresting
Set the redundant or error objects as not extensible

12. Randoop: Feedback-directed Random Test Generation
Sequence generation for a method
To execute a method m(T1,…,Tk), we need to find existing sequences that generate the parameters
For each Ti
Primitive type, randomly select a value from a pool
Reference type, select a extensible (non-redundant and legal) value v from existing sequences, or use null
Generate a new sequence, including the method m and the required sequences to generate the parameters

13. Randoop: Feedback-directed Random Test Generation
The Randoop algorithm
Repeat until timeout
Randomly select a method m(T1,…,Tk)
Generate a new sequence for m
Execute the new sequence
Check simple oracles, set the extensible flags of the objects
Output the generated sequences as unit tests

14. Outline Automated Test Generation of Unit Tests
Randoop: Feedback-directed Random Test Generation
Randoop Experience
Potential Improvements

15. Randoop Experience Subject programs Interested Metrics Java.util
Berkeley DB
Interested Metrics
Number of tests
Code coverage
Error detection

16. Randoop Experience Experiments on Java.util Demo
7 classes in Java.util, used in other researchers’ experiments
Run Randoop 60 seconds
4834 tests
50.8% instruction coverage
195 failed tests, many false positives!
Demo

17. Randoop Experience Experiments on Berkeley DB Demo
Java edition of Berkeley DB, com.sleepycat.je
Run Randoop 60 seconds
Around 20,000 tests
30% instruction coverage
300 failed tests, many false positives!
Need to prepare some environments, e.g., files in the disk
Can more tests achieve higher coverage?
Run Randoop 120 seconds: out of memory
Demo

18. Outline Automated Test Generation of Unit Tests
Randoop: Feedback-directed Random Test Generation
Randoop Experience
Potential Improvements

19. Potential Improvements
The problems of Randoop
Not enough code coverage
Difficult to generate interesting sequences quickly before running out of memory
Too large test suites
Need to perform test selection (reduction)

20. Thank you!