1. Concolic Modularity Testing Derrick Coetzee University of California, Berkeley CS 265 Final Project Presentation

2. Concolic Testing A white-box improvement on random testing Observe how inputs are used at runtime Compute new inputs designed to force program to execute a different path Effective in testing program correctness on a variety of inputs

3. More than correctness Analogy with type systems: – Some types prevent bad inputs – Some types prevent abstraction violations Can concolic testing also detect abstraction violations? – Detect implementation dependence – Find portability issues – Help enforce modularity

4. Detect implementation dependence Forgetting to check for error return codes – e.g. Assuming malloc() never returns NULL Relying on the behavior of a particular implementation – e.g. Assuming result of malloc() is always word- aligned Not coping with improbable events – e.g. Assuming hash functions always return distinct values on distinct inputs.

5. A simple approach Introduce a new symbolic variable whenever the function is invoked Depending on its value, modify the result void* my_malloc(size_t size) { void* result = malloc(size); int simulate_out_of_memory; CREST_int(simulate_out_of_memory); if (simulate_out_of_memory) { result = NULL; } return result; }

6. A simple approach Doesn’t always work – May lead to an infeasible sequence of results int my_hash(char* s) { int result; CREST_int(result); return result; } my_hash("hello") → 0 /* OK */ my_hash("world") → 0 /* OK */ my_hash("hello") → 1 /* Not OK */

7. Generalizing the approach In general, the behavior of functions is constrained by their specification – e.g. Hash functions must be pure With concolic testing, invariants can be enforced directly in the source language – e.g. By adding an input/output history

8. Generalizing the approach string_int_map history; int my_hash(char* s) { if (contains_key(history, s)) { return lookup(history, s); } else { int result; CREST_int(result); add_pair(history, s, result); return result; }

9. A more complex example A sort function invokes a comparator (≤,=) Comparator is constrained to be a total order – a = a – if a = b then b = a – if a = b and b = c then a = c – if a ≤ b and b ≤ a then a = b – if a ≤ b and b ≤ c then a ≤ c Can use input/output history to enforce all these properties in the source language

10. Liveness properties Specifications may constrain sequences of operations – e.g. if malloc() returns NULL, it will continue to return NULL for arguments of the same size or larger until free() is called Easy to support by maintaining state across calls – May even help reduce the search space

11. Experiments: malloc test Took the CREST grep 2.2 example Instrumented all malloc() calls – Possibly return NULL CREST enumerated 5621 tests – 2 were real bugs – Adding the liveness constraint: Reduced the number of tests to 2869 (49% less) Still found the 2 bugs

12. Experiments: sort Selection sort using a comparator function The comparator was constrained only to represent a total order For a list of 5 elements, CREST enumerated 541 different comparators If any of the constraints are removed, CREST finds an explicit comparator function that causes the sort to fail

13. Experiments: hash grep 2.2 uses an internal hash function for fast state matching Constrained hash function only to be pure Enumerated 1472 tests on test input – grep was robust against hash collisions – Removing pure constraint did not produce errors but produced extra states