1. Testing Concurrent Programs
Rice Computer Science Club
Mathias Ricken
Rice University
October 4, 2007

2. Moore’s Law *

3. *

4. Timeliness CPU clock frequencies stagnate
Multi-Core CPUs provide additional processing power
Multiple threads needed to use multiple cores
Writing concurrent programs is difficult!

5. Programming Examples

6. Unit Testing Unit tests… Effective with a single thread of control
Test a part, not the whole program
Occur earlier
Automate testing
Serve as documentation
Prevent bugs from reoccurring
Help keep the shared repository clean
Effective with a single thread of control

7. Foundation of Unit Testing
Unit tests depend on deterministic behavior
Known input, expected output… Success  correct behavior Failure  flawed code
Outcome of test is meaningful

8. Problems Due to Concurrency
Thread scheduling is nondeterministic and machine-dependent
Code may be executed under different schedules
Different schedules may produce different results
Known input, expected output… Success  correct behavior in this schedule, may be flawed in other schedule Failure  flawed code
Success of unit test is meaningless

9. Possible Solutions Programming Language Features Lock-Free Algorithms
Ensuring that bad things cannot happen
May restrict programmers
Lock-Free Algorithms
Ensuring that if bad things happen, it’s ok
May limit data structures available
Comprehensive Testing
Testing if bad things happen in any schedule
Does not prevent problems, but does not limit solutions either

10. Contributions Improvements to JUnit
Detect exceptions and failed assertions in threads other than the main thread
Annotations for Concurrency Invariants
Express complicated requirements about locks and threads
Tools for Schedule-Based Execution
Record, deadlock monitor
Random delays, random yields

11. Improvements to JUnit Uncaught exceptions and failed assertions
Not caught in child threads

12. Sample JUnit Tests public class Test extends TestCase {
public void testException() {
throw new RuntimeException("booh!"); }
public void testAssertion() {
assertEquals(0, 1); }
Both tests fail.
Both tests fail.
if (0!=1)
throw new AssertionFailedError();

13. Problematic JUnit Tests
Main thread
public class Test extends TestCase {
public void testException() {
new Thread(new Runnable() {
public void run() {
throw new RuntimeException("booh!"); }
}).start();
new Thread(new Runnable() {
public void run() {
throw new RuntimeException("booh!"); }
}).start();
throw new RuntimeException("booh!");
Child thread
Main thread
Child thread
spawns
uncaught!
end of test
success!

14. Problematic JUnit Tests
Main thread
public class Test extends TestCase {
public void testException() {
new Thread(new Runnable() {
public void run() {
throw new RuntimeException("booh!"); }
}).start();
new Thread(new Runnable() {
public void run() {
throw new RuntimeException("booh!"); }
}).start();
throw new RuntimeException("booh!");
Child thread
Uncaught exception, test should fail but does not!

15. Improvements to JUnit Uncaught exceptions and failed assertions
Not caught in child threads
Thread group with exception handler
JUnit test runs in a separate thread, not main thread
Child threads are created in same thread group
When test ends, check if handler was invoked

16. Thread Group for JUnit Tests
Test thread
public class Test extends TestCase {
public void testException() {
new Thread(new Runnable() {
public void run() {
throw new RuntimeException("booh!"); }
}).start();
new Thread(new Runnable() {
public void run() {
throw new RuntimeException("booh!"); }
}).start();
throw new RuntimeException("booh!");
Child thread
invokes
checks
TestGroup’s Uncaught Exception Handler

17. Thread Group for JUnit Tests
Test thread
public class Test extends TestCase {
public void testException() {
new Thread(new Runnable() {
public void run() {
throw new RuntimeException("booh!"); }
}).start();
new Thread(new Runnable() {
public void run() {
throw new RuntimeException("booh!"); }
}).start();
throw new RuntimeException("booh!");
Child thread
spawns and waits
resumes
Main thread
failure!
check group’s handler
spawns
end of test
Test thread
uncaught!
invokes group’s handler
Child thread

18. Improvements to JUnit Uncaught exceptions and failed assertions
Not caught in child threads
Thread group with exception handler
JUnit test runs in a separate thread, not main thread
Child threads are created in same thread group
When test ends, check if handler was invoked
Detection of uncaught exceptions and failed assertions in child threads that occurred before test’s end
Past tense: occurred!

19. Child Thread Outlives Parent
Test thread
public class Test extends TestCase {
public void testException() {
new Thread(new Runnable() {
public void run() {
throw new RuntimeException("booh!"); }
}).start();
new Thread(new Runnable() {
public void run() {
throw new RuntimeException("booh!"); }
}).start();
throw new RuntimeException("booh!");
Child thread
spawns and waits
resumes
Main thread
failure!
check group’s handler
spawns
end of test
Test thread
uncaught!
invokes group’s handler
Child thread

20. Child Thread Outlives Parent
Test thread
public class Test extends TestCase {
public void testException() {
new Thread(new Runnable() {
public void run() {
throw new RuntimeException("booh!"); }
}).start();
new Thread(new Runnable() {
public void run() {
throw new RuntimeException("booh!"); }
}).start();
throw new RuntimeException("booh!");
Child thread
check group’s handler
spawns and waits
resumes
Main thread
success!
spawns
Too late!
Test thread
end of test
uncaught!
invokes group’s handler
Child thread

21. Improvements to JUnit Child threads are not required to terminate
A test may pass before an error is reached
Detect if any child threads are still alive
Declare failure if test thread has not waited
Ignore daemon threads, system threads (AWT, RMI, garbage collection, etc.)
Previous schedule is a test failure
Should be prevented by using Thread.join()

22. Enforced Join public class Test extends TestCase {
Test thread
public class Test extends TestCase {
public void testException() {
new Thread(new Runnable() {
public void run() {
throw new RuntimeException("booh!"); }
});
t.start(); … t.join();
Thread t = new Thread(new Runnable() {
public void run() {
throw new RuntimeException("booh!"); }
});
t.start(); … t.join(); …
throw new RuntimeException("booh!");
Child thread

23. Improvements to JUnit Child threads are not required to terminate
A test may pass before an error is reached
Detect if any child threads are still alive
Declare failure if test thread has not waited
Ignore daemon threads, system threads (AWT, RMI, garbage collection, etc.)
Previous schedule is a test failure
Should be prevented by using Thread.join()

24. Testing ConcJUnit
Replacement for junit.jar or as plugin JAR for JUnit 4.2
Available as binary and source at
Results from DrJava’s unit tests
Child thread for communication with slave VM still alive in test
Several reader and writer threads still alive in low level test (calls to join() missing)

25. Conclusion Improved JUnit now detects problems in other threads
Only in chosen schedule
Needs schedule-based execution
Annotations ease documentation and checking of concurrency invariants
Open-source library of Java API invariants
Support programs for schedule-based execution

26. Future Work Schedule-Based Execution
Replay given schedule
Generate possible schedules
Dynamic race detection
Probabilities/durations for random yields/sleeps
Extend annotations to Floyd-Hoare logic
Preconditions, postconditions
Representation invariants

27. Many Thanks To… My advisor My committee members NFS and Texas ATP
Corky Cartwright
My committee members
Walid Taha
Bill Scherer
NFS and Texas ATP
For partially providing funding
Rice Computer Science Club

28. Extra Slides

29. Tractability of Comprehensive Testing
Test all possible schedules
Concurrent unit tests meaningful again
Number of schedules (N)
t: # of threads, s: # of slices per thread
detail

30. Extra: Number of Schedules
Product of s-combinations
For thread 1: choose s out of ts time slices
For thread 2: choose s out of ts-s time slices …
For thread t-1: choose s out of 2s time slices
For thread t-1: choose s out of s time slices
Writing s-combinations using factorial
Cancel out terms in denominator and next numerator
Left with (ts)! in numerator and t numerators with s!
back

31. Tractability of Comprehensive Testing
If program is race-free, we do not have to simulate all thread switches
Threads interfere only at “critical points”: lock operations, shared or volatile variables, etc.
Code between critical points cannot affect outcome
Simulate all possible arrangements of blocks delimited by critical points
Run dynamic race detection in parallel
Lockset algorithm (e.g. Eraser by Savage et al)

32. Critical Points Example
Local Var 1
All accesses protected by lock
lock access unlock
lock access unlock
Thread 1
Shared Var
Lock
Local variables don’t need locking
All accesses protected by lock
All accesses protected by lock
Thread 2
lock access unlock
Local Var 1

33. Fewer Schedules Fewer critical points than thread switches
Reduces number of schedules
Example: Two threads, but no communication  N = 1
Unit tests are small
Hopefully comprehensive simulation is tractable
If not, heuristics are still better than nothing

34. Limitations Improvements only check chosen schedule
A different schedule may still fail
Requires comprehensive testing to be meaningful
May still miss uncaught exceptions
Specify absolute parent thread group, not relative
Cannot detect uncaught exceptions in a program’s uncaught exception handler (JLS limitation)
details

35. Extra: Limitations May still miss uncaught exceptions
Specify absolute parent thread group, not relative (rare)
Koders.com: 913 matches ThreadGroup vs. 49,329 matches for Thread
Cannot detect uncaught exceptions in a program’s uncaught exception handler (JLS limitation)
Koders.com: 32 method definitions for uncaughtException method
back

36. Extra: DrJava Statistics
Unit tests
passed
failed
not run
Invariants
met
% failed
KLOC
“event thread”
2004
736
610 36 90
5116
4161
965
18.83%
107 1
2006
881
34412
30616
3796
11.03
129 99
back

37. Concurrency Invariants
Has to be called in event thread
TableModel, TreeModel
May not be called in event thread
invokeAndWait()
Have to acquire readers/writers lock
AbstractDocument
DrJava’s documents

38. Invariants Difficult to Determine
May be found in
Javadoc comments
Only in internal comments
Whitepapers
Often not documented at all
Errors not immediately evident
Impossible to check automatically

39. Java Annotations
Add invariants as annotations @NotEventThread public static void invokeAndWait( Runnable r) { … }
Process class files
Find uses of annotations
Insert bytecode to check invariants at method beginning

40. Advantages of Annotations
Java Language constructs
Syntax checked by compiler
Easy to apply to part of the program
e.g. when compared to a type system change
Light-weight
Negligible runtime impact if not debugging (slightly bigger class files)
Automatic Checking

41. Predicate Annotations
In annotation definition, specify static boolean Java method
Method must be callable from every context  completely static and public
Data in annotation, method arguments and value of this passed when method invoked

42. Predicate Annotation Example
@PredicateLink(value=Predicates.class,
method="example",
arguments=true)
ExampleAnnotation {
String foo; }
Refers to Predicates.example
Definition

43. Predicate Annotation Example
Usage
public class TestCode {
@ExampleAnnotation(foo="test")
public void test(int param) { … } } …
TestCode t = new TestCode(); t.test(5);
Call

44. Predicate Annotation Example
public class Predicates {
public static boolean example(
Object this0,
int param,
String foo) {
return (foo.length()<param); }

45. Predicate Annotation Example
@PredicateLink(value=Predicates.class,
method="example",
arguments=true)
ExampleAnnotation {
String foo; }
public class TestCode {
@ExampleAnnotation(foo="test")
public void test(int param){…} } …
TestCode t = new TestCode();
t.test(5);
public class Predicates {
public static boolean example(
Object this0,
int param,
String foo) {
return (foo.length()<param); // this0==t, param==5, foo=="test" }

46. Invariant Annotation Library
@OnlyThreadWithName
@NotNullArgument
etc. (ca. 80 annotations)

47. Problem: Multiple Annotations
Java does not allow the same annotation class multiple times
@OnlyThreadWithName("foo")
@OnlyThreadWithName("bar") // error
void testMethod() { … }
Conjunctions, disjunctions and negations?

48. Annotation Subclasses?
Let annotation extend a supertype?
Invariant { }
OnlyThreadWithName
extends Invariant { String name(); }
And extends Invariant {
Invariant[] terms(); }
Subtyping not allowed for annotations

49. Generic Annotations? Write @And as generic annotation?
And<T> {
T[] terms(); }
OnlyThreadWithName {
String name();
Generics not allowed in annotations

50. Work-Around Different meta-annotation, @Combine
@Combine(Combine.Mode.AND)
SeveralNames {
OnlyThreadWithName[] value(); }
@OnlyThreadWithName("bar")})
void testMethod() { … }

51. Combine Annotations May only contain invariant annotations
Predicate annotations
Combine annotations
Arrays of the above
Predicate method automatically generated
Calls member predicate methods
Accumulates using AND, OR or NOT
NOT first negates, then uses AND
Default mode is OR
De Morgan’s Law: NOT (a OR b) = (NOT a) AND (NOT b)

52. Invariant Inheritance
Invariants on a method
Apply to the method and all overriding methods in subclasses
Invariants on a class
Apply to all methods introduced in that class or subclasses
 Methods can have invariants defined elsewhere
All annotations describe requirements for the client (and, due to subclassing, for subclasses)
Allows frameworks to describe requirements
Description “thread-safe” is often wrong

53. Invariant Subtyping
To maintain substitutability, subclasses may not strengthen invariants
Invariants can be modeled as special input parameter
Tuple of invariants (“record” in λ calculus [Pierce])
Subtyping rules for records declare the “wider” record as subtype
In function types, parameter types are contravariant
I0 = {}, I1 = {inv1}, I2 = {inv1,inv2}, I2 <: I1 <: I0
F0 = I0 → ·, F1 = I1 → ·, F2 = I2 → ·, F0 <: F1 <: F2

54. Invariant Subtyping Analyze methods with invariants as parameter
Invariants subtyping: A B C
IA = {}, IB = {inv1}, IC = {inv1,inv2}; IC <: IB <: IA FA = IA → ·, FB = IB → ·, FC = IC → ·; FA <: FB <: FC
Java subtyping: C <: B <: A
class A {
void f()
{ … }; }
class B
extends A {
@Inv1
void f()
{ … }; }
class C
extends B {
@Inv2
void f()
{ … }; }

55. Detection of Subtyping Problems
If Java subtyping and invariant subtyping disagree (A <: B but B A)
Substitutability not maintained
Statically emit warning
Detect if client subclasses do not use framework classes as prescribed
Safer multithreaded frameworks

56. Java API Annotations Started to annotate methods in Java API
30 whole classes, 44 individual methods
Community project at
Anyone can suggest annotations
Vote on suggested annotations
Browse by class or annotation type
Annotations can be extracted into XML
Share annotations
Add checks without needing source code

57. Testing Invariant Checker
Annotated two DrJava versions
3/26/2004
9/2/2006
Ran test suite, logged invariant violations
2004: 18.83% failed
2006: 11.03% failed
2006 version easier to annotate
Better documentation of invariants