1. Refactoring to Concurrency (Java Edition)
Prepared for: Boston Java Meetup Group
February 2011
Presented at Coverity; Copyright 2010 Concurrency Magic

2. Copyright 2010, Concurrency Magic
Overview
Step 1: Establish Performance Criteria
Step 2: Annotate Existing Class Thread Safety
Step 3: Increase Thread Safety (as needed)
Step 4: Use Tasks instead of loops/steps
Step 5: Use Thread Pool
Step 6: Increase Concurrency
Copyright 2010, Concurrency Magic

3. Common Concurrency Problems
Data Race Condition
Deadlock
Liveliness
Memory Visibility
And, “Heisenbugs”
POINT: Much can be statically checked
** CDROM MAGIC TRICK **
Race Condition:
Atomic operations
Lock (synchronized)
Spin Locks
Versioning
Lock-free (LoadLinked/StoreConditional or CompareAndSwap)
Dirty read, phantom Read
Lost locks (no release)
Deadlock:
Recovery
Time-out and retry
Lock ordering (multi-tier)
Open Calls
Liveliness:
Starvation, Priority Inversion
Heisenbugs
as opposed to bohrbug
Copyright 2010, Concurrency Magic
Presented at Coverity; Copyright 2010 Concurrency Magic

4. Solving Sudoku Puzzles
Example Problem
Solving Sudoku Puzzles

5. Single Threaded Sudoku (4x43
There are 12 “Houses”
4 Rows
4 Columns
4 Boxes
Each Value (1 - 4) must appear exactly once in each House
Algorithm:
Look for “Singles”
(Cells where only one value is valid) 2 1 1 3 4 2 2 1 3 4 4 2 1 3

6. Concurrent Sudoku: Thread per value1 2 X * 1 X X X * X X
Thread Algorithm:
4 Threads. Each considers one value
Mark which cells cannot have that value
Look for Singles for that value 2 X X X 1 X X 2 X X X 1 X 2 X X 3 4 3 * X X X * 4 X X X 4 X X X 3 X X X X X 4 X X X X X

7. Alternative Algorithm #1: Thread per cell2 1 2 1 2 1 2
Thread Algorithm:
16 threads each consider one cell and its 4 “candidates“
Look at neighbor cells in 3 “houses”
Remove invalid candidates 3 4 3 4 3 4 3 4 1 2 1 2 1 3 3 4 3 4 1 2 1 2 1 2 4 3 4 3 4 3 4 1 2 1 2 1 2 3 3 4 3 4 3 4

8. Alternative Alogorithm #2: Thread per “House”3 4 1 4 2 1 3 4 2 2 3 1 4 3 3 4 1 2 1 4 2 1 4 2 4 2 3 3 4 1 2 1 3 1 4 4 2 1 2 2 1 3 4 2 1

9. Alternative Algorithm #3: Thread per “Candidate”1 2 1 2 1 2 1 2
Thread Algorithm:
64 Threads. Each considers one candidate
Look at neighbor cells in 3 “houses”
Remove if invalid 3 4 3 4 3 4 3 4 1 2 1 2 1 3 3 4 3 4 1 2 1 2 1 2 4 3 4 3 4 3 4 1 2 1 2 1 2 3 3 4 3 4 3 4

10. Copyright 2010, Concurrency Magic
Package Architecture
Copyright 2010, Concurrency Magic

11. Step 1: Establish Criteria
“If you don’t know where you’re going, how will you know when you gotten there?”

12. Success Criteria Examples
Good Examples:
Response time < 1ms for typical queries
Process >100 queries per second
Deterministic Results
Bad Examples:
Keep all processors busy
Keep all threads busy
Avoid waiting for locks
Copyright 2010, Concurrency Magic

13. Copyright 2010, Concurrency Magic
Performance Test
Copyright 2010, Concurrency Magic

14. Timing Java Apps (1st attempt)
Start timer
Run algorithm A
Stop timer
Report elapsed time
Run algorithm B
. . .
Run algorithm Z
Copyright 2010, Concurrency Magic

15. Timing Java Apps (2nd attempt)
Repeat 1000 times
Start timer
Repeat 100 times (with 100 different puzzles)
Run algorithm A
Stop timer
Report min/ave/max times (divided by 100,000)
Run algorithm B
. . .
Copyright 2010, Concurrency Magic

16. Timing Java Apps (3rd attempt)
Repeat 1000 times
Start timer A
Repeat 100 times: Run algorithm A
Stop timer A
Start timer B
Repeat 100 times: Run algorithm B
Stop timer B
. . .
Start timer Z
Repeat 100 times: Run algorithm Z
Stop timer Z
Report min/ave/max (divided by 100,000)
Copyright 2010, Concurrency Magic

17. Timing Java Apps (4th attempt)
Start JVM
Repeat 1000 times
Start timer
Repeat 100 times (with 100 different puzzles)
Run algorithm A
Stop timer
Report min/ave/max times (divided by 100,000)
Stop JVM
Repeat for each algorithm
Copyright 2010, Concurrency Magic

18. Copyright 2010, Concurrency Magic
Statistics Gathering
Best Time
Ave Time
Nickname
Algorithm ms ms
STP
Pencil ms ms
STS
Scanner
Performance results solving 100 puzzles 1000 times. Sample Goal: cut time in half using 4 cores
Copyright 2010, Concurrency Magic

19. Step 2: Annotate Type Safety
Assign Type Safety Annotation to every class that might be used concurrently

20. Concurrency Categories for Classes
Immutable
Thread-Safe
Thread-Friendly
Thread-Neutral
Thread-Hostile
POINT: There are established rules that can be statically checked
Copyright 2010, Concurrency Magic
Presented at Coverity; Copyright 2010 Concurrency Magic

21. New Package: Annotations
Immutable { } ThreadSafe { } ThreadFriendly { } ThreadHostile { } ThreadNeutral { } GuardedBy { String value(); }
Copyright 2010, Concurrency Magic

22. Copyright 2010, Concurrency Magic
Value
@Immutable
public enum Value { UNKNOWN(0, '.'), ONE(1, '1'), TWO2, '2'), FIFTEEN (15, 'F'), SIXTEEN (16, 'G'); final private int number; final private char name; final static private Map<Character,Value> charMap = new HashMap<Character, Value>(); final static private List<Value> intList = new ArrayList<Value>(16); . . .
IMMUTABLE
Copyright 2010, Concurrency Magic
Presented at Coverity; Copyright 2010 Concurrency Magic

23. Copyright 2010, Concurrency Magic
Location
@ThreadNeutral
public class Location { private int row; private int col; private int offset; private int boxSize; . . .
ThreadNeutral
Copyright 2010, Concurrency Magic
Presented at Coverity; Copyright 2010 Concurrency Magic

24. Copyright 2010, Concurrency Magic
ValueSet
@ThreadNeutral
public class ValueSet { private Value value = Value.UNKNOWN; private EnumSet<Value> includedValues; . . .
ThreadNeutral
Copyright 2010, Concurrency Magic
Presented at Coverity; Copyright 2010 Concurrency Magic

25. Copyright 2010, Concurrency Magic
Puzzle
@ThreadHostile
public class Puzzle implements Cloneable { public static final List<Integer> SUPPORTED_SIZES = Arrays.asList(4, 9, 16); static int size = 0; static int boxSize = 0; static int gridSize = 0; private static List<List<Location>> rows; private static List<List<Location>> cols; private static List<List<Location>> boxes; private static List<List<Location>> houses; private static List<Location> allLocations; private List<Value> values; . . .
ThreadHostile
Copyright 2010, Concurrency Magic
Presented at Coverity; Copyright 2010 Concurrency Magic

26. Step 3: Increase Thread Safety (as needed)
Modify classes or create new classes with higher levels of atomicity
Copyright 2010, Concurrency Magic

27. Concurrency in Java Your Thread Neutral Code Your Thread Safe Code
Executer, Future, ConcurrentMap
java.util.concurrent
CountDownLatch, Semaphore
java.util.concurrent
AtomicInteger, AtomicLongArray
java.util.concurrent.atomic
Lock (ReentrantLock), Condition
java.util.concurrent.lock
POINT: As of 1.5, Java has firm foundation for MT
Spend whatever time is left on each entry
“Happens Before”
Instruction Re-ordering (CPU)
Instruction Re-ordering (compiler)
Cache coherence
CAS
Load-Link/Store-Conditional
ABA Problem
ATOMIC
public final boolean weakCompareAndSet(int expect, int update)
AtomicBoolean, AtomicInteger, AtomicLong, AtomicReference<V>
AtomicIntegerArray, AtomicLongArray, AtomicReferenceArray<E>
AtomicIntegerFieldUpdater<T>, AtomicLongFieldUpdater<T>, AtomicReferenceFieldUpdater<T,V> (reflection-based)
AtomicMarkableReference<V> (object reference + mark bit using "boxed" [reference, boolean] pairs
AtomicStampedReference<V> (object reference + integer "stamp“ using "boxed" [reference, integer] pairs.)
LOCKS
Interfaces: Condition, Lock, ReadWriteLock
Classes: AbstractQueuedSynchronizer, LockSupport, ReentrantLock, ReentrantReadWriteLock
CONCURRENT INTERFACES
BlockingQueue<E> Queue that waits for non-empty/non-full
Callable<V> Task that returns a result and may throw an exception.
CompletionService<V> Asynchronous tasks
ConcurrentMap<K,V> (with putIfAbsent)
Delayed Mix-in action upon after a given delay.
Executor Executes submitted Runnable tasks.
ExecutorService Executor with Futures
Future<V> Result of an asynchronous computation.
RejectedExecutionHandler For tasks that cannot be executed by a ThreadPoolExecutor.
ScheduledExecutorService ExecutorService that can schedule commands to run after delay or periodically.
ScheduledFuture<V> Delayed result-bearing action
ThreadFactory Creates new threads
CONCURRENT CLASSES
Executors = ExecutorService, ScheduledExecutorService, Future
Implementations: ThreadPoolExecutor, ScheduledThreadPoolExecutor, FutureTask
Queues: ConcurrentLinkedQueue, BlockingQueue, LinkedBlockingQueue, ArrayBlockingQueue, SynchronousQueue, PriorityBlockingQueue, and DelayQueue
Synchronizers: Semaphore, CountDownLatch, CyclicBarrier, Exchanger
Concurrent Collections: ConcurrentHashMap, CopyOnWriteArrayList, and CopyOnWriteArraySet.
Note: java.util.Hashtable and Collections.synchronizedMap(new HashMap()) are synchronized
But ConcurrentHashMap is "concurrent“, thread-safe, but not governed by a single exclusion lock
AbstractQueuedSynchronizer
java.util.concurrent.lock
compareAndSwap, AtomicReference
java.util.concurrent.atomic
“happens before”
Java JVM
Copyright 2010, Concurrency Magic
Presented at Coverity; Copyright 2010 Concurrency Magic

28. Copyright 2010, Concurrency Magic
Location
final public class Location { final private int row; final private int col; final private int offset; final private int boxSize; // Getters but not Setters
Now IMMUTABLE
Copyright 2010, Concurrency Magic
Presented at Coverity; Copyright 2010 Concurrency Magic

29. Copyright 2010, Concurrency Magic
Puzzle Hierarchy
Copyright 2010, Concurrency Magic

30. Copyright 2010, Concurrency Magic
Puzzle (Base Class)
public abstract class Puzzle implements Cloneable { /** List of Puzzle sizes supported (4x4, 9x9 and 16x16). */ public static final List<Integer> SUPPORTED_SIZES = Arrays.asList(4, 9, 16); /** Puzzle table used when this Puzzle was created. */ protected final PuzzleTables tables; /** Size of this Puzzle (4, 9 or 16). */ protected final int size;
Copyright 2010, Concurrency Magic

31. Basic Puzzle (Thread Neutral)
@ThreadNeutral public final class BasicPuzzle extends Puzzle { /** Table containing current Values for each cell in grid. */ private final List<Value> values;
Copyright 2010, Concurrency Magic

32. Copyright 2010, Concurrency Magic
BasicPuzzle get/set
@Override public Value getValue (Location loc) { return values.get(loc.getOffset()); } public void setValue (Location loc, Value v) { values.set(loc.getOffset(), v);
Copyright 2010, Concurrency Magic

33. Copyright 2010, Concurrency Magic
SynchronizedPuzzle
@ThreadSafe public class SynchronizedPuzzle extends Puzzle private final List<Value> public synchronized Value getValue (Location loc) { return values.get(loc.getOffset()); } public synchronized void setValue (Location loc, Value v) { values.set(loc.getOffset(), v);
Copyright 2010, Concurrency Magic

34. LockingPuzzle.LockableValue
@ThreadSafe public final class LockingPuzzle extends Puzzle { /** Value that can be locked */ private class LockableValue { private Value value; public LockableValue() { value = Value.UNKNOWN;} public synchronized Value get() { return value; } public synchronized void set(Value v) { value = v; } }
Copyright 2010, Concurrency Magic

35. LockingPuzzle get/set
@GuardedBy("itself") private final List<LockableValue> public Value getValue (Location loc) { return values.get(loc.getOffset()).get(); } public void setValue (Location loc, Value v) { values.get(loc.getOffset()).set(v);
Copyright 2010, Concurrency Magic

36. Copyright 2010, Concurrency Magic
AtomicPuzzle
@ThreadSafe public final class AtomicPuzzle extends Puzzle { AtomicReferenceArray<Value> public Value getValue (Location loc) { return values.get(loc.getOffset()); } public void setValue (Location loc, Value v) { values.set(loc.getOffset(), v); . . .
Copyright 2010, Concurrency Magic

37. Copyright 2010, Concurrency Magic
ImmutablePuzzle
@Immutable public final class ImmutablePuzzle extends Puzzle { private final List<Value> public Value getValue (Location loc) { return values.get(loc.getOffset()); } public void setValue (Location loc, Value v) { throw new IllegalArgumentException( "setValue is not implemented for ImmutablePuzzle");
Copyright 2010, Concurrency Magic

38. New class: GenericScanSolver
public class ScanSolver extends Solver { /** Puzzle currently being solved. */ final BasicPuzzle puzzle; ... public class GenericScanSolver extends Solver { final Puzzle puzzle;
Copyright 2010, Concurrency Magic

39. Use SynchronizedPuzzle
ST_SCANNER_WTH_SYNCHRONIZED_PUZZLE("STS.S") { public String getName() { return "ST Scanner with SynchronizedPuzzle"; } public Solver newSolver(Puzzle input, Verbosity verbosity) { Puzzle p = SynchronizedPuzzle.makeCopy(input); Solver s = GenericScanSolver.newSolver(p, verbosity); return s; } },
Copyright 2010, Concurrency Magic

40. Copyright 2010, Concurrency Magic
Use LockingPuzzle
ST_SCANNER_WTH_LOCKING_PUZZLE("STS.L") { public String getName() { return "ST Scanner with LockingPuzzle"; } public Solver newSolver(Puzzle input, Verbosity verbosity) { Puzzle p = LockingPuzzle.makeCopy(input); Solver s = GenericScanSolver.newSolver(p, verbosity); return s; } },
Copyright 2010, Concurrency Magic

41. Step 4: Use Tasks Convert loops and/or separate steps into Tasks.
Execute Tasks from a Queue.
(Still Single Threaded)
Copyright 2010, Concurrency Magic

42. Copyright 2010, Concurrency Magic
Creating Executor
private boolean runAllTests() { // Create pool of threads to execute tasks ExecutorService exec = Executors.newSingleThreadExecutor(); try { for (PuzzleResults r : results) { if (runOneTest(exec, r) != true) { // Report error return false; } } finally { exec.shutdown(); return true;
Copyright 2010, Concurrency Magic

43. Pass (unused) Executor
public static Solver newSolver(Executor exec, Puzzle input, Verbosity verbosity) { return new ScanSolver(input, verbosity); }
Copyright 2010, Concurrency Magic

44. Using Tasks in seekHiddenSingles
protected void seekHiddenSingles() { final CountDownLatch latch = new CountDownLatch(unsolvedValues.count()); for (final Value v : unsolvedValues.getAllValues()) { Runnable task = new Runnable() { public void run() { boolean foundAll = seekOneHiddenSingle(v); if (foundAll) { unsolvedValues.removeValue(v); } latch.countDown(); }; exec.execute(task); try { latch.await(); ... Error handling
Copyright 2010, Concurrency Magic

45. Replace Single Threaded Executor with Multi-threaded Executor
Step 5: Use a Thread Pool
Replace Single Threaded Executor with Multi-threaded Executor
Copyright 2010, Concurrency Magic

46. Copyright 2010, Concurrency Magic
Creating Thread Pool
private boolean runAllTests() { // Create pool of threads to execute tasks ExecutorService exec = Executors.newFixedThreadPool(numberOfThreads); try { for (PuzzleResults r : results) { if (runOneTest(exec, r) != true) { // Report error return false; } } finally { exec.shutdown(); return true;
Copyright 2010, Concurrency Magic

47. Step 6: Increase Concurrency
In this case,
rather than solve one Puzzle in parallel,
solve different Puzzles in parallel
Copyright 2010, Concurrency Magic

48. Concurrent Sudoku #5: Thread per puzzle3 4 3 1 3 4 2 4 1 2 3 4 1 2 1 4 1 2 1 2 3 4 4 3 2 2 4 1 2 1 3 2 1 3 2 3 4 4 3 3 4 2 1 4 1

49. Task to Solve one Puzzle
class PuzzleSolver implements Callable<Solvability> { private final SolverType solverType; private final PuzzleInfo info; public PuzzleSolver (SolverType s, PuzzleInfo info) { … } public Solvability call () { … } }
Copyright 2010, Concurrency Magic

50. Submit Tasks to Executor
protected boolean solveOneSetOfPuzzles(SolverType s) { ExecutorService exec = Executors.newSingleThreadExecutor(); for (PuzzleInfo info : puzzles) { Callable<Solvability> task = new PuzzleSolver(s, info); exec.submit(task); } exec.shutdown(); // Error handling
Copyright 2010, Concurrency Magic

51. Copyright 2010, Concurrency Magic
Use a Thread Pool
protected boolean solveOneSetOfPuzzles(SolverType s) { // ExecutorService exec = // Executors.newSingleThreadExecutor(); ExecutorService exec = Executors.newFixedThreadPool(numberOfThreads);
Copyright 2010, Concurrency Magic

52. Copyright 2010, Concurrency Magic
Summary
Step 1: Establish Performance Criteria
Step 2: Annotate Existing Class Thread Safety
Step 3: Increase Thread Safety (as needed)
Step 4: Use Tasks instead of loops/steps
Step 5: Use Thread Pool
Step 6: Increase Concurrency
Copyright 2010, Concurrency Magic