1. Applications of Synchronization Coverage A.Bron,E.Farchi, Y.Magid,Y.Nir,S.Ur Tehila Mayzels 1

2. Outline o Motivation o Introducting Coverage o Good Coverage Model o Synchronized Coverage Model o ConTest Implementation o How does it help? o Summary & Conclusions 2

3. Motivation Software Development Life Cycle Design Implementation (Coding ) TestingDeployment Test plan Implementation Regression execution Debugging Code fixing Verify? 3

4. Testing - Issues o Where to focus? o Should I add more tests? o Are there irrelevant tests? o When to stop testing? o Is it good enough? 4

5. Introducting Coverage Definition: A metric for measuring what percentage of code/functionality has been exercised by a test suite (regression suite). For a given system, define a set of testing tasks. For each task, check if it was actually performed ( covered ) in some test(s). Measure the fraction of tasks performed. Helps to: o Find uncovered code. o Create good regression suite. o Test the quality of the testing. o Guide code review. 5

6. Sequential Coverage Metrics 1. Control flow based: o Statement coverage – most popular o Branch coverage 2. Data flow based: o Define-Use 3. Input based 4. Specification based 6

7. Characteristics Of a Good Coverage Model o Tasks are statically generated from the code and are well understood by the user. o Almost all tasks must be coverable (or at least a simple review is possible for the cases in which they are not). o Uncovered tasks should yield an action item. o Some action is taken upon 100% coverage. 7

8. Moving to Concurrent World… Testing is much more complicated: o Non determinism: difficult to debug and replay. o Sequential coverage models are not useful for interleaving space. o Bugs appear only in specific configurations: difficult and expensive to find, and worse – commonly found by users. 8

9. Example – How much is 1+10+100+1000? static final int NUM = 4; static int Global = 0; public static void main(String[] argv) { Global = 0; threads[0] = new Adder(1); threads[1] = new Adder(10); threads[2] = new Adder(100); threads[3] = new Adder(1000); // Start Threads for(int i=0;i<NUM;i++) { threads[i].start(); } try{ Thread.sleep(1000); } catch(Exception exc) {} System.out.println(Global); } class Adder extends Thread { int Local; Adder(int i){ Local = i; { public void add() { example.Global += Local; } public void run() { add(); } Based on an example from Testing and Debugging Concurrent Software by Shmuel Ur 9

10. Example – Cont. add method Java Source code public void add() { example.Global += Local; { Byte Code Method void add() 0 getstatic #3 3 aload_0 4 getfield #2 7 iadd 8 putstatic #3 11 return 10

11. Synchronization Coverage: 1 st Attempt Concurrent pair of events: Piece of code1 Piece of code2 execution False – executed by the same thread True – executed by different threads Problems: 1.Too hard – too many tasks, difficult to cover and not very important. 2.Analysis is not automatic – which tasks are coverable? 11

12. Synchronization Coverage: 2 nd Attempt Shared variables Var X Problem – like before: Analysis is not automatic – which tasks are coverable? 12

13. Synchronization Coverage: The Chosen Attempt o Looking for interesting situations. o Handle statements with synchronization primitives. Con current Test ing Tool 13

14. ConTest Implementation o Supports Java and C/Pthread. o The code is instrumented, a list of the tasks is computed. o In test runtime, it keeps a representation of the synchronization objects, e.g., in order to know when a lock is held. o Noise injection mechanisms (yields and sleeps) help obtaining the tasks. o Can be adopted for any set of synchronization primitives. o Task definition for each primitive depends on semantics. 14

15. Mutual Exclusion – synchronize (most important) Blocked – A thread waited there, because another thread held the lock. Blocking - A thread holding the lock there caused another thread to wait. Example: synchronized(lock1) { counter++; updateDB(); } synchronized(lock1) { counter--; updateDB(); } blocking blocked blocking blocked 15

16. Mutual Exclusion – synchronize (most important) Blocked – A thread waited there, because another thread held the lock. Blocking - A thread holding the lock there caused another thread to wait. Example: synchronized(lock1) { counter++; updateDB(); } synchronized(lock1) { counter--; updateDB(); } blocking blocked blocking blocked 16

17. Mutual Exclusion – tryLock() Failed – Another thread holds the lock. Succeed – Succeeded to get the lock. Lock lock =...; if (lock.tryLock()) { try { // manipulate protected state } finally { lock.unlock(); } } else { // perform alternative actions } 17

18. Wait On Condition – wait() Repeated – wait was called at least twice in the same block, in the same run, by the same thread. public synchronized void foo() { while(!cond) { try { wait(); } catch (InterruptedException e) {} } System.out.println(cond happened!); } 18

19. Semaphore wait – acquire() Blocked – The semaphore was not immediately available. Not-blocked – The semaphore was immediately available. private Semaphore writeLock = new Semaphore(1); public void getWriteLock() throws InterruptedException { writeLock.acquire(); } public void releaseWriteLock() { writeLock.release(); } 19

20. Semaphore-try-wait – tryAcquire() Succeeded – Succeeded to get the semaphore. Failed – Another thread holds the semaphore. 20

21. Notify – notify()/notifyAll()/ signal()/signalAll() Had-target – There was another thread in a corresponding wait. Had-no-target – There was no other thread in a corresponding wait. Do you remember the Lost Notify bug from Kerens lecture? What will it cover here? 21

22. More synchronization tasks? Interrupted task for wait(): Ways to stop wait(): Notify Interrupt ( if in use ) Timeout Spuriously in some systems 22

23. Which task cannot be covered here? public static void main (String args[]) {... synchronized (lock) { new Thread(...).start();... } 23

24. Coverage Requirements o Tasks are statically generated from the code and are well understood by the user. Each synchronization primitives leads one or two tasks. Understanding is less trivial than for statement coverage, but not too difficult for a concurrent programmer. 24

25. Coverage Requirements o Almost all tasks must be coverable Exceptions are rare. A synchronized block is written in order to make threads wait. If it cant happen, perhaps its redundant. If it can – maybe its because the test is complicated. 25

26. Coverage Requirements o Uncovered tasks should yield an action item. If the synchronization is redundant, remove it. Otherwise, some interleaving scenario has not been tested. Strengthen the test. If the task cannot be covered, remove the task. 26

27. Coverage Requirements o Some action is taken upon 100% coverage This can be one of the criteria for exiting the unit test when testing a concurrent program. 27

28. How Does It Help? Guiding Review Dead code Synchronization statement is unnecessary Missing tests Synchronization statement is suspected as unnecessary Very hard to test Missing application code required 28

29. And After The Review: Action Items Write more tests Remove synchronization statements Change part of the design Rewrite part of the code Fix bugs!!! Further review 29

30. Summary & Conclusions o Coverage is a useful testing technology. o Verifying concurrent programs is difficult. o Synchronization Coverage helps testing the concurrent code. o ConTest - a concurrent testing tool that helps defining the tasks, running regression suite and gaining 100% coverage. 30

31. 31

32. 32

33. Coverage Results – W/O ConTest Taken from Testing and Debugging Concurrent Software by Shmuel Ur 33

34. Coverage Results – with ConTest Taken from Testing and Debugging Concurrent Software by Shmuel Ur 34