1. 2008 - Nov 1 Deadlock Hunter Final Presentation Software Engineering Lab. Winter 07/08 Supervised by: Yonatan Kaspi Yonatan Kaspi Introduced by: Jamil Shehadeh Husam Khshaiboun

2. 22008 - Nov Agenda Introduction (10 min) Gaston & Alphonso (5 min) The Deadlock Hunter (20 min) Script & Statistics (5 min) One More Example (3 min) Conclusions (2 min)

3. 3 Agenda Introduction (10 min) Gaston & Alphonso (5 min) The Deadlock Hunter (20 min) Script & Statistics (5 min) One More Example (3 min) Conclusions (2 min)

4. 42008 - Nov Introduction Deadlock - Refreshment Deadlock refers to a specific condition when two or more processes are each waiting for another to release a resource, or more than two processes are waiting for resources in a circular chain This situation may be likened to two people who are drawing diagrams, with only one pencil and one ruler between them. Deadlocks are more frequent in multithreaded programs because different threads use the same parent process resources. They are hard to find due to the low probability for the deadlock execution path to occur, since the threads are executed in the same “secure” path most of the time.

5. 52008 - Nov Introduction Deadlock Hunter Project Our goal is to build a tool that will force the operating system scheduler to make the execution path be truly random. We will do this by examining the byte code of a Java program and adding small code sections in places that effect the execution path. Our job is to locate the deadlock, not to fix it.

6. 62008 - Nov Introduction Strategy Dealing with threads in any language can be done using specific commands. Dealing with threads in any language can be done using specific commands. Using java enables us to be Platform independent. Using java enables us to be Platform independent. The specific commands will be searched for in java byte code using a parser. The specific commands will be searched for in java byte code using a parser. Code sections will be added automatically in these places in order to force randomization in the execution path so that the probability for a deadlock to occur increases. Code sections will be added automatically in these places in order to force randomization in the execution path so that the probability for a deadlock to occur increases. A Java Code with a hidden deadlock will be used as a client. A Java Code with a hidden deadlock will be used as a client. The deadlock hunter will build a new instrumented byte code according to the code sections that the parser find interesting. The deadlock hunter will build a new instrumented byte code according to the code sections that the parser find interesting.

7. 72008 - Nov Introduction Apache BCEL BCEL stands for Byte Code Engineering Library Intended to give users a convenient possibility to analyze, create, and manipulate (binary) Java class files. Classes are represented by objects which contain all the symbolic information of the given class: methods, fields and byte code instructions, in particular. Using it we can: Get an abstraction of the byte code Manipulate instruction lists Manipulate methods Look for keywords in the byte code Define new classes

8. 82008 - Nov Introduction Main Obstacles BCEL is a problematic tool Complicated user interface. Complicated user interface. Unintuitive usage. Unintuitive usage. Lets you deal with the corners. Lets you deal with the corners. No debugging ability after code instrumentation. Eclipse debug on a class file is not possible (overwrites the instrumented class file). Eclipse debug on a class file is not possible (overwrites the instrumented class file). Debug with prints only. Debug with prints only. Very few helping sources. Impossible to work continuously Impossible to work continuously Apache manual and FAQ don’t always help Apache manual and FAQ don’t always help Even Google search is not always effective Even Google search is not always effective

9. 92008 - Nov Agenda Introduction (10 min) Gaston & Alphonse (5 min) The Deadlock Hunter (20 min) Script & Statistics (5 min) One More Example (3 min) Conclusions (2 min)

10. 102008 - Nov Gaston & Alphonse A classical deadlock Gaston & Alphonso are two gentle and friendly fellows. They must bow one to the other anytime they meet. Each one of them bows and continue bowing until the other bows back. What if both of them bow at the same time?

11. 112008 - Nov Gaston & Alphonse The Original Code public void bow(Client bowed) { synchronized(this) { System.out.println(bowed.getName()+":"+ this.name +" has bowed to me!"); bowed.bowBack(this); }} public void bowBack(Client bowed) { synchronized(this) { System.out.println(bowed.getName()+":"+ this.name+" has bowed back to me!"); }} public static void main (String[] args) { final Client alphonse = new Client("Alphonse"); final Client gaston = new Client("Gaston"); new Thread(new Runnable() { public void run() { alphonse.bow(gaston); } }).start(); new Thread(new Runnable() { public void run() { gaston.bow(alphonse); } }).start(); }

12. 122008 - Nov Gaston & Alphonse Hide The Deadlock public void bow(Client bowed) { synchronized(this) { System.out.println(bowed.getName()+":"+ this.name +" has bowed to me!"); bowed.bowBack(this); }} public void bowBack(Client bowed) { synchronized(this) { System.out.println(bowed.getName()+":"+ this.name+" has bowed back to me!"); }} public static void main (String[] args) { final Client alphonse = new Client("Alphonse"); final Client gaston = new Client("Gaston"); new Thread(new Runnable() { public void run() { alphonse.bow(gaston); } }).start(); wait(10); new Thread(new Runnable() { public void run() { gaston.bow(alphonse); } }).start(); }

13. 132008 - Nov Gaston & Alphonse How to find the deadlock The idea is to enter other wait functions randomly anywhere before a synchronize keyword appears. Then, we can shake the stability of the regular execution path. We always enter the same byte code of an invariant function that includes a call of other modifiable function. To do that. “Some” BCEL is needed.

14. 142008 - Nov Agenda Gaston & Alphonso (5 min) The Deadlock Hunter (20 min) Script & Statistics (5 min) One More Example (3 min) Conclusions (2 min)

15. 152008 - Nov The DeadLock Hunter Entered method “Enter” method: public void enter(){ entered(1000);} “Entered” method: synchronized void entered(int time) { try{ if(time != 10){ Random rand = new Random(); double rnd = rand.nextDouble(); if(rnd > 0.3) wait(time);} else wait(10); } catch(Exception e) {e.printStackTrace();} } Client byte code “Enter” Function Byte Code

16. 162008 - Nov The Deadlock Hunter Main code flow Get the Client class file Create new Instruction list for each method Insert the “enter” function call where needed Update exception table handler Loop over its methods Deal with branch instructions Dump new class file

17. 172008 - Nov The Deadlock Hunter Original byte code of bow method 0: aload_0[42](1) 1: dup[89](1) 2: astore_2[77](1) 3: monitorenter[194](1) 4: getstatic[178](3) 23 7: new[187](3) 29 10: dup[89](1) 11: aload_1[43](1) 12: invokevirtual[182](3) 31 15: invokestatic[184](3) 33 18: invokespecial[183](3) 39 21: ldc[18](2) 41 23: invokevirtual[182](3) 43 26: aload_0[42](1) 27: getfield[180](3) 13 30: invokevirtual[182](3) 43 33: ldc[18](2) 47 35: invokevirtual[182](3) 43 38: invokevirtual[182](3) 49 41: invokevirtual[182](3) 52 44: aload_1[43](1) 45: aload_0[42](1) 46: invokevirtual[182](3) 57 49: aload_2[44](1) 50: monitorexit[195](1) 51: goto[167](3) -> return 54: aload_2[44](1) 55: monitorexit[195](1) 56: athrow[191](1) 57: return[177](1)

18. 182008 - Nov The DeadLock Hunter Getting Started Get the class object using its path: JavaClass clazz = Repository.lookupClass(args[0]); Extract the methods: Method[] methods = clazz.getMethods(); Get “bow” method: Method bow=methods[2];

19. 192008 - Nov The DeadLock Hunter Get and Parse instruction lists Create a constant pool generator: ConstantPoolGen cp = new ConstantPoolGen(clazz.getConstantPool()); Get instruction list of a method: InstructionList ilbow=new MethodGen(bow, clazz.getClassName(), cp).getInstructionList(); Get InstructionHandle: InstructionHandle[] bowhandle=ilbow.getInstructionHandles(); Parse and find “monitorenter”: if(bowhandle[i].getInstruction().toString().startsWith("monitorenter")) Find modification line “aload0”: aloadIdx=find_string(i+insertions*ilwork_func.getLength(), il, "aload_0");

20. 202008 - Nov The DeadLock Hunter Instruction list instrumentation Add a single instruction to a instruction list: il.append(bowhandle[i].getInstruction()); Add an instruction list to other instruction list: il.insert(il.getInstructionHandles()[aloadIdx], ilwork_func.copy()); Handling a branch instruction: if(ins[i].getInstruction() instanceof BranchInstruction){ BranchInstruction branch=(BranchInstruction)ins[i].getInstruction(); newil.append(branch); }

21. 212008 - Nov The DeadLock Hunter Remove Return in the Entered code 0: aload_0[42](1) 1: sipush[17](3) 2000 4: invokevirtual[182](3) 15 7: return[177](1) public InstructionList removeReturn(InstructionList Il){ InstructionList newIl= new InstructionList(); InstructionHandle[] ins=Il.getInstructionHandles(); for(int i=0; i<Il.getLength()-1; i++) if(ins[i].getInstruction() instanceof BranchInstruction){ BranchInstruction branch=(BranchInstruction)ins[i].getInstruction(); newIl.append(branch);} else if(ins[i].getInstruction() instanceof INVOKEVIRTUAL){ INVOKEVIRTUAL invoke=(INVOKEVIRTUAL)ins[i].getInstruction(); newIl.append(invoke);} else newIl.append(ins[i].getInstruction()); return newIl;} Before removing return: After removing return:

22. 222008 - Nov The DeadLock Hunter Exception table update The exception table marks handlers, i.e., code chunks, to be responsible for exceptions of certain types that are raised within a given area of the byte code. When there is no appropriate handler the exception is propagated back to the caller of the method. This table must be updated because adding instructions make an offset to the handler pointers and range line definitions. Get exception table: CodeException[] exp=methods[2].getCode().getExceptionTable(); Modify exception table according to the number of entered code lines for(int j=0; j<exp.length; j++) { int start=exp[j].getStartPC() int end=exp[j].getEndPC(); if (end>bc_length){ exp[j].setEndPC(exp[j].getEndPC()+ilprint_length); exp[j].setHandlerPC(exp[j].getHandlerPC()+ilprint_length); if (start>bc_length) exp[j].setStartPC(exp[j].getStartPC()+ilprint_length); }

23. 232008 - Nov The DeadLock Hunter Instrumented byte code of bow method 0: aload_0[42](1) 1: sipush[17](3) 1000 4: invokevirtual[182](3) 86 7: aload_0[42](1) 8: dup[89](1) 9: astore_2[77](1) 10: monitorenter[194](1) 11: getstatic[178](3) 23 14: new[187](3) 29 17: dup[89](1) 18: aload_1[43](1) 19: invokevirtual[182](3) 31 22: invokestatic[184](3) 33 25: invokespecial[183](3) 39 28: ldc[18](2) 41 30: invokevirtual[182](3) 43 33: aload_0[42](1) 34: getfield[180](3) 13 37: invokevirtual[182](3) 43 40: ldc[18](2) 47 42: invokevirtual[182](3) 43 45: invokevirtual[182](3) 49 48: invokevirtual[182](3) 52 51: aload_1[43](1) 52: aload_0[42](1) 53: invokevirtual[182](3) 57 56: aload_2[44](1) 57: monitorexit[195](1) 58: goto[167](3) -> return 61: aload_2[44](1) 62: monitorexit[195](1) 63: athrow[191](1) 64: return[177](1)

24. 242008 - Nov Agenda The Deadlock Hunter (20 min) Script & Statistics (5 min) One More Example (3 min) Conclusions (2 min)

25. 252008 - Nov Script and Statistics Script Code #!/bin/tcsh set count=0 set sum1=0 set till=100 set curr=0 while($count<$till) @ count++ echo $count java -classpath ~/.eclipse/test -noverify pack.Client > a.out & sleep 1 set curr = `wc -l a.out |cut -c 1` if($curr<4)then @ sum1++ endif End mv -f Client.class pack/ set count=0 set sum2=0 set curr=0 while($count<$till) @ count++ echo $count java -classpath ~/.eclipse/test -noverify pack.Client > a.out & sleep 3 set curr = `wc -l a.out |cut -c 1` \rm a.out if($curr<4)then @ sum2++ endif end echo "$sum1 deadlocks occured before code instrumentation" echo "$sum2 deadlocks occured after code instrumentation" Initialize Run original class Update sum if deadlock happens Run instrumented class Loop N times Get the instrumented class file Print output Loop N times Update sum if deadlock happens

26. 262008 - Nov Script and Statistics Script Output Before code instrumentation, 0 deadlocks occurred in 100 runs. After adding all waits randomly (probability 70%) in bow function only, 65 deadlocks occurred in 100 runs. After adding all waits randomly (probability 20%) in bow function only, 35 deadlocks occurred in 100 runs. Adding waits randomly in all functions resulted in 100 deadlocks in 100 runs !

27. 272008 - Nov Agenda Script & Statistics (5 min) One More Example (3 min) Conclusions (2 min)

28. 282008 - Nov One More Example The Client private Object lock1 = new Object (); private Object lock2 = new Object (); public void instanceMethod1 () { synchronized (lock1) { synchronized (lock2) { System.out.println("inside instanceMethod1"); }}} public void instanceMethod2 (){ synchronized (lock2){ synchronized (lock1){ System.out.println("inside instanceMethod2"); }}} public static void main (String[] args){ final Client2 first = new Client2(); final Client2 second = new Client2(); new Thread(new Runnable() { public void run() { first.instanceMethod1();} }).start(); wait(10); new Thread(new Runnable() { public void run() { second.instanceMethod2(); } }).start(); }

29. 292008 - Nov One More Example Statistics Before manipulating, 5 deadlocks occurred in 100 runs. After code instrumentation with probability 70% in all methods, 65 deadlocks occurred in 100 runs.

30. 302008 - Nov Agenda One More Example (3 min) Conclusions (2 min)

31. 312008 - Nov Conclusions The Client Execution path was randomized. A hidden deadlock has a higher probability to happen after code instrumentation. Project goal achieved: “Our goal is to build a tool that will force the operating system scheduler to make the execution path be truly random” “Our goal is to build a tool that will force the operating system scheduler to make the execution path be truly random” What next? Be unrelated with the java code completely: Be unrelated with the java code completely: Entered function should be an outer function Count deadlocks without relying on the clients prints Generalize to other interesting situations. (not only Sync. Statements) Generalize to other interesting situations. (not only Sync. Statements)

32. 322008 - Nov Thanks…