1. 1 Recommendation Systems for Code Reuse Tao Xie Department of Computer Science North Carolina State University Raleigh, USA

2. 2 2 Motivation Programmers commonly reuse APIs of existing frameworks or libraries –Advantages: Low cost and high efficiency of development –Challenges: Complexity and lack of documentation E.g., searching for information nearly ¼ of developer time [metallect.com] Frame works

3. Example Task from Eclipse Programming Task: How to parse code in a dirty editor of Eclipse? ? Query: “IEditorPart -> ICompilationUnit” Open Source Projects 1 2 N … … Extract MIS 1 MIS 2... … MIS k *MIS: Method-Invocation sequence, FMIS: Frequent MIS FMIS 1 FMIS 2 … FMIS n Recommend Mine PARSEWeb [Thummalapenta&Xie ASE 07]

4. 4 Scenario 1 While reusing APIs of existing open source frameworks or libraries, programmers often –know what type of object they need –but do not know how to write code for getting that object Query: “Source  Destination” How to use these APIs? Prospector [Mandelin et al. PLDI 05 ], XSnippet [Sahavechaphan&Claypool OOPSLA 06 ], PARSEWeb [Thummalapenta&Xie ASE 07]

5. 5 Example Task from Eclipse Programming Task: How to parse code in a dirty editor? Query: IEditorPart  ICompilationUnit Example solution from Prospector/PARSEWeb: IEditorPart iep =... IEditorInput editorInp = iep.getEditorInput(); IWorkingCopyManager wcm = JavaUI.getWorkingCopyManager(); ICompilationUnit icu = wcm.getWorkingCopy(editorInp); Difficulties: a. Needs an instance of IWorkingCopyManager b. Needs to invoke a static method of JavaUI for getting the preceding instance Prospector [Mandelin et al. PLDI 05 ], XSnippet [Sahavechaphan&Claypool OOPSLA 06 ], PARSEWeb [Thummalapenta&Xie ASE 07]

6. 6 Scenario 2 While reusing APIs of existing open source frameworks or libraries, programmers often –know what method call they need –but do not know how to write code before and after this method call Query: “Method name” How to use these APIs? MAPO [Xie&Pei MSR 05]

7. 7 Example Task from BCEL Programming Task: How to instrument the bytecode of a Java class by adding an extra method to the class? Query : org.apache.bcel.generic.ClassGen public void addMethod(Method m ) Example solution from MAPO: public void generateStubMethod(ClassGen c) InstructionList il = new InstructionList(); MethodGen m= genFromISList(il); m.setMaxLocals(); m.setMaxStack(); c.addMethod(m.getMethod()); System.out.println(“…”); … } MAPO [Xie&Pei MSR 05]

8. 8 Scenario 3 While reusing APIs of existing open source frameworks or libraries, programmers often –know structural context such as a class’ type, its parents, and fields’ types, a method’s signature, method or constructor callees –but do not know how to write code in this context Query: Structural context How to use these APIs? Strathcona [Holmes et al. 05], XSnippet [Sahavechaphan&Claypool OOPSLA 06 ]

9. 9 Example Task from HttpClient Programming Task: How to evolve a system to use a third party library, HttpClient, for handling http connections? Query : HttpClient, PostMethod classes Example solution from Strathcona: Strathcona [Holmes et al. 05], XSnippet [Sahavechaphan&Claypool OOPSLA 06 ]

10. 10 Steps in Recommenders Data collection/extraction Data preprocessing Data analysis/mining Result postprocessing Result representation

11. 11 Data Collection/Extraction From one or multiple local code repositories –Often followed by offline analysis or mining –Challenges: lack of relevant code examples –Ex.: Strathcona, Prospector, XSnippet From the whole open source world with a code search engine! –Often followed by on-the-fly analysis and mining –Challenges: only partial code files –Ex.: MAPO, PARSEWeb

12. 12 Exploiting A Code Search Engine Accepts queries including keywords of classes or/and method names Interacts with a code search engine such as Google code search to gather related code samples Stores gathered code samples (source files) in a local code repository (later being analyzed and mined) Challenges: gathered code samples are partial and not compilable as code search engines retrieve individual source files instead of entire projects PARSEWeb [Thummalapenta&Xie ASE 07]

13. 13 Available Code Search Engines Google Code Search http://www.google.com/codesearch http://www.google.com/codesearch Krugle: http://www.krugle.com/ http://www.krugle.com/ Koders: http://www.koders.com/http://www.koders.com/ Codase: http://www.codase.com/http://www.codase.com/ JExamples: http://www.jexamples.com/http://www.jexamples.com/ etc., Why not using just code search engines?

14. What are Developers Searching for? Assieme [Hoffmann et al. UIST 07] 339 sessions related to Java programming 15 million queries of Windows Live Search from May 2006. 117 API sessions (34.2%); 70 trouble-shooting sessions (20.6%)

15. 15 API-related Search Sessions 64.1% sessions contained queries that were merely descriptive but did not contain actual names of APIs, packages, types, or members. The remaining sessions contained –API or package names (12.8%), –Type names (17.9%) –Method names (5.1%). Among all these API-related sessions, 17.9% contained terms like “example”, “using”, or “sample code” Assieme [Hoffmann et al. UIST 07]

16. 16 An Example 4-Query Session java JSP current date java SimpleDateFormat using currentdate in jsp Assieme [Hoffmann et al. UIST 07]

17. Only compatible with new Java versions Why Not Use Web Search Engines? Requires installation of external library, but no link Code on pages essentially the same Contains no code examples parse xml java ©Raphael Hoffmann Assieme [Hoffmann et al. UIST 07]

18. Code Search Engines import javax.xml.parsers.*; import org.w3c.dom.*; public class JAXPSample { public static void main(String[] args) { String filename = "sample.xml"; try { DocumentBuilderFactory factory = DocumentBuilderFactory.newInstance(); DocumentBuilder parser = factory.newDocumentBuilder(); Document d = parser.parse(filename); } catch (Exception e) { System.err.println("Exception: " + e.getMessage()); } } } Index source code of open-source Projects (from compressed archive Files and CVS repositories) Code is parsed and terms in type names, variable names, etc. are weighted differently. ©Raphael Hoffmann Assieme [Hoffmann et al. UIST 07]

19. Why not use code search engines only? Irrelevant (An Emacs Lisp File!?!) Code is complicated, contains no comments related to query, and is more than 300(!) lines long Requires installation of external library, but no link Code on pages essentially the same parse xml java ©Raphael Hoffmann Assieme [Hoffmann et al. UIST 07]

20. Why not use code search engines only? MAPO [Xie&Pei MSR 06]

21. 21 Steps in Recommenders Data collection/extraction Data preprocessing Data analysis/mining Result postprocessing Result representation

22. 22 Fact Extraction Whole-program analysis: applicable when the whole code bases are available and compilable Partial-program analysis: applicable when only partial code samples are available and not compilable –When a code search engine is used

23. 23 Analysis of Partial Code Samples Not all code samples contain main method or driver code that can serve as an entry point –consider all public methods as entry points Deal with local method calls by inlining methods Deal with conditionals/loops by traversing control flow graphs Deal with unknown types with heuristics PARSEWeb [Thummalapenta&Xie ASE 07]

24. 24 Type Heuristics I Inferring fully qualified class names import javax.jms.QueueSession; import java.util.*; Public class test { public QueueSession qsObj; public Integer intObj; public Iterator iter; … -Fully qualified name of QueueSession is “javax.jms.QueueSession”, inferred through lookup of import statement -Fully qualified name of Integer is “java.lang.Integer”, inferred through loading of a class by appending “java.lang” to the class name -Cannot infer the fully qualified name of “Iterator” (incorporating domain knowledge of java.util helps) 24 PARSEWeb [Thummalapenta&Xie ASE 07]

25. 25 Type Heuristics II Infer the receiver type in expression “X.Y” – Lookup the declaration of X in local variables or member variables. If not, “X” is a class name and Y is a static member Infer the receiver type in expression “M1().Y” – Check the return type of M1() method declaration, if not available locally, the receiver type cannot be inferred 25 PARSEWeb [Thummalapenta&Xie ASE 07]

26. 26 Type Heuristics III Infer the return type of a method invocation in an assignment statement such as “ Queue qObj = createQueueSession()” –Lookup the type of the variable on the left hand side. The return type is the same as or a sub class of Queue Infer the return type of a method invocation in a return statement such as public QueueSession test() {... return connect.createQueueSession(false,int); } - Lookup the return type of the enclosing method declaration 26 PARSEWeb [Thummalapenta&Xie ASE 07]

27. 27 Type Heuristics IV Infer types with multiple method invocations Queue qObj = connect.m1(); Stack sObj = connect.m1().m2(); The receiver type of m2() can be inferred from the lookup of the return type of m1() 27 PARSEWeb [Thummalapenta&Xie ASE 07]

28. 28 Sequence Filtering Remove common Java library calls Remove sequences that contain no query words: ClassGen and addMethod InstructionList. () genFromISList(InstructionList) MethodGen.setMaxStack() MethodGen.setMaxLocals() MethodGen.getMethod() ClassGen.addMethod(Method) PrintStream.println(String) … public void generateStubMethod(ClassGen c) InstructionList il = new InstructionList(); MethodGen m= genFromISList(il); m.setMaxLocals(); m.setMaxStack(); c.addMethod(m.getMethod()); System.out.println(“…”); … } MAPO [Xie&Pei MSR 05]

29. Type Signature Graph Any path from h to w is a (h,w)-jungloid IFileCompilationUnit ICompilationUnit ASTNode IClassFile JavaCore.createCompilationUnitFrom() AST.parseCompilationUnit() supertype AST.parseCompilationUnit() JavaCore.createClassFileFrom() IJavaElementIResource supertype getResource() IContainer getParent() Prospector [Mandelin et al. PLDI 05 ]

30. Jungloids with Downcasts IDebugView debugger =... Viewer viewer = debugger.getViewer(); IStructuredSelection sel = (IStructuredSelection) viewer.getSelection(); JavaInspectExpression expr = (JavaInspectExpression) sel.getFirstElement(); IDebugView Viewer ISelection IStructuredSelection JavaInspectExpression Object getViewer() getSelection() getFirstElement() getInput() downcast Prospector [Mandelin et al. PLDI 05 ]

31. 31 Steps in Recommenders Data collection/extraction Data preprocessing Data analysis/mining Result postprocessing Result representation

32. 32 Data Analysis/Mining Some recommenders don’t use specific mining techniques to “abstract” or “generalize” common patterns but return relevant raw code samples –Prospector, Strathcona, XSnippet, PARSEWeb Data mining can be used to uncover hidden patterns –Association rules: CodeWeb [Michail ICSE 00] –Frequent subsequences: MAPO [Xie&Pei MSR 06] –Frequent partial orders: Apiator [Acharya et al. FSE 07]

33. 33 Association Rules KApplication reuse patterns CodeWeb [Michail ICSE 00]

34. #include void p ( ) { b ( ); c ( ); } void q ( ) { c ( ); b ( ); } void r ( ) { e ( ); f ( ); } void s ( ) { f ( ); e ( ); } int main ( ) { int i, j, k; a ( ); if ( i == 1) { f ( ); e ( ); c ( ); exit ( ); } else { if ( j == 1 ) p ( ); else q ( ); d ( ); if ( k == 1 ) r ( ); else s ( ); } Frequent SubSeq/Partial Order Consider APIs a, b, c, d, e, and f Apiator [Acharya et al. FSE 07]

35. #include void p ( ) { b ( ); c ( ); } void q ( ) { c ( ); b ( ); } void r ( ) { e ( ); f ( ); } void s ( ) { f ( ); e ( ); } int main ( ) { int i, j, k; a ( ); if ( i == 1) { f ( ); e ( ); c ( ); exit ( ); } else { if ( j == 1 ) p ( ); else q ( ); d ( ); if ( k == 1 ) r ( ); else s ( ); } 1 a  f  e  c 2 a  b  c  d  e  f 3 a  c  b  d  e  f 4 a  b  c  d  f  e 5 a  c  b  d  f  e a d c e b f a  b  d  e a  b  d  f a  c  d  e a  c  d  f (b) Static program traces (c) Frequent sequential patterns Support 4/5 (d) Frequent partial order R (a) Example code Consider APIs a, b, c, d, e, and f Frequent SubSeq/Partial Order Apiator [Acharya et al. FSE 07]

36. #include void p ( ) { b ( ); c ( ); } void q ( ) { c ( ); b ( ); } void r ( ) { e ( ); f ( ); } void s ( ) { f ( ); e ( ); } int main ( ) { int i, j, k; a ( ); if ( i == 1) { f ( ); e ( ); c ( ); exit ( ); } else { if ( j == 1 ) p ( ); else q ( ); d ( ); if ( k == 1 ) r ( ); else s ( ); } 1 a  f  e  c 2 a  b  c  d  e  f 3 a  c  b  d  e  f 4 a  b  c  d  f  e 5 a  c  b  d  f  e a d c e b f a  b  d  e a  b  d  f a  c  d  e a  c  d  f (b) Static program traces (c) Frequent sequential patterns support, 4/5 (d) Frequent partial order R(a) Example code Frequent SubSeq/Partial Order Consider APIs a, b, c, d, e, and f Apiator [Acharya et al. FSE 07]

37. 1 a  f  e  c 2 a  b  c  d  e  f 3 a  c  b  d  e  f 4 a  b  c  d  f  e 5 a  c  b  d  f  e a d c e b f a  b  d  e a  b  d  f a  c  d  e a  c  d  f (b) Static program traces (c) Frequent sequential patterns support, 4/5 (d) Frequent partial order R (a) Example code #include void p ( ) { b ( ); c ( ); } void q ( ) { c ( ); b ( ); } void r ( ) { e ( ); f ( ); } void s ( ) { f ( ); e ( ); } int main ( ) { int i, j, k; a ( ); if ( i == 1) { f ( ); e ( ); c ( ); exit ( ); } else { if ( j == 1 ) p ( ); else q ( ); d ( ); if ( k == 1 ) r ( ); else s ( ); } Frequent SubSeq/Partial Order Apiator [Acharya et al. FSE 07] MAPO [Xie&Pei MSR 05] MAPO Apiator

38. 38 Data Analysis/Mining Data collection/extraction Data preprocessing Data analysis/mining Result postprocessing Result representation

39. 39 Result Postprocessing When a third-party miner or learner isn’t used, this step may be considered part of the data analysis/mining step. Examples Result clustering Result ranking Result filtering

40. 40 Clustering and Ranking Candidate method sequences produced by the data analysis/mining step for query “Source  Destination” may be too many Solutions: Cluster similar sequences –Clustering heuristics are developed Rank sequences –Ranking heuristics are developed PARSEWeb [Thummalapenta&Xie ASE 07]

41. 41 Clustering Heuristics Method-invocation sequences with the same set of statements can be considered similar, although the statements are in different order. e.g., ''2 3 4 5'' and ''2 4 3 5 '' Method-invocation sequences with minor differences measured by an attribute cluster precision value can be considered similar. e.g., ''8 9 6 7'' and ''8 6 10 7 '' can be considered similar under cluster precision value one PARSEWeb [Thummalapenta&Xie ASE 07]

42. 42 Ranking Heuristics Heuristic 1: Higher frequency -> Higher rank Heuristic 2: Shorter length -> Higher rank Heuristic 3: Fewer package boundaries -> Higher rank PARSEWeb [Thummalapenta&Xie ASE 07]Prospector [Mandelin et al. PLDI 05 ]

43. 43 Query Splitting Lack of code samples that give candidate method- invocation sequences in the results of code search engines –Required method-invocation sequences are split among different source files Solution: –Split the user query into multiple queries –Compose the results for each split query PARSEWeb [Thummalapenta&Xie ASE 07]

44. 44 Query Splitting Example 1. User query: “org.eclipse.jface.viewers.IStructuredSelection->java.io.ObjectInputStream” Results: None 2. Query: “java.io.ObjectInputStream” Results: 3. Most used immediate sources are: java.io.InputStream, java.io.ByteArrayInputStream, java.io.FileInputStream 3. Three Queries to be fired: “org.eclipse.jface.viewers.IStructuredSelection-> java.io.InputStream” Results: 1 “org.eclipse.jface.viewers.IStructuredSelection-> java.io.ByteArrayInputStream” Results: 5 “org.eclipse.jface.viewers.IStructuredSelection-> java.io.FileInputStream” Results: None PARSEWeb [Thummalapenta&Xie ASE 07]

45. 45 Result Filtering Remove sequences that contain no query words: ClassGen and addMethod Compress consecutive calls of the same method into one, e.g., abbba  aba Remove duplicate frequent sequences after the compression, e.g., aba, aba  aba Reduce a seq if it is a subseq of another, e.g., aba, abab  abab MAPO [Xie&Pei MSR 06]

46. 46 Data Analysis/Mining Data collection/extraction Data preprocessing Data analysis/mining Result postprocessing Result representation

47. 47 Result Representation Display results in the tool user interface –Strathcona –XSnippet –PARSEWeb –MAPO –CodeBroker –Assieme

48. 48 Strathcona Strathcona [Holmes et al. 05]

49. 49 XSnippet XSnippet [Sahavechaphan&Claypool OOPSLA 06 ]

50. 50 PARSEWeb PARSEWeb [Thummalapenta&Xie ASE 07]

51. 51 PARSEWeb http://news.google.com/

52. 52 MAPO (new) MAPO [Xie&Pei MSR 06]

53. 53 MAPO (new) MAPO [Xie&Pei MSR 06]

54. CodeBroker Comments signature CodeBroker [Ye&Fischer ICSE 01] Information delivery that autonomously locates and presents software developers with task- relevant and personalized components. Active repository!!!

55. Assieme A hybrid search engine Index code snippets found on web pages Link them to required libraries and documentation Assieme [Hoffmann et al. UIST 07]

56. Assieme links to pages with snippets group pages with similar snippets links to required libraries Assieme [Hoffmann et al. UIST 07]

57. Example Evaluations of Recommenders Prospector Strathcona PARSEWeb

58. Prospector Experiment 1 (ranking test) hypothesis: –to find the desired code, the user needs to examine only top 5 candidate jungloids. result: –desired code in “top 5” 17 out 20 times (10 out of 20, in “top 1”) –remaining three fixable methodology: –used 20 real-world coding tasks –collected from FAQs, newsgroups, our practice, emails to us

59. Prospector Experiment 2 (user study) hypothesis: –Prospector-equipped programmers are better at solving API programming problems than other programmers methodology: –6 problems, each user did 3 with Prospector and 3 without –problems formulated not to reveal the query –sample problem: “The new Java channel IO system represents files as channels. How do I get a channel that represents a String filename?” –somewhat sparse data (10 users)

60. Experiment 2 (user study). Results. Prospector shortens development time –some problems solved only by Prospector users –when both groups succeeded, Prospector users 30% faster Prospector may help enable reuse –non-Prospector users sometimes reimplemented Prospector may help avoid making mistakes –mistakes applying code found on internet into own code The authors expect even stronger results on a more robust infrastructure.

61. Strathcona: User Study 2 developers were assigned 4 tasks on building a plug-in for Eclipse. Neither developers knew how to implement any of the tasks at hand. The results showed that the tool can deliver relevant and useful examples to developers. They also showed a developer can determine when the examples returned are not relevant. Table 2: Results from Evaluation: Useful Example Source Viewed Succeeded at Task Task 1 Subject 1 1 1 yes Subject 2 1 1yes Task 2 Subject 1 12 yes Subject 2 1 6 yes Task 3 Subject 1 0 2 yes Subject 20 6 yes Task 4 Subject 1 1 2 yes Subject 2 0 7 partially Strathcona [Holmes et al. 05]

62. Strathcona: Performance and Scalability As a test case for scalability, Eclipse 3.0 source was populated to the repository. The resulting amount of information in the repository is shown in Table1. On a Pentium 3 800 MHz 1024 MB RAM Server, a Pentium 3 1000 MHz 256 MB RAM Repository with Postgresql DB the performance numbers are: Table 1: Number of Structural Relations Classes 17,456 Methods 124,359 Fields 48,441 Inheritance Relations 15,187 Object Instant ions 43,923 Calls Relations 1,066,838 Total 1,316,204 –Less than 500 ms for building a structural context. –Less than 300 ms for displaying the example. –4 – 12 seconds server response time. Strathcona [Holmes et al. 05]

63. 63 PARSEWeb Evaluations Real Programming Problems: To address problems posted in developer forums Real Projects: To show that solutions recommended by PARSEWeb are –available in real projects –better than solutions recommended by related tools PROSPECTOR, Strathcona, and Google Code Search averagely

64. 64 Real Programming Problems Jakarta BCEL user forum, 2001 Problem : “How to disassemble java byte code” Query : “Code  Instruction” Solution Sequence: FileName:2_RepMIStubGenerator.java MethodName: isWriteMethod Rank:1 NumberOfOccurrences:1 Code,getCode() ReturnType:#UNKNOWN# CONSTRUCTOR,InstructionList(#UNKNOWN#) ReturnType:InstructionList InstructionList,getInstructions() ReturnType:Instruction Solution Sample Code : Code code; InstructionList il = new InstructionList(code.getCode()); Instruction[] ins = il.getInstructions();

65. 65 Real Programming Problems Dev 2 Dev Newsgroups, 2006 Problem : “how to connect db by sessionBean” Query : javax.naming.InitialContext  java.sql.Connection Solution Sequence : FileName:3 AddressBean.java MethodName:getNextUniqueKey Rank:1 NumberOfOccurrences:34 javax.naming.InitialContext,lookup(java.lang.String) ReturnType:javax.sql.DataSource javax.sql.DataSource,getConnection() ReturnType:java.sql.Connection

66. 66 Real Project: Logic Source File: LogicEditor.java SUMMARY-> PARSEWeb: 8/10, Prospector: 6/10, Strathcona: 5/10

67. 67 Comparison with Prospector 12 specific programming tasks taken from XSnippet approach. SUMMARY-> PARSEWeb: 11/12, Prospector: 7/12

68. 68 Comparison with Other Tools Percentage of tasks successfully completed by PARSEWeb, Prospector, and XSnippet

69. 69 Significance of Internal Techniques *Legend: Method inline: Method inlining Post Process: Sequence Post Processor Query Split: Query Splitter

70. 70 T. Xie Mining Program Source Code Questions? Bibliography on Mining Software Engineering Data http://ase.csc.ncsu.edu/dmse/ What software engineering tasks can be helped by data mining? What kinds of software engineering data can be mined? How are data mining techniques used in software engineering? Resources Available Data Mining Tools http://ase.csc.ncsu.edu/dmse/resources.html

71. Mining Partial Orders 71 Consider APIs a, b, c, d, e, and f Partial Order Partial Order with Transitive Reduction The extracted scenarios are fed to a partial order miner The partial order miner mines frequent closed partial order Closed Partial Order Apiator [Acharya et al. FSE 07]

72. XOpenDisplay XCloseDisplay XCreateWindow XGetWindowAttributes XCreateGC XSetForeground XGetBackground XMapWindow XChageWindowAttributes XMapWindow XSelectInput XGetAtomName XFreeGC XNextEvent Example Partial Order A usage scenario around XOpenDisplay API as a partial order. Specifications are shown with dotted lines. Apiator [Acharya et al. FSE 07]