Hipikat: A Project Memory for Software Development The CISC 864 Analysis By Lionel Marks

What’s the Typical Scenario for a New Developer? A new developer joins the team and knows little of the project An existing member offers advice and helps newcomer finish first tasks Information generally about basic ideas on the problem domain and pointers on using tools effectively Mentors give information on the eccentricities of the system and why things are done that way

What is the New Scenario? Team members are not co-located Hard for a newcomer to get up-to-speed on the project on the “virtual team” Using electronic media such as mailing lists, code repositories and work tracking programs, can create a “project memory” with Hipikat

What is Hipikat Good for? With the volume of information being so enormous, sifting through the documents would take too long Hipikat allows the user to search these archives Searching CVS for code to clone for your new method Searching configuration management tools for a history of changes, to help in setup maintenance or to redo a setup.

Related Work Group Memory  Design Assistant and TeamInfo had the information generated by humans specifically for their tool to query Recommender Systems  Remembrance Agent and Code Broker did not use multiple information sources

Related Work Programming from Examples  Reuse View Matcher and Explainer which has libraries of specifically created code to query, not existing code from the current system. Mining Software Repositories to help in SW Maintenance  CVSSearch – search comments from CVS commits  Expertise Browser and Expertise Recommender – recommend developers with expertise at certain parts of a project  Zimmerman et al., Ying et al. – found change patterns in repositories to recommend potentially relevant files when working on a change in the code of a project

How Hipikat Works 1. Hipikat forms the project memory from source code, documentation, and other electronic media such as messages and bug reports. All are put in a central database that can be queried. 2. It can then be queried to recommend artifacts that are relevant to the user’s current task. Information sources can be monitored periodically or continuously, and when they are committed to the database, then they can be included in searches

How Hipikat Works “A user makes a query by selecting an artifact in the Eclipse project workspace and choosing ‘Query Hipikat’ from the context menu.” (Interesting!) User can also search the Hipikat database through “Search Hipikat”

The Digg Effect Training the System (Increasing the rank of certain artifacts in queries) Just by clicking on the artifact (that it looks useful) Clicking a “thumbs-up” or “thumbs-down” Problems (Can anyone see them?)

The Digg Effect Training the System (Increasing the rank of certain artifacts in queries) Just by clicking on the artifact (that it looks useful) Clicking a “thumbs-up” or “thumbs-down” Problems (Can anyone see them?) Clicking on artifact  What if relevant on first sight, but not really upon inspection? Inc. rank of bad match Clicking on “thumb”  Asking to verify usefulness before even used the data. Can’t rank like this

Evaluation Criteria Precision vs. Recall In a perfect world, whatever the system recommends is exactly what you need Giving recommendations that have “examples of use” for relevant APIs rather than just giving the place in code where you must make your changes (i.e. “The solution”) (Interesting!)

Eclipse Newcomer Study Finding Participants Require adequate programming experience in Java Have experience developing large or medium-sized SW Very small pool of recruit-able computer science undergrads with the above criteria (Good point!) Also wanted to compare the solutions Got some of experienced members of the Eclipse Development team Asked them to perform the same tasks for baseline comparison

Eclipse Newcomer Study Results The first task, implementing the pop-up window when hover over the side of code to suggesting breakpoint properties Experts performed worst on handling the special cases 25% of experts got them 38% of newcomers got them perfectly right 63% of newcomers got it basically right Harder task 75% of experts solved it correctly 38% of newcomers got it right

Use of Hipikat More queries made on the harder task Almost all queries made within the first hour Generally once the person knew what file(s) to change, and had a general plan, no more queries were made. Criteria for finding a recommendation useful Is the problem report “interesting” Similar enough to current task – code re-use Learn relevant information For difficult coding problems, searched and found filenames that look potentially relevant Did not look at source code changes, but rather built their understanding from “scratch” looking at the file in an editor. (easier, did not want a “false lead” and have to continue searching)

My General Thoughts Good paper in general The paper did a good job at stating its facts and shortcomings. Would like to see it used in a different way, as a data miner for teaching the person about the system rather than providing assistance when in doubt (too much recall for when trying to perform a single task)

Likes and Dislikes of this Paper Likes  Its study of how people thought about the problem  The tool itself is an intriguing idea  The style of the paper – very readable! Dislikes  Asking that people take into account special cases when first coding something in a system – a bit of a reach  Saying that the participants must have worked on a medium to large SW system before. Many undergrads will go into industry working on their first large system ever. Would be interesting for your tool to get someone who is really new to these kinds of SW systems – see if they use Hipikat more