1. ARE YOU STUCK? by Jason Carter and Prasun Dewan Classification via Clustering Model Stuck Making Progress Stuck6016 Making Progress 8961392 Bottom Up Component Top Down Component Top Down Component Predict when programmers are stuck Imagine situations that indicate that programmers are stuck. Log the activities of programmers who explicitly indicate they are stuck Overall Accuracy: 92% Situations in which Programmers are Stuck Assumption: Decrease in Editing = Increase in other Actions Increase in Debugging = Stuck Increase in Navigating = Stuck Increase in Debugging + Increase Switching Task = Stuck Situations in which Programmers are Stuck Assumption: Decrease in Editing = Increase in other Actions Increase in Debugging = Stuck Increase in Navigating = Stuck Increase in Debugging + Increase Switching Task = Stuck Buttons used to Indicate Status Information extracted from logs X = Total of all events Buttons used to Indicate Status Information extracted from logs X = Total of all events Confusion Matrix - The rows represent the actual number of instances and the columns represent the predictions by the model. The blue diagonal represent cases where the model was correct. Stuck Could Use Help Making Progress EUREKA!!! Student Need Help? Professor Offer help to shy students Progress Measure students’ progress Identify problematic software components Improve programmers’ productivity Academia Industry Grrr… Hmm… is Bob stuck? Bob The greater the distance between developers, the more difficult it is to determine if they need help. Jim MOTIVATION Needle in a haystack Class Imbalance Distribution Problem Actual StuckPredicted Stuck No YesNo You may notice the accuracy is 75%, but the model never predicts that the programmer is stuck. This is because the majority of data used to construct the model has less stuck records than making progress records. This is called the class imbalance distribution problem which is often referred to as trying to find a “ needle in a hay stack ”. Accuracy: 75% Edit Percentage= (# of edits / X) * 100 Debug Percentage= (#of debugs + runs) / X * 100 Focus Percentage= (# of task switches / X) * 100 Navigation Percentage= (# of navigations / X ) * 100 Remove Percentage= (# of removals/ X) * 100 How to Identify Stuck Points Stuck points occur when edit percentage decreases, and other percentages increase. This graph illustrates programming behavior when Participant 1 was stuck. The greater the change, the more likely it is that a programmer is stuck. Cost Sensitive Model Stuck Making Progress Stuck5620 Making Progress 4311857 Oversampling Model Stuck Making Progress Stuck1101115 Making Progress 1582130 Overall Accuracy: 61% Overall Accuracy: 80% True Positive Accuracy: 73% True Positive Accuracy: 78%True Positive Accuracy: 90% Comparison of the Accuracy of Models Using Confusion Matrices Definitions Overall Accuracy True Positive Accuracy # of times a model predicted that a programmer was stuck total # of times a developer was actually stuck # of times a model predicted the correct situation total # of situations = = Each model was built using a different algorithm. Can you identify the Stuck Point? Cost sensitive Learning - uses a cost matrix to penalize the algorithm for misclassifying a record. Oversampling - replicates the class with the least number of records until the training set has an equal number of records in all classes Classification Via Clustering - applies a clustering algorithm to unlabeled data and then maps the found clusters back to classes according to the class distribution. University of North Carolina at Chapel Hill