28/06/2015Dr Andy Brooks1 MSc Software Maintenance MS Viðhald hugbúnaðar Fyrirlestrar 13 & 14 Do Visualizations Improve Program Comprehensibility?

28/06/2015Dr Andy Brooks2 Case Study Dæmisaga Reference Do Visualizations Improve Program Comprehensibility? Experiments with Control Structure Diagrams for Java, T D Hendrix et al., SIGCSE 2000 (© ACM), pp , 2000.

28/06/2015Dr Andy Brooks3 Why visualizations? A picture paints a thousand words. Graphical representations are used in the everyday world to good effective. “... the effectiveness of software visualization is still an open question and is certainly not universally accepted.”

28/06/2015Dr Andy Brooks4 The Control Structure Diagram (CSD) A graphical representation of the control structure and module-level organization of source code. The graphical representation supplements the source code without disrupting the normal layout of the source code.

28/06/2015Dr Andy Brooks5 public class CoinFlip { // Creates a Coin object, // flips it, and prints the results. public static void main (String[] args) { Coin myCoin = new Coin(); myCoin.flip(); System.out.println (myCoin); if (myCoin.isHeads()) System.out.println ("You win."); else System.out.println ("Better luck next time."); } } Some Java code.

28/06/2015Dr Andy Brooks6 ÏÕÖ× public class CoinFlip Ïϧ { Ïϧ // Creates a Coin object, Ïϧ // flips it, and prints the results. Ïϧ ÏϧÏÞßà public static void main (String[] args) ÏϧÏϧ { ÏϧÏϨ¹íÏ Coin myCoin = new Coin(); ÏϧÏϧ ÏϧÏϨ¹¹Ï myCoin.flip(); ÏϧÏϧ ÏϧÏϨ¹¹Ï System.out.println (myCoin); ÏϧÏϧ ÏϧÏϨ¹³´ if (myCoin.isHeads()) ÏϧÏϧÏ6¾¹¹Ï System.out.println ("You win."); ÏϧÏϧÏö´ else ÏϧÏϧÏȾ¹¹Ï System.out.println ("Better luck next time."); ÏϧÏÏ© } ÏÏ© } Some Java code with CSD.

28/06/2015Dr Andy Brooks7 public class ReverseNumber { // Reverses the digits of an integer mathematically. public static void main (String[] args) { int number, lastDigit, reverse = 0; Scanner scan = new Scanner (System.in); System.out.print ("Enter a positive integer: "); number = scan.nextInt(); do { lastDigit = number % 10; reverse = (reverse * 10) + lastDigit; number = number / 10; } while (number > 0); System.out.println (“Number reversed is " + reverse); } } Some Java code.

28/06/2015Dr Andy Brooks8 ÏÕÖ× public class ReverseNumber Ïϧ { Ïϧ // Reverses the digits of an integer mathematically. Ïϧ ÏϧÏÞßà public static void main (String[] args) ÏϧÏϧ { ÏϧÏϨ¹íÏ int number, lastDigit, reverse = 0; ÏϧÏϨ¹íÏ Scanner scan = new Scanner (System.in); ÏϧÏϨ¹¹Ï System.out.print ("Enter a positive integer: "); ÏϧÏϨ¹¹Ï number = scan.nextInt(); ÏϧÏϨ¹¹® do ÏϧÏϧÏÏ5 { ÏϧÏϧÏÏ7¹¹Ï lastDigit = number % 10; ÏϧÏϧÏÏ7¹¹Ï reverse = (reverse * 10) + lastDigit; ÏϧÏϧÏÏ7¹¹Ï number = number / 10; ÏϧÏϧÏÏ5 } ÏϧÏϧÏÏò while (number > 0); ÏϧÏϨ¹¹Ï System.out.println (“Number reversed is " + reverse); ÏϧÏÏ© } ÏÏ© } Some Java code with CSD.

28/06/2015Dr Andy Brooks9 // Computes and returns the greatest common divisor of // the two positive parameters. Uses Euclid's algorithm. private int gcd (int num1, int num2) { while (num1 != num2) if (num1 > num2) num1 = num1 - num2; else num2 = num2 - num1; return num1; } } Some Java code.

28/06/2015Dr Andy Brooks10 Ïϧ // Computes and returns the greatest common divisor of Ïϧ // the two positive parameters. Uses Euclid's algorithm. ÏϧÏÞßà private int gcd (int num1, int num2) ÏϧÏϧ { ÏϧÏϨ¹¹± while (num1 != num2) ÏϧÏϧÏÏ7¹³´ if (num1 > num2) ÏϧÏϧÏÏ5Ï6¾¹¹Ï num1 = num1 - num2; ÏϧÏϧÏÏ5Ïö´ else ÏϧÏϧÏÏ°ÏȾ¹¹Ï num2 = num2 - num1; ÏϧÏϧ Ïϧ¹Ĺ¹Ï return num1; ÏϧÏÏ© } ÏÏ© } Some Java code with CSD. CSD produced using the jGRASP IDE.

28/06/2015Dr Andy Brooks11 The Experiment Subjects were divided into two equal groups and provided with the same source code. Subjects were required to answer 12 questions about the code. The control group had plain source code while the intervention group had source code with CSD. control group - samanburðarhópur

28/06/2015Dr Andy Brooks12 Hypotheses H o : The CSD will not have a positive effect on program comprehensibility. H 1 : The CSD will have a positive effect on program comprehensibility. Andy says: Experimental researchers hope to be able to reject the null hypothesis (H o ) in favour of the alternative hypothesis (H 1 ).

28/06/2015Dr Andy Brooks13 Variables Independent variable ( óháð breyta ): code presentation (plain or CSD) Dependent variables: response time and response correctness

28/06/2015Dr Andy Brooks14 Experimental process All subjects received a 10 minute orientation to explain the task they were to perform. –Example short program, example questions, and how the questions were to be answered. Subjects were advised the task was similar to elements of a software inspection or debugging activity. The CSD subjects had an additional 5-10 minutes to explain the symbols of the CSD. Without sacrificing accuracy, subjects were asked to answer each question as quickly as possible.

28/06/2015Dr Andy Brooks15 Participants Participants were senior or graduate students taking an intensive programming course. –Developing and debugging non-trivial Java. Volunteers received extra credit points. Groups were balanced with respect to students´ ability on almost complete grade information. þátttakendur

28/06/2015Dr Andy Brooks16 Figure 4 Balance of groups prior to experiment. X indicates a non-letter grade graduate student. ©ACM

28/06/2015Dr Andy Brooks17 Participant loss 44 students volunteered. But some students were absent when the experiment started. For example, the CSD group had 4 A students compared to 2 A students in the control. Some data was eliminated from the analysis to ensure a proper balance. –So only data for 39 subjects were analysed.

28/06/2015Dr Andy Brooks18 The code The code was taken from a public domain graphics package written in Java. 183 source lines with several levels of control. A small number of control constructs were added for experimental purposes. Both groups were told they were inspecting real code. Both groups received the code in printed form to avoid any bias regarding familiarity with a program editor.

28/06/2015Dr Andy Brooks19 Question presentation Questions were presented as a sequence of web pages. Scripts recorded the subjects´ responses as well as response times. Response time was calculated from the time the question was displayed until the submit button was pressed.

28/06/2015Dr Andy Brooks20 Question design “The questions should be relevant to completing real comprehension tasks such as those found in inspection, testing, maintenance, and debugging activities.” “The questions should be universal, or as generally applicable, as possible.” “The questions should have single, objective answers.” –Answers were in terms of line numbers.

28/06/2015Dr Andy Brooks21 Representative questions 1.Where does the loop that begins on line 91 end? 2.How many variables and object instances are declared? 3.How many ways are there to exit the loop that begins on line 91? 4.To what line would control be transferred immediately after executing line 144? 5.How many syntactic levels deep is the most deeply nested statement? 6.How many conditions must be evaluted in order for line 152 to be excuted?

28/06/2015Dr Andy Brooks22 All experimental materials must be available to allow other researchers to replicate (or improve) the experiment. In this report, the authors choose to list 6 ‘representative’ questions rather than list all 12 questions actually used. Also, the authors will need to be contacted to obtain the code used.

28/06/2015Dr Andy Brooks23 Results Figure 5 Average time taken to respond per question A significant difference between overall response times was found with a p value of 0.06, but the actual statistical test used is not named. Only on question 12 does the control group perform better than the CSD group. ©ACM

28/06/2015Dr Andy Brooks24 Results Figure 6 Average time taken to respond correctly per question A significant difference between overall response times was found with a p value of , but the actual statistical test used is not named. The control group never performs better than the CSD group and there were no correct control group answers for questions 6, 7, and 12. ©ACM

28/06/2015Dr Andy Brooks25 Results Number of correct responses per question 45% of the CSD group´s responses were correct but only 26% of the control group´s responses were correct. The above figure is taken from another publication as Figure 7 of the case study does not show clearly the differences between the groups. ©IEEE

28/06/2015Dr Andy Brooks26 jGRASP Demonstrating the usefulness of visualization is one thing. To be practically useful, visualization must be incorporated in program editing tools. The jGRASP tool is a Java IDE that supports the CSD. Users can turn the CSD on or off at the click of a button.

28/06/2015Dr Andy Brooks27 Summary “The results of a controlled experiment indicate that the CSD can have a highly significant positive effect on human performance in program comprehension tasks.”

28/06/2015Dr Andy Brooks28 More than half the time, questions were answered incorrectly, even though subjects were told not to sacrifice accuracy. Why? If visualization is so useful, why were only 45% of the CSD group´s responses correct? The time/accuracy trade-off tells us that accuracy improves the longer you take performing a task. The data could have been analyzed for time/accuracy trade-offs.

28/06/2015Dr Andy Brooks29 Figure 6 suggest two trends are present: –the responses to questions 1-6 and, –the responses to questions What could have caused this? No real differences here? ©ACM

28/06/2015Dr Andy Brooks30 More control group subjects answered question 4 correctly. Why? Why could no control group subject provide a correct answer to questions 6, 7, and 12? ©IEEE