1. Evaluating the usability of the BlueJ environment. Marilyn Barallon Supervisor: Dr.Linda McIver

2. Outline. What is BlueJ. What is usability. How we measure usability. What has been done in the past. Studying BlueJ. What we found- a ‘picture’ of what students are doing with BlueJ Conclusion Further Work

3. What is BlueJ? A visual introductory programming environment to help teach JAVA and OO. Based on the Blue System developed at Sydney and Monash University. Currently being maintained by Deakin University, Australia and the University of Kent, UK. Currently being used by 412 institutions worldwide. Encourages an ‘objects first’ approach to teaching Object Orientation.

4. What is BlueJ? (cont’d). Designed to be simple to use:  A simple main window: UML picture of the program ‘Workbench’ area that displays objects that can be interacted with.  Simple Debugger: Stack window Instance, local and static variable windows State of the program window (running, finished)  A simple syntax directed editor

5. What is usability? Subjective There are three main views from which usability can be observed and assessed (N Bevan and J Kirakowski et al):  Product-oriented view: features of the product  User-oriented view: mental effort and attitude of the user.  User-performance view: how the user interacts with the product. Our research focuses on an assessment of BlueJ’s usability based on the last two views.

6. Measuring Usability? Evaluation frameworks (examples):  Price et al. framework includes 6 broad and general categories: Scope Content Form Method Interaction Effectiveness  Phillips and Mehandjiska et al. framework included 5 categories, each of which was further broken down into specific subcategories: Learnability Efficiency Flexibility Robustness Feedback

7. What has been done? Only formal evaluations so far: surveys and data loggers. Problems with survey data:  Requires students to reflect on past interactions  Rank thoughts and/or feelings according to unique judgement of what is ‘very useful’  Never really used anything else, so how would they know? (How well did it help you learn JAVA?) Problems with program logger data:  Fails to tell us what was happening between compilations.  Or what were the ‘substantial edits’

8. Understanding the user. Usability assessment requires understanding of users. Surveys and data loggers do not provide a mechanism for understanding users. Video-taped ‘think-out-loud’ observational experiments  Directly observe and analyse ‘what’ students are doing, natural behaviour.  And ask ‘why’ at the time.  ‘Discuss’ problems they encounter, at the time. (JAVA or interface problem)  Create an overall ‘picture’ of BlueJ user behaviour.

9. Problems with Observational Experiments. Researcher Bias Reactive behaviour Limited degree to which we can generalise behaviour to other people. In order to mitigate these risks, long term observational studies and training could be done.

10. Studying BlueJ Observations of students using BlueJ. Better understanding of users. A first step towards a comprehensive usability study. Detailed picture of student understanding of BlueJ and JAVA.

11. Studying BlueJ (cont’d) Observations involved debugging a program using BlueJ.  Two-hour video taped ‘think-out-loud’ sessions including a short questionnaire  Conducted 4 pilot study experiments, with final year students (1 from software engineering, 2 from digital systems, 1 computer science).  Conducted 5 final experiments with 2 nd years (Computing degrees) whom have completed cse1203- Programming with JAVA 2.

12. The Program. Program draws pictures using graphical output, making it easy to ‘see’ what is an error. Students were first shown what the correct output of the program should look like. 5 semantic bugs.

13. Problem Solving Abilities. Did not use BlueJ to explore the program.  Minimal exploratory behaviour of the program with the workbench.  Jumped straight into code. Lacked independent exploration and testing abilities.  Sought guidance whenever they could. As a result, poor hypothesis establishment and refinement of the behaviours of errors.

14. Can students use the debugger? One student. Common misconceptions and difficulties:  Setting breakpoints on non-executable lines of code.  Instantiation of the debugger window- (which method to call)  Did not understand how the debugger traces the program. Should I set one on each line? Will it go from here to there? Can I set a breakpoint on each line in this class and use the debugger to step through this class? Can I move backwards? These problems were seen in both final and 2 nd years.

15. Common Debugging Strategies. All students used code comparison at least once. Students appeared to edit ‘Suspicious’ statements for no apparent reason.  For example in a loop condition, change ‘<‘ to a ‘<=‘ We hypothesis that students were selecting particular statements to change based in the unfamiliar ways in which they were being used  For example in a loop structure, count+=2 instead of count +=1 Interestingly, most students expressed print statements as their preferred testing strategy.

16. Problems with BlueJ. Compiler and JAVA exception error messages.  Failed to take notice of them.  Failed to understand what they were telling them. Status bar:  Students failed to use it to distinguish between when the machine is busy and when it is free. ‘Remove’ option on the workbench  Problem for ‘new’ users learning OO, language syntax and BlueJ.

17. Conclusion. Final and 2 nd years do not possess independent debugging and problem solving ability. Students cannot use the debugger.  Shown the correct workings of the program beforehand.  Graphical output makes it easy to understand ‘what’ is happening.  Could ask questions at any time.

18. Conclusion (cont’d) Lack of understanding of JAVA and OO. Find ways of teaching and developing debugging and problem solving ability.  Enforcing use of the debugger. Need to make aware to students how it facilitates exploration. The way BlueJ handles its error messages, displays its status bar and object workbench need to be investigated further for possible usability issues and redesign.

19. Further Work. We can now move forward and construct a usability framework specific to BlueJ. Redesign the way error messages are displayed:  Relocate the error message area to another section of the screen.  Remove the error message area and use pop-up windows  Keep the error message area and better direct the users attention to it. Further investigation to determine what ‘new’ BlueJ users think the workbench ‘remove’ option does. Re-locate the status bar so that it encourages students to use it to determine when they should perform their next action.