1. stanford hci group / cs376 research topics in human-computer interaction http://cs376.stanford.edu Software Tools Matt Wright Center for Computer Research in Music and Acoustics (CCRMA), Stanford 14 October 2004

2. 2 Software Tools Readings  Past, Present, and Future of User Interface Software Tools, Brad Myers, Scott E. Hudson, Randy Pausch  Natural Programming Languages and Environments, Brad A. Myers, John F. Pane, Andy Ko

3. 14 October 2004 3 Software Tools Past, Present, and Future  Past: ideas that lived or died  Present: homogenous WIMP GUIs, mature tools to make them  Future: Ubicomp form factors and networking, recognition- based input, 3D, end-user programmability (!)

4. 14 October 2004 4 Software Tools Determinants of Success  The parts of the user interface that are addressed  Threshold and Ceiling  Path of Least Resistance  Predictability  Moving Targets

5. 14 October 2004 5 Software Tools What Worked  Window Managers (60s): drawing, input  Window Toolkits (Mac): components & framework  Event “languages” (“prog. model”?)  Interface builders: GUI GUI builders  Component Systems  Scripting Languages  Hypertext  OO

6. 14 October 2004 6 Software Tools Why They Worked  Good models (powerful and well-matched to problem domains)  Good factoring  Scripting + components + interface builder = platform

7. 14 October 2004 7 Software Tools What Didn’t Work (1/2)  User Interface Management Systems:  High-level, abstract descriptions of interfaces, system generates implementation  Designer wants to control low- level pragmatics  Standard UI elements do this factoring better  Formal Language Based Tools  Based on compiler tools (Regexp, CFG)  wrong model (dialog, sequence), too hard to use

8. 14 October 2004 8 Software Tools What Didn’t Work (2/2)  Constraints  Good for users (spreadsheets, drawing)  Bad for UI programmers (unpredictable, hard to use and debug)  Good for graphical layout (NeXTStep, HTML(?))  Model Based/Automatic Techniques  Very high level abstract description of interface  Declare I/O needs; heuristics select components, layout, etc.  Unpredictability, require new language, limited

9. 14 October 2004 9 Software Tools Why They Didn’t Work  “Impedence mismatch” between problem and solution  UIMS, Model-based too ambitious  Formal Lang. tools too limited  Constraints successful only at right “scope” of what’s constrained  Moving Target problem

10. 14 October 2004 10 Software Tools Futures: commodification  CS researchers less ahead of HW curve  Multimedia  Enable Ubicomp

11. 14 October 2004 11 Software Tools Future: Ubicomp  Deal with different I/O capabilities  Back to high-level, device- independent UI specification?  Need device prototyping tools  Networking, syncing  Interpersonal and intrapersonal  Standards vs. capitalism?  Paper emphasizes tools; I think representations are more important.

12. 14 October 2004 12 Software Tools Future: Recognition (1/2)  Gestures, handwriting, speech  Must be interpreted by software to identify content  “Tools will be needed that hide all of this complexity and provide an easy-to-use interface to programmers.”  I think that’s a fantasy. (E.g., real-time fundamental frequency estimation for interactive computer music.)

13. 14 October 2004 13 Software Tools Future: Recognition (2/2)  Uncertainty of input requires feedback  Interpretation of input requires deep knowledge of context  No obvious segmentation into discrete events

14. 14 October 2004 14 Software Tools Future: 3D  VRML not accepted  Forget tools, we still need paradigms:  Selection  Interaction  Navigation  Metaphors

15. 14 October 2004 15 Software Tools Future (?!?): End-User Programming, Customization, Scripting  Yay spreadsheets  MS Office now “totally” VB scriptable

16. 14 October 2004 16 Software Tools Misc. Future Issues  Users with worse dexterity, memory, etc.  Translucency, magic lens  Components: easy to use, hard to make  Sitting, both hands free, looking at UI?  Computer has user’s full attention?  Build evaluation support into tools

17. 14 October 2004 17 Software Tools [This space intentionally left blank]

18. 14 October 2004 18 Software Tools “Natural”  Ubicomp: pens are natural; typewriters are artificial.  Myers, Pane, Ko:  “language and environment… work in the way that people who do not have programming experience would expect.”  “create programming languages and environments that are more “natural,” or closer to the way people think about their tasks.”

19. 14 October 2004 19 Software Tools “Unnatural”?  Create thinking that empowers people to accomplish their tasks:  Indirection  Recursion  Matrices  Function as object  …

20. 14 October 2004 20 Software Tools The Big Idea  Design of programming languages and environments should be guided by Empirical Studies of Programmers (ESP) and HCI:  What prog. lang. constructs cause bugs?  What questions do we ask (ourselves) while debugging?  What ideas do fifth graders use to solve programming tasks on paper?  HCI applied to programming tools

21. 14 October 2004 21 Software Tools Programming as Adapting to the Computer  “[P]rogramming is the process of transforming a mental plan in familiar terms into one that is compatible with the computer”  Therefore computer should work the way we expect (i.e., the way we think).  Directness applied to prog. lang. design.  Straw man: Array sum by iteration in C.

22. 14 October 2004 22 Software Tools Some Things People Expect  Undo  Cut and Paste  Save As  Search and replace  Command and filename completion …none are “natural”; they came into our collective consciousness via computers

23. 14 October 2004 23 Software Tools Studies of “natural” problem solving  “Participants were presented with programming tasks and asked to solve them on paper using whatever text or diagrams they wanted to use.”  85-page survey of novice programming research: http://web.cs.cmu.edu/~pane/cmu-cs-96- 132.html

24. 14 October 2004 24 Software Tools Example task

25. 14 October 2004 25 Software Tools Results  Events cause action (“When pacman loses all his lives, it’s game over.”)  Aggregate operators > iteration  Boolean logic rare and often buggy. (E.g., “the cars with license plates from Georgia and Louisiana.”)  Layout in pictures, actions and behaviors in text

26. 14 October 2004 26 Software Tools HANDS Screenshot

27. 14 October 2004 27 Software Tools HANDS Programming Model  All data are on global, persistent, visible “cards”:  Front: name/value lists  Back: graphics or text for visual display  Cards in  Computation done by “Handy” handling events  System generates all events (collisions, clicks…)  Imperative event-handler code (“add 100 to the game’s score”)  Operations manipulate cards’ properties

28. 14 October 2004 28 Software Tools Success of HANDS  Fifth graders could learn in 3 hours (!) and then use to solve programming problems.  Control group used “limited” language without queries, aggregate operations, and data visibility:

29. 14 October 2004 29 Software Tools How We Debug  Hypothesizing what runtime actions caused failure  Observing data about a program’s runtime state  Restructuring data into different representations  Exploring restructured runtime data  Diagnosing what code caused faulty runtime actions  Repairing erroneous code to prevent such actions

30. 14 October 2004 30 Software Tools Questions at time of failure  (32%) Why did x happen?  (68%) Why didn’t x happen?  (0%) Other  WhyLine: a Workspace that Helps You Link Instructions to Numbers and Events

31. 14 October 2004 31 Software Tools Using the WhyLine

32. 14 October 2004 32 Software Tools

33. 14 October 2004 33 Software Tools WhyLine results  Master’s students, identical conditions  Average debugging time 20 seconds/bug instead of 155 seconds/bug. (!)  Group with WhyLine completed 40% more tasks in given 90 minutes.

34. 14 October 2004 34 Software Tools Good Ideas  Get real data on how (all kinds of) programmers make bugs, waste time…  Design programming tools that avoid these problems  …do these techniques scale to “real” programming?