1. stanford hci group / cs376 http://cs376.stanford.ed u Jeffrey Heer · 26 May 2009 User Interface Toolkits

2. Final Project Presentations Tuesday June 9, 3:30-6:30pm, 104 Gates 8 minute presentations  6 min for research, 2 min for questions  Start with an overview:  1 sentence statement of your research result  1 slide / 4 sentences of what you did and why  Rest of time on details. Assume audience is familiar with HCI issues: focus on your work 2

3. cryptonomicon.com/beginning.html In the beginning…

4. The Xerox Alto (1973)

5. Event-Driven UIs Old model (e.g., UNIX shell, DOS)  Interaction controlled by system, user queried for input when needed by system Event-Driven Interfaces (e.g., GUIs)  Interaction controlled by user  System waits for user actions and then reacts  More complicated programming and architecture

6. Periodic Table of Motif Widgets

7. 7 Toolkit Example: Java Swing  GUI toolkit with a widget set and an API

8. 8 User Interface Builders

9. Event Dispatch Loop Event Queue Queue of input events Event Loop (runs in dedicated thread) Remove next event from queue Determine event type Find proper component(s) Invoke callbacks on components Repeat, or wait until event arrives Component Invoked callback method Update application state Request repaint, if needed Mouse moved (t 0,x,y)

10. Event Dispatch Window Panel TextAreaPanel Button Label Event Queue Mouse moved (t 0,x,y) Mouse pressed (t 1,x,y,1) Mouse dragged (t 2,x,y,1) Key typed (t 3, ‘F1’) … (queues and dispatches incoming events in a dedicated thread) /* callback for TextArea */ public void mouseMoved(e) { // process mouse moved event }

11. 11 Success of Tools Today’s tools are highly successful  Window Managers, User Interface Toolkits, and Interface Builders are ubiquitous  Most software built using them  Are based on many years of HCI research:  Brad A. Myers. “A Brief History of Human Computer Interaction Technology.” ACM interactions. Vol. 5, no. 2, March, 1998. pp. 44-54.

12. Why Tools? The quality of the interfaces will be higher. Why?  Rapid prototyping.  Easier to incorporate changes motivated by evaluation.  Re-use affords investment in high quality tools.  Consistency of interface design.  Enable collaboration among specialists. 12

13. Why Tools? The user interface will be easier to create and maintain. Why?  Less code to write due to component re-use.  Better modularization, separation of concerns  Tools may abstract complex systems or algorithms.  Easier to port an application to different hardware or software environments if device dependencies are isolated in the user interface tool. 13

14. 14 What should tools do? Help design the interface given a specification of the tasks. Help implement the interface given a design. Help evaluate the interface after it is designed. Create easy-to-use interfaces. Allow the designer to rapidly investigate different designs. Allow non-programmers to create user interfaces. Provide portability across different machines and devices. Be easy to use themselves.

15. 15 Tools might do:  Provide sets of standard UI components  Guide the implementation  Help with screen layout and graphic design.  Validate user inputs  Handle user errors  Handle aborting and undoing of operations  Provide help and prompts  Deal with field scrolling and editing  Insulate the application from all device dependencies and the underlying software and hardware systems.  Support features in the interface that allow the end user to customize the interface.

16. Application Types What application domains are “deserving” of specialized toolkit support? 16

17. 17 Application Types Each has own unique UI style, and implementation challenges  Word processors  Drawing programs ( CAD/CAM)  Painting programs  Mail readers  Spreadsheets  Programming environments / code editors  WWW  Interactive games  Visualizations  Automated-teller machines (ATM)  Virtual Reality  Multi-media  Video  Animation  Controlling machinery

18. 18 DiamondSpin Toolkit  Toolkit for tabletop user interfaces  [Shen, Vernier, Forlines, Ringel] CHI ’04

19. 19

20. 20 Tabletop UI Needs  Multi-user support  Identity-aware widgets  Multiple menus  Public and personal spaces  Resolving conflicting actions  Arbitrary orientation of UI elements  Techniques to control orientation and layout  Rotation sensitive components

21. 21 iStuff Toolkit  Physical UI components for ubiquitous computing environments (multiple users, devices, and applications)  [Ballagas, Ringel, Stone, Borchers], CHI ‘03

22. 22 iStuff Design  iStuff components  Device + proxy (“smarts” are in the proxy)  PatchPanel  Translate between iStuff events and application-specific events  Run-time retargetable events  Address dimension mismatches

23. 23

24. 24 Discussion of Themes Address the useful & important aspects of UIs  Narrower tools have been more successful than ones that try to do “everything”  Do one thing well Threshold / Ceiling  Research systems often aim for high ceiling  Successful systems often seem to instead aim for a low threshold  Impossible to have both?

25. 25 Threshold & Ceiling Difficulty of Use Sophistication of what can be created Goal HyperCard Visual Basic Director (v6) HyperTalk xCmds Basic C Programming Lingo C Programming Programming in C MFC Click and Create

26. 26 Discussion of Themes, cont. Path of Least Resistance  Tools should guide implementers into better user interfaces  Goal for the future: do this more? Predictability  Programmers do not seem willing to release control  Especially when system may do sub- optimal things Moving Targets  Long stability of Macintosh Desktop paradigm has enabled maturing of tools

27. Evaluating User Interface Tools  “An API is a user interface where programmers are the users”  ?  Evaluate toolkit as you would a UI?  Factors  Expressiveness (Ceiling)  Development Rate (of skilled user)  Learning Rate (to become skilled)  Performance  Portability 27

28. 28 The Future of Interface Tools Supporting…  Prototyping  Collaboration  Evaluation  of interfaces built with tools  of tools themselves – how to prototype, test, iterate?  Emerging interface styles, such as  mobile  recognition-based UIs (speech, pens, vision)  multiple devices

29. 29 Current Research Challenges  Complex design space  e.g., do we have to update the toolkit every time someone creates a new sensor or actuator?  Ambiguous input  Speech, gestures, computer vision, etc. aren’t recognized as accurately as mouse clicks. Should the toolkit handle the recognition?

30. 30 Summary  Toolkits provide reusable interface components to simplify UI development  Toolkit trap: it’s tempting to only make UIs that the toolkit makes easy, instead of making what’s best for a specific app  Toolkit types:  WIMP (Garnet, Swing, Motif, etc)  Specialty (Phidgets, iStuff, Papier-Mache, DiamondSpin, GroupKit, Peripheral Displays Toolkit, etc)

31. Next Time: Adaptive Interaction You should use Sharon’s online paper viewer for the next set of readings. 31