1. 5-1 5. Computer Science and Human-Computer Interaction 5.1. Motivation 5.2. User Interface Software Concepts 5.2.1. Design Process for Interactive Systems 5.2.2. Interaction Techniques 5.2.3. Interaction Styles 5.2.4. User Interface Management Systems 5.2.5. User Interface Description Languages 5.3. User Interface Software Tools 5.3.1. Window Systems 5.3.2. Toolkits 5.3.3. UIMS 5.4. Future Directions

2. 5-2 5.1. Motivation Easier to use, but harder to program [Myers] Multiprogramming Real-time programming Robustness Testability Modularization Need to support iterative design and testing Software tools

3. 5-3 5.2. User Interface Software Concepts Dialogue independence Interface (syntax) vs. application (semantics, business logic) Model-View-Controller architecture Interaction techniques Basic interaction styles Design levels: conceptual, semantic, syntactic, lexical Generate UI from abstract description Early research in user interface management system (UIMS) and user interface description languages (UIDL) User interface software tools New interaction styles (non-WIMP)

4. 5-4 5.2.1. Design Process for Interactive Systems Partition into Four Levels [Foley & Wallace] Conceptual Semantic Syntactic Lexical

5. 5-5 5.2.2. Interaction Techniques Interaction task Interaction device Interaction technique A way of using a device to perform a generic HCI task Widget in a toolkit Two information processors communicating via constrained interface [Tufte] Goal: Increase bandwidth across channel

6. 5-6 5.2.3. Interaction Styles: Current Command Language Question and Answer Form-based Menu Natural Language Direct Manipulation

7. 5-7 Virtual and augmented reality Tangible interfaces Ubiquitous, pervasive, handheld Lightweight, tacit, passive, non- command, context-aware Affective computing Multi-modal interfaces Reality-based interfaces Brain-computer interaction Interaction Styles: Emerging

8. 5-8 Dialogue Independence [Hartson & Hix] User Interface Management System (UIMS) Approach [Olsen] Before After Levels of design: Conceptual [Johnson & Henderson], Semantic, Syntactic, Lexical 5.2.4. User Interface Management Systems Research Application and its own User Interface USER UIMS Application (Syntactic+Lexical) (Semantic)

9. 5-9 Classes of Programmers and Tool Users Runtime Application End User UIMS Tools User Interface Designer Application Programmer (UIDL)

10. 5-10 Same UIMS for many applications Different interfaces to same application Interface to null application (rapid prototyping) Dialogue Independence USER Application 1 Application 2 Application 3 UIMS User Interface 1 User Interface 2 User Interface 3 ApplicationUSER

11. 5-11 5.2.5. User Interface Description Languages How does the user interface designer describe the desired user interface to the UIMS or toolkit? Layout Behavior With a User Interface Description Language (UIDL) May be interactive Generate UI from abstract description UIDL describes user-visible behavior, but not implementation More in Section 5.3.3 Other uses of UIDL (evaluation)

12. 5-12 Goal: Support dialogue independence and separation of programming roles [Foley, van Dam, Feiner, & Hughes] 5.3. User Interface Software Tools Toolkit Application Window system/graphics package Operating system Hardware UIMS

13. 5-13 5.3.1. Window Systems Manage shared resources (pixels, keyboard, mouse), provide clipping, translation Tells client when exposed, resized, or other damage Could implement in different places Single built-in window manager or allow separate Dispatcher (N.B. Security classifications shown are simulated only.) AccessWindowMouseEvent (Event *e) {... FileWindowMouseEvent (Event *e) { If (e->type==MS_LEFT)...

14. 5-14 5.3.2. Toolkits Library of reusable routines for implementing human-computer interface features Basic components Set of widgets "Intrinsics" Does not try to describe or support user interface as a whole Standard look and feel Encourage consistency But imposes limits on interaction style

15. 5-15 Toolkits (cont'd) Examples Visual Studio (Windows) Interface Builder (Macintosh) GTK (X/Unix/Linux) Based on controls or "widgets" (N.B. Security classifications shown are simulated only.) AccessCheckBox (Boolean val) ClassifTypein (char *val) DisplayMsgFile ()

16. 5-16 5.3.3. UIMS Higher level than procedure library May include main program, provide external control of application Call semantics/business logic as subroutines Message to component software Design time, run time, and evaluation time tools [Myers] Module partitioning: Runtime vs. interface design time partitioning

17. 5-17 Model-View-Controller Design pattern, originally Smalltalk Found, in some form, in most UI toolkits Matches dialogue independence, syntactic/semantic split Extend to web browser USER ModelControllerView

18. 5-18 User Interface Description Language Designer creates it, UIMS executes it Specific language or technique suitable for describing user interfaces Describe user-visible behavior, but not implementation Goal: Generate a UI from high-level, abstract description Platform-independent if possible (Windows, Mac, X) Platform-type-independent if possible (desktop, cellphone, voice)! Model-based UI development [Paterno; Jacob, Limbourg, & Vanderdonckt] Current GUI practice Has evolved from general purpose UIDL into small, stable set of widgets and classes Change with new UI styles, portability needs Choice of design language at heart of UIMS

19. 5-19 Based on techniques for specifying static languages Both can be modified for interactive languages BNF send-cmd::= SEND DRAFTNAME RECIPNAME State Transition Diagram Alternate Form of State Transition Diagram start: SEND ->getdr getdr: DRAFTNAME ->getre getre: RECIPNAME ->end Basic Classes of UIDLs startgetdrgetreend SENDDRAFTNAMERECIPNAME

20. 5-20 Early UIMS/UIDL Research Language-based Menu Network [Kasik] BNF, grammar-based [Olsen] State transition diagram [Newman] Event language Declarative Non-language-based Interactive builder (bottom up) Current state of practice, use GUI techniques to specify GUI Plug together components with pre-defined interactive behavior Automatic generation (top down) [Foley] By demonstration [Myers] Other categories Constraint-based [Hudson] User-oriented [Moran]

21. 5-21 Example: Visual Programming for UI Layout vs. Behavior UIMS, with state diagrams for behavior [Jacob, “A state transition diagram language...”] Visual Basic, pre-programmed behavior

22. 5-22 Example: State Diagram as UIDL Command-driven Graphical Report Editor [Jacob, “Using formal specifications...”]

23. 5-23 UI Evaluation Directly from UIDL Evaluate interface without building system Emerging area, but lack underlying formal knowledge of UI design Analyze user interface to detect specific properties [Darlington et al.] Apply human performance models [Card] Automated screen design critique [Tullis] Prove safety-critical properties Screen mockup, prototyping tools “Slide show" of canned screens based on user inputs UIMS sans application (use stubs for semantics) Visual Basic, Powerpoint, plain HTML also used this way

24. 5-24 UI Tools: Standardization Early: Many desktop platforms No standardization Current: ~3 major platforms, standard widgets Settled into stable toolkits, widget libraries, GUI class libraries UIMS seems like overkill Emerging: Mobile, automobile Diverse platforms, no standardization Wider range of interaction techniques Model-based development, UIMS redux Future: Same issue for new non-WIMP interaction styles Still wider range of possible interactions

25. 5-25 UI Tools: Trends UI toolkits, libraries Become important, integral part of software development tool sets e.g., Macintosh Cocoa UI features are well integrated Good uptake despite unusual programming language Web browser as UI platform Cloud computing Chrome-only OS New tools for web UIs, but still clumsy, poorly integrated Making cross platform toolkits less significant

26. 5-26 UI Tools: Mobile Devices Basic HCI development and design principles Largely GUI, not new interaction style + gestures, GPS,... Use context: Full attention or not? Development tools Early: Clumsy tools, restricted UI designs Much like early GUIs, function key-driven (pre-mouse) GUI Current: UIs and tools very similar to standard GUI iPhone: Interface Builder (Mac, Objective C) Droid: Java + Swing (minor differences) UI Consistency Low level consistency High level models, UIDL could help Example: IMAP mail clients

27. 5-27 5.4. Future Directions: Emerging Interaction Style Parallel, Highly-interactive (non-WIMP) Lightweight, Context-aware, Non-command Continuous (+ Discrete) Tangible Interfaces Microprocessors, 3D forms Ubicomp, Pervasive, Context-aware Sensors, networks Reality-based Interaction Brain-computer Interaction

28. 5-28 Implications for Software CurrentFuture Single thread I/OParallel, asynchronous dialogues; may be interrelated Discrete tokensContinuous inputs and responses (plus discrete) Precise tokensProbabilistic input, not easily tokenized Sequence, not timeReal-time requirements, deadline-based Explicit user commands Passive monitoring of user

29. 5-29 Reality-Based Interaction Connects several emerging interaction styles Understand together as a new generation of HCI through RBI [Jacob et al., “Reality-Based Interaction...”] Computer interaction more like rest of world Exploit skills and expectations user already has about simple, everyday, non-digital world Reality + extensions Design tradeoffs:

30. 5-30 Tangible User Interfaces Simple, transparent mechanical structures Use knowledge of physical world to operate Augment physical objects with digital meaning Combine physical + digital representations Exploit advantages of each [Jacob et al., “A Tangible Interface...”] [Zigelbaum et al., The Tangible Video Editor...”]

31. 5-31 Technology for TUI Sensors Produce signal in response to change in surroundings Actuators Produce physical change in response to signal Microcontroller Communicates with devices and with main computer Other technologies RFID Computer vision... Wider range of I/O events than GUI Development methodology, tools None!

32. 5-32 A Visual Language for Modeling Tangible User Interfaces [Shaer] TAC paradigm Each TUI consists of token within a constraint Same object may sometimes be token, sometimes constraint Two tier model fits well Dialogue (states, storyboard) Interaction (especially continuous) TUIML

33. 5-33 Conclusions: 1. Research Background Background for following current research in user interface software Interaction styles Dialogue independence Four levels of design User interface management system (UIMS) User interface description language (UIDL) Goal: Generate UI from abstract description Types of software tools New interaction styles (non-WIMP) and requirements Easier to use, harder to program?

34. 5-34 Conclusions: 2. Practice What research ideas can be applied in building user interfaces today? Separate interface from application programming Model-View-Controller architecture Use specialized tools (UIMS) for leverage Use specialized languages (UIDL) for perspicuity and portability

35. 5-35 Sources Textbooks Olsen Foley, van Dam, Feiner, & Hughes Shneiderman, Plaisant, Cohen, & Jacobs History Early UIMS research [Buxton et al.; Foley and Wallace; Jacob; Kasik; Newman; Olsen] Classic surveys: ACM Transactions on Computer-Human Interaction [Myers, Hudson, & Pausch], Computing Surveys [Hartson & Hix] Current research ACM UIST Conference on User Interface Software and Technology (annual conference) ACM EICS Symposium on Engineering Interactive Computing Systems (conference) CADUI Conference on Computer-Aided Design of User Interfaces (conference) ACM Transactions on Computer-Human Interaction (journal) interactions (ACM magazine)

36. 5-36 References W. Buxton, M.R. Lamb, D. Sherman, and K.C. Smith, ''Towards a Comprehensive User Interface Management System,'' Computer Graphics 17(3) pp. 35-42 (1983). S.K. Card, T.P. Moran, and A. Newell, ''The Keystroke-Level Model for User Performance Time with Interactive Systems,'' Comm. ACM 23 pp. 396-410 (1980). J. Darlington, W. Dzida, and S. Herda, ''The Role of Excursions in Interactive Systems,'' International Journal of Man-Machine Studies 18 pp. 101-112 (1983). J. Foley, W.C. Kim, S. Kovacevic, and K. Murray, ''Defining Interfaces at a High Level of Abstraction,'' IEEE Software 6(1) pp. 25-32 (January 1989). J.D. Foley, A. van Dam, S.K. Feiner, and J.F. Hughes, Computer Graphics: Principles and Practice, Addison- Wesley, Reading, Mass. (1990). J.D. Foley and V.L. Wallace, ''The Art of Graphic Man-Machine Conversation,'' Proceedings of the IEEE 62(4) pp. 462-471 (1974). H.R. Hartson and D. Hix, ''Human-computer Interface Development: Concepts and Systems for its Management,'' Computing Surveys 21(1) pp. 5-92 (1989). S.E. Hudson, ''Graphical Specification of Flexible User Interface Displays,'' Proc. ACM CHI'89 Human Factors in Computing Systems Conference pp. 105-114 (1989). R.J.K. Jacob, "A Specification Language for Direct Manipulation User Interfaces," ACM Transactions on Graphics, Vol. 5(4) pp. 283-317 (1986) http://www.cs.tufts.edu/~jacob/papers/tog.pdf]. R.J.K. Jacob, ''A State Transition Diagram Language for Visual Programming,'' IEEE Computer 18(8) pp. 51-59 (1985) [http://www.cs.tufts.edu/~jacob/papers/ieeecomputer.pdf]. R.J.K. Jacob, "Using Formal Specifications in the Design of a Human-Computer Interface," Communications of the ACM, Vol. 26(4) pp. 259-264 (1983) [http://www.cs.tufts.edu/~jacob/papers/cacm.pdf].http://www.cs.tufts.edu/~jacob/papers/cacm.pdf R.J.K. Jacob, A. Girouard, L.M. Hirshfield, M.S. Horn, O. Shaer, E.T. Solovey, and J. Zigelbaum, "Reality- Based Interaction: A Framework for Post-WIMP Interfaces," Proc. ACM CHI 2008 Human Factors in Computing Systems Conference pp. 201-210, ACM Press (2008) [http://www.cs.tufts.edu/~jacob/papers/chi08.pdf].

37. R.J.K. Jacob, H. Ishii, G. Pangaro, and J. Patten, "A Tangible Interface for Organizing Information Using a Grid," Proc. ACM CHI 2002 Human Factors in Computing Systems Conference pp. 339-346, ACM Press (2002) [http://www.cs.tufts.edu/~jacob/papers/chi02.pdf]. R.J.K. Jacob, Q. Limbourg, and J. Vanderdonckt (Eds.), Computer- Aided Design of User Interfaces IV, Kluwer Academic Publishers, Dordrecht (2005). J. Johnson and A. Henderson, "Conceptual Models: Begin by Designing What to Design," interactions, Vol. 9(1) pp. 25-32 (January 2002). D.J. Kasik, ''A User Interface Management System,'' Computer Graphics 16(3) pp. 99-106 (1982). T.P. Moran, ''The Command Language Grammar: A Representation for the User Interface of Interactive Computer Systems,'' International Journal of Man-Machine Studies 15 pp. 3-50 (1981). B.A. Myers, ''User Interface Software Tools,'' ACM Transactions on Computer-Human Interaction 2(1) pp. 64- 103 (March 1995). B.A. Myers, Creating User Interfaces by Demonstration, Academic Press, Boston (1988). B. Myers, S.E. Hudson, and R. Pausch, "Past, Present, and Future of User Interface Software Tools," ACM Transactions on Computer- Human Interaction, Vol. 7(1) pp. 3-28 (March 2000). Also reprinted in Human- Computer Interaction in the New Millenium, ed. J.M. Carroll, Addison-Wesley/ACM Press, Reading, Mass., 2002, pp. 213-233. W.M. Newman, ''A System for Interactive Graphical Programming,'' Proc. Spring Joint Computer Conference pp. 47-54, AFIPS (1968). D.R. Olsen, Building Interactive Systems: Principles for Human-Computer Interaction, Course Technology Inc. (2009). D.R. Olsen, User Interface Management Systems: Models and Algorithms, Morgan Kaufmann, San Mateo, Calif. (1992). F. Paterno, Model-Based Design and Evaluation of Interactive Applications,, Springer-Verlag, London (2000). O. Shaer and R.J.K. Jacob, "A Specification Paradigm for the Design and Implementation of Tangible User Interfaces," ACM Transactions on Computer-Human Interaction (2009) [http://www.cs.tufts.edu/~jacob/papers/tochi.shaer.pdf]. B. Shneiderman, C. Plaisant, M. Cohen, and S. Jacobs, Designing the User Interface: Strategies for Effective Human-Computer Interaction, Addison-Wesley, Reading, Mass. (2009). E.R. Tufte, ''Visual Design of the User Interface,'' IBM Corporation, Armonk, N.Y. (1989). 5-37

38. T.S. Tullis, ''A Computer-Based Tool for Evaluating Alphanumeric Displays,'' Proc. INTERACT '84 Conference, B. Shackel, ed. (1984). J. Zigelbaum, M. Horn, O. Shaer, and R.J.K. Jacob, "The Tangible Video Editor: Collaborative Video Editing with Active Tokens," Proc. TEI 2007 First International Conference on Tangible and Embedded Interaction pp. 43-46 (2007) [http://www.cs.tufts.edu/~jacob/papers/tei07.zigelbaum.pdf]. 5-38