Past, Present and Future of User Interface Software Tools Brad A. Myers (work with Scott E. Hudson and Randy Pausch) Human-Computer Interaction Institute School of Computer Science Carnegie Mellon University http://www.cs.cmu.edu/~bam bam@cs.cmu.edu

Introduction User Interface Software Tools Help developers design and implement user interfaces Focus on Tools, but influenced by future UIs Today’s tools are highly successful Window Managers, Toolkits, Interface Builders ubiquitous Most software built using them Are based on HCI research Brad A. Myers. “A Brief History of Human Computer Interaction Technology.” ACM interactions. Vol. 5, no. 2, March, 1998. pp. 44-54. http://www.cs.cmu.edu/~amulet/papers/uihistory.tr.html Future tools must be different

What Worked Window Managers and Toolkits Event Languages Graphical, Interactive Tools Component Architectures Hypertext Object Oriented Programming

Window Managers Multiple (tiled) windows in research systems of 1960’s: NLS, etc. Overlapping introduced in Alan Kay’s thesis (1969) Smalltalk, 1974 at Xerox PARC Successful because multiple windows help users manage scarce resources: Screen space and input devices Attention of users Affordances for reminding and finding other work

Toolkits A collection of widgets (“controls”) Menus, scroll bars, text entry fields, buttons, etc. Toolkits help with programming Help maintain consistency among UIs Key insight of Macintosh toolbox Successful partially because address common, low-level features for all UIs

Event Languages Create programs by writing event handlers Many UIMSs used this style Univ. of Alberta (1985), Sassafras (1986), etc. Now used by HyperCard, Visual Basic, Lingo, etc. Toolkits with call-backs or action methods are related Advantages: Natural for GUIs since generate discrete events Flow of control in user’s hands rather than programmer’s Discourages moded UIs Won’t work well in future

Graphical Interactive Tools Create parts of user interface by laying out widgets with a mouse Examples: Menulay (1983), Trillium (1986), Jean-Marie Hullot from INRIA to NeXT Now: HyperCard, Interface Builders, Visual Basic’s layout editor, resource editors, “constructors” Advantages: Graphical parts done in an appropriate, graphical way Accessible to non-programmers

Component Architectures Create applications out of components which are separately developed and compiled In UI software, each component controls an area of the screen Example: drawing component handles picture inside a document Invented by Andrew research project at CMU (1988) Now: OLE, OpenDoc, ActiveX, Java Beans

Hypertext Ted Nelson named it in 1965 and developed a Hypertext system at Brown University Important systems: NLS (1967), Hyperties (1986) World-Wide Web Phenomenal success due to: Ease of use of Mosaic browser Support for embedded graphics Support for easy authoring Tim Berners-Lee used the hypertext idea to create the World Wide Web in 1990 at the government-funded European Particle Physics Laboratory (CERN).

Object Oriented Programming Success of OO owes much to UI software field Popularized by Smalltalk GUI elements (widgets) seem like objects Have state, accept events (messages) Rise parallels GUIs C++ with Windows 3.1

What Hasn’t Caught On User Interface Management Systems Separation of UI and application is harder than expected Formal Language-Based Tools Grammars and Transition networks are not appropriate for direct manipulation Constraints Declare once, system maintains Get to be unpredictable and hard to control Model-Based and Automatic Techniques Results are not as good as hand-created Sometimes, no savings of effort

Future Prospects and Visions Important Trends Ubiquitous Computing Move to recognition-based interfaces End-user customization and scripting

Ubiquitous Computing Computation embedded in many kinds of devices Digital pagers and cell phones, Palm Pilots, CrossPads, laptops, wall-size displays, “smart” rooms Next wave: easy communication with radio E.g., BlueTooth: www.bluetooth.com Significant Implications for tools Moving target problem Tools for coordinating multiple, distributed, communicating devices “Multi-computer” user interfaces

More problems: Varying Input and Output Today’s Desktop screens vary by a factor of 2.5 in size and a factor of 4 in pixels Tomorrow’s screen will vary by factors of 100 in size and a factor of 625 in pixels Cell phone to Stanford’s wall (3796 x 1436 pixels)

Varying I/O Capabilities, cont. Interaction techniques for desktop will not work No room on small devices Can’t reach menubar on wall-size devices Want to run same application on different devices

Need for Prototyping Devices User interface will be in hardware Rapid design and prototyping needed for hardware Pragmatics and usability cannot be evaluated from a simulation on a screen

Move to recognition-based interfaces Speech, gestures, camera-based vision Multimodal interaction User will pick which modality to use Use multiple modalities at same time Today, programming these requires knowing about Hidden-Markov Models, grammars, feature vectors, etc. Need tools to hide these complexities

Fundamental Differences of Recognition-based UIs Input is uncertain Recognition can make errors Requires monitoring, feedback, correction Do not produce discrete events GUI event model no longer works Interpreting input requires deep knowledge of data Context of the application “Move the red truck to here” Need a architecture based on accessible application data structures “Reflection”, “Open Data Model”

More Assumptions of Today’s Tools Skill and Dexterity of users Older users Makes single, fixed library of widgets untenable Non-overlapping and opaque components Preclude translucency, magic lens interactions

Operating Systems Considerations What is in the OS? Window Manager? Toolkit? Communication? Scripting facilities? Need ever increasing services for applications Need more access to low-level information E.g., hardware buttons, whether on network Ideally, API to support competition and research into these components

Conclusions Research in tools necessarily trails innovation in UI design Due to consolidation on desktop metaphor, significant progress in tools UI design poised for radical changes New opportunities and challenges for tools

Past, Present and Future of User Interface Software Tools Thank you! Past, Present and Future of User Interface Software Tools Brad A. Myers (work with Scott E. Hudson and Randy Pausch) Human-Computer Interaction Institute School of Computer Science Carnegie Mellon University http://www.cs.cmu.edu/~bam bam@cs.cmu.edu

Talk given with SlideShow Commander Result of the Pebbles research project http://www.cs.cmu.edu/~pebbles SlideShow Commander to be commercialized by: Available soon! Synergy Solutions, Inc. http://www.synsolutions.com/ Brad Myers 23