Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #0 Jeffrey Nichols and Brad A. Myers Carnegie Mellon University October 30, 2002 Generating Remote Control Interfaces for Complex Appliances The Fifteenth Annual Symposium on User Interface Software & Technology (UIST) 2002 Paris, France

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #1 The Problem Appliances are too complex

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #2 The Problem, cont. Each complex appliances has its own idiosyncratic interface! Stereo systems Telephones VCRs Alarm Clocks … Increasingly Computerized Low Usability

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #3 Our Solution Separate the interface from the appliance! Handheld becomes personal universal controller (PUC) Key Features Interface-independent appliance specification Automatic generation of GUI and speech interfaces Specifications Control Feedback

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #4 Automatic Generation of UIs Benefits All interfaces consistent for the user With conventions of handheld Even from multiple manufacturers Addresses hotel alarm clock problem! Multiple modalities (GUI + Speech UI) Can take into account user preferences Will work on special purpose devices (for disabled)

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #5 Important Work By Others INCITS V2 Standardization Effort Alternative Interface Access Protocol (AIAP) [Zimmermann, CHI 2002] User Interface Modeling Language (UIML) Xweb (now ICE) [Olsen Jr., UIST 2000] Stanford iRoom, iCrafter [Ponnekanti, Ubicomp 2001] Speakeasy [Newman, UIST 2002]

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #6 - Comm. Protocol- Interface Generators- Specification Lang.Architecture XML-based - Appliance Adaptors

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #7 Language Design Approach Create reference interfaces AIWA Shelf Stereo AT&T Telephone/Answering Machine Test interfaces with subjects Users twice as fast and made half the errors with reference interfaces as compared to manufacturers interfaces Analyze interfaces for functional information

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #8 Language Elements State Variables and Commands Represent functions of appliance State variables have types Boolean, Enumeration, Integer, String, etc. Variables sufficient for most functions but not all seek button on a Radio Label Information One label not suitable everywhere The optimal label length changes with screen size Speech interfaces may benefit from pronunciation and text-to-speech information

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #9 Language Elements, cont. Group Tree Specify organization of functions We use n-ary tree with variables or commands at leaves

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #10 Language Elements, cont. Dependency Information Formulas that specify when a variable or command is active in terms of other state variables Equals, Greater Than, Less Than Linked with logical operators (AND, OR) For example, true AM

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #11 Interface Generators Generators for Two Modalities Graphical Implemented for PocketPC in Java 1.1 Uses dependency information to generate panel structure of interface Speech Implemented using Universal Speech Interface (USI) techniques [Rosenfeld 2001] Uses dependency information to disambiguate shortcut words (e.g. play) and resolve pre-conditions for a requested function (e.g. play CD)

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #12 Graphical Interface Generator Focuses on panel structure of user interface Small groups of controls have basic layouts Complexity comes from structure of groups Structure can be inferred from dependency info!

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #13 Inferring Structure Find sets of variables that are mutually exclusive Every variable in a set will never be active at the same time as a variable in another set Create structure with sets, using overlapping panels

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #14 Choosing Panel Types a)b)c)c) full screen tabbed partial screen

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #15 Making the Interface Concrete Finish conceptual layout Choose controls (decision tree) Choose row layouts (one column, two column, etc.) Allocate space Examine panel contents and choose sizes Instantiate and place controls

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #16 Generating Speech Interfaces Automatically build USI tree from dependencies Allows verbal navigation of functional groups Automatically generate grammar for parser Phrases for query and control What is playmode? Set playmode to play play Automatically generate language model and pronunciation for recognizer

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #17 Controlling Appliances We have built adaptors for many actual appliances Sony Digital Camcorder X10 Lighting Audiophase Shelf Stereo AudioReQuest MP3 player Software Media Players (WinAmp, WMP) Written specifications for others Elevator Telephone/Answering Machine

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #18 Demonstration

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #19 Future Work Improve quality of generated user interfaces Back-tracking, etc. Specification and generation of list interfaces Necessary for many appliances Adaptive interface generation New interfaces modeled on past interfaces General purpose appliance adaptor for HAVi and UPnP protocols

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #20 Acknowledgements PUC Project Members Brad A. Myers Michael Higgins Joseph Hughes Thomas K. Harris Roni Rosenfeld Mathilde Pignol Stefanie Shriver Peter Lucas Kevin Litwack Funding National Science Foundation Pittsburgh Digital Greenhouse Microsoft Equipment Grants Mitsubishi (MERL) VividLogic Symbol Technologies Hewlett-Packard Lucent

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #21 User Interface Software & Technology (UIST) October 30, 2002 Thanks!

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #22

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #23 Determining Panel Structure Find states that others depend upon Search for the group that contains this state Determine mutually exclusive sets within group children Assign sets to panels Based upon types, tree properties Power Mode Band PlayStat e CDTrac k FMStation FMPreset s AMStation AMPreset s Radi o CDCD Mode == Any Mode == Radio Mode == CD

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #24 Choosing Panel Types Rules for Choosing Panels If… state has boolean type all controls are one set Create… Two full-screen overlapping panels.

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #25 Choosing Panel Types, cont. Rules for Choosing Panels If… state has enumerated type there is a set for every value of the state Create… a tabbed panel

Jeffrey Nichols User Interface Software & Technology (UIST) October 30, 2002 Slide #26 Choosing Panel Types, cont. Rules for Choosing Panels If… there are mutually exclusive sets of controls Create… overlapping panels a control for the state not on the overlapping panel