UNIFORM: Automatically Generating Consistent Remote Control User Interfaces Jeffrey Nichols, Brad A. Myers, Brandon Rothrock Human-Computer Interaction Institute School of Computer Science Carnegie Mellon University Conference on Human Factors in Computing Systems (CHI 2006) April 25, 2006

1 Motivation The number of computerized devices is increasing In the kitchen Ovens, blenders, microwaves, etc. In the living room Home entertainment systems: VCRs, televisions, stereos, etc. Around the house Thermostats, lighting, alarm clocks, outdoor watering systems, etc. In the car Stereos, climate controls, navigation systems, integrated cellular phones, etc. At the office Copiers, fax machines, printers, etc.

2 Motivation, cont. Appliance interfaces could be more usable if their interfaces were consistent Almost all appliances have some unique features Question: How might additional functionality be presented to the user while still providing a consistent interface?

3 UNIFORM Generates personally consistent user interfaces Personal Mobile Devices Personal Universal Controller (PUC) system Appliances Specifications Control State Feedback

4 UNIFORM Copier A Copier B Consistency disabled – Copier A used first

5 UNIFORM Copier A Copier B Consistency enabled – Copier A used first Consistent Copier B

6 Overview Motivation Previous Work on Consistency Specification Authoring Studies Requirements for Consistency Architecture Knowledge Base and Finding Similarity Generating Consistent Interfaces Discussion and Future Work

7 Related Work Theory of Consistency 1988 workshop at CHI was unable to define consistency [Nielsen 88, Grudin 89] Loosely defined as doing similar things in similar ways [Reisner 90] Different dimensions of consistency [Kellogg 87] Work on consistency as knowledge transfer (e.g. [Keiras & Polson 85, Polson 88] )

8 Related Work, cont. Systems addressing Consistency Some tools can automatically test for consistency GLEAN uses a GOMS-based approach [Kieras 95] Sherlock generates metrics for interpretation by designer [Mahajan 97] Previous automatic generation systems have ensured consistency within an application or family of applications ITS [Wiecha 90] PUC [Nichols 02] S UPPLE [Gajos 05]

9 Specification Authoring Study Study of Authoring Appliance Specifications How can specifications vary for different appliances with similar functions? How can specifications vary for different authors? Two User Studies Study #1: An expert user (me!) wrote specs for three different VCRs Study #2: Three subjects wrote specifications for the same VCR All subjects instructed to: Include all functions Be faithful to the existing appliance design Low End: Panasonic VCR Mid-Range: Samsung DVD-VCR High-End: Mitsubishi DVCR

10 Results of Both Studies Functions Some were identical Others were similar but had different labels Others had very different specifications Structure Structure for the first study was very similar across appliances Top-level structure was the same for the second study, but many differences were found at lower levels Results shows that specifications can differ substantially Analysis of results showed important distinction between functional and structural consistency

11 Consistency in UNIFORM We define a consistent interface to be one that incorporates elements and organization that are familiar to the user from previous interfaces We have refined this definition into six requirements, based on previous research and our authoring study

12 Requirements for Consistency 1. Interfaces should manipulate functions in the same way 2. Interfaces should locate similar functions in the same place 3. Interfaces should use similar labels for similar functions 4. Interfaces should maintain a similar visual appearance

13 Requirements for Consistency 1. Interfaces should manipulate functions in the same way 2. Interfaces should locate similar functions in the same place 3. Interfaces should use similar labels for similar functions 4. Interfaces should maintain a similar visual appearance 5. Usability of unique functions is more important than consistency

14 Requirements for Consistency 1. Interfaces should manipulate functions in the same way 2. Interfaces should locate similar functions in the same place 3. Interfaces should use similar labels for similar functions 4. Interfaces should maintain a similar visual appearance 5. Usability of unique functions is more important than consistency 6. Users must be able to choose to which appliance consistency is ensured

15 Overview Motivation Previous Work on Consistency Specification Authoring Studies Requirements for Consistency Architecture Knowledge Base and Finding Similarity Generating Consistent Interfaces Discussion and Future Work

16 UNIFORM Architecture Knowledge Base UNIFORM Interface Generation PUC Appliance Specification Personally Consistent User Interface

17 UNIFORM Architecture Knowledge Base Previous Specifications Functional Mappings Between Specifications Previously Generated UIs Mapping Phase 1 Abstract Consistency Phase 3 Concrete UI Building Phase 4 Output: Consistent User Interface Concrete Consistency Phase 5 2 Abstract UI Building Phase Input: Appliance Specification UNIFORM Interface Generation

18 UNIFORM Architecture Mapping Phase 1 Abstract Consistency Phase 3 Concrete UI Building Phase 4 Output: Consistent User Interface Concrete Consistency Phase 5 2 Abstract UI Building Phase Input: Appliance Specification UNIFORM Interface Generation Knowledge Base Previous Specifications Functional Mappings Between Specifications Previously Generated UIs

19 UNIFORM Architecture Knowledge Base Previous Specifications Functional Mappings Between Specifications Previously Generated UIs UNIFORM Interface Generation Mapping Phase 1 Abstract Consistency Phase 3 Concrete UI Building Phase 4 Output: Consistent User Interface Concrete Consistency Phase 5 2 Abstract UI Building Phase Input: Appliance Specification

20 UNIFORM Architecture Knowledge Base Previous Specifications Functional Mappings Between Specifications Previously Generated UIs Mapping Phase 1 Abstract Consistency Phase 3 Output: Consistent User Interface Concrete Consistency Phase 5 Input: Appliance Specification UNIFORM Interface Generation Concrete UI Building Phase 4 2 Abstract UI Building Phase

21 UNIFORM Architecture Knowledge Base Previous Specifications Functional Mappings Between Specifications Previously Generated UIs Concrete UI Building Phase 4 Output: Consistent User Interface 2 Abstract UI Building Phase Input: Appliance Specification UNIFORM Interface Generation Mapping Phase 1 Abstract Consistency Phase 3 Concrete Consistency Phase 5

22 Knowledge Base Functional Mappings We support six types of mappings, based on the results of the specification authoring studies general, state, node, group, list, and template Specifies that a function in one specification is similar to a function in another Mappings are grouped within the knowledge base in mapping graphs Knowledge Base Previous Specifications Functional Mappings Between Specifications Previously Generated UIs

23 Mapping Graphs Group semantically similar functions across appliances One graph for each function Power Volume Media Controls Etc. Node counts Store generation history Used to find basis for consistency Edges Track whether consistency can be ensured Choice in consistency supported by manipulating the node counts Knowledge Base Previous Specifications Functional Mappings Between Specifications Previously Generated UIs 3 1 DVD Player State: Mode { Play, Stop, Pause } Command: Next Track Command: Previous Track Answering Machine State: Mode { Play, Stop, Pause } Command: Play New Messages Cheap VCR Command: Play Command: Stop Command: Pause Command: Rewind Command: F-Fwd Command: Record Mitsubishi VCR State: Mode { Play, Stop, Pause, Rewind, F-Fwd, Record } 0 0

24 Where do mappings come from? Three ways mappings may be found Automatically extracted during mapping phase Challenging because of limited semantic information in specifications We have implemented two mapping algorithms, one based on work in automatic schema mapping from the database community Current best algorithm finds about 60% of mappings We are also exploring other techniques Specified by the user Manually entered as text Interaction techniques? Downloaded from the Internet Manufacturers? Internet communities? (e.g. Input: Appliance Specification Mapping Phase 1 Abstract Consistency Phase 3 Concrete UI Building Phase 4 Output: Consistent User Interface Concrete Consistency Phase 5 2 Abstract UI Building Phase

25 Abstract UI Building Phase Rules from previous PUC system Transforms specification into abstract user interface Abstract user interface includes: Tree organization Control choice Input: Appliance Specification Mapping Phase 1 Abstract Consistency Phase 3 Concrete UI Building Phase 4 Output: Consistent User Interface Concrete Consistency Phase 5 2 Abstract UI Building Phase

26 Abstract Consistency Phase Most of the work to ensure consistency happens in this phase Operates on the abstract user interface We have found that most consistency can be ensured at the abstract level Also enables many of our rules to cross platforms with little or no additional work This phase is made up of two parts: Functional sub-phase Similar functions are assigned similar labels Similar functions are converted, if necessary, to a consistent representation Abstract Structural sub-phase Similar functions are placed in similar groups Input: Appliance Specification Mapping Phase 1 Abstract Consistency Phase 3 Concrete UI Building Phase 4 Output: Consistent User Interface Concrete Consistency Phase 5 2 Abstract UI Building Phase

27 Abstract Structural Sub-Phase This sub-phase has two parts: Moving Similar functions are moved to previous groups Re-ordering Within each group, functions are re-ordered to match previous order

28 Moving BaseStatusSetupClockChannel Base Samsung DVD-VCR Base Mitsubishi DVCR

29 Moving BaseStatusSetupClockChannel Base Setup Samsung DVD-VCR Base Setup Mitsubishi DVCR

30 Moving BaseStatusSetupClockChannel Base Setup Clock Samsung DVD-VCR Base Status Clock Mitsubishi DVCR

31 Moving BaseStatusSetup Base Setup Clock Samsung DVD-VCR Base Status Clock Mitsubishi DVCR Clock Channel Status

32 Moving BaseStatusSetupClock Channel Base Status Clock Channel Samsung DVD-VCR Base Setup Channel Mitsubishi DVCR Channel Setup Unique functions are moved with their group but never by themselves

33 Re-Ordering

34 Re-Ordering When TimeDate ChannelConflictSpeedSourceType StartStop Length VCR+DayConflictChannelSpeed Timed Recordings for the Samsung DVD-VCR Timed Recordings for the Mitsubishi DVCR

35 Re-Ordering ConflictVCR+ChannelWhenSpeed WhenType ChannelSpeedConflict Timed Recordings for the Samsung DVD-VCR Timed Recordings for the Mitsubishi DVCR WhenTypeChannelSpeedConflict

36 Re-Ordering StartStopVCR+DayConflictChannelSpeed Timed Recordings for the Samsung DVD-VCR Timed Recordings for the Mitsubishi DVCR TimeDate ConflictChannelSourceWhenLengthType Speed

37 Concrete UI Building Phase Rules from previous PUC system Transforms abstract user interface into concrete user interface Concrete user interface includes: Organization necessary for screen size Panel Structure Instantiations of UI objects Positions and sizes of controls Input: Appliance Specification Mapping Phase 1 Abstract Consistency Phase 3 Concrete UI Building Phase 4 Output: Consistent User Interface Concrete Consistency Phase 5 2 Abstract UI Building Phase

38 Concrete Consistency Phase Differences in the available labels or number of controls may have created inconsistencies For example, a rule may have not added organization in this interface that was in the previous interface The side-panel orientation in a previous interface might have been chosen to allow a long label to fit Rules in this phase address these problems provided that the usability of unique functions is not affected For example, side-panel orientation might not change if needed for a large label or control Input: Appliance Specification Mapping Phase 1 Abstract Consistency Phase 3 Concrete UI Building Phase 4 Output: Consistent User Interface Concrete Consistency Phase 5 2 Abstract UI Building Phase

39 Overview Motivation Previous Work on Consistency Specification Authoring Studies Requirements for Consistency Architecture Knowledge Base and Finding Similarity Generating Consistent Interfaces Discussion and Future Work

40 Discussion and Future Work Evaluation of Generated Interfaces (in progress) Users will perform ~10 tasks with two all-in-one copier interfaces (HP or Canon) Three conditions Actual Appliance vs. PUC vs. Uniform

41 Discussion and Future Work How else might we deal with unique functions? Better semantic information More detailed modeling? Better knowledge of consistency Elaborate and improve on requirements for consistency Dimensions of consistency and relative importance More operational knowledge More consistency rules Adding new organization Support for consistency ghosts and other forms of transformational consistency [Richter 2006]

42 Acknowledgements Co-Authors Brad A. Myers Brandon Rothrock Thesis Committee Scott Hudson John Zimmerman Dan Olsen Jr. Funding National Science Foundation Microsoft General Motors Intel Pittsburgh Digital Greenhouse Equipment Grants Mitsubishi (MERL) PUC Project Members Duen Horng Chau Kevin Litwack Thomas K. Harris Michael Higgins Joseph Hughes Thomas Psik Roni Rosenfeld Rajesh Seenichamy Pegeen Shen Htet Htet Aung Mathilde Pignol Suporn Pongnumkul Jeffrey Stylos Stefanie Tomko Peter Lucas

Thanks for listening! For more information…

UNIFORM: Automatically Generating Consistent Remote Control User Interfaces Jeffrey Nichols, Brad A. Myers, Brandon Rothrock Human-Computer Interaction Institute School of Computer Science Carnegie Mellon University Conference on Human Factors in Computing Systems (CHI 2006) April 25, 2006

45 Questions Have you thought about using an approach where, given that you have 4-5 appliances, you generate all of the possible consistent interfaces and then find the right one (either by the user or through some kind of optimization or metric) – Allen Cypher What happens when I start with a simple interface and then get more complex over time? Does the consistent structure cause everything to become hidden? – Margaret Burnett Can you handle the case where on one appliance the function is a button and on another the function is multiple steps? – Henry Lieberman

46 Generated Phone Examples Copier A Copier B Consistency disabled

47 Generated Phone Examples Copier ACopier B Consistency enabled - Copier A used first

48 Generated Phone Examples Copier B Copier A Consistency enabled - Copier B used first

49 Functional Sub-Phase Inspects each function in the new specification Determines whether there is a previous function with which to ensure consistency (using mapping graph) Rules make changes to the abstract interface to ensure consistency Eight rules covering a number of potential functional differences Always ensure that previous labels are used May change the controls Two sorting functions from different copier interfaces

50 Copier A Copier B UNIFORM Consistency disabled – Copier B used firstConsistency disabled – Copier A used first

51 Copier A Copier B UNIFORM Consistency disabled – Copier B used first Consistent Copier A Consistency enabled – Copier B used first

52 Motivation, cont.

53 Abstract Consistency Phase Mitsubishi DVCR without consistency Samsung DVD-VCR without consistency Samsung DVD-VCR consistent with Mitsubishi DVCR