stanford hci groupoct 21, 2008 Björn Hartmann Loren Yu, Abel Allison, Yeonsoo Yang Scott R. Klemmer Design as Exploration Creating Interface Alternatives through Parallel Authoring and Runtime Tuning

2 Exploration of alternatives is central to design.

3 Prototypes for the Microsoft mouse From Moggridge, Designing Interactions, Ch2 MAP THE DESIGN SPACE

4 From Design Secrets: Products 2 CommunicatE

Tohidi et al, CHI 2006 TEST 5

How can tools support the creation of user interface alternatives? 6 Color picker #1 Color picker #2 Color picker #3

“[D]esigners frequently wanted to have multiple designs side-by-side [...] However […] there is no built-in way in today’s implementation tools to have two versions of a behavior operating side-by- side.” Myers et al., VL/HCC

Adobe Flash Arduino / Processing Adobe Dreamweaver Xcode (iPhone) Interaction Designers Write Code 8

Programming requires working with two representations. 9 Editor: author behavior … … … … … … … Runtime: observe behavior

Outline  Requirements for Alternatives of Programmed Interactions  Juxtapose for Desktop Interactions: System & Evaluation  Alternatives off the Desktop  Related & Future Work 10

3 Requirements for Programmed Interactions  Manage parameter variations  Manage code alternatives  Access variations & alternatives at runtime 11 (based on 3 interviews & code inspection)

 Manage parameter variations  Manage code alternatives  Access variations & alternatives at runtime 12 Processing code brought in by an interviewee 3 Requirements for Programmed Interactions

 Manage parameter variations  Manage code alternatives  Access variations & alternatives at runtime 13 3 Requirements for Programmed Interactions

Juxtapose: Source alternatives… 14

Juxtapose: Source alternatives… 15

… are executed in parallel, 16

and tuned through an generated UI. 17

Parallel Editing 18 Juxtapose editor for ActionScript 2. Generates Flash files.

Parallel Editing 19

Parallel Editing Linked Editing: Toomim

Parallel Editing Linked Editing: Toomim

Example: Rethinking the Progress Bar 22 Harrison et al., UIST 2007 Progress profiles Progress bar test application

23

Parallel Input: Event Echoing 24

Implementing Parallel Editing: The Cursor Correspondence Problem 25 Straightforward solution: Longest common subsequence algorithm diff(A,B,…,N) Run during idle times Doc. A Doc. BDoc. C

Limits of Parallel Editing & Execution  Level of granularity for editing: Extension to multi-file projects is not trivial.  Can the user make sense of behaviors running in parallel? 26

Limits of Event Echoing 27 Yes No Delete this file? Ok Delete this file? Yes No

Parameter Tuning 28

Example: Tuning Phosphor 29 Baudisch, UIST 2006

Example: Tuning Phosphor 30 Baudisch, UIST 2006

Generating Tuning Interfaces Juxtapose User Library User’s Application: //… Number A = 10; Boolean B = true; User’s Application: //… Number A = 10; Boolean B = true; User Process Juxtapose Process Inspect Send variable info XB A 31

Generating Tuning Interfaces Juxtapose User Library User’s Application: //… Number A = 10; Boolean B = true; User’s Application: //… Number A = 10; Boolean B = true; User Process Juxtapose Process Update Send tuning message XB A 32

What parameter ranges should be chosen? ? 0-100? 0-255? Number.minValue – Number.maxValue? 33 Any nontrivial program property is undecidable.

Take a guess, then give control to the user. 34 At authoring time: source annotations

Limits of Tuning  Which variables matter? Juxtapose extracts only globals 35

Limits of Tuning  Which variables matter? Juxtapose extracts only globals  Are variables read again? Juxtapose uses callbacks var tunable = 5; var callback= function(varName,oldVal,newVal){ 04 redraw(); 05 return newVal; 06 } 07 this.watch(′tunable′,callback);

Understanding User Strategies & Measuring Benefits  Questions:  How does Juxtapose fit into interaction prototyping practice?  Can we quantify the benefits?  Method:  Lab evaluation, N=18, students  minute sessions with design tasks & survey 37

38 Mapping Task Given ActionScript code that loads a multilayered map, develop navigation options for a handheld GPS prototype for pedestrians and bicyclists.

Alternatives for Map Navigation & Rendering 39

40 Participants’ Strategies  Linked vs. unlinked editing with multiple alternatives applied by all participants.  Alternatives used both for different scenarios, as well as “scratch space”  Snapshots important to save state

41 Suggested improvements  Automate code restructuring for callbacks  Introduce direct manipulation for parameters in user’s application (when possible)

42 Tree Matching Task Search in 4D parameter space. Given recursive drawing code for this: Produce these:

43

44

45 p<0.01 p~0.01 not significant

Outline  Requirements for Alternatives of Programmed Interactions  Juxtapose for Desktop Interactions: System & Evaluation  Alternatives off the Desktop  Related & Future Work 46

Juxtapose Mobile 47 Alternative 1 Alternative 2 Alternative 3

48

Juxtapose Runtime on PC Juxtapose Mobile Juxtapose Wrapper User’s Flash Application 49 Juxtapose Wrapper User’s Flash Application TCP/IP var 1 Alt. 1 Alt. 2 var 2

Juxtapose for Microcontrollers 50 Arduino: 8bit Atmel AVR RISC chip programmed in C with gcc

51 How might one implement variable tuning in a language without reflection?

Parse code and build symbol table Var. NameTypePointer “dly”int&dly “blink”boolean&blink int dly = 25; boolean blink=false; void setup() {...} //… 52

Emit table into code & compile void initVarTable(void) { varTable[1].varName = PSTR("dly"); varTable[1].varType = VAR_TYPE_INT; varTable[1].varPtr = &dly; varTable[2].varName = PSTR("blink"); varTable[2].varType = VAR_TYPE_BOOLEAN; varTable[2].varPtr = &blink; } // plus a bunch of communication code... Auto-generated table 53

At Runtime Serial message: tune “dly” value 100 *(varTable[“dly”].varPtr) = 100; X dlyblink 54

So What’s Different? Target PlatformHow are alternatives executed? Why? How do users interact with alternatives? In parallelSequentially or in parallel In parallel, because commodity hardware is (relatively) cheap Sequentially (parallelism is possible, but less appealing) Sequentially, because custom hardware is expensive to build Sequentially 55

What’s Important? A structured approach to alternatives eliminates cruft and enables more exploration. 56

What’s Important? A structured approach to alternatives eliminates cruft and enables more exploration. Tool support requires connecting authoring and execution environments. 3 Techniques:  Linked editing to manage code alternatives  Execute set of programs side-by-side  Auto-generate tuning interface 57

Related Work 58 Design Galleries, Marks, SIGGRAPH 97 Spreadsheets for Images, Levoy, SIGGRAPH 94 Spreadsheets for Visualization, Chi, IEEE CGA 98 Parameter Spectrums & Parallel Pies, Terry, UIST 02 & CHI 04 Subjunctive Interfaces, Lunzer, TOCHI 08 TEAM STORM, Hailpern, CHI 07 (only first authors listed) Partials, Terry, PhD Thesis 05 Amulet Inspector, Myers, IEEE ToSE 97 JPie, Goldman,Sci. of Comp. Prog. 98 ChucK, Wang, NIME 04 Adobe Image Foundation Tookit, 08 Alternatives in Other DomainsAugmented Development Tools

Related Work 59 Language-level support Partials, Terry, PhD Thesis 05 Amulet Inspector, Myers, IEEE ToSE 97 JPie, Goldman,Sci. of Comp. Prog. 98 ChucK, Wang, NIME 04 Adobe Image Foundation Tookit, 08 Augmented Development Tools

Related Work 60 Alternatives are generated automatically from formal specification Design Galleries, Marks, SIGGRAPH 97 Spreadsheets for Images, Levoy, SIGGRAPH 94 Spreadsheets for Visualization, Chi, IEEE CGA 98 Parameter Spectrums & Parallel Pies, Terry, UIST 02 & CHI 04 Subjunctive Interfaces, Lunzer, TOCHI 08 TEAM STORM, Hailpern, CHI 07 Alternatives in Other Domains

Related Work 61 Exploit spatial syntax Design Galleries, Marks, SIGGRAPH 97 Spreadsheets for Images, Levoy, SIGGRAPH 94 Spreadsheets for Visualization, Chi, IEEE CGA 98 Parameter Spectrums & Parallel Pies, Terry, UIST 02 & CHI 04 Subjunctive Interfaces, Lunzer, TOCHI 08 TEAM STORM, Hailpern, CHI 07 Alternatives in Other Domains

Current Work: Revisions for Interaction Design 62 Revisions & comments for text in Microsoft Word Revisions & comments for interactions in d.tools

stanford hci groupoct 21, Björn Hartmann Loren Yu, Abel Allison, Yeonsoo Yang Scott R. Klemmer Supported by: NSF grant IIS Equipment from Intel & Nokia SAP Stanford Graduate Fellowship

64 At runtime: editable min/max values Take a guess, then give control to the user.

Implementing Parallel Editing: Incremental Structure Tracking 65 Doc. A User inserts text Empty block created Blocks sub- divided User deletes text Doc. B Doc. A Doc. B Doc. A Doc. B Doc. A Doc. B INSERTDELETE

Future Work Graphics + Code? 66

67

68

69

70

Tangible Control 71

Switching alternatives Load Alternative2.hex Alternative 2 Bootloader 72

73 p<0.001 (paired two-tailed Student’s t)

Buxton, Sketching User Experiences

How can interaction design tools support creation and management of user interface alternatives? 75

Juxtapose Runtime on PC Juxtapose Mobile Juxtapose Wrapper User’s Flash Application Open socket, Send variable info 76 Juxtapose Wrapper User’s Flash Application TCP/IP Open socket, send variable info var 1 Alt. 1 Alt. 2 var 2

Juxtapose Mobile Juxtapose Wrapper User’s Flash Application Tuning messages For Alternative 2 77 Juxtapose Wrapper User’s Flash Application Tuning messages For Alternative 1 TCP/IP Juxtapose Runtime on PC var 1 Alt. 1 Alt. 2 var 2

Adding Tuning to the Arduino IDE 78

Tuning without Reflection Parse source to extract global variable declarations Emit variable table + wrapper code into source Compile modified source 79

Related Work 80 Design Galleries, Marks, SIGGRPAH 97 Parameter Spectrums & Parallel Pies, Terry, UIST 02 & CHI 04 Subjunctive Interfaces, Lunzer, TOCHI 08 Spreadsheets for Images, Levoy, SIGGRAPH 94 Spreadsheets for Visualization, Chi, IEEE CGA 98 TEAM STORM, Hailpern, CHI 07 Partials, Terry, PhD Thesis 05 Amulet Inspector, Myers, IEEE ToSE 97 Jpie, Goldman,Sci. of Comp. Prog. 98 ChucK, Wang, NIME 04 Adobe Image Foundation Tookit, 08 (only first authors listed) Mixed initiative

Redrawn from Buxton, Sketching User Experiences Generate Options, Select. Repeat.

Limits of Parallel Editing  Cognitive challenge: Changes to alternatives are not visible during editing  Level of granularity for editing: Extension to multi-file projects not trivial 82

Beyond the Desktop 83