1 High Performance Pen Interfaces Ken Hinckley Microsoft Research October 22 nd,

2 Pen Computing Now?

3 Project #1: Stitching Ken Hinckley Gonzalo Ramos Francois Guimbretiere Patrick Baudisch Marc Smith

4 Still an immature technology – Flaky, hard for users to configure, slow, … Most research has focused on small-screen presentation of web pages What are the completely new & unique affordances of wireless connectivity? Odd phrase: not what it is, but what it lacks Wireless Networks

5 Wireless Network 2004 = Horseless Carriage 1904? Are Wireless Networks really just wired networks without the wires? Or are they something completely different?

6 Wireless Device Soup: Which links does the user want?

7 Users need techniques to intuitively form bridges between devices How do users name the devices to connect? What is type / purpose of the connection? Parameters? (Who copies what, to where?) Name That Device

8 Pen stroke that spans displays Move the pen Cross over bezel Finish stroke on nearby tablet System infers connection Stitching

9 1.How is a connection established? 2.What type of connection is required? 3.What information is shared? 4.What is the spatial relationship between devices? 5.How do users share physical space? Explored in context of photo sharing app Central Design Questions

10 AVI’04 ACM Advanced Visual Interfaces 2004 VIDEO

11 1. Establishing a Connection Recognizes timing & dynamics of pen trajectory There is nothing special about the pen! Wireless signal strength determines nearby devices Tablet 1 Tablet 2 ΔtΔt

12 2. Type of Connection User selects a multi-device command e.g., Pie menu at end of stitching gesture

13 3. What Information to Share? Many commands may need operands Lasso to select, continue to other display to stitch.

14 4. What is Spatial Relationship: Automatic Screen Calibration Fits a line to absolute coords of user’s pen stroke

15 5. How Do Users Share Physical Space? Sociology  Proxemics: How people use space Invisible bubble that surrounds an individual

16 6. How Do Users Share Physical Space? Sociology  Proxemics: How people use space Invisible bubble that surrounds an individual Homework assignment: Sit right next to someone at airport (when it is not necessary to do so) Time how long it is before the other person leaves

17 5. How Do Users Share Physical Space? Sociology  Proxemics: How people use space Invisible bubble that surrounds an individual

18

19

20 Don’t require contact : touching is taboo “Do they have to be right next to one another?”: 10-40cm “wide screen would be nice for collaboration” “no two faces trying to peek at only one screen” Proxemic Lessons for Stitching

21 Don’t require contact “Do they have to be right next to one another?”: 10-40cm “wide screen would be nice for collaboration” “no two faces trying to peek at only one screen” Gives users flexibility to be involved, or not: mood, task, … Intimate Spaces: Combining screens. For close collaboration with friend or trusted colleague Personal Spaces: Tablets can be separated by up to 30” yet still possible to stitch to give files to colleague, etc. Social Spaces: Once connected, “transporter” can be used to give files to a user beyond arm’s reach Proxemic Lessons for Stitching

22 Don’t require contact “Do they have to be right next to one another?”: 10-40cm “wide screen would be nice for collaboration” “no two faces trying to peek at only one screen” Gives users flexibility to be involved, or not Intimate Spaces: Combining screens. For close collaboration with friend or trusted colleague Personal Spaces: Tablets can be separated by up to 30” yet still possible to stitch to give files to colleague, etc. Social Spaces: Once connected, “transporter” can be used to give files to a user beyond arm’s reach Support varying relative body orientation Proxemic Lessons for Stitching

23 Proxemics: Orientation of Spaces 6 conversations Communication patterns change wrt orientation Stitching supports: Shoulder-to-shoulder Face-to-face At 90 angles Relative frequency:

24 Multiple Devices: Cooperative Stitching Ongoing work User 1 User 2 User 3 User 4

25 Multiple Devices: Cooperative Stitching Ongoing work User 1 User 2 User 3 User 4

26 Project #2: Scriboli Ken Hinckley Patrick Baudisch Gonzalo Ramos Francois Guimbretiere

27 4 trips (+ some xtra) for copy/paste Round trip problem on Tablet PC

28 4 trips (+ some xtra) for copy/paste Round trip problem

29 4 trips (+ some xtra) for copy/paste Round trip problem

30 4 trips (+ some xtra) for copy/paste Round trip problem

31 4 trips (+ some xtra) for copy/paste Round trip problem

32 4 trips (+ some xtra) for copy/paste Round trip problem

33 4 trips (+ some xtra) for copy/paste Round trip problem

34 4 trips (+ some xtra) for copy/paste Round trip problem

35 No right click, kbd shortcuts to avoid this! Round trip problem

36 GEdit – Kurtenbach & Buxton Delete group – end stroke inside lasso Move – end stroke outside. Objects move on PenUp. Copy – add final “C” to move One of the first works considering pen phrases Every stroke is gesture Only 3 cmds Group vs. single obj. inconsistent Delete one obj.

37 SKETCH – Brown University Adding 1 more gesture breaks everything Specific to 3D, difficult to learn how to use

38 Marking Menus PromptedBlind Scale independence Self-revealing: prompts w/ available cmds Rapid expert use: blind marking Menu appears only after 1/3 second delay Size of mark does not matter

39 FlowMenu – Francois Guimbretiere Crossing in  out chooses Item… submenu Crossing out  in chooses Move command Dragging continues Integrates command selection with continuous parameter manipulation But no “blind marking,” size of gesture matters

40 Tivoli Tivoli Xerox PARC We wanted quick, informal feel of scribbling

41 Learning Cursive building blocks of writing as basis for pen UI? not natural handwriting recognition

42 The Deep Structure of Handwriting The “scribble” of Scriboli Phasic muscle contractions: fast, brief, usually repetitive Pre-programmed, internally driven “The conversion of those […] O’s to real words and sentences happens when small variations in the amount of verticality or laterality […] are introduced into the movement. The extreme orderliness and predictability of individual handwriting is due to this mode of generating and controlling the movement.” F. R. Wilson, The Hand, p. 163

43 Design Philosophy of Scriboli RAPID Fast motions, repeatable, no waiting / dwelling, minimal demands on visual attention UNAMBIGUOUS No guessing games No reco unless I tell the system otherwise (There is almost no reco in Scriboli) EXPRESSIVE Supports many commands, easy to grow design Mechanisms for a wide span of pen apps, not restricted to specific niche application

44 Grammar for pen input Scriboli proposes fundamental building blocks Links together object, verb, & indirect object in fast, fluid, unambiguous command phrases Written text separates words sentence start sentence end delimits clause “The cat sat on the mat”Punct. space capitals period comma object verb indirect obj. Scriboli Equivalent Pen up/pen down Gesture button down Gesture button up Pigtail (delimiter) Scope (lasso, line, …) Marking menu Crossing manip. phase

45 Ink vs. Gesture Who does the recognition? Ink: strokes intended primarily for human Gesture: strokes intended for computer Virtually all pen systems make this distinction at some level Very seductive to attempt to do automatically Many systems spend most or all effort just on this Any pen gesture might occur in freeform input Scriboli uses explicit ink/gesture mode switch

46 Ink/Gesture study w/ U. Washington 5 Techniques: Barrel Button Tap+hold NPH button Pen Pressure Pen Eraser

47 Grammar for pen input Scriboli proposes fundamental building blocks Links together object, verb, & indirect object in fast, fluid, unambiguous command phrases Written text separates words sentence start sentence end delimits clause “The cat sat on the mat”Punct. space capitals period comma object verb indirect obj. Scriboli Equivalent Pen up/pen down Gesture button down Gesture button up Pigtail (delimiter) Scope (lasso, line, …) Marking menu Crossing manip. phase

48 Delimiters: Single-stroke lasso selection + marking Initial part of stroke is the scope Pigtail is the delimiter Tail interpreted as the mark

49 What is a Delimiter? A delimiter is “something different” about the pen input stream Determines lexical structure of pen stroke Pigtail only intended as delimiter during Gesture mode Not an automatically inferred mode switch

50 Grammar for pen input Scriboli proposes fundamental building blocks Links together object, verb, & indirect object in fast, fluid, unambiguous command phrases Written text separates words sentence start sentence end delimits clause “The cat sat on the mat”Punct. space capitals period comma object verb indirect obj. Scriboli Equivalent Pen up/pen down Gesture button down Gesture button up Pigtail (delimiter) Scope (lasso, line, …) Marking menu Crossing manip. phase

51 Direct manipulation phase Copy-paste in place Zero round trips Continuing past outer border turns into dragging (like FlowMenu) A single stroke supports: Scope Command activation Dragging (direct manip) FlowMenu cannot support all 3

52 Direct manip phase preserves scale-independence Menu activation +333 ms visual menu pops up blind marking check for crossing cross boundary dragging Possible to have both: boundary crossing (like FlowMenu) scale-independence (like marking menus) Scale-ind. only matters for blind marking Dragging naturally lasts longer than 1/3 second.

53 Grammar for pen input Scriboli proposes fundamental building blocks Links together object, verb, & indirect object in fast, fluid, unambiguous command phrases Written text separates words sentence start sentence end delimits clause “The cat sat on the mat”Punct. space capitals period comma object verb indirect obj. Scriboli Equivalent Pen up/pen down Gesture button down Gesture button up Pigtail (delimiter) Scope (lasso, line, …) Marking menu Crossing manip phase

54 Scriboli Video VIDEO

55 Delimiters performance study Pigtail Timeout drag+hold during lasso Multi-stroke acquire selection handle after lasso Button Hit button during lasso to delimit scope vs. mark RI: Repeated Invocation Same command 24x

56 Learning effects Timeout plateaus quickly Button gets worse Pigtail, multi-stroke follow similar trend Pigtail problems: Fail to accept Heading in wrong direction Curl too much  choose wrong direction

57 Pigtail self-revelation technique Timeout easy to learn but tedious Pause  Scriboli shows how to draw pigtail Allows experienced users to shortcut timeout Helps novices learn the pigtail (?)

58 ONGOING: Phrasing together multiple gesture strokes Automatic means: hard to group arbitrary strokes across various application domains Time delay: “Time outs are lame” Scriboli uses muscular tension on Gesture button (lit.: may help reduce mode errors)

59 Separating the pig from the tail “+” symbol indicates gesture is extension of current command phrase Solves “screen edge” problem very nicely

60 Disjoint scopes

61 Grammar for pen input Scriboli proposes fundamental building blocks Links together object, verb, & indirect object in fast, fluid, unambiguous command phrases Written text separate words sentence start sentence end delimits clause “The cat sat on the mat”Punct. space capitals period comma object verb indirect obj. Scriboli Equivalent Pen up/pen down Gesture button down Gesture button up Pigtail (delimiter) Scope (lasso, line, …) Marking menu Crossing manip. phase

62 Crossing Scope (line) Draw stroke through object Crossing can have different semantics than lassoing Brings up crossing menu e.g. the crossing line specifies right edge of icon for alignment

63 Alignment (Multiple to edge)

64 Alignment (Multiple to edge)

65 Alignment (Multiple to edge)

66 Crossing Scope design properties For cmds that require edge and/or axis Only possible to invoke with requisite spatial information Limits possible errors of syntax Tremendous economy of design e.g. 1 Align cmd instead of 6

67 Alignment (Individual Edges)

68 Alignment (Individual Edges)

69 Alignment (Individual Edges)

70 Alignment (Individual Edges)

71 Alignment (Various edges)

72 Alignment (Various edges)

73 Alignment (Various edges)

74 Alignment (Various edges)

75 Alignment (Various edges)

76 Alignment (Various edges)

77 Grammar for pen input Scriboli proposes fundamental building blocks Links together object, verb, & indirect object in fast, fluid, unambiguous command phrases Written text separate words sentence start sentence end delimits clause “The cat sat on the mat”Punct. space capitals period comma object verb indirect obj. Scriboli Equivalent Pen up/pen down Gesture button down Gesture button up Pigtail (delimiter) Scope (lasso, line, …) Marking menu Crossing manip. phase

78 Future work Are there building blocks missing? What are alternative designs for each part? What other phrases can be envisioned? Written text separates words sentence start sentence end delimits clause “The cat sat on the mat”Punct. space capitals period comma object verb indirect obj. Scriboli Equivalent Pen up/pen down Gesture button down Gesture button up Pigtail (delimiter) Scope (lasso, line, …) Marking menu Crossing manip. phase

79 Questions?