2. large displays are like regular sized displays, only larger, right? patrick baudisch microsoft research visualization and interaction research

3. large screens and multimon

4. ... are coming [Jon Peddie Research Dec, 2002 N=6652] No Multimon 30% Plan to Use Multimon 38% Use Multimon 32% information mural [Guimbretière, Winograd] on large screens optical flow helps navigation [Tan 2001] large screens help productivity tasks [Czerwinski 2003] multi-monitor setups: access palette windows in Photoshop, CAD… [Grudin 2001]

5. building a large display focus plus context screens helpI cant find my mouse cursor high-density cursor helpI cant reach my stuff anymore drag-and-pop context where there is no room for context halo mouse motion mouse motion fill-in cursors current frame fill-in cursors previous frame

6. Hardware –At least one hi-res display –At least one larger low- res display Software –scaling of the display content is preserved –resolution varies

8. setup

9. application scenarios video

10. field study: users & tasks Subjects taskDocument/view Static documents Web designerPage: 800 pixel Mechanical engineerPolybot segment: 5cm Graphic designerPoster: 1m Architect in remodelingBuilding: 50m Photogrammetry (2)Highway 2 miles Geographic info. systemCounty: 80km Chip designers (2)Wafer: 12cm Dynamic Air traffic ctrl. tool builderZone: 50km Ego shooter gamerSurrounding: 360º Submarine ROV op.Surrounding: 360º Strategy gamers (2)Map: 30k pixel Smallest detailRatio Table detail: 1 pixel800 Clearance: 0.03mm2,000 Align: 0.5mm2,000 Accuracy: 1cm5,000 Accuracy: 1 inch100,000 Land boundaries: 0.5m160,000 Grid: 0.5 m 240,000 Plane distance in 25m steps2,000 Aiming: 0.1º3,000 Use arms: 1mm/0.05º8,000 Aiming: 1 pixel30,000

11. focus plus context screen Visualization Same # of pixels fisheye 5 5 overview plus detail 4 4 Display technology homogeneous resolution 4 4 wall-size, hi-res display 4 4 What participants used Available to ½ of participants … and current solutions

12. experiment 1: 3 interfaces: focus plus context screen overview + detail homogeneous 2 tasks 12 subjects from Xerox PARC Within subjects, counter-balanced Same number of pixels

13. task 1: closest hotel 8 maps per interface F+C screen and O+D use same magnification factor

14. task 2: verify connections Verify a different set of 24 connections on the board

15. 0 100 200 300 400 500 600 700 Map taskBoard task zooming+panning overview+detail focus+context screen results Average task completion times in seconds 21% faster 36% faster manually zooming takes time visually switching reorientation visually more ambiguous

16. experiment 2: driving simulation 120 sec sequence 100 fields of nails; 30 rocks; tradeoff

17. results Mean number of collisions subjects caused Sweet spot: flight simulation, unmanned vehicles… 0 5 10 15 20 25 run-over nailsrocks hit overview+detail focus+context screen Error rate only 1/3 of two- monitor setup Subjects preferred the f+c interface

18. so it worked really well with content that already was focus & context… …but what about the computer desktop? how to view peripheral content in high-res problem only because periphery is low-res? not really… how to view a detail on a huge display wall? it is just hard to see detail if located far away the distinction of screen space into focus and context regions is always there (focus plus context screens only emphasize it) ktop? but how about the computer desktop?

19. keeping the mouse working on a large screen, cursor is further away from user longer distances higher mouse acceleration temporal aliasing: 500 pixels jumps lack of visual continuity & weak stimulus users lose track of the cursor

20. the problem will get worse yes, but wont faster computers make this problem go away? NO: cursor update is limited by screen refresh rate screen refresh rate has actually decreased (LCDs) larger screens + lower refresh rate status quo future: even larger screens problem will get worse

21. how it works previous cursor position current cursor position mouse motion fill-in cursors current frame fill-in cursors previous frame inserts cursor image between actual cursor positions the mouse cursor appear more continuous

22. this is not the mouse trail the windows mouse trail… makes mouse trail last longer drawback: cursor images lag behind...is not high- density cursor hd cursor makes mouse trail denser lag-free: mouse stops => cursor stops video

23. benefits previous cursor position mouse motion current cursor position fill-in cursors current frame fill-in cursors previous frame mouse motion 1.mouse cursor appear more continuous easier to track the cursor 2.higher visual weight easier to re-acquire the cursor

24. designs alternatives acceleration reference: exponential acceleration a b d e f c h g frame

25. designs alternatives a b d e f c h g frame acceleration motion blur with higher weight

26. designs alternatives a b d e f c h g frame acceleration temporal super-sampling vs. motion blur chose discreet version 1. latest cursor position is always shown blur-free and in full opacity 2. appearance that users are familiar with today 3. computationally less expensive chose discreet version 1. latest cursor position is always shown blur-free and in full opacity 2. appearance that users are familiar with today 3. computationally less expensive

27. designs alternatives a b d e f c h g frame acceleration density = detectability vs. intrusiveness

28. transfer function mouse speed distance between cursor images onset threshold (configurable) cursor trail provides no speed cues hd cursor has no effect transfer function (configurable)

29. designs alternatives a b d e f c h g frame acceleration optional cursor growth

30. user study conducted pre-study to define interface candidates interfaces: control vs. high-density cursor (conservative, tripleDensity, plusScaling) fitts law task triple-mon: button located at 5 to 40 distance participants: 7 external participants, 5 coworkers hypotheses high-density cursor faster the greater the distance the greater the effect tripleDensity and plusScaling faster than conservative

31. results time % relative to regular cursor 90 92 94 96 98 100 102 target distance (mm) 1252505007501000 regular mouse cursor high-density cursors speedup up to 7% conservative + scale +3-dense short distance

32. subjective satisfaction Condition Liked Most Liked Least control 07 HD_conservative 23 HD_tripleDensity 20 HD_plusScaling 60 most participants did not notice that cursor was different! did that condition use a different mouse acceleration?…

33. lesson we learned: display frame rate is not a hard limit

34. but how to see details? so this gets the mouse to the peripherynice …but what if the user uses touch for input? … or if user needs to see content in detail? lets focus on a specific case for a moment: extend basic actions drag-and-drop and picking

35. scenario 1: long distances dragging is designed for small screens… … but becomes time-consuming on large screens

36. scenario 2: dragging + bezels dragging across bezels in display wall is no problem for the mouse… …but a big problem when using pen/touch input

37. drag-and-pop: demo users starts dragging icon towards a distant folder or application icons of compatible type come towards mouse cursor user drops icon with minimal motion targets retract drag-and-pop works across bezels

38. drag-and-pop generalizes direct manipulation bring content to the user let the user interact with it send content back

39. scenario 1: long distances dragging on large screens

40. scenario 2: dragging + bezels dragging across bezels in display wall

41. the displays we used…

42. design

43. selecting candidates initialize all icons are candidates filter eliminate icons with non-matching file types eliminate icons that are too close eliminate icons outside target angle if necessary, restrict to some hard limit

44. preserving layout snap to grid eliminate empty rows and columns translate back place center of bounding box in front of user closer for experts

45. the rubber band animation did not work frozen motion blur narrow midriff suggests elasticity clue for distance simplified version

46. getting it out of the way to rearrange icons on the desktop (overloaded): any mouse motion moving away from the popped-up icons de-activates drag-and-pop introduce flick gesture into mouse motion

47. pre-study 3 layouts for study: sparse (11), frame (28), cluttered (35) 15 single, 6 dual, and 4 triple monitor users overall resolutions 800,000 pixels to 3,900,000 pixels (= 66% more than the display wall used in the experiment).

48. user study participants: 2 female, 5 male dynaWall 3 Smartboard 15 long (4.5m) 3 x 1024x768 pixels native code not stable enough Macromedia Flash version task: drag icons into matching folder highlighting disappeared when started each desktop: 11-35 icons + 10 icons to be filed

49. results faster with drag-and-pop error rate higher with drag-and-pop most of the effect caused by the bezels Control Drag-and-pop 3.7 times speedup Control Drag-and-pop

50. subjective satisfaction > 6 (out of 7) I liked using drag-and-pop I always understood what was happening when drag-and-pop was on, I would use drag-and-pop for large displays. < 3 for It took a long time to get used to drag-and-pop It was hard to control what the targets did when drag-and-pop was on. drag-and-pop interface causes less manual stress and fatigue than the control interface lesson learned: the shortest connection between two points on a display wall is not a straight line (fixed)

51. drag-and-pick problem launch app or open file drag-and-pick user drags background all icons in that direction move to the cursor user drags % releases mouse over it target is activated

52. lesson we learned: screen space and interaction dont have to be the same this was only for moving files and launching apps. need to solve this for the general case

53. can we push this further? (unless the app has fixed focus of attention) bring content to the user on demand thats greatso maybe the periphery does not need to be full resolution after all can we push this even further? can we create a visual periphery without any peripheral screen space?

54. a first attempt… city lights

55. + the problem

56. halo

57. cinematography 1.entry and exit points 2.point of view arrow-based techniques 3.partially out of the frame halo rings are familiar, graceful degradation

58. streetlamps aura visible from distance aura is round overlapping auras aggregate fading of aura indicates distance what we changed smooth transition sharp edge disks rings dark background light background

59. app designers can use color texture arc thickness

60. user study

61. interfaces arc/arrow fading off scale 110-300m/cm map as backdrop readability ok same selectable size hypothesis: halo faster halo ring distance from display border legend

62. pre-study to define tasks 8 participants (6 GPS users, 2 PDA users) informal interviews 10-40 minutes 4 tasks to be used in study

63. 1. locate task click at expected location of off-screen targets had to simulate on PC

64. 2. closest task click arrow/arc or off-screen location closest to car

65. 3. traverse task click all five targets so as to form shortest path

66. 4. avoid task click on hospital farthest away from traffic jams

67. procedure 12 participants within subject design, counterbalanced four training maps per interface/task, then eight timed maps questionnaire

68. task completion time TaskArrow interfaceHalo interface Locate20.1 (7.3)16.8 (6.7) Closest9.9 (10.1)6.6 (5.3) Traverse20.6 (14.1)16.8 (8.7) Avoid9.2 (4.7)7.7 (5.8) 33% 16%

69. error rate TaskArrow interfaceHalo interface Locate23.5 pixels (21.6)28.4 pixels (33.8) Closest22% (42%)21% (41%) Traverse97.4 pixels (94.7)81.0 pixels (96.7) Avoid15% (35%)14% (34%) participants underestimated distances by 26% participants saw ovals (gestalt laws?) we can compensate for that: width += 35%

70. subjective preference

71. results halo 16%-33% faster than arrows no split attention distortion-free space scale independent no need to annotate distance perceive all rings at once [treisman & gormican] limitation: max number or rings

72. access off-screen content make halo arcs (or city lights) clickable to pan display to that location vs. preserve users spatial memory by using drag-and-pop instead: bring peripheral content to PDA screen

73. conclusions distance to peripheral space and human peripheral vision make peripheral content hard to read making periphery all hi-res does not solve this problem we need to support users using appropriate visualization and interaction techniques mouse motion mouse motion fill-in cursors current frame fill-in cursors previous frame

74. read more about it focus plus context screens: UIST 2001, CHI 2002 SIGGRAPH 2002 demo high-density cursor Interact 2003 drag-and-pop Interact 2003 city lights CHI 2003 halo CHI 2003, UIST 2003 demo

75. thanks! try it out www.patrickbaudisch.com thanks to: focus plus context screens: victoria bellotti, nathan good, paul stewart, pamela schraedley, michael brueckner, rich gold, high-density cursor: ed cutrell, george robertson, & VIBE drag-and-pop: & ed cutrell, dan robbins, mary czerwinski, peter tandler, ben bederson, and alex zierlinger halo: ruth rosenholtz, polle zellweger, jock mackinlay, lance good, and mark stefik

77. Seamless integration of displays a b

78. Context No reflections on focus screen Focus

79. If I had to commercialize today… Build all-analog immersive video link Immersive telepresence, Remote operated vehicles, drones Immersive VR Remote medicine

80. ContextWall

81. Extra

82. (a) locate(b) closest (d) avoid(c) traverse

83. How does it work?

84. The scaling software Display image on two display units of different resolution Similar to two-headed display but display units are overlapping and one of them has to be scaled down (Related work Flux capacitor [Dr. Emmett Brown, 1985])

85. Linux/VNC app focus context input server clip scale viewer

86. Image viewer context Photo shop scale ACDsee.gif mouse fork input focus

87. related work acquiring distant targets move cursor with eye gaze (Sibert and Jacob, 2000), Magic pointing (Zhai et al., 1999) flick snaps cursor to target (Dulberg et al. (1999) sticky icons capture cursor (Swaminathan and Sato, 1997) throwing gets across long distances (Geißler, 1998) expanding targets save space on screen (McGuffin and Balakrishnan, 2002) drag-and-pop (baudisch et al 2003) enhance detectability of the mouse cursor for radar animation (Microsoft, Steve Bathiche) cursor growth (Kensington Mouseworks 2001) mouse trail for slow response LCDs (e.g. MS Windows) liveCursor points in the direction of its motion (Ben Bederson) motion blur and temporal supersampling reduce temporal aliasing, such as stroboscope, e.g. wheel spokes rendering a scene multiple times (Dachille and Kaufman, 2000) improve the perceived responsiveness of graphics apps (Conner and Holden, 1997) help users anticipate motion (Chang, 1993; Thomas & Calder, 2001)

88. design goals for users who track the cursor enhance the predictability of the cursor path enhanced trail density/continuous blur smooth interpolation of the cursor path preservation of trail density as a cue for cursor speed. for users who reacquire the cursor increase the detectability of the cursor (visual weight) enhanced trail density enhanced cursor opacity and cursor scaling. preserve responsiveness

89. designs alternatives a b d e f c h g frame acceleration distance between cursor images as cue for mouse speed

90. designs alternatives a b d e f c h g frame acceleration smooth interpolation

91. bezier interpolation cursor position 1. linear interpolation 2. attraction point 3. interpolate

92. pre-study goal: define interfaces for user study participants: 14 coworkers informal procedure try out high-density cursor try out different settings (density, onset…) choose favorite setting resulting interface parameters 12-17 pixels/frame vs. 35 pixels/frame distance = sqrt(n) cursor growth on or off

93. touch/pen input breaks touch/pen-input + multimon touch and pen input renaissance PDAs Tablets Liveboards / Smartboards multi-display systems DynaWall, iRoom Smartboard wall connect tablet to external screen …

94. scenario 1: tables + screen tablet users scribble with pen… but filing icons into folder on external monitor requires mouse

95. scenario 1: tablet + screen filing icons into folder on external monitor

96. related work techniques for transferring information drag-and-drop avoids hidden clipboard (e.g. Xerox Star) hyper­dragging (Rekimoto, 1999) pick-and-drop (Rekimoto, 1997)+ take-and-put (Streitz et al., 2001) overcome large distances magic pointing (Zhai et al., 1999) requires an indirect input device gesture input techniques (Rubine, 1991) throwing (Geißler, 1998) for reliable target acquisition? laser pointers to acquire targets on a Smartboard (Myers et al. 2002) mouse-based interaction techniques lodestones and lay lines (Jul, 2002) flick (Dulberg et al., 1999) sticky icons (Swaminathan and Sato 1997)

97. related work driving directions vs. route planning aids overview-plus-detail focus-plus-context pointing into off-screen space

98. inside applications… drag-and-pop works even if target is occluded clipped closed (folder) use the concept to file emails?

99. intrusion border handle space for arcs… and for corner arcs reserve space for content

100. arc length = distance

101. handling many objects find best (restaurant): relevance cut-off see all (dangers): merge arcs