large displays are like regular sized displays, only larger, right? patrick baudisch microsoft research visualization and interaction research
large screens and multimon
... are coming [Jon Peddie Research Dec, 2002 N=6652] No Multimon 30% Plan to Use Multimon 38% Use Multimon 32% information mural [Guimbretière, Winograd] on large screens optical flow helps navigation [Tan 2001] large screens help productivity tasks [Czerwinski 2003] multi-monitor setups: access palette windows in Photoshop, CAD… [Grudin 2001]
building a large display focus plus context screens helpI cant find my mouse cursor high-density cursor helpI cant reach my stuff anymore drag-and-pop context where there is no room for context halo mouse motion mouse motion fill-in cursors current frame fill-in cursors previous frame
Hardware –At least one hi-res display –At least one larger low- res display Software –scaling of the display content is preserved –resolution varies
setup
application scenarios video
field study: users & tasks Subjects taskDocument/view Static documents Web designerPage: 800 pixel Mechanical engineerPolybot segment: 5cm Graphic designerPoster: 1m Architect in remodelingBuilding: 50m Photogrammetry (2)Highway 2 miles Geographic info. systemCounty: 80km Chip designers (2)Wafer: 12cm Dynamic Air traffic ctrl. tool builderZone: 50km Ego shooter gamerSurrounding: 360º Submarine ROV op.Surrounding: 360º Strategy gamers (2)Map: 30k pixel Smallest detailRatio Table detail: 1 pixel800 Clearance: 0.03mm2,000 Align: 0.5mm2,000 Accuracy: 1cm5,000 Accuracy: 1 inch100,000 Land boundaries: 0.5m160,000 Grid: 0.5 m 240,000 Plane distance in 25m steps2,000 Aiming: 0.1º3,000 Use arms: 1mm/0.05º8,000 Aiming: 1 pixel30,000
focus plus context screen Visualization Same # of pixels fisheye 5 5 overview plus detail 4 4 Display technology homogeneous resolution 4 4 wall-size, hi-res display 4 4 What participants used Available to ½ of participants … and current solutions
experiment 1: 3 interfaces: focus plus context screen overview + detail homogeneous 2 tasks 12 subjects from Xerox PARC Within subjects, counter-balanced Same number of pixels
task 1: closest hotel 8 maps per interface F+C screen and O+D use same magnification factor
task 2: verify connections Verify a different set of 24 connections on the board
Map taskBoard task zooming+panning overview+detail focus+context screen results Average task completion times in seconds 21% faster 36% faster manually zooming takes time visually switching reorientation visually more ambiguous
experiment 2: driving simulation 120 sec sequence 100 fields of nails; 30 rocks; tradeoff
results Mean number of collisions subjects caused Sweet spot: flight simulation, unmanned vehicles… run-over nailsrocks hit overview+detail focus+context screen Error rate only 1/3 of two- monitor setup Subjects preferred the f+c interface
so it worked really well with content that already was focus & context… …but what about the computer desktop? how to view peripheral content in high-res problem only because periphery is low-res? not really… how to view a detail on a huge display wall? it is just hard to see detail if located far away the distinction of screen space into focus and context regions is always there (focus plus context screens only emphasize it) ktop? but how about the computer desktop?
keeping the mouse working on a large screen, cursor is further away from user longer distances higher mouse acceleration temporal aliasing: 500 pixels jumps lack of visual continuity & weak stimulus users lose track of the cursor
the problem will get worse yes, but wont faster computers make this problem go away? NO: cursor update is limited by screen refresh rate screen refresh rate has actually decreased (LCDs) larger screens + lower refresh rate status quo future: even larger screens problem will get worse
how it works previous cursor position current cursor position mouse motion fill-in cursors current frame fill-in cursors previous frame inserts cursor image between actual cursor positions the mouse cursor appear more continuous
this is not the mouse trail the windows mouse trail… makes mouse trail last longer drawback: cursor images lag behind...is not high- density cursor hd cursor makes mouse trail denser lag-free: mouse stops => cursor stops video
benefits previous cursor position mouse motion current cursor position fill-in cursors current frame fill-in cursors previous frame mouse motion 1.mouse cursor appear more continuous easier to track the cursor 2.higher visual weight easier to re-acquire the cursor
designs alternatives acceleration reference: exponential acceleration a b d e f c h g frame
designs alternatives a b d e f c h g frame acceleration motion blur with higher weight
designs alternatives a b d e f c h g frame acceleration temporal super-sampling vs. motion blur chose discreet version 1. latest cursor position is always shown blur-free and in full opacity 2. appearance that users are familiar with today 3. computationally less expensive chose discreet version 1. latest cursor position is always shown blur-free and in full opacity 2. appearance that users are familiar with today 3. computationally less expensive
designs alternatives a b d e f c h g frame acceleration density = detectability vs. intrusiveness
transfer function mouse speed distance between cursor images onset threshold (configurable) cursor trail provides no speed cues hd cursor has no effect transfer function (configurable)
designs alternatives a b d e f c h g frame acceleration optional cursor growth
user study conducted pre-study to define interface candidates interfaces: control vs. high-density cursor (conservative, tripleDensity, plusScaling) fitts law task triple-mon: button located at 5 to 40 distance participants: 7 external participants, 5 coworkers hypotheses high-density cursor faster the greater the distance the greater the effect tripleDensity and plusScaling faster than conservative
results time % relative to regular cursor target distance (mm) regular mouse cursor high-density cursors speedup up to 7% conservative + scale +3-dense short distance
subjective satisfaction Condition Liked Most Liked Least control 07 HD_conservative 23 HD_tripleDensity 20 HD_plusScaling 60 most participants did not notice that cursor was different! did that condition use a different mouse acceleration?…
lesson we learned: display frame rate is not a hard limit
but how to see details? so this gets the mouse to the peripherynice …but what if the user uses touch for input? … or if user needs to see content in detail? lets focus on a specific case for a moment: extend basic actions drag-and-drop and picking
scenario 1: long distances dragging is designed for small screens… … but becomes time-consuming on large screens
scenario 2: dragging + bezels dragging across bezels in display wall is no problem for the mouse… …but a big problem when using pen/touch input
drag-and-pop: demo users starts dragging icon towards a distant folder or application icons of compatible type come towards mouse cursor user drops icon with minimal motion targets retract drag-and-pop works across bezels
drag-and-pop generalizes direct manipulation bring content to the user let the user interact with it send content back
scenario 1: long distances dragging on large screens
scenario 2: dragging + bezels dragging across bezels in display wall
the displays we used…
design
selecting candidates initialize all icons are candidates filter eliminate icons with non-matching file types eliminate icons that are too close eliminate icons outside target angle if necessary, restrict to some hard limit
preserving layout snap to grid eliminate empty rows and columns translate back place center of bounding box in front of user closer for experts
the rubber band animation did not work frozen motion blur narrow midriff suggests elasticity clue for distance simplified version
getting it out of the way to rearrange icons on the desktop (overloaded): any mouse motion moving away from the popped-up icons de-activates drag-and-pop introduce flick gesture into mouse motion
pre-study 3 layouts for study: sparse (11), frame (28), cluttered (35) 15 single, 6 dual, and 4 triple monitor users overall resolutions 800,000 pixels to 3,900,000 pixels (= 66% more than the display wall used in the experiment).
user study participants: 2 female, 5 male dynaWall 3 Smartboard 15 long (4.5m) 3 x 1024x768 pixels native code not stable enough Macromedia Flash version task: drag icons into matching folder highlighting disappeared when started each desktop: icons + 10 icons to be filed
results faster with drag-and-pop error rate higher with drag-and-pop most of the effect caused by the bezels Control Drag-and-pop 3.7 times speedup Control Drag-and-pop
subjective satisfaction > 6 (out of 7) I liked using drag-and-pop I always understood what was happening when drag-and-pop was on, I would use drag-and-pop for large displays. < 3 for It took a long time to get used to drag-and-pop It was hard to control what the targets did when drag-and-pop was on. drag-and-pop interface causes less manual stress and fatigue than the control interface lesson learned: the shortest connection between two points on a display wall is not a straight line (fixed)
drag-and-pick problem launch app or open file drag-and-pick user drags background all icons in that direction move to the cursor user drags % releases mouse over it target is activated
lesson we learned: screen space and interaction dont have to be the same this was only for moving files and launching apps. need to solve this for the general case
can we push this further? (unless the app has fixed focus of attention) bring content to the user on demand thats greatso maybe the periphery does not need to be full resolution after all can we push this even further? can we create a visual periphery without any peripheral screen space?
a first attempt… city lights
+ the problem
halo
cinematography 1.entry and exit points 2.point of view arrow-based techniques 3.partially out of the frame halo rings are familiar, graceful degradation
streetlamps aura visible from distance aura is round overlapping auras aggregate fading of aura indicates distance what we changed smooth transition sharp edge disks rings dark background light background
app designers can use color texture arc thickness
user study
interfaces arc/arrow fading off scale m/cm map as backdrop readability ok same selectable size hypothesis: halo faster halo ring distance from display border legend
pre-study to define tasks 8 participants (6 GPS users, 2 PDA users) informal interviews minutes 4 tasks to be used in study
1. locate task click at expected location of off-screen targets had to simulate on PC
2. closest task click arrow/arc or off-screen location closest to car
3. traverse task click all five targets so as to form shortest path
4. avoid task click on hospital farthest away from traffic jams
procedure 12 participants within subject design, counterbalanced four training maps per interface/task, then eight timed maps questionnaire
task completion time TaskArrow interfaceHalo interface Locate20.1 (7.3)16.8 (6.7) Closest9.9 (10.1)6.6 (5.3) Traverse20.6 (14.1)16.8 (8.7) Avoid9.2 (4.7)7.7 (5.8) 33% 16%
error rate TaskArrow interfaceHalo interface Locate23.5 pixels (21.6)28.4 pixels (33.8) Closest22% (42%)21% (41%) Traverse97.4 pixels (94.7)81.0 pixels (96.7) Avoid15% (35%)14% (34%) participants underestimated distances by 26% participants saw ovals (gestalt laws?) we can compensate for that: width += 35%
subjective preference
results halo 16%-33% faster than arrows no split attention distortion-free space scale independent no need to annotate distance perceive all rings at once [treisman & gormican] limitation: max number or rings
access off-screen content make halo arcs (or city lights) clickable to pan display to that location vs. preserve users spatial memory by using drag-and-pop instead: bring peripheral content to PDA screen
conclusions distance to peripheral space and human peripheral vision make peripheral content hard to read making periphery all hi-res does not solve this problem we need to support users using appropriate visualization and interaction techniques mouse motion mouse motion fill-in cursors current frame fill-in cursors previous frame
read more about it focus plus context screens: UIST 2001, CHI 2002 SIGGRAPH 2002 demo high-density cursor Interact 2003 drag-and-pop Interact 2003 city lights CHI 2003 halo CHI 2003, UIST 2003 demo
thanks! try it out thanks to: focus plus context screens: victoria bellotti, nathan good, paul stewart, pamela schraedley, michael brueckner, rich gold, high-density cursor: ed cutrell, george robertson, & VIBE drag-and-pop: & ed cutrell, dan robbins, mary czerwinski, peter tandler, ben bederson, and alex zierlinger halo: ruth rosenholtz, polle zellweger, jock mackinlay, lance good, and mark stefik
Seamless integration of displays a b
Context No reflections on focus screen Focus
If I had to commercialize today… Build all-analog immersive video link Immersive telepresence, Remote operated vehicles, drones Immersive VR Remote medicine
ContextWall
Extra
(a) locate(b) closest (d) avoid(c) traverse
How does it work?
The scaling software Display image on two display units of different resolution Similar to two-headed display but display units are overlapping and one of them has to be scaled down (Related work Flux capacitor [Dr. Emmett Brown, 1985])
Linux/VNC app focus context input server clip scale viewer
Image viewer context Photo shop scale ACDsee.gif mouse fork input focus
related work acquiring distant targets move cursor with eye gaze (Sibert and Jacob, 2000), Magic pointing (Zhai et al., 1999) flick snaps cursor to target (Dulberg et al. (1999) sticky icons capture cursor (Swaminathan and Sato, 1997) throwing gets across long distances (Geißler, 1998) expanding targets save space on screen (McGuffin and Balakrishnan, 2002) drag-and-pop (baudisch et al 2003) enhance detectability of the mouse cursor for radar animation (Microsoft, Steve Bathiche) cursor growth (Kensington Mouseworks 2001) mouse trail for slow response LCDs (e.g. MS Windows) liveCursor points in the direction of its motion (Ben Bederson) motion blur and temporal supersampling reduce temporal aliasing, such as stroboscope, e.g. wheel spokes rendering a scene multiple times (Dachille and Kaufman, 2000) improve the perceived responsiveness of graphics apps (Conner and Holden, 1997) help users anticipate motion (Chang, 1993; Thomas & Calder, 2001)
design goals for users who track the cursor enhance the predictability of the cursor path enhanced trail density/continuous blur smooth interpolation of the cursor path preservation of trail density as a cue for cursor speed. for users who reacquire the cursor increase the detectability of the cursor (visual weight) enhanced trail density enhanced cursor opacity and cursor scaling. preserve responsiveness
designs alternatives a b d e f c h g frame acceleration distance between cursor images as cue for mouse speed
designs alternatives a b d e f c h g frame acceleration smooth interpolation
bezier interpolation cursor position 1. linear interpolation 2. attraction point 3. interpolate
pre-study goal: define interfaces for user study participants: 14 coworkers informal procedure try out high-density cursor try out different settings (density, onset…) choose favorite setting resulting interface parameters pixels/frame vs. 35 pixels/frame distance = sqrt(n) cursor growth on or off
touch/pen input breaks touch/pen-input + multimon touch and pen input renaissance PDAs Tablets Liveboards / Smartboards multi-display systems DynaWall, iRoom Smartboard wall connect tablet to external screen …
scenario 1: tables + screen tablet users scribble with pen… but filing icons into folder on external monitor requires mouse
scenario 1: tablet + screen filing icons into folder on external monitor
related work techniques for transferring information drag-and-drop avoids hidden clipboard (e.g. Xerox Star) hyperdragging (Rekimoto, 1999) pick-and-drop (Rekimoto, 1997)+ take-and-put (Streitz et al., 2001) overcome large distances magic pointing (Zhai et al., 1999) requires an indirect input device gesture input techniques (Rubine, 1991) throwing (Geißler, 1998) for reliable target acquisition? laser pointers to acquire targets on a Smartboard (Myers et al. 2002) mouse-based interaction techniques lodestones and lay lines (Jul, 2002) flick (Dulberg et al., 1999) sticky icons (Swaminathan and Sato 1997)
related work driving directions vs. route planning aids overview-plus-detail focus-plus-context pointing into off-screen space
inside applications… drag-and-pop works even if target is occluded clipped closed (folder) use the concept to file s?
intrusion border handle space for arcs… and for corner arcs reserve space for content
arc length = distance
handling many objects find best (restaurant): relevance cut-off see all (dangers): merge arcs