1. Interacting with Visualizations
Ware Chapter 10
University of Texas – Pan American
CSCI 6361, Spring 2014

2. Interacting with Visualizations - Introduction The very big picture
Best visualizations support productive interaction
Interactive visualizations
Not merely static representations of data
Though certainly has its place
Allows, e.g.,
Inspection of underlying data from the visualization
Transformation of data
Filtering – removal of data by some criteria
E.g., visual analytics systems we have seen clearly demonstrate use of highly interactive systems, indeed, across visual mappings
E.g., “Overview first, zoom and filter, then details on demand”
Shneiderman, 1996 (at class site)
Though in fact may see interesting detail, zoom out, find others, zoom in, …
VxInsight, Sandia Labs

3. Recall, Amplifying Cognition Norman, 1993
Humans think by interleaving internal mental action with perceptual interaction with the world
This interleaving is how human intelligence is expanded
Within a task (by external aids)
Across generations (by passing on techniques)
External graphic (visual) representations are an important class of external aids
Don Norman is an influential cognitive scientist
The power of the unaided mind is highly overrated. Without external aids, memory, thought, and reasoning are all constrained. But human intelligence is highly flexible and adaptive, superb at inventing procedures and objects that overcome its own limits. The real powers come from devising external aids that enhance cognitive abilities. How have we increased memory, thought, and reasoning? By the invention of external aids:
It is things that make us smart. (Norman, 1993, p. 43)
External Cognition

4. Introduction and Overview
Visualization as an “internal interface”
Interface between human and computer in a man-machine problem-solving system
Computer-based information system supports data gathering, calculation, and analysis
Augments investigator’s working memory
Provides visual markers for concepts
Reveals structural relationships between problem components
Some models of visualization – different takes on the same thing!
Overview, zoom, filter, details (Shneiderman)
Visualization Pipeline North (from Card et al.)
Knowledge crystalization (Card et al.)
Ware
Model human processor (Card et al.)
Motor processor – Ex: Fitts’ law
Viewing information spaces
Distortion techniques, fisheye views
Navigation and Exploration

5. Example: “Overview, zoom and filter, details on demand” - Shneiderman
VxInsight demonstrates:
“Overview, zoom and filter, details on demand”
Saw earlier when talking about text representations (visual mappings)
Again, visual analytics systems provide
Developed by Sandia Labs to visualize databases
“Elements of database can be “anything”
For IV “abstract”
e.g., document relations, company profiles
Example screens show grant proposals
Shows interactive capabilities

6. VxInsight: Overview
vvv

7. VxInsight Interaction paradigm (Shneiderman): Or (Ware) … Overview
Zoom
Filter
Details on demand
Browse
Search query
Or (Ware) …
Lowest level
Data manipulation loop
Intermediate
Exploration and navigation loop
Highest
Problem-solving loop

8. VxInsight - Overview Interaction paradigm Overview Zoom Filter
Details on demand
Browse
Search query

9. VxInsight - Zoom Interaction paradigm Overview Zoom Filter
Details on demand
Browse
Search query

10. VxInsightv - Details Interaction paradigm Overview Zoom Filter
Details on demand
Browse
Search query

11. VxInsightv - Query Interaction paradigm Overview Zoom Filter
Details on demand
Browse
Search query

12. Recall, Visualization Pipeline: Or, another take on interaction: Mapping Data to Visual Form
Raw
Information
Visual
Form
Dataset
Views
User
- Task
Data
Transformations
Mappings
View F
F -1
Interaction
Perception
Most fundamentally – Visualizations are:
“adjustable mappings from data to visual form to human perceiver”
Series of data transformations ( )
Multiple chained transformations
Human adjusts the transformations - interaction
Entire pipeline comprises an information visualization

13. Visualization Pipeline: Human might adjust any of the visualization Stages
Raw
Information
Visual
Form
Dataset
Views
User
- Task
Data
Transformations
Mappings
View F
F -1
Interaction
Perception
Data transformations (rarely):
Map raw data (idiosynchratic form) into data tables (relational descriptions including metatags)
Visual Mappings (sometimes):
E.g., table to graph
Transform data tables into visual structures that combine spatial substrates, marks, and graphical properties
View Transformations (very often):
E.g., zooming, …, changing viewpoint
Create views of Visual Structures by specifying graphical parameters such as position, scaling, and clipping

14. Ware: Interactive Visualization: Interlocking Feedback Loops – Quick Look
Process made up of interlocking feedback loops
Lowest level: Data manipulation loop
Objects selected and moved
Relies on eye-hand coordination
Requires delay-free interaction
Intermediate: Exploration & navigation loop
User finds way in large visual space
Searching a large data space part by part
Building a cognitive map of the data/simulation
Highest: Problem-solving loop
Forming and testing hypotheses about data
Refines hypotheses through augmented visualization
Repeat through cycles, revising or replacing visualization
New data added, problem reformulated, possible solutions identified
Visualization as external representation of problem
Extension of cognitive process
Exploration
and Navigation
Problem Solving
Data
Manipulation

15. Interactive Visualization Recall, Problem Solving, Cognitive Amplification, Knowledge Crystallization, (Card et al.)
Knowledge crystallization: Gather knowledge, make sense of it, use it in task
Task operations
Task
Forage for data
Write, decide, or act
Overview
Zoom
Filter
Details
Browse
Search query
Extract
Compose
Reorder
Cluster
Class
Average
Promote
Detect pattern
Abstract
Search for schema
Problem-solve
Read fact
Read comparison
Read patter
Manipulate
Create
Delete
Instantiate schema
Instantiate

16. Again, Ware’s Interlocking Feedback Loops
Interactive visualization
Process made up of interlocking feedback loops
Lowest level: Data manipulation loop
Objects selected and moved
Relies on eye-hand coordination
Requires delay-free interaction
Intermediate: Exploration & navigation loop
User finds way in large visual space
Searching a large data space part by part
Building a cognitive map of the data/simulation
Highest: Problem-solving loop
Forming and testing hypotheses about data
Refines hypotheses through augmented visualization
Repeat through cycles, revising or replacing visualization
New data added, problem reformulated, possible solutions identified
Visualization as external representation of problem
Extension of cognitive process
Exploration
and Navigation
Problem Solving
Data
Manipulation

17. 1) Interacting Feedback Loops, and 2) Knowledge Crystallization, …
Knowledge crystallization: Gather knowledge, make sense of it, use it in task
Different time spans
Problem Solving - outer
Longest time
Exploration and Navigation
Primary use of data and information visualizations
Occurs for all elements of problem solving, knowledge crystallization
Data Manipulation
Motor, etc.
Again, for all element of exploration and navigation
Forage for data
Write, decide, or act
Problem-solve
Instantiate schema
Search for schema
Task
Exploration and Navigation
Data
Manipulation

18. Interacting Feedback Loops – Another Way Ware’s account with “gear” metaphor
As “gears” …
Problem Solving
(knowledge
crystalization)
Exploration and Navigation
Data Manipulation
Exploration
and Navigation
Problem Solving
Data
Manipulation

19. Lowest Level: Data Manipulation Loop
Exploration
and Navigation
Problem Solving
Data
Manipulation

20. Lowest Level: Data Manipulation Loop
Visual-Manual Control Loop
Very carefully studied, for example …
Choice reaction time: Hick-Hyman law
Reaction time = a + b log2 (C)
2D positioning and selection: Fitts’ law – quickly, more later
Part of ISO standard
Protocols for evaluating user performance and comfort when using pointing devices with visual display terminals
Selection time = a + b log2 (D/W + 1.0)
Hitting smaller targets further away is harder
Adding latency severely increases difficulty
Fitts’ law, including lag
Mean time = a + b (Human Time + Machine lag) log2 (D/W + 1.0)
Control compatibility is important
Offset and scale is easy to deal with; rotation is hard
Reaction time in making choices
>= 160 msec per doubling of the numbers of choices
Faster if allowed to make mistakes
Exploration
and Navigation
Problem Solving
Data
Manipulation

21. Model Human Processor + Attention Recall
A “useful” big picture - Card et al. ’83 plus attention
Senses/input ® f(attention, processing) ® motor/output
Notion of “processors”
Purely an engineering abstraction
Detail next

22. Model Human Processor + Attention
Sensory store
Rapid decay “buffer” to hold sensory input for later processing
Perceptual processor
Recognizes symbols, phonemes
Aided by LTM
Cognitive processor
Uses recognized symbols
Makes comparisons and decisions
Problem solving
Interacts with LTM and WM
Motor processor
Input from cog. proc. for action
Instructs muscles
Feedback
Results of muscles by senses
Attention
Allocation of resources

23. Model Human Processor Recall
Card et al. ’83
An architecture with parameters for cognitive engineering …
Will see visual image store, etc. tonight
Memory properties
Decay time: how long memory lasts
Size: number of things stored
Encoding: type of things stored

24. Model Human Processor Motor Processor
tM = 70 (range 30-70)
For repetitive tasks without feedback
Tasks with feedback involve all:
Perceptual processor
Cognitive processor

25. Motor Processing Motor processor can operate in two ways:
1. Open-loop control
Motor processor runs a program by itself – no feedback about correctness
Maximum rate, cycle time is tM = Tmotor ~ 70 ms
Experiment: Scribble without looking and trying to stay in lines
2. Closed-loop control
Experiment: Looking at lines, draw within the lines
Muscle movements (or their effect on the world) are perceived by cognitive system and compared with desired result
Cycle time is Tprocess + Tcognitive + Tmotor ~ 240 ms
The motor processor can operate in two ways. It can run autonomously, repeatedly issuing
the same instructions to the muscles. This is “open-loop” control; the motor processor
receives no feedback from the perceptual system about whether its instructions are correct.
With open loop control, the maximum rate of operation is just Tm.
The other way is “closed-loop” control, which has a complete feedback loop. The
perceptual system looks at what the motor processor did, and the cognitive system makes a
decision about how to correct the movement, and then the motor system issues a new
instruction. At best, the feedback loop needs one cycle of each processor to run, or Tp + Tc
+ Tm ~ 240 ms.
Here’s a simple but interesting experiment that you can try: take a sheet of lined paper and
scribble a sawtooth wave back and forth between two lines, going as fast as you can but
trying to hit the lines exactly on every peak and trough. Do it for 5 seconds. The frequency
of the sawtooth carrier wave is dictated by open-loop control, so you can use it to derive
your Tm. The frequency of the wave’s envelope, the corrections you had to make to get
your scribble back to the lines, is closed-loop control. You can use that to derive your value
of Tp + Tc.

26. Fitts’s Law - demo Fitts’s Law
Fundamental law of human sensory-motor system
Fitts, P. M. (1954). The information capacity of the human motor system in controlling the amplitude of movement. Journal of Experimental Psychology, 47,
E.g., for direct (reach) and mouse use
Demo:
Best – won’t run on class box
OK – no line plotted

27. Fitts’s Law - demo Fitts’s Law “tele-actor” results from demo:
Fundamental law of human sensory-motor system
“tele-actor” results from demo:

28. Fitts’s Law Fitts’s Law
Fundamental law of human sensory-motor system
E.g., for direct (reach) and mouse use
The time to acquire a target is a function of distance to and width (size) of target
T = f (D, S)
Time T to move your hand to a target of size S at distance D away:
T = ReactionT + MotorT
= a + b * log2 (2 * D/S)
Depends only on index of difficulty log(2D/S)

29. Explananation of Fitts’s Law
Moving hand to a target is closed-loop control
Vs. open-loop control we saw for Card et al. model
Each (correction) cycle covers remaining distance D, with error εD
Smaller correction in position as get closer
(because there is less distance with which to correct)
Slower velocity
(because don’t go so fast with shorter distance)
We can explain Fitts’s Law by appealing to the human information processing model. Fitt’s
Law relies on closed-loop control. In each cycle, your motor system instructs your hand to
move the entire remaining distance D. The accuracy of that motion is proportional to the
distance moved, so your hand gets within some error εD of the target (possibly
undershooting, possibly overshooting). Your perceptual and cognitive processors perceive
where your hand arrived and compare it to the target, and then your motor system issues a
correction to move the remaining distance εD – which it does, but again with proportional
error, so your hand is now within ε2D. This process repeats, with the error decreasing
geometrically, until n iterations have brought your hand within the target – i.e., εnD ≤ ½ S.
Solving for n, and letting the total time T = n (Tp + Tc + Tm), we get:
T = a + b log (2D/S)
where a is the reaction time for getting your hand moving, and b = - (Tp + Tc + Tm)/log ε.
The graphs above show the typical trajectory of a person’s hand, demonstrating this
correction cycle in action. The position-time graph shows an alternating sequence of
movements and plateaus; each one corresponds to one cycle. The velocity-time graph
shows the same effect, and emphasizes that hand velocity of each subsequent cycle is
smaller, since the motor processor must achieve more precision on each iteration.

30. Implications of Fitts’s Law
Buttons, etc. should be reasonable size;
hard to click small targets.
Edges and corners of the computer display are easy to reach
Mac single menubar better than multiple Windows menubars
Also, pointer is "caught" at the edges
Popup menus can usually be opened faster than pull-down menus
User avoids movement
Pie menu items are typically selected faster than linear menu items
Small distance from the center of the menu
Wedge-shaped target areas are large

31. Power Law of Practice Time to do a task decreases with practice
Obviously
Involves all of perceptual-cognitive-motor system
Time Tn to do a task the nth time:
Decaying exponential rate
Tn = T1n α
α is typically
Example:
Novices get rapidly better at task with practice, but performance “levels off”
Though still increasing performance

32. Intermediate Level: Exploration, View Refinement and Navigation
Problem Solving
Data
Manipulation

33. Intermediate Level: Exploration, View Refinement and Navigation
View navigation important when data space is too large to fit on screen
Complex problem
Considers theories of pathfinding and map use, cognitive spatial metaphors, direct manipulation, visual feedback
Basic navigation control loop (below)
Left is human – cognitive and spatial model with which user understands data space and progress through it
Maintaining data space for some time may become encoded in long-term memory
Right is system – visualization may be updated and refined from data mapped into spatial model
Includes:
3D Locomotion and viewpoint control
Pathfinding
Focus + context
Exploration
and Navigation
Problem Solving
Data
Manipulation

34. 3D Locomotion and Viewpoint Control Navigation in 3D
Displaying data elements so looks like 3D landscape, vs. flat map, often used
Follows from Gibsonian orientation
Affordances
Properties of the world perceived in terms of potential for action (physical model, direct perception)
Problem with generalization to user interfaces/interaction
Nevertheless, important and influential
Have examined depth cues
Embed objects in space, navigate space
Flying viewpoint through the data space
Constrain user to useful parts of the space to reduce cognitive load of navigation
Surface of the ground
Walkways within power plant
Particular paths of interest
Examples
Web browser: Harmony
Clustering of text, Wise et al.

35. 3D Locomotion and Viewpoint Control: Spatial Metaphors
Evaluation
Exploration and Explanation
Cognitive and Physical Affordance
Task 1: Find areas of detail in the scene
Task 2: Make the best movie
3D environments: Hallway, extended terrain, closed object.
World-in-hand
Good for discrete objects
Poor affordances for looking scale changes – detail
Problem with center of rotation when extended scenes
Eye-in-hand
Easiest under some circumstances
Poor physical affordances for many views
Subjects sometimes acted as if model were actually present
Walking
Flying vehicle control
Hardest to learn but most flexible
Non-linear velocity control
Spontaneous switch in mental model
The predictor as solution

36. 3D Locomotion and Viewpoint Control: Wayfinding, Cognitive, and Real Maps
Worldlets
Can be rotated to facilitate recognition

37. Frames of Reference Egocentric, Exocentric
Use of maps implies ability to apply another perspective
To physical,
e.g., road map (view from above),
Or abstract
… another frame of reference
Egocentric
view from user
Exocentric
View from outside the user
Road map just one of many exocentric view
Movement of body (vs. eyes) affects orientation most
Pan, tilt, …, but not rotation, so dof constrained in practice

38. Frames of Reference Tethered view, world view
Various views illustrated

39. Mutiple Simultanous Views
Represent data space in different forms in different views
E.g., “spiral calendar”

40. Focus, Context, and Scale
Saw this earlier, here, in Ware

41. Focus, Context, and Scale
Problem of finding detail in larger context
Again, spatial navigation
Wayfinding problem may be considered as discovering specific objects in a larger context
Addressed by multiple views at differing spatial scales
Movement between views at different scales (and frames of reference)
Changing spatial scale
E.g., overview + detail
Addressed, also, by changing structural scale
E.g., collapsing lines of code in display of software systems

42. Focus, Context, and Scale: Overview and Detail
Fred Brooks’ GRIP project at UNC
Molecular structure solution, docking
Architectural walkthrough
Users always going from detail to overview
Then overview to detail…
Then detail to overview…
Options
Provide display of both
Provide easy, non-jarring switch between them
Multiple-Window Zoom with Callouts …

43. Focus+Context: Fisheye Views, 1
Detail + Overview
Keep focus, while remaining aware of context
Fisheye views
Physical, of course, also ..
A distance function. (based on relevance)
Given a target item (focus)
Less relevant other items are dropped from the display
Classic cover
New Yorker’s idea of the world

44. Focus+Context: Fisheye Views, 2
Detail + Overview
Keep focus while remaining aware of context
Fisheye views
Physical, of course, also ..
A distance function. (based on relevance)
Given a target item (focus)
Less relevant other items are dropped from the display
Or, are just physically smaller – distortion

45. Distortion Techniques, Generally
Distort space = Transform space
By various transformations
“Built-in” overview and detail, and landmarks
Dynamic zoom
Provides focus + context
Several examples follow
Spatial distortion enables smooth variation

46. Focus + Context, 1 Fisheye Views
Keep focus while remaining aware of the context
Fisheye views:
A distance function (based on relevance)
Given a target item (focus)
Less relevant other items are dropped from the display.
Demo of Fisheye Menus:

47. Focus + Context, 2 Bifocal Lens
Database navigation: An Office Environment for the Professional by R. Spence and M. Apperley

48. Focus + Context, 3 Distorted Views
The Table Lens: Merging Graphical and Symbolic Representations in an Interactive Focus + Context Visualization for TabularInformation by R. Rao and S. K. Card
A Review and Taxonomy of Distortion Oriented Presentation Techniques by Y. K. Leung and M. D. Apperley

49. Focus + Context, 4 Distorted Views
Extending Distortion Viewing from 2D to 3D by M. Sheelagh, T. Carpendale, D. J. Cowperthwaite, F. David Fracchia
Magnification and displacement:

50. Focus + Context, 5 Demo Alternate Geometry
The Hyperbolic Browser: A Focus + Context Technique for Visualizing Large Hierarchies by J. Lamping and R. Rao
Demo

51. Other Navigation Techniques: GeoZui3D, Zooming + 2 dof rotations
Translate point on surface to center
Then scale
Or translate and scale

52. View Refinement and Navigation (optional, from 2nd ed.)
Transparency:
When there is the perception of direct contact with the data, the interface becomes transparent
Big idea in interfaces
Temporal feedback rapid (< 1/10 second)
Response is compatible with interaction method
Interactive adjustment of ranges
Zoom in on data area of interest
Sometimes nonlinear mapping brings area of interest into range where patterns are easy to see (logarithmic)

53. Interaction vs. Animation
Ware comments:
Exploration (interaction) vs. Presentation (animation)
Flexibility vs. Efficiency
Active vs. Passive Participation
Immediacy of response and engagement
Control promotes understanding
Person moving learns more than partner watching
Active control increases sense of presence

54. End .