Information Visualization CSCI 6174: Open Problems in CS Fall 2011 Richard Fowler

Ya gotta visualize … I see what you mean … so, visualization can be considered not just a visual process, but a cognitive (thought) process as well And a very large part of human brain taken up with visual system and that part of the brain is still useful beyond “simply” getting an image of the world … which is in fact pretty complicated

Visualization is … Visualize: (Computer-based) Visualization: “To form a mental image or vision of …” “To imagine or remember as if actually seeing …” Firmly embedded in language, if you see what I mean (Computer-based) Visualization: “The use of computer-supported, interactive, visual representations of data to amplify cognition” Cognition is the acquisition or use of knowledge Card, Mackinlay Shneiderman ’98 Scientific Visualization: physical Information Visualization: abstract

Visualization is not New Cave guys, prehistory, hunting Directions and maps Science and graphs e.g, Boyle: p = vt … but, computer based visualization is new … and the systematic delineation of the design space of (especially information) visualization systems is growing nonlinearly

Visualization and Insight “Computing is about insight, not numbers” Richard Hamming, 1969 And a lot of people knew that already Likewise, purpose of visualization is insight, not pictures “An information visualization is a visual user interface to information with the goal of providing insight.”, (Spence, in North) Goals of insight Discovery Explanation Decision making

“Computing is about insight, not numbers” Numbers – states, %college, income: State % college degree income State % college degree income

“Computing is about insight, not numbers” Insights: What state has highest income?, What is relation between education and income?, Any outliers? State % college degree income State % college degree income

“Computing is about insight, not numbers” Insights: What state has highest income?, What is relation between education and income?, Any outliers?

Not about Useless Visual Stuff - Clutter “3d” adds nothing (at best)

Detrimental useless stuff USA Today

An Example, Challenger Shuttle Presented to decision makers To launch or not Temp in 30’s “Chart junk” Finding form of visual representation is important cf. “Many Eyes”

An Example, Challenger Shuttle With right visualization, insight (pattern) is obvious Plot o-ring damage vs. temperature

Insight … Some examples ….

A Classic Static Graphics Example Napolean’s Russian campaign N soldiers, distance, temperature – from Tufte

For what it’s worth … x

Visualization Pipeline: Mapping Data to Visual Form Raw Information Visual Form Dataset Views User - Task Data Transformations Mappings View F F -1 Interaction Perception Visualizations: “adjustable mappings from data to visual form to human perceiver” Series of data transformations Multiple chained transformations Human adjust the transformation Entire pipeline comprises an information visualization

Visualization Stages Data transformations: Visual Mappings: Raw Information Visual Form Dataset Views User - Task Data Transformations Mappings View F F -1 Interaction Perception Data transformations: Map raw data (idiosynchratic form) into data tables (relational descriptions including metatags) Visual Mappings: Transform data tables into visual structures that combine spatial substrates, marks, and graphical properties View Transformations: Create views of the Visual Structures by specifying graphical parameters such as position, scaling, and clipping

Information Structure Raw Information Visual Form Dataset Views User - Task Data Transformations Mappings View F F -1 Interaction Perception Visual mapping is starting point for visualization design Includes identifying underlying structure in data, and for display Tabular structure Spatial and temporal structure Trees, networks, and graphs Text and document collection structure Combining multiple strategies Impacts how user thinks about problem - Mental model

A “Taxonomy” of Visualization Space Physical Data 1D, 2D, 3D Multiple Dimensions, >3 Trees Networks Interaction Dynamic Queries Interactive Analysis Overview + Detail Focus + Context Fisheye Views Bifocal Lens Distorted Views Alternate Geometry Data Mapping: Text Text in 1D Text in 2D Text in 3D Text in 3D + Time Higher-Level Visualization InfoSphere Workspaces Visual Objects

1D Linear Data

1D Linear Data

2D Map Data

3D World Data

Multiple Dimensions > 3 “Straightforward” 1, 2, 3 dimensional representations E.g., time and concrete Can extend to more challenging n-dimensional representations Which is at core of visualization challenges E.g., Feiner et al., “worlds within worlds”

Temporal Data

Trees, Networks, and Graphs Connections between /among individual entities Most generally, a graph is a set edges connected by a set of vertices G = V(e) “Most general” data structure Graph layout and display an area of iv Trees, as data structure, occur … a lot E.g., Cone trees

Tree/Hierarchical Data Workspaces The Information Visualizer: An Information Workspace by G. R. Robertson, S. K. Card, J. M. Mackinlay, 1991 CACM

Networks E.g., network traffic data

Visualization of NSFNET Cox, D. & Patterson, R., NCSA, 1992

Routes of the Internet, 1/15/05 The opte project Earlier snapshot in permanent collection of NY Museum of Modern Art

3-d hyperbolic tree of web sites using Prefuse

Abstract – Non-physical Concept map Graph of “conceptual” information From Berners-Lee’s proposal to CERN for what is now called www, March 1989 Manual “graph drawing” http://www.nic.funet.fi/index/FUNET/history/internet/w3c/proposal.html

FYI - Demo http://thejit.org/

Text and Document Collection Structure Derivation of relationships upon which display is to be based a challenge E.g., Wise et al

Text and Document Collection Structure, e.g., Galaxy of News

Overview Strategies Typically useful, or critical, to have “feel” for all data Then, allows closer inspection in “context” of all data Overview + detail, focus + context Known from the outset of visualization Bifocal Lens Database navigation: An Office Environment for the Professional by R. Spence and M. Apperley Shneiderman mantra “overview first, zoom and filter, details on demand”

Focus+Context: Fisheye Views, 1 Detail + Overview Keep focus, while remaining aware of context Fisheye views Physical, of course, also .. A distance function. (based on relevance) Given a target item (focus) Less relevant other items are dropped from the display Classic cover New Yorker’s idea of the world

Focus+Context: Fisheye Views, 2 Detail + Overview Keep focus while remaining aware of context Fisheye views Physical, of course, also .. A distance function. (based on relevance) Given a target item (focus) Less relevant other items are dropped from the display Or, are just physically smaller – distortion

Focus + Context – Spatial Distortion Selectively reduce complexity as f(user’s viewpoint) Spatial distortion Project network on distorted space Viewing “lens”

Focus + Context – Spatial Distortion Selectively reduce complexity as f(user’s viewpoint) Spatial distortion Project network on distorted space Viewing “lens” Seamless transition

Focus + Context – Hyperbolic View Again, selectively reduce complexity as f(user’s viewpt.) Smooth change during interaction

Focus + Context – Hyperbolic View Also, in 3 space Demo

3-d hyperbolic tree of web sites using Prefuse

Tools

IBM’s Many Eyes Multiple visualizations

IBM’s Many Eyes Visualization types

IBM’s Many Eyes Life expectancy vs. health care costs http://manyeyes.alphaworks.ibm.com/manyeyes/visualizations/life-expectancy-vs-per-capita-annu

Visualization Pipeline: Mapping Data to Visual Form Raw Information Visual Form Dataset Views User - Task Data Transformations Mappings View F F -1 Interaction Perception Visualizations: “adjustable mappings from data to visual form to human perceiver” Series of data transformations Multiple chained transformations Human adjust the transformation Entire pipeline comprises an information visualization

Visualization Stages Data transformations: Visual Mappings: Raw Information Visual Form Dataset Views User - Task Data Transformations Mappings View F F -1 Interaction Perception Data transformations: Map raw data (idiosynchratic form) into data tables (relational descriptions including metatags) Visual Mappings: Transform data tables into visual structures that combine spatial substrates, marks, and graphical properties View Transformations: Create views of the Visual Structures by specifying graphical parameters such as position, scaling, and clipping

Information Structure Raw Information Visual Form Dataset Views User - Task Data Transformations Mappings View F F -1 Interaction Perception Visual mapping is starting point for visualization design Includes identifying underlying structure in data, and for display Tabular structure Spatial and temporal structure Trees, networks, and graphs Text and document collection structure Combining multiple strategies Impacts how user thinks about problem - Mental model

Information Vis Systems at UTPA

Information Vis Systems at UTPA Data mining, VAS - Visual Analysis System - Hubs and authorities Text visualization, ATV - Abstract Text Viewer - Tag clouds and such Clinician’s tool for personality, DID-TM Dissociative Identity Disorder – Trait Mapper, Visualizing personality Reading: Knowledge domain citation and semantic structure Knowledge worker’s tool Selectively varying density in graph visualization Perceiving organization Reports available on web site

Data mining, VAS - Visual Analysis System

Data mining, VAS - Visual Analysis System Hubs and authorities Emphasizes effort on data Collection and transformation to form dataset Raw Information Visual Form Dataset Views User - Task Data Transformations Mappings View F F -1 Interaction Perception

Data Mining: Hubs and Authorities Attempt to overcome shortcomings of text indexed search engines Graph and cluster based approach Link structure of WWW “latent human annotation” Link to page implicit “endorsement” of page Web as directed graph Based on link structure, characterizes pages as: 1. “Authorities” - best sources of information - high indegree (refined) 2. “Hubs” - provide collections of links to authorities - high outdegree (refined)

The System Goal: Allow user to rapidly and incrementally assess utility of web pages Data mining (hubs and authorities) Visualization Filtering User interaction tools

System Architecture Goal: Allow users to systematically and incrementally access web pages User Query User Interact Query Results Filter for Display Search Engine W W W Fetch Pages Layout Pages Hub Scores

User Query User Interact Query Results Filter for Display Raw Information Visual Form Dataset Views User - Task Data Transformations Mappings View F F -1 Interaction Perception Goal: Allow users to systematically and incrementally access web pages User Query User Interact Query Results Filter for Display Search Engine W W W Fetch Pages Layout Pages Hub Scores

Example Screen Only pages of highest hub and authority scores Red: Hubs Blue:Authorities User can select pages

Example Screen - Detail

ATV - Abstract Text Viewer Text Visualization Tag cloud from infovis wiki

ATV - Abstract Text Viewer Text Visualization Electronic presentation of text for a generation Ubiquitous Manuals, web document/pages, books, … Surprisingly few tools for augmenting ATV: Text reading tool for electronic documents Uses well-known and novel techniques

ATV Electronic Presentation Techniques Overview + Detail Facilitates orientation and navigation Works for spatial data and text Abstract text’s content and use to organize Enhance reader’s efficiency and effectiveness Use existing elements: HTML tags Use system derived elements: keywords, …

Paragraph View ATV is a browser Left for structure & content, “overview” Right for enhanced text, “detail”

HTML Structure View Headings reveal structure (outline) Entire document available

Link View All links (navigation elements) available

Word Frequency View Crawler reads domain Words above threshold in domain listed Overview of domain

Word Frequency View Words with frequencies > 2 thresholds displayed

Detail (Text) Window Darkness of text = f(relatedness) to entire document Similarity of paragraph to entire document

Detail (Text) Window Word search provided

ATV Conclusions Testbed for implementing and testing text abstraction and viewing techniques Currently provides tools targeting HTML documents Extension to non-marked documents Platform for usability testing

DID-TM

Clinician’s tool for personality, DID-TM Dissociative Identity Disorder – Trait Mapper Visualizing personality Tool for clinician use Manage complexity of case history Show visually state and progress of client in integrating identities Well known visualization techniques E.g., parallel coordinates Novel techniques E.g., coding of communication and shift over time

DID-TM Personality profiles Identity communication graph Stored and indexed clinician’s notes

Visualizing Knowledge Domain Structure Knowledge worker’s (or anyone’s) tool Yet again, managing large amounts of information -Tools for organizing knowledge domain E.g., scientist (or student) learning about a new domain Become acquainted with literature or find new relations and information Citeseer Exploring and retrieving information Visual representation of citation network Visual representations of semantic similarity of documents Similar to Document Explorer

Network Visualization

Visualizing Knowledge Domain Structure Exploring and retrieving information Visual representation of citation network Relationships among documents as shown by citations (references) Visual representations of document semantic similarity network Semantic document network Again, relations, now based on content similarity Raw Information Visual Form Dataset Views User - Task Data Transformations Mappings View F F -1 Interaction Perception

Extracting and Organizing Content Raw Information Visual Form Dataset Views User - Task Data Transformations Mappings View F F -1 Interaction Perception Networks: 1. Citations form graph 2. Document similarity Word cooccurrence Similarity of Documents Compare all pairs of documents Use distance matrix to derive network Network density varies

Displaying the Networks: Node Positioning using Spring Layout Raw Information Visual Form Dataset Views User - Task Data Transformations Mappings View F F -1 Interaction Perception Physical spring analog “Spring Embedder” algorithm Can vary spring length, strength, elastic properties E.g., document similarity Example at right in 3D Interaction by movement

Network Visualization Visualizing Knowledge Domain Citation and Semantic Structure Citeseer Visualization 1,138 documents from Citeseer collection Citation network Nodes are documents, links are citations (references) Here, links are weighted by document similarity

Citeseer Visualization - Query Query “information visualization” Results used to form citation graph and visual representation

Citeseer Visualization – Results as MST Minimum cost spanning tree (graph) used to represent query results

Earlier Network Display & Interaction Tools Overviews Nodes of highest degree Landmarks: Visible, selectable Bookmarks Set & return to viewpoint Fluid motion Network density selectable Anchors User-defined selectable Signposts Anchor labeled with overview nodes Global orientation at level of local detail Expand and Collapse Nodes Color

Display & Interaction Tools, cont. Stereo Viewing LCD glasses Head tracked, “look around” Compromise immersion for text tasks

Selective Density in Network Viewing

Overview Reducing and managing network density for visualization Varying structural density Distorted space display techniques Deriving quantitative metrics from documents From which document network created Pathfinder networks Path length limited minimum cost networks A new hybrid representation to selectively vary density

Internet alliances and partnershships, 2002 www.orgnet.com/netindustry.html

Trade relationships, 1992 www.

Reducing and Managing Density Focus + context techniques “selectively” reduce density User’s view affects display Spatial distortion techniques Use same network, but change space upon which it is projected Change network structure itself, depending on where focus is E.g., Furnas’ 1986 account of fevs, display as f(distance from focus) Threshold techniques Display only links with weights > some value As part of structure derivation (network formation) E.g., minimum cost spanning tree (MCST) Limiting case for connected graph Pathfinder networks

Focus + Context – Hyperbolic View Again, selectively reduce complexity as f(user’s viewpt.) Smooth change during interaction

Threshold Reduce complexity by eliminating links < some threshold Not necessarily preserve connectivity http://www.g3tvu.co.uk/Network,_Radio_Link_and_Route_Styles.htm

Varying and Reducing Density As shown, can vary display space and locally (selectively) reduce density Distortion techniques Also, can reduce density globally (overall) Link weight threshold, as shown Minimum spanning trees Here, Pathfinder networks Goal of work is to create representation that uses structural (vs. display) manipulation to provide global context and local detail

PfNets – Path Length Limited For some data set of distances Here, data are provided by human subjects Document network uses interdocument distances Construct network that is sufficiently dense that any node can be reached from any other node in q links q = n-1 Schvaneveldt et al., 1989

PfNets – Path Length Limited Smaller q Denser graph Schvaneveldt et al., 1989

Graph Display Considerations Graph display issues critical in visualization And a field in itself Force directed layout Widely used E.g., prefuse Works well for sparse graphs Shows global relations well Not so well for dense

PfNets for Global Context and Local Detail Combine sparse pfnet (inf, n-1) with more dense at point of interest Provide detail + context

Hybrid Pfnets Sparse overview.TIF

Hybrid Pfnets Dense

Hybrid Pfnets Dense zoomed in

Hybrid Pfnets Combined

Perception of Organization

Perception of Organization Self organizing systems Simple rules, complex behaviors Social insects Ants, bees Flocks of birds Fairly well modeled with few constraints Coherence (cohesion) of flock Distance from another individual Direction What are the roles of the elements of organization used by people?

Perception of Organization Coherence (cohesion) of flock - Distance from another individual Direction - Stereoscopy

Perception of Organization Coherence (cohesion) of flock - Distance from another individual Direction - Stereoscopy

BONUS! Immersive interfaces, prescence, … New research effort

Introduction The “best” interfaces, and all systems, typically find their task utility through engagement (etc.) appropriate for the task This idea is at the core of arguments for the use of direct manipulation interfaces All of the following are interrelated: Immersion, engagement, presence, virtual reality 3D display and interaction devices In field of CS and HCI: “spatial interfaces” Will introduce the idea of presence

Immersion, “Virtual Reality”, and Virtual Environments Immersive interfaces High sensory immersion – visual, auditory, haptic, proprioceptive “Virtual reality”, or, virtual environments “Virtual reality is a technology that is used to generate a simulated environment in digital form... Using the equipment, users are immersed in a totally virtual world.” Working definition – an immersive interactive system In context of “virtual reality”, immersion usually = spatial immersion Note: “Immersion” (and engagement and presence) is a continuum Text ... Visual and 3d .. Stereo ... HMD… “jacked in” Cyberspace Term coined by Gibson in Neuromancer … and in the 21st century, the Matrix

Immersion and Virtual Reality “The mind has a strong desire to believe that the world it perceives is real” – Jaron Lanier, among others For example, “illusion” (perception) of depth (for spatial immersion) Stereo parallax Head motion parallax Object motion parallax Texture scale Interaction: grab and move an object Proprioceptive cues: when you reach out and see a hand where you believe your hand to be, you accept the hand as your own Often you will accept what you see as “real” even if graphics poor Constellation of cues

Presence “The Aesthetic Impression of 3D Space” Sense of presence Vividly 3d Actually present in the world Sense of being there Holodeck … Presence has to do as much with engagement, as visual information E.g., one can be “in the world”, when reading Here, one sees, or visualizes, the world 3D depth cues are those elements that enhance feeling of 3 (vs. 2) dimensions in a display, From occlusion to stereoscopic display

Presence “The Aesthetic Impression of 3D Space” Immersive interfaces term used to describe interfaces/devices which lead toward immersion (sense of presence, engagement) in the virtual environment presented on the display Virtual reality interfaces term used similarly to immersive interfaces Degree of immersion conventional desktop screen fishtank virtual reality (semi-immersive workbench) immersive virtual reality augmented reality with video or optical blending … number of cues …

Immersive and 3D Interfaces Teleoperation Virtual and augmented reality Immersion and VR – contribution of components … Survey of 3D displays Surround screen displays - CAVE Input devices - Data glove Data walls Workbenches Hemispherical display Head-mounted displays Arm-mounted displays Virtual retinal display Autostereoscopic displays

Sutherland’s 1960’s equipment Ultimate display, 1965 Sword of Damocles – 1st HMD Actual camera-like metal shutters

Virtual and Augmented Reality Augmented reality shows real world with an overlay of additional overlay Knowlton (1975) Partially-silvered mirror over keyboard Programmable labels Tactile feedback

Augmented Reality, 2 Enables users to see real world with an overlay of additional interaction Situational awareness See through glasses Typically, add text+images to real world Very sensitive to head tracking, when used

Surround-screen displays Pro less obtrusive headgear multi-user? better stereo Con occlusion problem missing sides

Surround screen displays – CAVE, 1 A room with walls and/or floor formed by rear projection screens Head tracking Stereo Light scattering problems Visual immersion Field of view is 100% possible, ~200 degrees

Surround screen displays – CAVE, 2 Typical size: 10’ x 10’ x 10’ room 2 or 3 walls are rear projection screens Floor is projected from above User is tracked He/she also wears stereo shutter goggles… Uses “wand” to manipulate Projects 3D scenes for viewer’s point of view on walls Walls vanish, user perceives full 3D scene So, view is only correct for that viewer Cost is fairly high

UTPA Immersive Systems Lab ~Summer, 2012 Proj. 27’ 13’ Security area 21’ 6’ TV storage ~8’ CAVE Physiological Measurement Equipment Computers Front Projection Screen Development

Questions? .