1. Information Visualization and Immersive Interfaces CSCI 6174: Open Problems in CS Fall 2013 Richard Fowler

2. An example Hans Rosling's 200 Countries, 200 Years, 4 Minutes http://www.youtube.com/watch?v=jbkSRLYSojo

3. To visualize is … I see what you mean … –so, visualization can be considered not just a visual process, but a cognitive (thought) process as well …. and a very large part of human brain taken up with visual system –and that part of brain is useful beyond “simply” getting an image of the world

4. Visualization is … Visualize: –“To form a mental image or vision of …” –“To imagine or remember as if actually seeing …” –Firmly embedded in language, if you see what I mean (Computer-based) Visualization: –“The use of computer-supported, interactive, visual representations of data to amplify cognition” Cognition is the acquisition or use of knowledge Card, Mackinlay Shneiderman ’98 –Scientific Visualization: physical –Information Visualization: abstract

5. Visualization is not New Cave guys, prehistory, hunting Directions and maps Science and graphs –e.g, Boyle: p = vt … but, computer based visualization is new –Allows interactive exploration and can be embedded in the work flow –… and the systematic delineation of the design space of (especially information) visualization systems is growing nonlinearly

6. Visualization and Insight “Computing is about insight, not numbers” –Richard Hamming, 1969 –And a lot of people knew that already Likewise, purpose of visualization is insight, not pictures –“An information visualization is a visual user interface to information with the goal of providing insight.”, (Spence, in North) Goals of insight –Discovery –Explanation –Decision making

7. “... insight, not numbers” Numbers – states, %college, income : State % college degree income

8. “... insight, not numbers” Insights : –What state has highest income? –What is relation between education and income? –Any outliers? State % college degree income

9. “… insight, not numbers” Insights : –What state has highest income?, –What is relation between education and income?, –Any outliers?

10. Not about Useless Visual Stuff - Clutter Here, “3d” adds nothing –(at best)

11. Detrimental, Useless Stuff USA Today So, what information is this trying to convey?

12. An Example, Challenger Shuttle Presented to decision makers prior to launch –To launch or not –Temp in 30’s “Chart junk” Finding form of visual representation is important –cf. “Many Eyes”

13. An Example, Challenger Shuttle With right visualization, insight (pattern) is obvious –Plot o-ring damage vs. temperature

14. Insight, through Visualization … Some examples ….

15. Insight, through Visualization … What accounts for US deficit? –Economic recovery measures –TARP, Fannie, and Freddie “bailouts” –Wars in Iraq and Afghanistan –Bush-era tax cuts –Economic downturn –Other (longer standing) things

16. For what it’s worth …

17. A “Taxonomy” of Visualization Topics Card, Mackinlay, and Schneiderman Space Physical Data 1D, 2D, 3D Multiple Dimensions, >3 Trees Networks Interaction Dynamic Queries Interactive Analysis Overview + Detail Focus + Context Fisheye Views Bifocal Lens Distorted Views Alternate Geometry Data Mapping: Text Text in 1D Text in 2D Text in 3D Text in 3D + Time Higher-Level Visualization InfoSphere Workspaces Visual Objects

18. 1D - Linear Data Software Visualization

19. 2D - Map Data Presidential votes 2008 on Map

20. 3D World Data – Scientific Visualization e.g., Molecule Modeling – Augmented Physical Representation

21. Multiple Dimensions > 3 What does visual representation for > 3 dimensions look like? “Straightforward” 1, 2, 3 dimensional representations –E.g., time and concrete Can extend to more challenging n- dimensional representations –Which is at core of visualization challenges E.g., Feiner et al., “Worlds within Worlds”

22. Temporal Data Baby names across years

23. Trees, Networks, and Graphs Connections between /among individual entities Most generally, a graph is a set edges connected by a set of vertices –G = V(e) –“Most general” data structure Graph layout and display an area of iv Trees, as data structure, occur … a lot –E.g., Cone trees

24. Routes of the Internet, 1/15/05 Earlier snapshot in permanent collection of NY Museum of Modern Art

25. Abstract, Non-physical, Network E.g., concept map –Graph of “conceptual” information What’s this one? From Tim Berners- Lee’s proposal to CERN for what is now called w w w, March 1989 Manual “graph drawing” http://www.nic.funet.fi/index/FUNET/history/internet/w3c/proposal.html

26. Text and Document Collection Structure Derivation of relationships upon which display is to be based a challenge E.g., Wise et al.

27. Overview Strategies Typically useful, or critical, to have “feel” for all data –Then, system allows closer inspection in “context” of all data –Overview + detail, focus + context Known from the outset of visualization –Bifocal Lens Database navigation: An Office Environment for the Professional by R. Spence and M. Apperley Shneiderman mantra –“overview first, zoom and filter, details on demand”

28. Focus+Context: Fisheye Views, 1 Detail + Overview –Keep focus, while remaining aware of context Fisheye views –Physical, of course –Also, distance function (based on relevance) –Given a target item (focus) –Less relevant items dropped from display –Classic cover New Yorker’s idea of the world

29. Focus+Context: Fisheye Views, 2 Detail + Overview –Keep focus while remaining aware of context Fisheye views –Physical, of course –Also, a distance function (based on relevance) –Given a target item (focus) –Less relevant items dropped from display –Or, are just physically smaller – distortion

30. Focus + Context – Spatial Distortion Selectively reduce complexity as: f(user’s viewpoint) Spatial distortion –Project network on distorted space Viewing “lens”

31. Focus + Context – Spatial Distortion Selectively reduce complexity as: f(user’s viewpoint) Spatial distortion –Project network on distorted space Viewing “lens” Seamless transition

32. Focus + Context – Hyperbolic View Again, selectively reduce complexity as f(user’s viewpoint) As with distorted lens, space is non-linear, here, hyperbolic Smooth change during interaction

33. 3-d Hyperbolic Tree of Web Sites Using Prefuse

34. Visualization Pipeline: Mapping Data to Visual Form, cf. North paper Visualizations: –“adjustable mappings from data to visual form to human perceiver” Series of data transformations –Multiple chained transformations –Human adjust the transformation Entire pipeline comprises an information visualization Raw Information Visual Form Dataset Views User - Task Data Transformations Visual Mappings View Transformations F F -1 Interaction Visual Perception

35. Visualization Stages Data transformations: –Map raw data (idiosynchratic form) into data tables (relational descriptions including metatags) Visual Mappings: –Transform data tables into visual structures that combine spatial substrates, marks, and graphical properties View Transformations: –Create views of the Visual Structures by specifying graphical parameters such as position, scaling, and clipping Raw Information Visual Form Dataset Views User - Task Data Transformations Visual Mappings View Transformations F F -1 Interaction Visual Perception

36. Information Structure Visual mapping is starting point for visualization design Includes identifying underlying structure in data, and for display –Tabular structure –Spatial and temporal structure –Trees, networks, and graphs –Text and document collection structure –Combining multiple strategies Impacts how user thinks about problem - Mental model Raw Information Visual Form Dataset Views User - Task Data Transformations Visual Mappings View Transformations F F -1 Interaction Visual Perception

37. A Recent (2012) System

38. Information visualization for task of document retrieval Document collection is organized as clusters (visually represented) –User views and interacts with the cluster representations IV Interaction: –Overview –Flock-based clustering –Zoom, spatially –Filter –Text match, query –Markers –Details on demand –Document itself –Tunable

39. Swarm (Flock) based Clustering for Information Visualization and Retrieval Reynold’s technique for animation of flocks of birds –Computational model of group motion Particle system –Standard part of many toolkits http://www.youtube.com/watch?v=-8KZBGAAM30 Agent-based approach –Individual elements each apply rules for motion, as particle systems, etc. Reynolds (1987). Flocks, herds, and schools: A distributed behavioral model, Computer Graphics, 21(4), 25-34.

40. Flock-based Movement Individual agents utilize only three rules –Agents adjust their directions of movement and velocities - steer –Form single group Movement perceptually similar to flock of birds, other social biological group –Alignment: Steer towards average direction of movement of nearby agents –Separation: Steer to avoid being too close to nearby agents –Cohesion: Steer to move toward the center position of nearby agents Rules are added for obstacles, …, edge of screen, …

41. Flock - based Clustering: Multiple Groups – Bird Species, Document Clusters, … Rules described so far form a single group –As, e.g., with a single species of birds Can add additional rules that consider similarity or a priori categorization among agents to form multiple groups “Multi-species flocking” –Birds of same species form groups and moved together –Agents (birds) described by feature vector, e.g., length of beak, color, size Flock-based document clustering –Vector Space Model –Similarity of documents determined by weighted number of terms in common –Essentially, a biologically (or animation) inspired “Iterative clustering technique”

42. System Information visualization for task of document retrieval –Overview –Flock-based clustering –Zoom, spatially –Filter –Text match, query –Markers –Details on demand –Document itself –Tunable

43. System’s Flock-based Clustering E.g., 500 documents –Clusters apparent Uses standard metrics of document similarity with velocity vector change based on similarity Also considers thresholds of similarity and dissimilarity –Provides tunable parameters for arranging spatial display (Cui et al., 2006) –Now,

44. Identifying Clusters of Interest Term Match Term match to mark documents to guide exploration –Matching documents’ color is changed –Multiple term matches

45. Identifying Clusters of Interest Term Match Term match to mark documents to guide exploration –Matching documents’ color is changed –Multiple term matches

46. Identifying Clusters of Interest Term Match Term match to mark documents to guide exploration –Matching documents’ color is changed –Multiple term matches

47. Identifying Clusters of Interest Term Match Term match to mark documents to guide exploration –Matching documents’ color is changed –Multiple term matches

48. Identifying Clusters of Interest Query Match Term match to mark documents to guide exploration –Matching documents’ color is changed –Multiple term matches Query match to mark documents

49. Viewing Individual Documents Zoom, then select individual element –Title is displayed

50. Viewing Individual Documents Select title to display individual document text –Also, can select from list retrieved in query

51. Interactively Adjusting Clusters In practice, many clustering techniques make use of “tuning ” System allows user to change parameters interactively –Weights for “basic” rules and other motion parameters Avoidance, cohesion, alignment Velocity minimum, maximum Limit of steering –Range for consideration –Thresholds for similarity and dissimilarity

52. Stereoscopic Display Perception / extraction of structure facilitated by stereoscopic display –Reveals z component of 3d arrangement –Head-tracked stereo. Structure from motion Challenge of practical stereoscopy in context of task performance Two monitor desktop

53. Stereoscopic Display Multi-screen display Perception / extraction of structure facilitated by stereoscopic display –Reveals z component of 3d arrangement –Head-tracked stereo. Structure from motion Challenge of practical stereoscopy in context of task performance

54. Immersive Interfaces, Prescence, …

55. Introduction The “best” interfaces, and all systems, typically find their task utility through engagement appropriate for the task –This idea is at the core of arguments for the use of direct manipulation interfaces All of the following are interrelated: –Immersion, engagement, presence, virtual reality –3D display and interaction devices In field of CS and HCI: –“spatial interfaces”, “immersive interfaces”

56. Examples of Immersive Interfaces Tiled display wall Head mounted display Surround screen projection Spherical projection

57. Immersion, “Virtual Reality”, and Virtual Environments Immersive interfaces –High sensory immersion – visual, auditory, haptic, proprioceptive “Virtual reality”, or, virtual environments –“Virtual reality is a technology that is used to generate a simulated environment in digital form… users are immersed in a totally virtual world.” –Working definition – an immersive interactive system In context of “virtual reality”, immersion = spatial immersion –Different literatures use word in different ways Note: “Immersion” (and engagement and presence) is a continuum –Text... Visual and 3d.. Stereo... HMD… “jacked in” –Cyberspace Term coined by Gibson in Neuromancer, 1984 … and in the 21 st century, the Matrix

58. Immersion and Virtual Reality “The mind has a strong desire to believe that the world it perceives is real” – Jaron Lanier, VR “pioneer”, … among others For example, “illusion” (perception) of depth (for spatial immersion) Stereo parallax Head motion parallax Object motion parallax Texture scale Interaction: grab and move an object Proprioceptive cues: –When you reach out and see a hand where you believe your hand should be, you accept the hand as your own Often user will accept what see asn “real” even if graphics poor Constellation of cues

59. optional Components for Perception of 3D for Spatial Immersion, Cutting, 1996 Perception of 3 (vs. 2) dimensions strongest element of spatial immersion –Perception of 3d from depth cues See figure Other elements –Integration important –Visual display types –Stereoscopic display –Head position sensing –Hand-position sensing –Force feedback –Sound input and output –Other sensations

60. Presence “The Aesthetic Impression of 3D Space” Sense of presence –Actually present in the world –Vividly 3d –Sense of being there –Holodeck … –In game literature, often called “immersion”, which is fine, but confusing for us Presence has to do with engagement, as well as visual information –E.g., one can be “in the world”, when reading –Here, one sees, or visualizes, the world

61. Immersive and 3D Interfaces Degree of immersion –conventional desktop screen Some 3D displays: –Data walls –Workbenches –Hemispherical display –Head-mounted displays –Arm-mounted displays –Surround screen displays - CAVE –Virtual retinal display –Autostereoscopic displays But first, … about stereoscopic display –A “compelling” 3D depth cue

62. 3D Interfaces, Stereopsis “Discovery” of Stereopsis, 1838 Charles Wheatstone –Prolific scientist, Wheatstone bridge “… the mind perceives an object of three- dimensions by means of the two dissimilar pictures projected by it on the two retinae…”, 1838 Contributions to the Physiology of Vision.—Part the First. On some remarkable, and hitherto unobserved, Phenomena of Binocular Vision. "Philosophical Transactions" of the Royal Society of London, Vol. 128, pp. 371 – 394, 1838. http://www.stereoscopy.com/library/wheatstone-paper1838.html What the left and right eye see (retinal image)

63. Getting Different Images to Each Eye Wheatstone’s Stereoscope “The stereoscope is represented by figs. 8. and 9; the former being a front view, and the latter a plan of the instrument. A A' are two plane mirrors, about four inches square, inserted in frames, …” Below, modern mirror stereoscope using computer monitors

64. Getting Different Images to Each Eye these days Commodity televisions –120 hz display rate –Alternately, left then right eye image –Active lcd glasses, alternately block left eye, then right eye image lcd lens is alternately transparent, then opague 60 hz left eye, 60 hz right eye Theaters –Polarized glasses

65. Getting Different Images to Each Eye Sutherland’s 1960’s equipment “Ultimate display”, 1965 Sword of Damocles – 1 st HMD –Actual camera-like shutters –Actual camera-like metal shutters

66. Immersive 3D Displays (head mounted displays) Head position is tracked, and display updated for user view –As if “in a virtual (cg) world” Head mounted display, NASA Ames, Fisher et al., mid-1980’s

67. Also, Virtual and Augmented Reality Augmented reality shows real world with an overlay of additional overlay Knowlton (1975) Partially-silvered mirror over keyboard Programmable labels Tactile feedback

68. Also, Virtual and Augmented Reality, 2 Enables users to see real world with an overlay of additional interaction –Situational awareness See through glasses Typically, add text+images to real world Very sensitive to tracking These days, “mobile apps” are appearing

69. Immersive 3D Displays Surround Screen Systems Essentially same multi-surface display and interaction paradigm used today as 1992 Orders of magnitude less hardware and software cost

70. Immersive 3D Displays Surround Screen Systems Space with walls and/or floor formed by rear projection screens –Stereoscopic –Head and position tracking So can change 3D view (projection) as user moves, or moves head Projects 3D scenes for viewer’s point of view on walls –Walls “vanish”, User perceives full 3D scene –View only correct for one viewer Visual immersion –Field of view is 100% possible

71. Immersive 3D Displays Surround Screen Systems Typical size: 10’ x 10’ x 10’ 2 or 3 walls are rear projection screens –To save space, mirrors often used to “fold” projection –Floor projected from above

72. UTPA Immersive Systems Lab ~Summer, 2013 … or so Proj. 27’ 13’ Security area 21’ 6’ TV storage ~8’ CAVE Physiological Measurement Equipment Computers Front Projection Screen Proj. Development

73. Current Immersive Display Projects Information visualization in immersive environments Role of stereoscopy, display size, and narrative on presence Role of personality factors in presence Gallery installation Getting things to work … –Clusters, projectors, software

74. Questions?