Saliency-guided Enhancement for Volume Visualization Youngmin Kim and Amitabh Varshney Department of Computer Science University of Maryland at College Park

2 Motivation The volume datasets have grown in complexity Visible Human Project 13GB ~ 60GB National Library of Medicine (NIH) Richtmyer-Meshkov Instability Simulation 2 TB (= 7.5GB * 273 time steps) Lawrence Livermore National Laboratory Human visual capabilities remain fixed The need to draw visual attention to appropriate regions in their visualization

3 Motivation We can draw viewer attention in several ways Obtrusive methods like arrows or flashing pixels Distracts the viewer from exploring other regions Principles of visual perception used by artists and illustrators Gently guide to regions that they wished to emphasize

4 Contributions A new saliency-based enhancement operator Guides visual attention in volume visualization without sacrificing local context Considers the influence of each voxel at multiple scales Augments the existing visualization pipeline Enhances regional visual saliency Validation by eye-tracking-based user study Our method elicits greater visual attention

5 Related Work - Saliency Computation and Evaluation Computational models for image [Itti et al. PAMI 98] and mesh [Lee et al. SIGGRAPH 05] Evaluation by predicting eye movements [Parkhurst et al. 02], [Privitera and Stark PAMI 00] Use of eye movements Volume composition [Lu et al. EuroVis 06] Abstractions of photographs [DeCarlo and Santella SIGGRAPH 02, NPAR 04] Use of Saliency Progressive visualization [Machiraju et al., 01] Importance-based enhancement [Rheingans and Ebert TVCG 01] Interior and exterior visualization [Viola et al. TVCG 05] Generalizing focus+context [Hauser Dagstuhl 03] Saliency has not been used for guiding visual attention Mesh Saliency

6 Related Work – Transfer Functions Transfer Functions map the physical appearance to the local geometric attributes such as: Gradient magnitude [Levoy CG&A 88] First and second derivatives [Kindlmann and Durkin Volume Rendering 98] Multi-dimensional transfer functions [Kindlmann et al. Vis 03], [Kniss et al. TVCG 02], [Kniss et al. Vis 03], [Machiraju et al. 01] Have played a crucial role in informative Visualization Difficult to emphasize (or deemphasize) regions specified exclusively by locations in a volume

7 Overview Saliency Field Enhancement Operators Emphasis Field Saliency Enhancement Saliency-enhanced Volume Rendering Validation by eye-tracking based user study Transfer Functions Saliency Field by User Input Emphasis Field Computed Enhancement Operators Saliency-enhanced Volume Rendering Validation by eye- tracking device Saliency Enhancement

8 S ( v ) = | G ( C, v, σ ) – G ( C, v, 2σ )| Basic idea from Saliency Computation Saliency map is: Mesh saliency based on curvature values Image saliency based on intensity and color In general, saliency may be defined on a given scalar field C : Mean curvature

9 Emphasis Field Computation Mesh Saliency: S ( v ) = G ( C, v, σ) – G ( C, v, 2 σ) We introduce the concept of an Emphasis Field E to define a Saliency Field S in a volume S ( v ) = G ( E, v, σ ) – G ( E, v, 2σ ) Given a saliency field, can we design some scalar field that will generate it? Known Unknown KnownUnknown

10 Emphasis Field Computation Expressible as simultaneous linear equations Saliency Enhancement Operator ( C -1 ) C E = S, which implies E = C -1 S Given a saliency field S, the enhancement operator C -1 will generate the emphasis field E where c ij is the difference between two Gaussian weights at scale σ and at scale 2σ for a voxel v j from the center voxel v i =

11 Emphasis Field Computation We like to use enhancement operators at multiple scales σ i Let E i be the emphasis field at scale σ i Compute this by applying the enhancement operator C i -1 on the saliency field S Final emphasis field is computed as the summation of E i

12 Emphasis Field in Practice A system of simultaneous linear equations in n variables Generally, can handle arbitrary saliency regions and values Computationally expensive: O(kn 2 ) or O(n 3 ) Alleviate this by solving a 1D system of equations Given a saliency field Solve 1D system of equations at multiple scales and sum them up Approximate results using piecewise polynomial radial functions [Wendland 1995] Interpret results to be along the radial dimension Assume spherical regions of interest (ROI)

13 Visualization Enhancement Emphasis Fields can alter visualization parameters in several ways Various rendering stylizations and effects possible We outline a couple of possibilities Brightness Widely used to elicit visual attention by artists Modulate the Value parameter in the HSV model as follows: –V new (v) = V(v)(1+ E (v)), where –λ - E (v) λ + –Used 0.4 λ and 0.15 λ Saturation Can modulate Saturation instead of Value if the latter is not effective (for instance, in regions already very bright)

14 Gaussian-based vs. Saliency-guided Enhancement Previous Gaussian-based Enhancement of a Volume Volume Illustration [Rheingans and Ebert TVCG 01] Importance-based regional enhancement We use a Gaussian fall-off from the boundary of ROI

15 Visualization Enhancement - Brightness Traditional Volume Rendering Gaussian-based Enhancement Saliency-guided Enhancement Traditional Volume Rendering Gaussian-based Enhancement Saliency-guided Enhancement

16 Visualization Enhancement - Saturation Traditional Volume Rendering Saliency-guided Enhancement Increasing brightness diminishes the appearance of blood vessels at the center of the Sheep Heart model

17 User Study Validated results by an eye-tracking-based user study Hypotheses: The eye fixations increase over the region of interest (ROI) in a volume by the saliency-guided enhancement compared to the traditional volume visualization (Hypothesis H1) the Gaussian-based enhancement (Hypothesis H2)

18 User Study – Experimental Design Eye-tracker and General Settings ISCAN ETL-500 Records eye movements at 60Hz 17-inch LCD monitor With a resolution of 1280x1024 Placed at a distance of 50cm (19.7) from the subjects Eye-tracker Calibration Desired accuracy of 30 pixels Two-step calibration process Standard calibration with 5 points Look and click on 13 points –Triangulation and interpolation with 4 corner points Accuracy test on 16 random points

19 User Study – Experimental Design Extracting fixations from raw points Raw points: all points from the eye-tracker Saccade Removal Velocity > 15°/sec Fixation combining Filter out the points which stay less than 100ms within 15 pixels Average eye locations within 15 pixels and 100ms

20 User Study – Experimental Design Image Ordering 10 users (who passed the accuracy tests) Total of 20 images: 4 models * (1 original + 2 regions * 2 different enhancement methods (Gaussian, Saliency)) Each user saw 12 images out of these 20 images 4 models * (1 original + 2 altered)) Enhanced different regions with different methods Placed similar images far apart to alleviate differential carryover effects Randomized the order of regions and the order of enhancement types (Gaussian and saliency-based) to counterbalance overall effects Duration 12 trials (images), each of which takes 5 seconds

21 User Study – Result I Traditional Volume RenderingTraditional Volume Rendering With Fixation Points Saliency FieldGaussian-based EnhancementGaussian-based Enhancement With Fixation Points Saliency-guided Enhancement With Fixation Points Saliency-guided Enhancement

22 User Study – Result II Traditional Volume RenderingTraditional Volume Rendering With Fixation Points Saliency FieldGaussian-based EnhancementGaussian-based Enhancement With Fixation Points Saliency-guided Enhancement With Fixation Points Saliency-guided Enhancement

23 Data Analysis I The percentage of fixations on the ROI for the original, Gaussian- enhanced, and Saliency-enhanced visualizations

24 Data Analysis II A two-way ANOVA on the percentage of fixations for two conditions, regions and enhancement methods for each volume For regions, no statistically significant results as expected F(1,34) = ~ , p > 0.05 For enhancement methods, statistically significant results F(2,34) = ~ , p 0.01

25 Data Analysis III Carried out a pairwise t-test on the percentage of fixations before and after we applied enhancement techniques for each model Found a statistically significant difference in the percentage of fixations with saliency-guided enhancement for all the models H1 H2 H1 H2 Hypothesis H1: More fixations than the traditional Hypothesis H2: More fixations than the Gaussian

26 Conclusions Introduced the concept of the Emphasis Field for selective visual emphasis (or de-emphasis) Developed the computational framework to generate the Emphasis Field from a given Saliency Field Illustrated the use of the Emphasis Field in Visualization Validated its ability to successfully guide visual attention to desired regions Saliency-guided Enhancement provides a powerful tool to help scientists, engineers, and medical researchers explore large visual datasets

27 Future Work Measure comprehensibility of the volume rendered images Explore other appearance attributes such as opacity and texture detail Generalize to handle time-varying datasets with multiple superposed scalar and vector fields Identify the relative importance of various scales

28 Acknowledgments Datasets: Stefan Roettger (University of Erlangen) and Dirk Bartz (University of Tuebingen) Discussions: David Jacobs, François Guimbretière, Derek Juba, and Robert Patro (University of Maryland) Eye-tracker: François Guimbretière The Anonymous Referees Supported by NSF grants: CCF , CCF , CNS , and IIS

29 Questions ?? Supplemental material in the DVD-ROM Lab: Project: Images: