Presentation is loading. Please wait.

Presentation is loading. Please wait.

A 2D Flow Visualization User Study Using Explicit Flow Synthesis and Implicit Task Design VisWeek 2011 Zhanping Liu Shangshu Cai J. Edward Swan II Robert.

Similar presentations


Presentation on theme: "A 2D Flow Visualization User Study Using Explicit Flow Synthesis and Implicit Task Design VisWeek 2011 Zhanping Liu Shangshu Cai J. Edward Swan II Robert."— Presentation transcript:

1

2 A 2D Flow Visualization User Study Using Explicit Flow Synthesis and Implicit Task Design VisWeek 2011 Zhanping Liu Shangshu Cai J. Edward Swan II Robert J. Moorhead II Joel P. Martin T. J. Jankun-Kelly University of Pennsylvania Kentucky State University University of California at Santa Barbara Mississippi State University Mississippi State University Lockheed Martin Corp. Army Research Lab Mississippi State University IEEE TVCG

3 Outline Explicit Flow Synthesis Diverse Evaluation Aspects Implicit Task Design VIS 2011 Experimental Components Synthetic Flow Datasets Flow Visualization Techniques Flow Analysis Tasks Brief Introduction Test Results Test Strategy Concluding Remarks

4 Flow Representation Geometry-based / glyph-based Texture-based / image-based arrow plots, streamlines, pathlines, streak lines, time lines stream ribbons, stream tubes, stream surfaces, streak surfaces, …… graphical primitives rendered for a sparse or discrete representation good survey by McLoughlin et al (EuroGraphics 09) topology-based methods use graphical primitives for a representation spot noise, LIC, UFLIC, LEA, IBFV, IBFVS, ISA, UFAC, …… texture convolution / advection for a dense continuous representation good survey by Laramee et al (Computer Graphics Forum 04) VIS 2011 Brief Introduction

5 Flow Visualization User Study NIH-NSF report on V isualization R esearch C hallenges ( J ohnson etc 06) different techniques may be advantageous in different aspects only a few have been evaluated to determine their effectiveness the best methods might not have been integrated into vis. systems domain scientists may not yet have access to cutting-edge techniques insufficient user feedback for visualization researchers and developers more user studies are needed to examine flow representations improve existing techniques design innovative techniques VIS 2011 bridge the long-lasting gaps between research, development, and deployment Brief Introduction

6 Flow Visualization User Study VIS 2011 Previous work 2D flow visualization user study (Laidlaw et al, TVCG 05) 3D flow visualization user study (Forsberg et al, TVCG 09) …… insufficient research on effective user study methodologies Brief Introduction

7 Flow Visualization User Study VIS 2011 given a user study framework or platform for evaluating flow visualization techniques distorted by various bias issues the data collected and the analysis results are distorted too, failing to provide objective conclusions flow visualization techniques Previous work Brief Introduction

8 Flow Visualization User Study Previous work 2D flow visualization user study (Laidlaw et al, TVCG 05) 3D flow visualization user study (Forsberg et al, TVCG 09) …… insufficient research on effective user study methodologies VIS 2011 There is more to a flow visualization user study than the scenarios being considered the techniques being evaluated the flow features being examined the specific yet usually ad-hoc conclusions being drawn Brief Introduction e.g., surface flows, volume flows, time-varying flows, …… e.g., UFLIC, LEA, IBFV, IBFVS, ISA, UFAC, …… e.g., separation, attachment, vortex core, periodic orbit, ……

9 Flow Visualization User Study Conducting objective 2D flow visualization user studies even with traditional and well-known techniques remains an open problem requires valid methodologies an anti-bias platform refines our understanding of some 2D flow vis. techniques offers quantitative support for qualitative evidence or anecdotal advice in terms of the effectiveness of flow vis. techniques VIS 2011 Brief Introduction that is necessary for carrying out convincing flow visualization user studies with more complex configurations helps formulate a general framework

10 VIS 2011 Brief Introduction Our 2D Flow Visualization User Study FlowVUS motivated by the necessity for and significance of effective flow visualization user study methodologies builds on Laidlaw et als work features new strategies and important improvements explicit flow synthesis implicit task design flow data bias task design bias

11 VIS 2011 Brief Introduction Our 2D Flow Visualization User Study FlowVUS given a user study framework or platform for evaluating flow visualization techniques equipped with anti-bias methodologies the data collected and the analysis results are convincing, leading to a better understanding of techniques flow visualization techniques

12 VIS 2011 Brief Introduction Our 2D Flow Visualization User Study FlowVUS Major contributions explicit flow synthesis combats data-related bias by automatically generating many flows with similar topological complexities but with different structures implicit task design reduces task-related bias by designing sample-free pattern-based flow analysis tasks that require thorough investigation of the flow direction diverse evaluation perspectives involve representation continuity, visual intuition, image contrast, and color mapping when selecting a set of representative vis. techniques hybrid timing strategy uses two timing schemes (fixed duration / variable duration) to help reveal subtle differences in vis. effectiveness between techniques refined statistical analysis processes outliers + R yan REGWQ post-hoc homogeneous subset tests

13 VIS 2011 Experimental Components Synthetic Flow Datasets Flow Visualization Techniques 2D Flow Visualization User Study Pipeline Flow Analysis Tasks Synthetic Flow Datasets Flow Visualization Techniques Flow Analysis Tasks three fundamental components of a typical flow vis. user study

14 VIS 2011 Experimental Components Synthetic Flow Datasets Previous work on flow vis. user study uses implicit flow synthesis samples randomly selected and the associated vectors randomly assigned a flow field is generated by vector interpolation between the samples the topology of the resulting flow is unpredictable number of critical points, the locations, the types & overall complexity A single dataset would introduce learning effect unacceptable Synthetic datasets are used for user study in medical imaging Multiple datasets may incur data-dependent bias (in flow complexity) Data-dependent bias can be suppressed to an acceptable degree by synthesizing flows with similar topological complexities implicit flow synthesis

15 VIS 2011 Experimental Components Synthetic Flow Datasets employs parameterized placement and configuration of critical points provides great flexibility and control in creating pseudo flow fields B asis V ector F ield (BVF) flow synthesis method by van Wijk (TOG 02) a BVF is governed by a critical point with some parameters the entire flow results from the combination of multiple BVFs a survey and initialization-analysis-editing by Zhang et al (TOG 06) Explicit Flow Synthesis FlowVUS BVF FlowVUS is the first user study to value and apply explicit flow synthesis based on BVF for fast automatic generation of many synthetic flows centers and foci Explicitly Specified Critical Points (ESCPs) saddles derived from the interaction among centers and foci uses a force composition-attenuation method to govern the influence of an ESCP ( with rotational force and radial force ) or a BVF on an arbitrary point

16 VIS 2011 Experimental Components Synthetic Flow Datasets

17 VIS 2011 Experimental Components Synthetic Flow Datasets Layout templates to synthesize flows with diverse structures yet with a relatively balanced layout of a fixed number of ESCP s + a slightly varying number of saddles to maintain nearly the same topological complexity between many flows a primary ESCP is randomly placed & configured in each blue block and its mirror ESCP is placed based on a symmetry type yet with the sink / source type & clockwise/counter-clockwise orientation possibly different they may be geometrically symmetric but topologically asymmetric location radial force rotational force ESCP parameters force attenuation sink / source type clockwise / counter-clockwise orientation to generate x- / y- / center-symmetric and dubiously asymmetric flows so as to support our pattern-based implicit flow analysis task design 4 pairs of x-symmetric ESCP placement blocks 4 pairs of y-symmetric ESCP placement blocks 4 pairs of center-symmetric ESCP placement blocks blue block: for primary ESCP placement; gray block: for mirror ESCP placement

18 VIS 2011 Experimental Components Synthetic Flow Datasets symmetric flows versus asymmetric flows asymmetric x-symmetric asymmetric center-symmetric asymmetric y-symmetric

19 VIS 2011 Experimental Components Flow Visualization Techniques direction the positive and negative directions tangent to the flow orientation the positive direction of the flow only (e.g., oriented LIC ) velocity magnitude a scalar quantity Primitive flow characteristics The most important a vector quantity providing the fundamental info that distinguishes a flow field from a scalar field and hence governs why / how flow visualization differs very much from scalar visualization in the working mechanism how well a flow vis. technique delineates the general, directional info largely determines its effectiveness in conveying specific flow features An informal classification Direct Feature-Extraction Based (DFEB) e.g., topology extraction Indirect User-Exploration Based ( IUEB) e.g., flow lines and LIC many flow features ( e.g., critical points ) visually recognizable from them direction from the flow reconstruction or visual analysis perspective our focus

20 VIS 2011 Experimental Components Flow Visualization Techniques need more user studies than DFEB techniques do due to the human factors user exploration visual analysis mental reconstruction IUEB techniques 54 candidates 3 families hedgehogs streamlines LIC selected through a thorough intra- and inter-family investigation representative of many geometry-based and texture-based techniques in terms of the aforementioned four major visual / evaluation aspects configured via iterative internal tests for optimal visualization results 7 techniques involve several major visual factors representation continuity (e.g., 0D / 1.5D / 2D) visual intuition image contrast color mapping FlowVUS evaluation aspects

21 VIS 2011 Experimental Components Flow Visualization Techniques ArrowCMArrowCWStreamCM StreamCWBasicLICEnhancedLICOrientedLIC

22 VIS 2011 Experimental Components Flow Analysis Tasks impossible & unnecessary to enumerate specific / complex flow features and then design many flow analysis tasks (how many studies are enough?) Some essential points in order to reduce task-related bias, flow analysis tasks may take an indirect / implicit way and a testable form the performance of an average participant in visual flow analysis is expected to reflect the effectiveness of the IUEB technique (being used) in conveying the flow direction the general fundamental information flow analysis tasks in a user study are not necessarily real or practical flow analysis tasks are the way instead of (or at least more than) the goal for example, synthetic tasks are often used for psychological user studies by devising some seemingly irrelevant yet intrinsically coupled questions ( do not directly ask the user to check the flow direction at a point) ( questions are easy to understand but challenging to answer correctly)

23 VIS 2011 Experimental Components Flow Analysis Tasks used in previous work and susceptible to bias a typical example directly ask to check the flow direction at a point the participant is shown a randomly placed circle (of which the center is hence a random sample) and asked to click on the point along the circle that a particle advected from the center is to hit Explicit sample-based tasks a methodology advocated and formulated in this paper to suppress bias use a simple form but indirectly require thorough investigation of the flow Implicit pattern-based tasks mouse pointing & clicking, irrelevant of judgment, affect the test result the complexity of a flow usually varies with the location more difficult to do this task in turbulent areas than in laminar areas the selection of the circles radius may further compound this issue critical point recognition detect patterns globally/across the whole domain critical point classification match patterns locally/around an area of interest

24 VIS 2011 Experimental Components Flow Analysis Tasks Implicit task design to relieve non-expert participants from understanding complex, possibly domain-specific details in the form of easy-to-understand yet difficult-to-answer questions requiring intensive analysis of flow directions using specific real tasks about well-known flow features critical point recognition (CPR) critical point classification (CPC) involving in-depth flow structures identification of separatrices identification of periodic orbits creating general synthetic tasks to reduce data-related bias resulting from flow sampling and mouse point-and-click operations such as symmetric pattern categorization (SPC) to examine the flow direction both globally and locally to check the entire pattern: x-/y-/z-/center-symmetric or asymmetric

25 VIS 2011 Experimental Components Flow Analysis Tasks Very challenging synthetic tasks two or three critical points (centers, foci, and saddles) combined with a variety of configurations to define some Composite Templates (CTs) CT-based CPR-like pattern recognition CT-based CPC-like pattern classification checking if flow A and flow B have a CT pattern in common judging if flow A is a rotational version of flow B determining if flow A is exactly part of flow B The selected implicit tasks CPR + CPC + SPC integration of 2 real tasks and 1 synthetic task to demonstrate the types the balance between the overall challenge degree and the test duration some synthetic tasks mentioned above would require more test time

26 VIS 2011 The Input Test Strategy 7M images generated using the selected 7 techniques to visualize M synthetic flows involving N x-symmetric, N y-symmetric, N center-symmetric, and optionally N asymmetric flows M = 3N or 4N (e.g., N = 30) depending on the expected complexity and time duration of the test Ground truth one record per synthetic flow symmetry type of the overall pattern the location and type of every ESCP ( center / focus ) from the synthesizer the location of every derived saddle from N ewton- R aphson root-finding Task Session 1 CPR task (recognizing ALL critical points from an image) or <= 30 CPC tasks or <= 30 (without asymmetric flows) / 40 (with asymmetric flows) SPC tasks

27 VIS 2011 Task Management Test Strategy 1 set = (1 CPR session + 1 CPC session + 1 SPC session) for one technique 1 cycle = 7 sets (one for each technique) 1 test = 3 cycles 1 session = 1 CPR task or (<= 30) CPC tasks or (<= 30/40) SPC tasks = 21 sets = 63 sessions for each participant use 7 techniques thrice to produce 7 × 3 = 21 images (for 21 randomly- selected flows), with 1 image for each CPC session (3 per technique) use each technique to produce 30 images (for 30 randomly-selected flows), with 1 randomly-selected critical point marked per image (with 10 marked for each critical point type: center, focus, saddle), for each CPC session use each technique to visualize 30 or 40 randomly-selected flows (creating 10 images for each symmetry / asymmetric type) for each SPC session 21 CPR sessions + 21 CPC sessions + 21 SPC sessions = 63 sessions with a bank of images pre-generated for one time, 63 sessions are created using TestGen upon each test and are then delivered in random order

28 VIS 2011 Hybrid Timing Test Strategy Effectiveness metrics the effectiveness of a visualization technique is usually reflected by answer correctness and response time a more effective technique allows the user to get a correct answer faster given a fixed amount of time, more correct answers tend to result from a more effective technique than from a less effective technique Variable-duration session mouse click positions and response time are recorded for a session flow analysis (for recognizing a single critical point) is relatively quick the answer is precision-critical (despite a considerable error tolerance) seeks to curb the participant from hastiness and excessive inaccuracy Fixed-duration session as many tasks as possible are presented to the participant one by one in a fixed amount of time (30s) and radio-button choices are recorded flow analysis is relatively slow and judgment-intensive intended to push the participant to accomplish more tasks for CPR for CPC & SPC (response time on average) this hybrid timing strategy helps reveal the subtle differences in visualization effectiveness that may exist between techniques

29 VIS 2011 Test Strategy CPR Critical Point Recognition

30 VIS 2011 Test Strategy CPC Critical Point Classification

31 VIS 2011 Test Strategy SPC Symmetric Pattern Categorization

32 VIS 2011 Test Results Basic Facts 4 CFD experts + 16 graduate students in science & engineering expert and non-expert participants were not compared herein 5079 CPR trials CPC trials SPR trials were recorded Processing Outliers the response time and the (CPR) location error each showed a skewed normal distribution in terms of the histogram outliers were determined case by case by investigating the tails of the distributions and noting values after conspicuous gaps each outlier was replaced with the median of the cells responses the absolute differences in response time for CPR / CPC / SPC turned out to be small, regardless of the statistical differences a higher priority assigned to correctness than to response speed to provide correctness-over-response-sorting (CORS) when evaluating the seven techniques in the overall visualization effectiveness

33 VIS 2011 Test Results Statistical Analysis C hi-square tests and ANOVA (univariate analysis of variance ) calculating post-hoc homogeneous subsets using R yan REGWQ tests FlowVUS Results CPR (Critical Point Recognition) response time mean time (in seconds) to recognize a critical point (5079 trials, F(6,115.3) = 19.9, p < 0.001) means with the same letter are not significantly different at p 0.05 (Ryan REGWQ post- hoc hst)

34 VIS 2011 Test Results FlowVUS Results CPR (Critical Point Recognition) answer incorrectness CORS sorting by CPR effectiveness in decreasing order EnhancedLIC - StreamCM - BasicLIC - OrientedLIC - StreamCW - ArrowCM - ArrowCW 336 errors, χ 2 (6) = 132, p < 0.001

35 VIS 2011 Test Results FlowVUS Results CPC (Critical Point Classification) response time mean time (in seconds) to classify a critical point (7467 trials, F(6,116.2) = 30.9, p < 0.001) means with the same letter are not significantly different at p 0.05 (Ryan REGWQ post- hoc hst)

36 VIS 2011 Test Results FlowVUS Results CPC (Critical Point Classification) answer incorrectness CORS sorting by CPC effectiveness in decreasing order EnhancedLIC - StreamCW - StreamCM - BasicLIC - OrientedLIC - ArrowCW - ArrowCM 753 errors, χ 2 (6) = 772, p < 0.001

37 VIS 2011 Test Results FlowVUS Results SPC (Symmetric Pattern Categorization) response time mean time (in sec.s) to categorize a symmetric pattern (4948 trials, F(6,123.1) = 8.74, p < 0.001) means with the same letter are not significantly different at p 0.05 (Ryan REGWQ post- hoc hst)

38 VIS 2011 Test Results FlowVUS Results SPC (Symmetric Pattern Categorization) answer incorrectness CORS sorting by SPC effectiveness in decreasing order EnhancedLIC - StreamCM - BasicLIC - OrientedLIC - StreamCW – ArrowCM - ArrowCW 323 errors, χ 2 (6) = 70.1, p < 0.001

39 VIS 2011 Test Results CORS Sorting by CPR effectiveness CORS Sorting by CPC effectiveness CORS Sorting by SPC effectiveness EnhancedLIC StreamCMStreamCWStreamCM BasicLICStreamCMBasicLIC OrientedLICBasicLICOrientedLIC StreamCWOrientedLICStreamCW ArrowCMArrowCWArrowCM ArrowCWArrowCMArrowCW a texture-based dense representation with accentuated flow streaks (EnhancedLIC) enables intuitive perception of the flow a geometry-based integral representation with uniform density control (StreamCM or StreamCW) exploits visual interpolation to facilitate mental reconstruction of the flow color mapping has a considerable influence on a geometry-based flow representation

40 VIS 2011 Concluding Remarks Key Points Explicit flow synthesis Implicit task design Diverse evaluation perspectives Hybrid timing strategy Refined statistical analysis to reduce data-related bias template-based parameterized placement & configuration of critical points automatic synthesis of diverse flows with similar topological complexities to suppress task-related bias pattern-based (real tasks + synthetic tasks) the way more than the goal representative techniques representation continuity visual intuition image contrast color mapping variable-duration session fixed-duration session to reveal the subtle differences in vis. effectiveness between techniques processes outliers + Ryan REGWQ post-hoc homogeneous subset tests to reduce data-related bias to suppress task-related bias Explicit flow synthesis Implicit task design Two important methodologies / concepts proposed as part of our anti-bias framework for conducting objective flow vis. user studies

41 VIS 2011 Concluding Remarks Limitations & Lessons FlowVUS is bias-resistant but not bias-free Varying a-priori familiarity with techniques Varying a-priori familiarity with flow features Real flows needed for introducing techniques Care needed for predicting the time duration bias is pervasive throughout the whole pipeline of a user study and hence we cannot totally eliminate it while we need to reduce it cannot let it be some participants were not familiar with the LIC s upon the training session more user studies are needed to disseminate the latest vis. techniques care needs to be taken when evaluating more sophisticated / current ones some participants needed extra help with some features during the training session many challenges facing an evaluation involving more complex features synthetic flows are needed for formal tests while real flows (particularly with contextual boundaries) are needed, besides real flows, for the training session

42 VIS 2011 Concluding Remarks Future Plans Anti-bias methodologies user studies might otherwise be non-convincing & even worse misleading probably one way to help you judge between 2 contradicting conclusions as of now more important than scenarios, techniques, features, conclusions require much research (e.g., explicit flow synthesis & implicit task design ) end users might not care about the underlying working mechanism they are interested in the resulting images and the associated visual aspects (such as image contrast, color map, intuition, continuity, etc) neither possible nor necessary to evaluate every existing vis. technique Evaluation aspects representative visualization techniques provide general guidelines for visualization research ( algorithm design ) Interesting topics user studies on streamline placement algorithms user studies on surface flow visualization techniques user studies on volume flow visualization techniques to adopt the conclusions of a user study without necessary anti-bias methods? controversial view controversial view

43 Thank you for your time and attention!


Download ppt "A 2D Flow Visualization User Study Using Explicit Flow Synthesis and Implicit Task Design VisWeek 2011 Zhanping Liu Shangshu Cai J. Edward Swan II Robert."

Similar presentations


Ads by Google