LInfoVis Winter 2011 Chris Culy Evaluation of visualizations

LInfoVis Winter 2011 Chris Culy Levels: perception and interpetation Visual/perceptual how well can people perceive the distinctions the visualization intends? Interpretation how well can people interpret the visualization?

LInfoVis Winter 2011 Chris Culy Levels: Use Use in isolation how accurately can people use the visualization for a particular task in isolation? how quickly can people use the visualization for a particular task in isolation? Use as part of a broader goal how accurately can people use the visualization for a particular task as part of a broader goal? how quickly can people use the visualization for a particular task as part of a broader goal?

LInfoVis Winter 2011 Chris Culy Levels: Satisfaction how satisfied are people with the visualization? e.g. easy/hard, "cool", etc. ≠ how well they can use it how useful is the visualization for what they want to do? how well do they "get it"? people may use/prefer a more difficult tool if it's "cool" if they already know it – there's a learning curve for a new tool if it's cheaper

LInfoVis Winter 2011 Chris Culy Goals Formative evaluation Goal is to get specific information to improve the software Done during the development cycle Often informal Summative evaluation Goal is to get general information about how the software performs Done at the end of (a) development cycle Often (more) formal Can also be used to improve the next iteration

LInfoVis Winter 2011 Chris Culy Some principles for experiments There should be a specific purpose for the experiment What are you evaluating? Experiments should be task based The user should be introduced to the software, especially something new The user should not be “interfered with” during the experiment

LInfoVis Winter 2011 Chris Culy Some techniques for experiments "instrumenting" the program tracking clicks, etc, and then trying to analyse the patterns also for tracking speed, accuracy

LInfoVis Winter 2011 Chris Culy Some techniques for experiments "instrumenting" the user various means for perception eye tracking shows where people are paying attention

LInfoVis Winter 2011 Chris Culy Some techniques for experiments Observation with note taking, timing of the user's actions User feedback pre/post session questionnaire "think aloud protocol": user explains what they're doing and why as they're doing it explicit questions during demo (informal) post session interview (“debriefing”)

LInfoVis Winter 2011 Chris Culy Formal vs. informal Formal Several subjects (6+) Experimental design (comparison) Often statistical analysis of results Experiment usually recorded Informal, "guerilla" testing Few subjects (3-4) Quick, informal test to get user feedback on s.t.

LInfoVis Winter 2011 Chris Culy Techniques for deployed software "Bug" reports When is a bug not a bug? Feature requests Interviews (or narratives) with "customers" Case studies (positive = “success stories”) Observation of "customers" using the software

LInfoVis Winter 2011 Chris Culy Interpreting users' reactions Users can only react in terms of what they know/assume. They don't analyse what they do, so their suggestions tend to be too concrete. They often make specific suggestions that aren't necessarily the best solutions, since they don't know what the possibilities are They are honestly trying to be helpful