Evaluation Methods - Summary

How to chose a method? Stage of study – formative, iterative, summative Pros & cons Metrics – depends on what you want to measure Qualitative vs. quantitative Research perspective – CS vs. psychology vs. sociology

Pros & Cons Realism Precision Generalizability Time & cost Researcher expertise

Methods Survey Interview Controlled-lab experiment In-lab observation Controlled field experiment Field observation study Heuristic Evaluation Cognitive Walkthrough Experimental simulation GOMS Computer simulation Formal theory

Survey Pros – Easy to get a large number of responses. – Quick and easy to conduct. – Highly generalizable. Cons – Self-selection. – Participants often only offer enough information to answer the question. – Can miss details. – Low in realism and precision.

Interviews Pros – Quick and easy to conduct. – Gives designer quick feedback on a range of ideas. – Can get a person’s initial reaction to an idea. – Can get detailed information from a person. Cons – Often takes place away from natural setting. – Question wording or interviewer “body language” can bias answers. – High probability of false positives and missed problems (e.g., users may not have a clear idea of how an app will be used). – Can miss details if interviewer doesn’t know what issues to draw out.

Controlled Lab Experiment Pros – Provides precise, quantifiable data. – Easier to draw inferences from data. – Relatively quick. – Can get a medium-sized number of participants. Cons – Short duration of a lab experiment may not be enough to allow users to become accustomed to an app. – Not a natural setting – interaction may not be normal.

In-lab Observation Pros – Relatively quick. – Can get a medium-sized number of participants. Cons – Observations are subjective and error prone. – Short duration of lab observation is not enough time for user to get accustomed to using the interface. – Not a natural setting – interaction may not be normal.

Controlled Field Experiment Pros – Less intrusive than most other evaluation methods. – Provides more precise data than field observation. – Can observe natural behavior of user (though some part of the system will be controlled/unnatural). Cons – More intrusive than field observation. – Less natural than field observation.

Field Observation Study Pros – Only way to observe natural behavior of user & interaction between user & tools. Cons – Difficult and time consuming. – Hard to get permission to observe people. – Observations are subjective and error prone. – Cannot make strong interpretations from observations. – Not very generalizable.

Heuristic Evaluation Pros – Quick and easy. Cons – Nielson’s heuristics may not be as relevant to non- GUIs. – Results in false positives in missed problems, especially when experts are not part of target audience.

Cognitive Walkthrough Pros – Quick and easy. Cons – Results in false positives and missed problems when evaluator is different from target audience.

Experimental Simulation Pros – Still fairly precise. – More realistic than in-lab experiment. Cons (same as lab exp.) – Short duration of a lab experiment may not be enough to allow users to become accustomed to an app. – Not a natural setting – interaction may not be normal.

GOMS Pros – Predict human performance before committing to a specific design in code or running user studies – Many studies have validated the model (it works) Cons – Assumes error-free, skilled user behavior – No formal recipe for how to perform analysis – Significant time investment

Computer Simulation Creating a complete & closed system that models the operation of the concrete system without users. Example: – geophysical process going on in connection with the eruption of Mount St. Helens – Simulation of pilot during plane crash

Computer Simulation Pros – Supposedly high in realism (depends on accuracy of data/system replication) Cons – Low in precision & generalizability

Formal Theory– This is a dissertation Formulating general relations (propositions, hypothesis, or postulates) among a number of variables of interest. Pros – Relatively generalizable Cons – Not realistic or precise

How to chose a method? Stage of study Pros & cons – Realism – Precision – Generalizability – Time & cost Researcher expertise Metrics Qualitative vs. quantitative Research perspective

Methods Survey Interview Controlled-lab experiment In-lab observation Controlled field experiment Field observation study Heuristic Evaluation Cognitive Walkthrough Automated observation user study Experimental simulation GOMS Computer simulation Formal theory

Early Stage – these are the best Survey Interview Controlled-lab experiment In-lab observation Controlled field experiment Field observation study Heuristic Evaluation Cognitive Walkthrough Automated observation user study Experimental simulation GOMS Computer simulation Formal theory

Iterative & Summative Stages – these are the best Survey Interview Controlled-lab experiment In-lab observation Controlled field experiment Field observation study Heuristic Evaluation Cognitive Walkthrough Automated observation user study Experimental simulation GOMS Computer simulation Formal theory

Realism – these are the best Survey Interview Controlled-lab experiment In-lab observation Controlled field experiment Field observation study Heuristic Evaluation Cognitive Walkthrough Automated observation user study Experimental simulation GOMS Computer simulation Formal theory

Precision – these are the best Survey Interview Controlled-lab experiment In-lab observation Controlled field experiment Field observation study Heuristic Evaluation Cognitive Walkthrough Automated observation user study Experimental simulation GOMS Computer simulation Formal theory

Generalizability – these are the best Survey Interview Controlled-lab experiment In-lab observation Controlled field experiment Field observation study Heuristic Evaluation Cognitive Walkthrough Automated observation user study Experimental simulation GOMS Computer simulation Formal theory

Time & Cost – these are the best Survey Interview Controlled-lab experiment In-lab observation Controlled field experiment Field observation study Heuristic Evaluation Cognitive Walkthrough Automated observation user study Experimental simulation GOMS Computer simulation Formal theory

Researcher Perspective – Whoever is doing will choose whatever they want Survey Interview Controlled-lab experiment In-lab observation Controlled field experiment Field observation study Heuristic Evaluation Cognitive Walkthrough Automated observation user study Experimental simulation GOMS Computer simulation Formal theory

Metrics: examples Traditional GUIs: – efficiency (time to complete task) – accuracy (# of errors) – simplicity Peripheral Displays: – awareness (recall) – distraction (dual-task behavior) – aesthetics