1. Evaluation Methodologies
CS160 Discussion Section
Evaluation Methodologies
April

2. Evaluation Methodologies
Evaluation through expert analysis
Cognitive walkthrough
Heuristic evaluation
Model-based evaluation (GOMS)‏
Evaluation through user participation
Lab studies
Field studies

3. Choosing Evaluation Method
Comparative evaluation of Evaluation Methods is hard because of the many uncontrolled variables.
Factors distinguishing evaluation techniques:
Stage/cycle at which the evaluation is carried out.
Style of evaluation
Level of subjectivity or objectivity of the technique
Type of measures provided
The information provided
The immediacy of the response
The level of interference implied
The resources required

4. Design vs. Implementation
Evaluation at Design
Need to be quick and cheap
Design experts only
Analytic
Evaluation of Implementation
A physical artifact exist, in the form of something ranging from a paper mock-up to a full implementation
More comprehensive
Bring in users as participants
Participatory design involves users throughout the design process
Cognitive walkthrough are expert based and analytic but can be used to evaluation implementation as well as design

5. Laboratory vs. Field Studies
Laboratory studies
Pro: allow controlled experimentation and observation
Con: losing something of the naturalness of the user's environment.
Dominate early states of design.
Field studies:
Pro: retains authenticity of the user's environment
Con: allows less control over user activity (i.e., Introducing confounding factors).
Conducted after a concrete implementation

6. Subjective vs. Objective
Subjective techniques
Rely heavily on the interpretation of the evaluator, e.g., cognitive walkthrough or think aloud; interviews/questionnaires inherently subjective.
Pro: powerful if used correctly, providing information that may not be available from more objective methods.
Con: Evaluator bias
Increasing the number of evaluators reduces the risk of bias.

7. Subjective vs. Objective
Object techniques
Provide similar information for anyone correctly carrying out the procedure (repeatable result), eg., controlled experiments.
Pro: repeatable results, not dependent on the persuasion of the particular evaluator; avoid bias and provide comparable results.
Con: may not give detailed feedback on user experience.

8. Qualitative vs. Quantitative Measures
Non-numerical, more difficult to analyze
Can provide important detail that cannot be quantified.
Subjective techniques tend to be qualitative.
Qualitative information can be mapped onto a scale: questionnaires seek for qualitative information but answers are represented quantitatively.
Quantitative
Numeric, easily analyzable using statistical techniques.
Objective techniques tend to be quantitative.

9. Information provided Low-level High-level
Allowing design decisions to be made (which font or color to use)‏
Controlled experiment: an experiment to measure a particular aspect of the interface
High-level
User's general impression of the system.
Answer questions such as “Is the system usable?”
Questionnaire, interview techniques.

10. Immediacy of response and Intrusiveness
When is user's response recorded?
During interaction, eg., think aloud.
More intrusive: process of measurement can actually alter the way user works.
Sensitivity on the part of the evaluator can help to reduce this.
Automatic logging is an exception, but the amount of useful information is limited.
Post interaction, e.g., interview, post task walkthrough
Recollection is liable to suffer from bias in recall and reconstruction

11. Resources
Resources to consider: equipment, time, money, participants, expertise of evaluator and context.
Often have to pick the tactic that will produce the most effective and useful information for the system under consideration.
Some techniques are more reliant on evaluator expertise than others, e.g., formal analytic techniques.
Heuristic evaluation requires less understanding of user goal structures etc.
Sometimes it's not possible to gain access to the intended users or it's not feasible to test the system in its intended environment.

12. Summary Evaluation is integral to the design process
Evaluation should take place throughout the design life cycle.
Early evaluation helps to minimize the cost of early design errors. Most techniques used at this stage are analytic and involve experts.
Once a system has been developed (prototype or full), experimental and observational techniques can be used to get both quantitative and qualitative information.

13. Q&A
Poster session
Implementation

14. Administrivia
Notify us by tomorrow if you intend to participate in the competition.
Section will be canceled next week.
Make available your project implementation online.