1. 1 Human-Computer Interaction Usability Evaluation: 2 Expert and Empirical Methods

2. 2 Lecture Overview Expert evaluationExpert evaluation Empirical evaluationEmpirical evaluation ObservationalObservational Survey evaluationSurvey evaluation Experimental evaluationExperimental evaluation

3. 3 Expert Evaluation Strongly diagnostic Strongly diagnostic Overview of whole interface Overview of whole interface Few resources needed (except £ for experts) Few resources needed (except £ for experts) Cheap Cheap High potential return - detects significant problems High potential return - detects significant problems Restrictions in role playing Restrictions in role playing Subject to bias Subject to bias Problems locating experts Problems locating experts Cannot capture real user behaviour Cannot capture real user behaviour Advantages Disadvantages

4. 4 Observational Evaluation Quickly highlights difficulties Quickly highlights difficulties Verbal protocols valuable source of information Verbal protocols valuable source of information Can be used for rapid iterative development Can be used for rapid iterative development Rich qualitative data Rich qualitative data Observation can affect user activity and performance levels. (Hawthorne Effect) Observation can affect user activity and performance levels. (Hawthorne Effect) Analysis of data can be time-consuming and resource- consuming Analysis of data can be time-consuming and resource- consuming Advantages Disadvantages

5. 5 Observational Evaluation (Cont’d) - Cooperative Evaluation (Monk et al., 1992) Intended for use by non-HF expertsIntended for use by non-HF experts User as collaborator in evaluation, not simply as subjectUser as collaborator in evaluation, not simply as subject User begins by thinking aloudUser begins by thinking aloud Evaluator can answer and ask questionsEvaluator can answer and ask questions Evaluation session produces a protocolEvaluation session produces a protocol Protocol transcription and analysis can be time- consumingProtocol transcription and analysis can be time- consuming

6. 6 Survey Evaluation Addresses users’ opinions and understanding of interface. Addresses users’ opinions and understanding of interface. Can be made to be diagnostic Can be made to be diagnostic Can be applied to users and designers Can be applied to users and designers Questions can be tailored to the individual Questions can be tailored to the individual Rating scales lead to quantitative results Rating scales lead to quantitative results Can be used on a large group of users Can be used on a large group of users User experience is important User experience is important Low response rates (especially by post) Low response rates (especially by post) Possible interviewer bias Possible interviewer bias Possible response bias Possible response bias Analysis can be complicated and lengthy Analysis can be complicated and lengthy Interviews very time- consuming Interviews very time- consuming Advantages Disadvantages Interviews Questionnaires

7. 7 Experimental Evaluation Powerful method (dependent on the effects investigated) Powerful method (dependent on the effects investigated) Quantitative data for statistical analysis Quantitative data for statistical analysis Can compare different groups of users Can compare different groups of users Reliability and validity good Reliability and validity good Replicable Replicable High resource demands High resource demands Requires knowledge of experimental method Requires knowledge of experimental method Time spent on experiments can mean evaluation is difficult to integrate into design cycle Time spent on experiments can mean evaluation is difficult to integrate into design cycle Tasks can be artificial and restricted Tasks can be artificial and restricted Cannot always generalize to full system in typical working situation Cannot always generalize to full system in typical working situation Advantages Disadvantages

8. 8 Guidelines for Experimental Design Decide measures of interest and hypothesesDecide measures of interest and hypotheses Develop a set of representative tasksDevelop a set of representative tasks Run a pilot studyRun a pilot study Determine experimental designDetermine experimental design Within-subjects or between-subjectsWithin-subjects or between-subjects Dependent and independent variablesDependent and independent variables Select sample(s) of typical subjects (size?)Select sample(s) of typical subjects (size?) Experimental design to eliminate unwanted variables and effectsExperimental design to eliminate unwanted variables and effects Explain to subjects and run experiments(s)Explain to subjects and run experiments(s) Collect objective and subjective dataCollect objective and subjective data Compute statistics and analyseCompute statistics and analyse

9. 9 General Points Evaluation is relevant throughout developmentEvaluation is relevant throughout development Different methods most suited at different stages - rule-of-thumb:Different methods most suited at different stages - rule-of-thumb: Early design - paper-based only - analytic / expertEarly design - paper-based only - analytic / expert Prototype development - observational / experimentalPrototype development - observational / experimental Late development - surveyLate development - survey A mix of objective and subjective measures is desirableA mix of objective and subjective measures is desirable

10. 10 Lecture Review Expert evaluationExpert evaluation Empirical evaluationEmpirical evaluation ObservationalObservational Survey evaluationSurvey evaluation Experimental evaluationExperimental evaluation