Methodology and Explanation XX50125 Lecture 1: Part I. Introduction to Evaluation Methods Part 2. Experiments Dr. Danaë Stanton Fraser

Methodology and Explanation 2007 Goals of this unit To introduce research methods in human computer interaction To explore qualitative and quantative methods To gain experience in putting these methods into practice To gain a critical understanding of analytic techniques

Methodology and Explanation 2007 Unit structure Lectures each focusing on methods of analysis Practical sessions putting these methods into practice

Methodology and Explanation 2007 Course text and slides Rogers, Y., Sharp, H., and Preece, J Interaction Design: Beyond Human Computer Interaction. Wiley. Slides available online at:

Methodology and Explanation 2007 Assessment 2 assessed courseworks: 1 st coursework (report) 2 nd coursework (report and presentation)

Methodology and Explanation 2007 Evaluation Methods Controlled experiments Ethnography Expert review Usability testing Surveys/questionnaires Data logging Interviews and focus groups

Methodology and Explanation 2007 Not just desktop computing. What other kinds of technologies are we evaluating?

Methodology and Explanation 2007 Direct manipulation & graphical user interfaces

Methodology and Explanation 2007 Collaborative Desktop

Methodology and Explanation 2007 Immersive technologies

Methodology and Explanation 2007 Tangible interfaces

Methodology and Explanation 2007 Wearable, handheld and mobile devices

Methodology and Explanation 2007 Embedded interfaces

Methodology and Explanation 2007 Evaluate: which method to choose Design or implementation? Laboratory or field studies? Subjective or objective? Qualitative or quantitative? Information provided? Immediacy of response? Intrusiveness? Resources?

Methodology and Explanation 2007 Part II Experiments: A brief guide

Methodology and Explanation 2007 Aim To answer a question or test an hypothesis that predicts a relationship between two or more events, known as variables. E.g. Will spatial knowledge be superior following exploration of a VR simulation of a building or from exploring a model of a building?

Methodology and Explanation 2007 Variables Such hypotheses are tested by manipulating one or more of the variables. The variable that is manipulated is called the independent variable (the conditions to test this variable are setup independently before the experiment starts). In the study mentioned media type (VR vs model) is the independent measure The dependent variable would be accuracy of spatial information (ie time to reach point a from point b) because the hypothesis is that the time to carry out the task depends on the media explored.

Methodology and Explanation 2007 Variables and Conditions In order to test an hypothesis the experimenter sets up the experimental conditions e.g. Condition 1 Pretest Explore model Post test Condition 2 Pretest Explore VR Post test Control condition – against which to compare the results?

Methodology and Explanation 2007 Participants Between participants Two drawbacks – no. of participants needed, individual differences Advantage – no order effects Within participants Counterbalancing required Matched pairs on characteristics such as gender and expertise Difficult to match across all variables

Methodology and Explanation 2007 Practicalities Where will the experiment be carried out? How will the equipment be set-up? How does one introduce participants to study? What scripts are need to standardise the procedure? Always include pilot study

Methodology and Explanation 2007 Data Collection and Analysis Data – performance measures are taken e.g. response times, no. of errors Use graphs The data should be averaged across conditions to examine any differences Statistical tests such as t-tests and ANOVAs can reveal whether the differences are significant Software packages such as SPSS are often used If there is no significant difference then the hypothesis is refuted.

Methodology and Explanation 2007 Data captured Record quantitative results However also gather process data e.g. dialogue turns, gesture and non verbal behaviour

Methodology and Explanation 2007 Preparation for Practical 1 Topic: Examining the use of gestures in collaboration. In preparation for the practical please read: 1.Kraut, R. E., Fussell, S. R., and Siegel, J. (2003). Visual Information as a Conversational Resource in Collaborative Physical Tasks. Human-Computer Interaction,Vol. 18, pp. 13– 49 Lawrence Erlbaum Associates, Inc. df 2. Kirk, D. and Stanton Fraser, D. (2005). The Effects of Remote Gesturing on Distance Instruction. Computer Supported Collaborative Learning (CSCL 2005). Taiwan. May.