1. AMSc Research Methods Research approach IV: Experimental [1] Jane Reid jane@dcs.qmul.ac.uk

2. Jane Reid, AMSc RM, QMUL, 4/12/01 2 Overview of lectures Tuesday 4th December [Jane] –Lecture on experimental design –Introduction to experimental design exercise Thursday 6th December [Jane] –Experimental design presentations Tuesday 11th December [Marie-Luce] –Lecture on statistical testing –Introduction to statistical testing exercise Thursday 13th December [Marie-Luce] –Statistical testing presentations

3. Jane Reid, AMSc RM, QMUL, 4/12/01 3 Background Aim: to develop and/or test theories about human characteristics and behaviour Origins in experimental psychology Experimental work can be carried out at different stages of the system development process, e.g. –Design, i.e. to provide requirements capture information –Implementation, i.e. for formative evaluation of prototypes –Evaluation stage, i.e. for final evaluation, in terms of functionality or usability, of finished system

4. Jane Reid, AMSc RM, QMUL, 4/12/01 4 Scenario Imagine you wish to test 2 different schemes for helping young children to learn new vocabulary The 2 schemes are: –Cards with the individual words written on them (“W” cards”) –Cards with the individual words and an illustrative picture on them (“W+P” cards) You want to examine how effective these cards are at different recall times of 2 minutes, 10 minutes and 30 minutes

5. Jane Reid, AMSc RM, QMUL, 4/12/01 5 Experimental hypotheses [1] A theory developed about a relationship between 2 or more factors is an experimental hypothesis Hypotheses are developed in pairings: –A hypothesis that predicts the presence of an effect or association is an alternative hypothesis E.g. The type of vocabulary scheme employed will have an effect on the number of words recalled –A hypothesis that predicts the absence of an effect or association is a null hypothesis E.g. The type of vocabulary scheme employed will have no effect on the number of words recalled

6. Jane Reid, AMSc RM, QMUL, 4/12/01 6 Experimental hypotheses [2] A hypothesis can predict an effect or association: –In one direction - a one-tailed hypothesis E.g. The W+P vocabulary scheme will result in more words being recalled than the W scheme (W+P > W) –In either direction - a two-tailed hypothesis E.g. The type of vocabulary scheme employed will result in a different number of words being recalled (W+P > W or W > W+P, i.e. W <> W+P)

7. Jane Reid, AMSc RM, QMUL, 4/12/01 7 Hypothesis testing [1] An experiment is designed to answer one (or more) experimental hypothesis(es) A hypothesis cannot be proved; it can only be upheld on the basis of the evidence presented Dangers of hypothesis testing: –Type I error The null hypothesis is incorrectly rejected, i.e. the alternative hypothesis is incorrectly upheld –E.g. finding that W+P > W, when really W = W+P

8. Jane Reid, AMSc RM, QMUL, 4/12/01 8 Hypothesis testing [2] –Type II error The null hypothesis is incorrectly upheld, i.e. the alternative hypothesis is incorrectly rejected –E.g. finding that W = W+P when really W > W+P

9. Jane Reid, AMSc RM, QMUL, 4/12/01 9 Variables [1] An experimenter attempts to eliminate unwanted effects from confounding variables, i.e. factors other than those being investigated –E.g. age, family background, educational background, gender... Independent variables –Can sometimes be manipulated directly by the experimenter E.g. vocabulary scheme –Can sometimes only be manipulated indirectly by assignment of participants E.g. gender

10. Jane Reid, AMSc RM, QMUL, 4/12/01 10 Variables [2] –Each manipulation of an independent variable is an experimental condition E.g. W and W+P vocabulary schemes Dependent variables –These are the variables thought to be affected by changes in the independent variables E.g. number of words recalled after a set time –The effect on the dependent variables is measured during the experiment

11. Jane Reid, AMSc RM, QMUL, 4/12/01 11 Experimental design [1] The basis for the experiment Differential designs: –Examination of the difference in effect of 2 or more experimental conditions (of an independent variable) on a dependent variable E.g. The effect of choosing the W or W+P vocabulary scheme on the number of words recalled Correlational designs: –Examination of a possible association between 2 variables E.g. Any association between the ability to learn new vocabulary and the ability to spell

12. Jane Reid, AMSc RM, QMUL, 4/12/01 12 Experimental design [2] –Not possible to form a hypothesis about which variables have an effect on which other variables, so: No concept of independent/dependent variables A cause-and-effect relationship does not always exist between the variables anyway

13. Jane Reid, AMSc RM, QMUL, 4/12/01 13 Designs with one independent variable [1] Using a control condition: –The 2 experimental conditions are the control condition and the independent variable E.g. a standard, existing vocabulary scheme using no cards –The effect of the independent variable is measured against the baseline of the control condition results Using 2 experimental conditions: –The effects of the 2 conditions are measured against each other E.g. W and W+P vocabulary schemes

14. Jane Reid, AMSc RM, QMUL, 4/12/01 14 Designs with one independent variable [2] Using 3 or more experimental conditions: –Possible to examine trends across 3 or more conditions E.g. The effectiveness of a vocabulary scheme after different amounts of recall time: 2 minutes, 10 minutes, 30 minutes

15. Jane Reid, AMSc RM, QMUL, 4/12/01 15 Designs with 2 or more independent variables Each independent variable may have a separate effect - a main effect –E.g. effect of vocabulary scheme choice, effect of recall time There may also be an interaction effect due to the combination of 2 or more independent variables –E.g. One vocabulary scheme (W) may lead to better recall after a short period of time (2 minutes) while the other (W+P) may lead to better recall after a longer period of time (10 minutes) Extension to 3 or more independent variables is possible, but such designs may quickly become complex –E.g. Include the effect of gender in the experiment

16. Jane Reid, AMSc RM, QMUL, 4/12/01 16 Participants [1] When different participants are assigned to each experimental condition, this is an unrelated or between- subjects design –E.g. Different children assigned to each vocabulary scheme –Advantage: 2 closely related experimental conditions can be tested with different participants –Disadvantage: possible bias in results due to unwanted effects from other variables

17. Jane Reid, AMSc RM, QMUL, 4/12/01 17 Participants [2] When the same participants are assigned to each experimental condition, this is a related or within- subjects design –E.g. The same children perform under all 3 recall conditions –Advantage: no confounding variables; possible to study an effect over time –Disadvantage: care needed in experimental design to avoid unwanted learning effects, e.g. by using counterbalancing

18. Jane Reid, AMSc RM, QMUL, 4/12/01 18 Participants [3] When different participants are assigned to each experimental condition, but matched on important characteristics, this is a matched subjects design, and is treated as a related subjects design –E.g. Children assigned to different groups are matched on age, gender... –Advantage: attempt to combine the advantages of the related and unrelated subjects designs –Disadvantage: very difficult (impossible?) to identify and match on all significant variables

19. Jane Reid, AMSc RM, QMUL, 4/12/01 19 Data measurement For statistical analysis, data must be: –Quantitative –Objective Levels of measurement: –Nominal - categories of data –Ordinal - rank ordered data –Interval - numerical data, measured on a continuous scale, with equal intervals between measurement points –Ratio - interval data with an absolute zero

20. Jane Reid, AMSc RM, QMUL, 4/12/01 20 Measures For naturally quantitative data: –Results in interval or ratio data –E.g. time, number of errors For qualitative data that can be measured in a quantitative way (or classified as such by the experimenter) –Results in ordinal data –E.g. confidence, satisfaction For qualitative data –Results in nominal data –E.g. preference for one interface over another

21. Jane Reid, AMSc RM, QMUL, 4/12/01 21 Instruments [1] Questionnaires: –Structured: closed (fixed) questions and response categories –Semi-structured: some closed questions and some open (flexible) questions –Unstructured: open questions –Can be administered before or after the experiment (pre- and post- questionnaires) –Advantage: experimenter does not need to be present –Disadvantage: no interaction, so no discussion - unexpected data may be missed

22. Jane Reid, AMSc RM, QMUL, 4/12/01 22 Instruments [2] Interviews: –Similar possibilities to questionnaires –Advantage: interaction, so unexpected information may be captured –Disadvantage: participants may be less willing to express themselves honestly Observation: –Experimenter observes the participant during the experiment, and may video- or audio-tape the experiment –Advantage: can pick up on transitory events, e.g. hesitations –Disadvantage: Hawthorne effect, i.e. participants behave differently when they are aware of being observed

23. Jane Reid, AMSc RM, QMUL, 4/12/01 23 Instruments [3] Think aloud protocols: –Participants are asked to “think aloud” during the experiment –Advantage: can pick up on thoughts and feelings of the participant –Disadvantage: an unnatural process, so may distract the participant from his task Automatic collection of data: –E.g. software logging

24. Jane Reid, AMSc RM, QMUL, 4/12/01 24 Data analysis Descriptive statistics –Describe the raw data in numerical or graphical form Inferential statistics –Statistical tests and significance testing –Allow conclusions (inferences) to be drawn from results and generalisations to be made BRING A CALCULATOR WITH YOU NEXT WEEK!