1. Today: Assignment 2 back on Friday
True Experiments: Single-Factor Design
Today’s readings:
The base paper
What did you find in your domain of interest
Research paper

2. It’s a matter of control
Quasi Experiment
True Experiment
Selection of subjects for the conditions
Observe categories of subjects
If the subject variable is the IV, it’s a quasi experiment
Don’t know whether differences are caused by the IV or differences in the subjects
Random assignment of subjects to condition
Manipulate the IV
Control allows ruling out of alternative hypotheses

3. Other features
In some instances cannot completely control the what, when, where, and how
Need to collect data at a certain time or not at all
Practical limitations to data collection, experimental protocol

5. Validity
Internal validity is reduced due to the presence of controlled/confounded variables
But not necessarily invalid
It’s important for the researcher to evaluate the likelihood that there are alternative hypotheses for observed differences
Need to convince self and audience of the validity

6. External validity
If the experimental setting more closely replicates the setting of interest, external validity can be higher than a true experiment run in a controlled lab setting
Often comes down to what is most important for the research question
Control or ecological validity?

7. Terminology Factors: Independent Variables (IVs) of an experiment
Level: particular value of an IV
Condition: a group or treatment (technique)
e.g., Condition 1: old system, Condition 2: new system
Treatment: a condition of an experiment
Subject: participant (can also think more broadly of data sets that are ‘subjected’ to a treatment)

8. Factors to Treatments
At least 1 Factor (IV) has to vary to have an experiment
Effect of screen size and input technique on performance (speed, accuracy)
An IV must always have at least 2 levels
Condition refers to a particular way that subjects are treated
Between subject: experimental conditions are the same as the groups
Within subjects: only 1 group, that experiences every condition (can be many conditions in an experiment)
Mixed: some variables are between, some within

9. Experimental designs
Between subjects: Different participants - single group of participants is allocated randomly to the experimental conditions.
Within subjects: Same participants - all participants appear in both conditions.
Takes care of individual differences
Matched participants - participants are matched in pairs, e.g., based on expertise, gender, etc.
Compromise – groups not likely to be equal, but can match on the factors you think might most impact results 9 9

10. Within-subjects It solves the individual differences issues
But raises other problems:
Need to look at the impact of experiencing the two conditions
Will they get tired? Gain practice? Learn what is expected?
Need to control for order and sequence effects?

11. Order Effects
Changes in performance resulting from (ordinal) position in which a condition appears in an experiment (always first?)
Arises from warm-up, learning, fatigue, etc.
Effect can be averaged and removed if all possible orders are presented in the experiment and there has been random assignment to orders

12. Sequence effects
Changes in performance resulting from interactions among conditions (e.g., if done first, condition 1 has an impact on performance in condition 2)
Effects viewed may not be main effects of the IV, but interaction effects
Can be controlled by arranging each condition to follow every other condition equally often

13. Counterbalancing
Controlling order and sequence effects by arranging subjects to experience the various conditions (levels of the IV) in different orders
Self-directed learning: investigate the different counterbalancing methods
Randomization
Block Randomization
Reverse counter-balancing
Latin squares and Greco squares (when you can’t fully counterbalance)

14. Between, within, matched participant design 14 14

15. True Experiment – Single Factor Design
Images & additional notes text from:
True Experiment – Single Factor Design

16. Experimental Design: spot the flaw
One-Group Post-Test-Only Design
Group of subjects are given a treatment (x)
Single factor – only one IV
Then tested on the dependent variables (observation – o)
What’s the problem?
Failure of internal validity: Cause and effect - don’t know what the DV measurements were before the treatment (was there a change?), don’t know that the treatment caused the DV score (was it something else that also occurred?)
Survey of motivational program in which people engage in several activities, some of which are humiliating and/or exhausting
Survey partiparticipants to evaluate effect of training
Most report experience was worthwile and feel better about themselves than before training
But survey is worthless because we don’t know how they felt before training – so we don’t know if they really changed
No assurance that the training caused the change and not something else (a break in their normal routine)

17. Experimental Design: spot the flaw
Post-Test-Only, non-equivalent control groups
Non-random (N) allocation of subjects into groups
One group is given the treatment, one doesn’t receive it (different levels to each group)
Post-test: measure the DV
What’s the problem?
Addition of control condition is an attempt to get that baseline against which to compare the DVs after treatment (compare with/without treatment)
Attempting to improve the study by adding a control condition
Can match the subjects of the control condition on as many variables as possible
But the control group is NOT equivalent in every way
The non-randomness of the allocation can cause inequalities (self-selection may reveal different biases)
Consider as a quasi-experiment N X O

18. Experimental Design: spot the flaw
One-Group Pre-Test-Post-Test Design
Single group (within subjects)
Pre-test: measure the DVs
Give the treatment
Post-test: Re-measure the DVs
What’s the problem?
Another way of improving upon the one-group post-test only design
Individual differences are controlled for
But possibly threats to internal validity
But still don’t know if there are other explanations for any difference involved other than the treatment (would need to think hard about confounding variables that are not related to the individual), also issues related to the temporal nature introduced (learning, changes over time, etc.)
May not be clear what aspects of the treatment caused the change
A pre-post test design requires that you collect data on study participants’ level of performance before the intervention took place (pre-), and that you collect the same data after the intervention took place (post-). This study design only looks at one group of individuals who receive the intervention, which is called the treatment group. The pre-post test design allows you to make inferences on the effect of your intervention by looking at the difference in the pre-test and post-test results. However, interpreting the pre-test and post-test difference should be done with caution since you cannot be sure that the differences in the pre-test and the post-test are causally related to the intervention.
While the pre-post test design will allow you to measure the potential effects of an intervention by examining the difference in the pre-test and post-test results, it does not allow you to test whether this difference would have occurred in the absence of your intervention. For example, perhaps the effect of improved academic achievement is due to the students getting used to taking a test rather than the use of educational software.

19. Two-Group, Pre &Post-Test Design
Two groups:
Between subjects: random allocation
Treatment
Pre-test and Post-test: measure the DV
Trying to achieve control over threats to validity (but can never rule out all)
Need to have enough subjects (many users, many data runs) to achieve acceptable power
To get the true effects of the program or intervention, it is necessary to have both a treatment group and a control group. As the names suggest, the treatment group receives the intervention. The control group, however, gets the business-as-usual conditions, meaning they only receive interventions that they would have gotten if they had not participated in the study. By having both a group that received the intervention and another group that did not, researchers control for the possibility that other factors not related to the intervention (e.g., students getting accustomed to a test, or simple maturation over the intervening time) are responsible for the difference between the pre-test and post-test results. It is also important that both the treatment group and the control group are of adequate size to be able to determine whether an effect took place or not. While the size of the sample ought to be determined by specific scientific methods, a general rule of thumb is that each group ought to have at least 30 participants.

20. Within-subjects (repeated measures)
Similar to the one-group pre-test-post-test design
It solves the individual differences issues
But raises other problems:
Need to look at the impact of experiencing the two conditions
Will they get tired? Gain practice? Learn what is expected?
Need to control for order and sequence effects

21. Counter balanced

22. Determining effect of IV on the DV
Advantage of single factor design:
Easy to analyze (only one IV)
Fewer conditions (as many as number of levels)
Simple experimental design
BUT
Do not know how results would change for other levels of the controlled variables
Only measuring at one level of the controlled variable
Will the results hold for other levels? Or are there interactions between the IV and other variables?

23. Solutions?
Is repeating the experiment for another level of the controlled variable a valid solution? Why or why not?