1. Design (3): quasi-experimental and non-experimental designs
Learning outcomes
State characteristics of quasi-experimental design
State and define different types of quasi-experimental design and their analysis
State characteristics of non-experimental design
State and define different types of non-experimental design and their analysis

2. Outline Quasi-experimental designs Non-experimental designs
The meaning of treatment
Some elementary quasi-experimental designs
Non-experimental designs
Predictive versus explanatory research
Categorical IVs
Continuous IVs
Categorical and continuous IVs
Longitudinal research

3. Experiment Defining characteristics:
Establish cause-effect relationship between IV(s) and DV as stated in experimental hypothesis
Manipulation of IV(s)
Random assignment of participants to levels (categories) of the IV(s)
If no random assignment then quasi-experiment
If neither manipulation nor random assignment then non-experiment

4. Quasi-experimental designs
Quasi-experiment: as an experiment, except that participants are not randomly assigned to groups
Difficulties:
How to identify and separate the effects of treatments from the effects of all other factors affecting the DV?
Possible selection-treatment interaction: the groups may differ on a range of variables that may affect the effect of treatment on the DV
Meaning of treatment: what is it about the treatment that presumably caused group difference on the DV?
Greater clarity is possible when researcher is involved in design and administration of treatment
Poor examples of ‘treatment’, involving ‘global settings’ as treatment: compensatory education programmes (e.g. Head Start), institutions (public versus state schools)
Non-experimental design: no treatment administered

5. Two quasi-experimental designs
Non-equivalent control group
Interrupted time series

6. Non-equivalent control group
Difference with ‘Treatment-control. Pre-measures and post-measures’ design: no randomization  possible selection bias  other threats to internal validity even more likely, e.g. maturation
Design without a pre-test is preferable (as in experimental designs)
The further apart treatment and control group on pre-test, the more likely are effects of
selection and
Interaction of selection with other factors

7. Non-equivalent control group (2)
Analysis:
No single preferred analysis
Some recommend multiple analyses
Others stress importance of correct model specification as a basis for analysis, i.e. correct identification of IV(s) and extraneous variable(s), and relationships among them and with the DV
Two main approaches to analysis
Regression adjustment
Difference scores

8. Regression adjustment
Adjust post-test score for initial differences between NE groups on pre-test based on regression analysis
Mathematically equivalent to analysis of co-variance
However, threats to validity
Measurement issue: factor structures in the two comparison groups may not be equal and may change over time
pre-test and post-test should measure the same construct in both groups, and pre-test and post-test should measure the same construct
Regression artefacts
The further the two groups apart on the pre-test the greater the threat of regression towards the mean

9. Analysis of difference scores
Procedure
Per participant subtract pre-test score from post-test score – ‘raw-score’ difference
Calculate mean difference for both groups
Test difference between mean of difference for significance, e.g. using t test
Difference scores = specific case of regression adjustment
Same threats as those associated with RA
Additional threats follow

10. Analysis of difference scores - threats
Difficulties in interpretation and analysis
Participants in experimental and control groups may ‘grow’ at different rates. The larger the difference on the pre-test the more difficult to interpret the results.
Sensitivity and difficulty level of pre-test
When ceiling and/or floor effects occur on pre-test or post-test difference scores are not meaningful
‘Correlation with initial status’
Would expect a positive (imperfect) correlation between pre-test and post-test
However, with equal SD of pre-test and post-test, correlation will be negative  those scoring high on the pre-test tend to have smaller gain scores (or even decreasing scores) and those scoring low on pre-test tend to have larger gain scores

11. Interrupted time series (1)
Aims of collecting times-series data
Develop models to explain patterns that occur over time
Use models for forecasting
Interrupted time series
Series interrupted by some discrete event or intervention
Purpose: assess the effect of intervention; in what way has the intervention changed the time-series data?
Costly
Attrition rates can be high

12. Interrupted time series (2)
Simple interrupted time series
Major threat: history
Interrupted time series with non-equivalent control group
Attempts to control for history and other threats to internal validity
Degree to which threats to internal validity are controlled depends on comparability of the two NE groups
Analysis: time series modelling/analysis

13. Summary – quasi-experimental designs
In quasi-experimental designs participants are not randomly assigned to groups 
Difficult to identify and separate the effects of treatments from the effects of all other factors affecting the DV
Possible selection-treatment interaction
Two elementary quasi-experimental designs:
Non-equivalent control group
Interrupted time series

14. Predictive versus explanatory research
Predictive research
Aim: develop systems to predict criteria
Predictor(s) and criterion
Choice of predictor set does not require theory
Predictors selected until best sub-set is found
Explanatory research
Aim: test hypotheses to explain phenomena of interest
Independent variable's) – IV(s) (presumed cause) and dependent variable – DV (presumed effect)
Choice of predictor set based on theory
In the context of non-experimental designs, we will only consider explanatory research

15. Formulation and testing of models
Direction of inference:
From IVs (cause) to DVs (effect) in (quasi)-experimental research
From DVs to IVs in non-experimental research; chance of confusing IV and DV is far greater in non-experimental research
In (quasi)-experimental research, groups are compared that are exposed to different treatments
In non-experimental research groups are very frequently formed on the basis of the DV and differences among groups are attributed to some cause
Researcher tries to find out what variable explains observed differences
However, groups that are compared may not have been exposed to different treatments

16. Threats to internal validity in non-experimental designs
Use of non-probability samples:
Failure to use probability samples (random selection of participants from population and each has a non-zero probability of being selected)
This is not a threat to internal validity in experimental designs
Uncontrolled confounding variables
Control by
Subject selection (see above)
Statistical adjustment (same problems as with quasi-experimental design)
Exercise of controls in non-experimental research may have adverse effects by distorting relations among variables
Some elementary non-experimental designs follow

17. Categorical IV
IV: one or more broad classifications of people are used to explain the status of participants on some phenomenon of interest (presumed DV)
Probability sampling is a necessary, but not sufficient condition for valid inferences
Each grouping or class is treated as a separate population for comparison
Difficulty in offering IV as an explanation for observed differences on DV – ‘pseudo explanations’: ‘a sex difference is a question, not an answer’ (Baumeister)

18. Categorical IV (2) One IV
Analysis: unrelated t test or one-way independent measures analysis of variance
Multiple IVs: similarity with factorial designs in experimental research is only superficial
Influential extraneous variables may have been overlooked
The IVs are (usually) not independent of each other, as they are in experimental research

19. One continuous IV
Omission of relevant variables correlated with IV is a specification error and results in biased estimates of the effects of the IV
This is unavoidable in non-experimental research; minimize this shortcoming BY including major relevant variables in the design
Therefore non-experimental designs with one IV are almost certainly flawed
Analysis: simple regression analysis

20. Multiple continuous IVs
Correct model specification depends on theory of the phenomenon under investigation regarding the nature of the relations among the IVs

21. Multiple continuous IVs (2)
Single-stage models
DV affected by a set of intercorrelated IVs
Exogenous variable: variability assumed to be determined by causes outside the model
Endogenous variable: variability explained by exogeneous variables and possibly other endogenous variables
Analysis: (standard) multiple regression analysis
SES MA
MOT AA
error

22. Multiple continuous IVs (3)
Multi-stage models
One or more exogenous variables and two or more endogenous variables
Stages: the number of endogenous variables
Analysis: path analysis; also hierarchical multiple regression analysis
SES MA
MOT AA
error
SES MA
MOT AA
error

23. Categorical and continuous IVs
E.g. IVs: years of experience and gender; DV: salary
Years of experience as a control variable, investigating the effect of gender after adjusting for years of experience OR
Years of experience as a moderator of gender, investigating an attribute-treatment interaction
Analysis technique: analysis of co-variance, but interpretation different from that in experimental and non-experimental designs

24. Longitudinal research
Longitudinal: study a phenomenon repeatedly as it exists and evolves over time
Cross-sectional: measurement at a single moment in time
Disadvantages of longitudinal research:
Costly
Attrition
Effects of repeated measurement (testing)
The meaning of measures may change across time
Changes in personnel
History
Information diffusion
Cannot provide answers to pressing questions
Interest in research question and theoretical framework may have changed by the time an answer becomes available

25. Preparation for next practical class
Study key experimental research designs
Reading: Pedhazur: Ch. 13, 14
Clark-Carter: Ch. 1
Lecture notes

26. Summary – non-experimental designs
Participants are not randomly assigned to groups  (a) difficult to identify and separate the effects of treatments from the effects of all other factors and (b) possible selection-treatment interaction (as in QE designs)
No treatment is administered  difficult to identify and isolate the effect of IV(s)
Threats to internal validity include
Use of non-probability samples
Uncontrolled confounding variables
One or more categorical or continuous IVs or a combination of categorical and continuous IVs can be used