1. SESRI Workshop on Survey-based Experiments
Session 2: How and Why We Design Experiments
April 16, 2017
Doha, Qatar

2. Outline: Session 2 Features: why we use survey experiments
Elements: key concepts
Approaches: methods vs. substance

3. Why We Use Survey Experiments
Combine benefits of population-based surveys with experiments
External validity/generalizability (from population-based surveys)
Internal validity/causal inference (from experimental design)

4. External Validity/ Generalizability
Does the study allow me to make inferences about populations of interest?
Key is representativeness
Not all survey experiments embedded in representative survey samples, limits external validity
Threats to external validity: unrealistic treatment

5. Internal Validity/Causal Inference
Does the study allow me to conclude that the treatment caused the outcome?
Key is randomization, ruling out other causes
Threats to internal validity: failures of randomization, experimental effects, trend effects

6. Three Conditions for Establishing Causality
1. Covariation
2. Temporal order
3. Elimination of rival hypotheses/ alternative explanations

7. Elements Population-based surveys Randomization
Treatment/independent variable
Effect/outcome/dependent variable

8. Population-based Surveys
“Strictly speaking, population-based survey experiments are more experiment than survey.” (Mutz 2011, p. 3)
Survey does not need to be representative.

9. Population-based Surveys
Target population: set of units/people about which inferences are to be made.
Who is eligible and ineligible
Representativeness
Probability (random) vs non-probability samples
Approaches
Simple, stratified, clustered, post-stratification
Challenges
Need to give an example of Eligibility

10. Randomization AKA random assignment
Each participant has an equal probability of being assigned to each group
Ensures that groups are equivalent in expectation
Challenges

11. Clicker Question 3
How do you know if the experimental design was effective?
Treatment group and control group are same size
Dependent variable has different values in treatment group and control group
There is equivalence among covariates
Subjects were randomly assigned to groups

12. Random Assignment Example: Selected Sample
PID
AGE 1 35 2 40 3 85 4 19 5 77 6 56 7 88 8 22 9 64 10 31

13. Random Assignment Example: Participant Age
Method: Select odds and evens (from unsorted list)
Method: Random numbers table
Random Assignment 1
Treatment
Control 35 19 64 22 77 31 85 40 88 56
Random Assignment 2
Treatment
Control 31 19 35 22 40 56 64 77 85 88
Avg age: 69.8
Median: 77
Avg age: 33.6
Median: 31
Avg age: 51
Median: 40
Avg age: 52.4
Median: 56

14. Treatment/ Independent Variable
In survey experiments, researcher controls assignment of the treatment.
Treatment and control groups are identical in expectation pre-treatment, therefore any differences post-treatment are attributed to the treatment/independent variable.
In analysis, no need to “control for” other causes/ mediating factors.

15. Effect/ Outcome/ Dependent Variable
Dependent variable is the outcome to be explained.
In a survey experiment, any statistically significant differences in the dependent variable between the treatment and control group is attributed to the treatment.
May be differences of means, proportions, distributions, etc.

16. Approaches
Methods-based experiments
Substantive experiments

17. Methods-based Survey Experiments
Seek to improve the survey process
Test hypotheses about how changing aspects of surveys affects responses
Challenge - what is the “true” response?

18. Substantive Survey Experiments
Seek to test substantive/behavioral hypotheses
Often lines between methods and substantive experiments are blurred