1. Experimental, Quasi-Experimental, and Ex Post Facto Designs
By Lebna Valliparampil Thomas, Kun Lin, and Arman Dehpanah

2. Outline External Validity & Internal Validity Experimental Designs
Ex Post Facto Designs
Quasi Experimental Designs

3. Validity Accuracy, meaningfulness and credibility of your method.
Types:
Conclusion Validity
Construct Validity
External Validity
Internal Validity

4. Validity Accuracy, meaningfulness and credibility of your method.
Types:
Conclusion Validity
Construct Validity
External Validity
Internal Validity

5. External Validity
It is the degree to which your conclusions will hold for other persons in other places and times. (how generalizable your conclusions are)
There are two approaches to show something is generalizable:
Sampling method
Proximal similarity method

6. External Validity Sampling method
Get sample from the population that you want to generalize from
Conduct research and make conclusions on the sample
Generalize conclusions to the population.
Problems:
At the time of study, perhaps you don’t know who you want to generalize the research to
You may not be able to draw a sample with a good representation from the population
Impossible to sample across all times

7. External Validity - Sampling

8. External Validity Proximal Similarity Method
Think of different generalizability contexts.
Develop a theory about which contexts are similar and which ones are different
Place different contexts in terms of their relative similarity and form a gradient of similarity
For eg: imagine different settings that have people similar to what the study needs and settings that settings that have people that are less similar

9. External Validity – Proximal Similarity

10. External Validity Proximal Similarity Method
Think of different generalizability contexts.
Develop a theory about which contexts are similar and which ones are different
Place different contexts in terms of their relative similarity and form a gradient of similarity
We conclude that we can generalize the study to persons, places and time similar to the study
For eg: imagine different settings that have people similar to what the study needs and settings that settings that have people that are less similar

11. External Validity
Threats: how you might be going wrong while generalizing
People
Place
Time
Critics could argue that result is due to :
-unusual type of people who were in the study
-unusual place you did the study in (perhaps you did your educational study in a college town with lots of high-achieving educationally-oriented kids)
-you did your study in a peculiar time. For instance, if you did your smoking cessation study the week after the Surgeon General issues the well-publicized results of the latest smoking and cancer studies, you might get different results than if you had done it the week before.

12. External Validity Improving External Validity
If using sampling : use a sample that is representative of the population
If using proximal similarity:
Describe ways your contexts and others differ
Provide lots of data about degree of similarity between groups of people, place and time. Map out similarity using concept mapping.
Do study in a variety of places, with different people at different times
Select at random, try to prevent drop out

13. Internal Validity
Relevant in studies that establish a causal relationship
Key question – can you attribute the observed outcome to your program and not another possible cause.
For eg: you want to state that an intervention program caused a particular outcome.
It just says that what you did in the study caused what you observed.
For example:

14. Internal Validity How do you establish a causal relationship:
Temporal Precedence: The cause has to happen before the effect.
Some relationships are cyclical.
Eg: unemployment and inflation
Covariation of cause and effect:
If X then Y
If not X then Not Y
No plausible alternate explanations
When inflation is high -> employers lay off and causes -> unemployment
When everyone is employed, more demand of good -> inflation till supply increases
Correlation does not imply causation
For instance, if you did your smoking cessation study the week after the Surgeon General issues the well-publicized results of the latest smoking and cancer studies, you might get different results than if you had done it the week before.
Show that there is no third variable or a missing variable. Rule out other plausible explanations of the outcome.
Use an apt research design For ex: use a control group.

15. Internal Validity Single group Designs: Post test only Pre test and
let's imagine that we are studying the effects of a compensatory education program in mathematics for first grade students on a measure of math performance such as a standardized math achievement test.
In the post-only design, we would give the first graders the program and then give a math achievement posttest. We might choose not to give them a baseline measure because we have reason to believe they have no prior knowledge of the math skills we are teaching. It wouldn't make sense to pretest them if we expect they would all get a score of zero. In the pre-post design we are not willing to assume that they have no prior knowledge. We measure the baseline in order to determine where the students start out in math achievement. We might hypothesize that the change or gain from pretest to posttest is due to our special math tutoring program.

16. Internal Validity Single Group threats
History threats: a specific event or chain of event happened after the study started that caused the outcome.
Maturation threat
Testing threat: only happens in pretest and post test design.
Instrumentation threat: only happens in pretest and post test design. Change in testing instrument caused the outcome.
Mortality threat: caused by drop out of people
Regression threat: wont get a true estimate of how well the program did when the sample is not random - “ you can only go up from here phenomena”
How to rule out the threats: through an appropriate research design – use a control group
History threat: the kids watched Sesame street and learned more math
Maturation threat: the kids matured or grew up through the time the program took place and are better at math now
Testing threat: Pretest primed them for the program and they were able to do better than they would have otherwise
Instrumentation threat: they give an altered test, sometimes the post test just happened to be easier
Mortality threat: maybe the ones that dropped out are the ones that had the low pretests
Regression analysis: if you pick the lowest 10% in the pretest to do your study on – they can only get better from there

17. Internal Validity Multiple Group:
Involves at least two groups – program group and control group
Multiple Group Threats:
Selection bias: if outcomes vary because of the differences in the two groups rather than because of the program
Selection threat: any factor other than the program that causes the post test results.
Selection - history threat, Selection -maturation threat, Selection - testing threat, Selection - instrumentation threat, Selection - mortality threat, Selection - regression threat.
One group gets the program and other group no program - control group One group gets program and other group standard or another program - Comparing two programs for their relative outcomes.
In these cases, the issue is how comparable the groups are before the program.
If they are comparable and the only difference is the program, then the post test difference can be attributed to the program

18. Internal Validity
Social Interaction Threat: caused by social pressures and human interactions involved can lead to post test differences
Diffusion or imitation of treatment:
equalizes the outcomes for both groups
Compensatory rivalry:
control group develops a competitive attitude towards the program group and tries to be better
Resentful demoralization:
exaggerates the effect of the program because students in the control group get mad and give up
Compensatory equalization of treatment:

19. Research Designs

20. ”One must bear in mind that new facts are no discovered by the one who first observes them. They are discovered by the one who uses an excellent technique and is able to establish them with a full range of evidence, and in so doing convincing everyone.”
---- Santiago Ramón y Cajal

21. Classification of Research Designs
Exploratory or formal
Observational or communication based
Experimental or ex post facto
Descriptive or causal
Cross-sectional or longitudinal
Case or statistical study
Field, laboratory or simulation

22. Experimental or Ex Post Facto
In an experiment the researcher attempts to control and/or manipulate the variables in the study. Experimentation provides the most powerful support possible for a hypothesis of causation
With an ex post facto design, investigators have no control over the variables in the sense of being able to manipulate them. Report only what has happened or what is happening. Important that researches do not influence variables
These two are just two big categories of research designs. They have many subcategories and different designs that we will introduce later.

23. Important Terminology
Independent variable
Variable the researcher manipulates
Dependent variable
Variable that is potentially influenced by the independent variable
Confounding variable
”confounder”
“lurking variable”
“third variable”
“missing variable”
“alternative explanation” …
Before we officially start talking about the specific designs, there are some important terminology to clarify.

24. Independent Variable(s) Dependent Variable(s)
Confounding Variable
Unexamined variable that is or might be correlated with both an independent variable and a dependent variable.
Must be controlled if conclusions about cause-and-effect relationships are desired, or it would influence the internal validity.
Independent Variable(s)
Dependent Variable(s)

25. Independent Variable(s) Dependent Variable(s)
Control
Independent Variable(s)
Dependent Variable(s)
Independent variable(s)
Confounding variable(s)
How to control “independent variable” depends on the research questions. About how to control “confounding variable”, it comes with some general suggestions. But some of them are like a double-edged sword.

26. Controlling for Confounding Variables
Keep some things constant
Include a control group
Conduct a double-blind experiment
Randomly assign people to groups
Use one or more pretests to assess equivalence before the treatment(s)
Expose participants to all experimental treatments
Statistically control for confounding variables
(In LO textbook: checklists about identifying confounding variable, and about the identifying bias and threats for the external validity.)

27. Example
One study found that PhD students were more than twice as likely to have mental-health difficulties than the general highly educated population (from doi: /d )

28. PhD Example
“a simple, urgent truth: that many PhD students and postdoctoral researchers are overworked and overstressed — and their mental health is suffering because of it.”

29. Mental Health Issue(s)
Research Question
RQ1: Are PhD students more likely to have mental-health difficulties?
Hypothesis: PhD students are not more likely to have mental-health difficulties.
Assumption: We have valid sampling approach.
PhD
Mental Health Issue(s)
Aging
Other Confounding Variables

30. Projects
Deadlines
Literature Review
Increasing Stipend
Better Insurance

31. Research Question
RQ2: Is there any good solution to PhD students’ mental health issue?
Hypothesis: Providing PhD students with a higher living stipend or a better health insurance could decrease their mental health issue.
Higher Living Stipend, or Better Health Insurance
Mental Health Issue Decrease
Aging
Other Confounding Variables

32. Experiment Design Family

33. Experiment Design Family
A format table for upcoming presentation
Tx: treatments, intervention
Obs: observation
––: nothing occurs (no treatment or placebo)
Exp: a previous experience, not controllable independent variable

34. Pre-Experimental Design
Such designs are helpful only for forming tentative hypotheses that should be followed up with more controlled studies.
Design 1: One-Shot Experimental Case Study
Design 2: One-Group Pretest-Posttest Design
Design 3: Static Group Comparison

35. Pre-Experimental Design
Design 1: One-Shot Experimental Case Study
To show that one event (a treatment) precedes another event (the observation)
Group 1
PhD
Mental Test
Single observation: Real change?
Root of many misconceptions
if A then B
(if A then B) & (if not A then not B)

36. Pre-Experimental Design
Design 2: One-Group Pretest-Posttest Design
To show that change occurs after a treatment
Group 1
Pretest
Posttest
PhD
How about confounding variables? Single group threat: Maturation threat

37. Pre-Experimental Design
Design 3: Static Group Comparison
To show that a group receiving a treatment behaves differently than a group receiving no treatment
Group 1
Group 2
PhD ––
Mental Test
Selection bias: equivalent groups?
No systematic selection process
No test before treatment

38. True Experimental Design
Randomization: People or other units of study are randomly assigned to groups.
Design 4: Control-Group Pretest–Posttest Design
Design 5: Solomon Four-Group Design
Design 6: Control-Group Posttest-Only Design
Design 7: Within-Subjects Design

39. True Experimental Design
Design 4: Control-Group Pretest–Posttest Design
To show that change occurs following, but only following, a particular treatment.
Pretest
Posttest
Group 1
Group 2 ––
Testing threat: the effect of pretest

40. True Experimental Design
Design 5: Solomon Four-Group Design
To investigate the possible effect of pretesting
Pretest
Posttest
Group 1
Group 2 ––
Posttest
Group 3
Group 4 –– –– ––
Two groups vs. four groups

41. True Experimental Design
Design 6: Control-Group Posttest-Only Design
To determine the effects of a treatment when pretesting cannot or should not occur
Posttest ––

42. True Experimental Design
Design 7: Within-Subjects Design
All participants are exposed to all experimental treatments and any control conditions.
To compare the relative effects of different treatments for the same participants
Useful only when effects of each treatment are temporary and localized.
concept_1
Group 1
illustrated
no-illustrated
concept_2

43. Ex Post Facto Design Design 8: Simple Ex Post Facto Design
To show the possible effects of an experience that occurred, or a condition that was present, prior to the investigation
Group 1
PhD
Mental Test
Group 2 ––
Mental Test
Lose control:
dependent variable
confounding variables

44. Factorial Design Design 9: Two-Factor Experimental Design
To show that two groups are equivalent with respect to the dependent variable prior to the treatment, thus eliminating initial group differences as an explanation for posttreatment differences
Group 1
Tx_1
Tx_2
Posttest
Group 2
Tx_1 ––
Posttest
Group 3 ––
Tx_2
Posttest
Group 4 –– ––
Posttest

45. Combined Design 10: Combined Experimental and Ex Post Facto Design
To study the possible effects of an experimenter manipulated variable, a previously existing condition, and the interaction between the two

46. Wait...
Questions?

47. Quasi-Experimental Design
Similar to Experimental Designs, BUT there is no Random Assignment.
Can be used when randomness is either impossible or impractical.
Does not control for all confounding variables.
Cannot completely rule out some alternative explanations for the results.
Should be considered when interpreting data and results.
Internal Validity
Ease of Usability
Control

48. Quasi-Experimental Design
Design 11: Nonrandomized Control-Group Pretest–Posttest Design
Design 12: Regression-Discontinuity Design
Design 13: Simple Time-Series Design
Design 14: Control-Group Time-Series Design
Design 15: Reversal Time-Series Design
Design 16: Alternating-Treatments Design
Design 17: Multiple-Baseline Design

49. Design 11: Nonrandomized Control-Group Pretest–Posttest Design
The most frequently used design in social research.
Selects groups that are as similar as possible (such as two comparable classrooms or schools).
The two groups are unlikely to be as similar as random assignment.
Internal validity threat of Selection

50. Design 11: Nonrandomized Control-Group Pretest–Posttest Design
Observing both groups to make sure they are (almost) equivalent with respect to the dependent variable prior to the treatment.
Eliminating initial group differences as an explanation for posttreatment differences.

51. Back to the Example of PhD Students Mental Health Issue

52. Projects
Deadlines
Literature Review
Increasing Stipend
Better Insurance

53. x: treatment (program) group (Students with increased stipend)
o: control (comparison) group (Students with no change in stipend)
Selection Bias

54. Interpreting Different Outputs
Selection-Maturation?
Selection-History?
Selection-Regression?

55. Interpreting Different Outputs
Selection-Maturation?
Selection-History?
Selection-Regression?

56. Interpreting Different Outputs
This is referred to as Cross-over pattern and shows the effectiveness of the program

57. Design 12: Regression-Discontinuity Designs
Participants are assigned to program or comparison groups solely on the basis of a cutoff score on a pre-program measure.
Appropriate when we wish to target a program or treatment to those who most need or deserve it.
Comparable to randomized experiments in terms of internal validity.
All individuals on one side of the cutoff are assigned to one group.
All individuals on the other side of the cutoff are assigned to the other.
Needs a continuous quantitative pre-program measure
Grouping based on a cutoff score

58. Back to our example Projects Deadlines Literature Review
Increasing Stipend
Better Insurance

59. Measuring PhD students mental health at two different points in time
(Before increasing the stipend)
After increasing the stipend

60. So...

61. Design 13: Simple Time-Series Design
In its simplest form, consists of making a series of observations (i.e., measuring the dependent variable on several occasions), introducing an intervention or other new dynamic into the system, and then making additional observations.
Shows that for a single group, change occurs during a lengthy period only after the treatment has been administered
Baseline Data
Internal validity threats?

62. Design 14: Control Group Time-Series Design
Adding a control group to the previous design
Decreasing the internal validity threat

63. Design 15: Reversal Time-Series Design
Uses a within-subjects approach as a way of minimizing internal validity threats.
The treatment is sometimes present, sometimes absent.
Measures the dependent variable at regular intervals.

64. Design 16: Alternating-Treatments Design
A variation on the reversal time-series design.
Involves two or more different forms of treatment.
Showing that different treatments have different effects.
Involves sequentially administering different treatments at different times and comparing their effects against the possible consequences of non-treatment. X X X

65. Design 17: Multiple-Baseline Designs
The previous two designs are based on the assumption that the effects of any single treatment are temporary and limited to the immediate circumstances.
Requires at least two groups.
Prior to the treatment, baseline data are collected for all groups, and then the treatment itself is introduced at a different time for each group.
Shows the effect of a treatment by initiating it at different times for different groups.
Involves tracking two or more groups or individuals over time for a lengthy period of time, as well as initiating the treatment at different times for different groups.

66. Real-world Example
Instances of Risky Behavior on Slides and Climbers by Grade Level Reprinted from “Decreasing Children’s Risk Taking on the Playground” by A. Heck, J. Collins, and L. Peterson, 2001, Journal of Applied Behavior Analysis, 34, p. 351.

67. Advances in Quasi-Experimentation
The Role of Judgment
The Case for Tailored Designs
The Crucial Role of Theory
Attention to Program Implementation
The Importance of Quality Control
The Advantages of Multiple Perspectives
Evolution of the Concept of Validity
Development of Increasingly Complex Realistic Analytic Models

68. Conclusion
We gain greater confidence in our research findings when a study is repeated over and over again—perhaps with a different population, in a different setting, or with slight variations on the treatment implementation.
Experimental and ex post facto studies typically begin with specific research hypotheses, and subsequent statistical analyses should be conducted to test them.
We covered several research designs here but there are many other variations and types. These research designs can be considered as a starting point. You can use them in different forms or even mix them to create your own experiments.
Read Sample Dissertation (page 225 of the textbook) for a detailed analysis of the methodology section in the thesis.

69. Finally, You do not want to take the research design lightly!
Refer to wired.com: “Fantastically Wrong: Why People Once Thought Mice Grew Out of Wheat and Sweaty Shirts”
Link

70. To get the most out of your PhD, attending following events at CDM are highly recommended:
Welcoming new PhD students on Monday October 14th,
4 to 5 pm, CDM kitchen on 8th floor
Learning all about PhD in the PhD 101 event on Friday October 18th, 3 to 5 pm, CDM room 708
Presenting your research at SCD Symposium, sometime in May 2020