Presentation is loading. Please wait.

Presentation is loading. Please wait.

Experimental, Quasi-Experimental, and Ex Post Facto Designs

Similar presentations


Presentation on theme: "Experimental, Quasi-Experimental, and Ex Post Facto Designs"— Presentation transcript:

1 Experimental, Quasi-Experimental, and Ex Post Facto Designs
By Lebna Valliparampil Thomas, Kun Lin, and Arman Dehpanah

2 Outline External Validity & Internal Validity Experimental Designs
Ex Post Facto Designs Quasi Experimental Designs

3 Validity Accuracy, meaningfulness and credibility of your method.
Types: Conclusion Validity Construct Validity External Validity Internal Validity

4 Validity Accuracy, meaningfulness and credibility of your method.
Types: Conclusion Validity Construct Validity External Validity Internal Validity

5 External Validity It is the degree to which your conclusions will hold for other persons in other places and times. (how generalizable your conclusions are) There are two approaches to show something is generalizable: Sampling method Proximal similarity method

6 External Validity Sampling method
Get sample from the population that you want to generalize from Conduct research and make conclusions on the sample Generalize conclusions to the population. Problems: At the time of study, perhaps you don’t know who you want to generalize the research to You may not be able to draw a sample with a good representation from the population Impossible to sample across all times

7 External Validity - Sampling

8 External Validity Proximal Similarity Method
Think of different generalizability contexts. Develop a theory about which contexts are similar and which ones are different Place different contexts in terms of their relative similarity and form a gradient of similarity For eg: imagine different settings that have people similar to what the study needs and settings that settings that have people that are less similar

9 External Validity – Proximal Similarity

10 External Validity Proximal Similarity Method
Think of different generalizability contexts. Develop a theory about which contexts are similar and which ones are different Place different contexts in terms of their relative similarity and form a gradient of similarity We conclude that we can generalize the study to persons, places and time similar to the study For eg: imagine different settings that have people similar to what the study needs and settings that settings that have people that are less similar

11 External Validity Threats: how you might be going wrong while generalizing People Place Time Critics could argue that result is due to : -unusual type of people who were in the study -unusual place you did the study in (perhaps you did your educational study in a college town with lots of high-achieving educationally-oriented kids) -you did your study in a peculiar time. For instance, if you did your smoking cessation study the week after the Surgeon General issues the well-publicized results of the latest smoking and cancer studies, you might get different results than if you had done it the week before.

12 External Validity Improving External Validity
If using sampling : use a sample that is representative of the population If using proximal similarity: Describe ways your contexts and others differ Provide lots of data about degree of similarity between groups of people, place and time. Map out similarity using concept mapping. Do study in a variety of places, with different people at different times Select at random, try to prevent drop out

13 Internal Validity Relevant in studies that establish a causal relationship Key question – can you attribute the observed outcome to your program and not another possible cause. For eg: you want to state that an intervention program caused a particular outcome. It just says that what you did in the study caused what you observed. For example:

14 Internal Validity How do you establish a causal relationship:
Temporal Precedence: The cause has to happen before the effect. Some relationships are cyclical. Eg: unemployment and inflation Covariation of cause and effect: If X then Y If not X then Not Y No plausible alternate explanations When inflation is high -> employers lay off and causes -> unemployment When everyone is employed, more demand of good -> inflation till supply increases Correlation does not imply causation For instance, if you did your smoking cessation study the week after the Surgeon General issues the well-publicized results of the latest smoking and cancer studies, you might get different results than if you had done it the week before. Show that there is no third variable or a missing variable. Rule out other plausible explanations of the outcome. Use an apt research design For ex: use a control group.

15 Internal Validity Single group Designs: Post test only Pre test and
let's imagine that we are studying the effects of a compensatory education program in mathematics for first grade students on a measure of math performance such as a standardized math achievement test. In the post-only design, we would give the first graders the program and then give a math achievement posttest. We might choose not to give them a baseline measure because we have reason to believe they have no prior knowledge of the math skills we are teaching. It wouldn't make sense to pretest them if we expect they would all get a score of zero. In the pre-post design we are not willing to assume that they have no prior knowledge. We measure the baseline in order to determine where the students start out in math achievement. We might hypothesize that the change or gain from pretest to posttest is due to our special math tutoring program.

16 Internal Validity Single Group threats
History threats: a specific event or chain of event happened after the study started that caused the outcome. Maturation threat Testing threat: only happens in pretest and post test design. Instrumentation threat: only happens in pretest and post test design. Change in testing instrument caused the outcome. Mortality threat: caused by drop out of people Regression threat: wont get a true estimate of how well the program did when the sample is not random - “ you can only go up from here phenomena” How to rule out the threats: through an appropriate research design – use a control group History threat: the kids watched Sesame street and learned more math Maturation threat: the kids matured or grew up through the time the program took place and are better at math now Testing threat: Pretest primed them for the program and they were able to do better than they would have otherwise Instrumentation threat: they give an altered test, sometimes the post test just happened to be easier Mortality threat: maybe the ones that dropped out are the ones that had the low pretests Regression analysis: if you pick the lowest 10% in the pretest to do your study on – they can only get better from there

17 Internal Validity Multiple Group:
Involves at least two groups – program group and control group Multiple Group Threats: Selection bias: if outcomes vary because of the differences in the two groups rather than because of the program Selection threat: any factor other than the program that causes the post test results. Selection - history threat, Selection -maturation threat, Selection - testing threat, Selection - instrumentation threat, Selection - mortality threat, Selection - regression threat. One group gets the program and other group no program - control group One group gets program and other group standard or another program - Comparing two programs for their relative outcomes. In these cases, the issue is how comparable the groups are before the program. If they are comparable and the only difference is the program, then the post test difference can be attributed to the program

18 Internal Validity Social Interaction Threat: caused by social pressures and human interactions involved can lead to post test differences Diffusion or imitation of treatment: equalizes the outcomes for both groups Compensatory rivalry: control group develops a competitive attitude towards the program group and tries to be better Resentful demoralization: exaggerates the effect of the program because students in the control group get mad and give up Compensatory equalization of treatment:

19 Research Designs

20 ”One must bear in mind that new facts are no discovered by the one who first observes them. They are discovered by the one who uses an excellent technique and is able to establish them with a full range of evidence, and in so doing convincing everyone.” ---- Santiago Ramón y Cajal

21 Classification of Research Designs
Exploratory or formal Observational or communication based Experimental or ex post facto Descriptive or causal Cross-sectional or longitudinal Case or statistical study Field, laboratory or simulation

22 Experimental or Ex Post Facto
In an experiment the researcher attempts to control and/or manipulate the variables in the study. Experimentation provides the most powerful support possible for a hypothesis of causation With an ex post facto design, investigators have no control over the variables in the sense of being able to manipulate them. Report only what has happened or what is happening. Important that researches do not influence variables These two are just two big categories of research designs. They have many subcategories and different designs that we will introduce later.

23 Important Terminology
Independent variable Variable the researcher manipulates Dependent variable Variable that is potentially influenced by the independent variable Confounding variable ”confounder” “lurking variable” “third variable” “missing variable” “alternative explanation” Before we officially start talking about the specific designs, there are some important terminology to clarify.

24 Independent Variable(s) Dependent Variable(s)
Confounding Variable Unexamined variable that is or might be correlated with both an independent variable and a dependent variable. Must be controlled if conclusions about cause-and-effect relationships are desired, or it would influence the internal validity. Independent Variable(s) Dependent Variable(s)

25 Independent Variable(s) Dependent Variable(s)
Control Independent Variable(s) Dependent Variable(s) Independent variable(s) Confounding variable(s) How to control “independent variable” depends on the research questions. About how to control “confounding variable”, it comes with some general suggestions. But some of them are like a double-edged sword.

26 Controlling for Confounding Variables
Keep some things constant Include a control group Conduct a double-blind experiment Randomly assign people to groups Use one or more pretests to assess equivalence before the treatment(s) Expose participants to all experimental treatments Statistically control for confounding variables (In LO textbook: checklists about identifying confounding variable, and about the identifying bias and threats for the external validity.)

27 Example One study found that PhD students were more than twice as likely to have mental-health difficulties than the general highly educated population (from doi: /d )

28 PhD Example “a simple, urgent truth: that many PhD students and postdoctoral researchers are overworked and overstressed — and their mental health is suffering because of it.”

29 Mental Health Issue(s)
Research Question RQ1: Are PhD students more likely to have mental-health difficulties? Hypothesis: PhD students are not more likely to have mental-health difficulties. Assumption: We have valid sampling approach. PhD Mental Health Issue(s) Aging Other Confounding Variables

30 Projects Deadlines Literature Review Increasing Stipend Better Insurance

31 Research Question RQ2: Is there any good solution to PhD students’ mental health issue? Hypothesis: Providing PhD students with a higher living stipend or a better health insurance could decrease their mental health issue. Higher Living Stipend, or Better Health Insurance Mental Health Issue Decrease Aging Other Confounding Variables

32 Experiment Design Family

33 Experiment Design Family
A format table for upcoming presentation Tx: treatments, intervention Obs: observation ––: nothing occurs (no treatment or placebo) Exp: a previous experience, not controllable independent variable

34 Pre-Experimental Design
Such designs are helpful only for forming tentative hypotheses that should be followed up with more controlled studies. Design 1: One-Shot Experimental Case Study Design 2: One-Group Pretest-Posttest Design Design 3: Static Group Comparison

35 Pre-Experimental Design
Design 1: One-Shot Experimental Case Study To show that one event (a treatment) precedes another event (the observation) Group 1 PhD Mental Test Single observation: Real change? Root of many misconceptions if A then B (if A then B) & (if not A then not B)

36 Pre-Experimental Design
Design 2: One-Group Pretest-Posttest Design To show that change occurs after a treatment Group 1 Pretest Posttest PhD How about confounding variables? Single group threat: Maturation threat

37 Pre-Experimental Design
Design 3: Static Group Comparison To show that a group receiving a treatment behaves differently than a group receiving no treatment Group 1 Group 2 PhD –– Mental Test Selection bias: equivalent groups? No systematic selection process No test before treatment

38 True Experimental Design
Randomization: People or other units of study are randomly assigned to groups. Design 4: Control-Group Pretest–Posttest Design Design 5: Solomon Four-Group Design Design 6: Control-Group Posttest-Only Design Design 7: Within-Subjects Design

39 True Experimental Design
Design 4: Control-Group Pretest–Posttest Design To show that change occurs following, but only following, a particular treatment. Pretest Posttest Group 1 Group 2 –– Testing threat: the effect of pretest

40 True Experimental Design
Design 5: Solomon Four-Group Design To investigate the possible effect of pretesting Pretest Posttest Group 1 Group 2 –– Posttest Group 3 Group 4 –– –– –– Two groups vs. four groups

41 True Experimental Design
Design 6: Control-Group Posttest-Only Design To determine the effects of a treatment when pretesting cannot or should not occur Posttest ––

42 True Experimental Design
Design 7: Within-Subjects Design All participants are exposed to all experimental treatments and any control conditions. To compare the relative effects of different treatments for the same participants Useful only when effects of each treatment are temporary and localized. concept_1 Group 1 illustrated no-illustrated concept_2

43 Ex Post Facto Design Design 8: Simple Ex Post Facto Design
To show the possible effects of an experience that occurred, or a condition that was present, prior to the investigation Group 1 PhD Mental Test Group 2 –– Mental Test Lose control: dependent variable confounding variables

44 Factorial Design Design 9: Two-Factor Experimental Design
To show that two groups are equivalent with respect to the dependent variable prior to the treatment, thus eliminating initial group differences as an explanation for posttreatment differences Group 1 Tx_1 Tx_2 Posttest Group 2 Tx_1 –– Posttest Group 3 –– Tx_2 Posttest Group 4 –– –– Posttest

45 Combined Design 10: Combined Experimental and Ex Post Facto Design
To study the possible effects of an experimenter manipulated variable, a previously existing condition, and the interaction between the two

46 Wait... Questions?

47 Quasi-Experimental Design
Similar to Experimental Designs, BUT there is no Random Assignment. Can be used when randomness is either impossible or impractical. Does not control for all confounding variables. Cannot completely rule out some alternative explanations for the results. Should be considered when interpreting data and results. Internal Validity Ease of Usability Control

48 Quasi-Experimental Design
Design 11: Nonrandomized Control-Group Pretest–Posttest Design Design 12: Regression-Discontinuity Design Design 13: Simple Time-Series Design Design 14: Control-Group Time-Series Design Design 15: Reversal Time-Series Design Design 16: Alternating-Treatments Design Design 17: Multiple-Baseline Design

49 Design 11: Nonrandomized Control-Group Pretest–Posttest Design
The most frequently used design in social research. Selects groups that are as similar as possible (such as two comparable classrooms or schools). The two groups are unlikely to be as similar as random assignment. Internal validity threat of Selection

50 Design 11: Nonrandomized Control-Group Pretest–Posttest Design
Observing both groups to make sure they are (almost) equivalent with respect to the dependent variable prior to the treatment. Eliminating initial group differences as an explanation for posttreatment differences.

51 Back to the Example of PhD Students Mental Health Issue

52 Projects Deadlines Literature Review Increasing Stipend Better Insurance

53 x: treatment (program) group (Students with increased stipend)
o: control (comparison) group (Students with no change in stipend) Selection Bias

54 Interpreting Different Outputs
Selection-Maturation? Selection-History? Selection-Regression?

55 Interpreting Different Outputs
Selection-Maturation? Selection-History? Selection-Regression?

56 Interpreting Different Outputs
This is referred to as Cross-over pattern and shows the effectiveness of the program

57 Design 12: Regression-Discontinuity Designs
Participants are assigned to program or comparison groups solely on the basis of a cutoff score on a pre-program measure. Appropriate when we wish to target a program or treatment to those who most need or deserve it. Comparable to randomized experiments in terms of internal validity. All individuals on one side of the cutoff are assigned to one group. All individuals on the other side of the cutoff are assigned to the other. Needs a continuous quantitative pre-program measure Grouping based on a cutoff score

58 Back to our example Projects Deadlines Literature Review
Increasing Stipend Better Insurance

59 Measuring PhD students mental health at two different points in time
(Before increasing the stipend) After increasing the stipend

60 So...

61 Design 13: Simple Time-Series Design
In its simplest form, consists of making a series of observations (i.e., measuring the dependent variable on several occasions), introducing an intervention or other new dynamic into the system, and then making additional observations. Shows that for a single group, change occurs during a lengthy period only after the treatment has been administered Baseline Data Internal validity threats?

62 Design 14: Control Group Time-Series Design
Adding a control group to the previous design Decreasing the internal validity threat

63 Design 15: Reversal Time-Series Design
Uses a within-subjects approach as a way of minimizing internal validity threats. The treatment is sometimes present, sometimes absent. Measures the dependent variable at regular intervals.

64 Design 16: Alternating-Treatments Design
A variation on the reversal time-series design. Involves two or more different forms of treatment. Showing that different treatments have different effects. Involves sequentially administering different treatments at different times and comparing their effects against the possible consequences of non-treatment. X X X

65 Design 17: Multiple-Baseline Designs
The previous two designs are based on the assumption that the effects of any single treatment are temporary and limited to the immediate circumstances. Requires at least two groups. Prior to the treatment, baseline data are collected for all groups, and then the treatment itself is introduced at a different time for each group. Shows the effect of a treatment by initiating it at different times for different groups. Involves tracking two or more groups or individuals over time for a lengthy period of time, as well as initiating the treatment at different times for different groups.

66 Real-world Example Instances of Risky Behavior on Slides and Climbers by Grade Level Reprinted from “Decreasing Children’s Risk Taking on the Playground” by A. Heck, J. Collins, and L. Peterson, 2001, Journal of Applied Behavior Analysis, 34, p. 351.

67 Advances in Quasi-Experimentation
The Role of Judgment The Case for Tailored Designs The Crucial Role of Theory Attention to Program Implementation The Importance of Quality Control The Advantages of Multiple Perspectives Evolution of the Concept of Validity Development of Increasingly Complex Realistic Analytic Models

68 Conclusion We gain greater confidence in our research findings when a study is repeated over and over again—perhaps with a different population, in a different setting, or with slight variations on the treatment implementation. Experimental and ex post facto studies typically begin with specific research hypotheses, and subsequent statistical analyses should be conducted to test them. We covered several research designs here but there are many other variations and types. These research designs can be considered as a starting point. You can use them in different forms or even mix them to create your own experiments. Read Sample Dissertation (page 225 of the textbook) for a detailed analysis of the methodology section in the thesis.

69 Finally, You do not want to take the research design lightly!
Refer to wired.com: “Fantastically Wrong: Why People Once Thought Mice Grew Out of Wheat and Sweaty Shirts” Link

70 To get the most out of your PhD, attending following events at CDM are highly recommended:
Welcoming new PhD students on Monday October 14th, 4 to 5 pm, CDM kitchen on 8th floor Learning all about PhD in the PhD 101 event on Friday October 18th, 3 to 5 pm, CDM room 708 Presenting your research at SCD Symposium, sometime in May 2020

71


Download ppt "Experimental, Quasi-Experimental, and Ex Post Facto Designs"

Similar presentations


Ads by Google