Experimental Design making causal inferences Richard Lambert, Ph.D.

The Gold Standard What are the differences between these types of studies? What are the differences between these types of studies? True experiments True experiments Observational studies Observational studies Surveys Surveys

Observational Studies Intact groups, not randomly assigned Intact groups, not randomly assigned If there is a treatment, it may not be under the control of the researcher If there is a treatment, it may not be under the control of the researcher The goal is often just to look at relationships between variables The goal is often just to look at relationships between variables Correlation not causality Correlation not causality

Surveys Population is defined Population is defined Sample is selected from the population so that is can be representative of the population – SRS, Stratified RS Sample is selected from the population so that is can be representative of the population – SRS, Stratified RS The focus is on estimation of parameters, not the effects of a treatment The focus is on estimation of parameters, not the effects of a treatment

Surveys Compare and contrast SRS and Stratified RS Compare and contrast SRS and Stratified RS Both give everyone in the population an equal chance of selection AND give every possible sample of size n an equal probability of selection Both give everyone in the population an equal chance of selection AND give every possible sample of size n an equal probability of selection For Stratified RS, sampling is done within homogeneous strata For Stratified RS, sampling is done within homogeneous strata

Surveys For Stratified RS, we divide the population into homogeneous subgroups according to some variable we expect to be related to the survey questions For Stratified RS, we divide the population into homogeneous subgroups according to some variable we expect to be related to the survey questions We then sample within strata We then sample within strata Compare and contrast Stratified SRS and Blocking Compare and contrast Stratified SRS and Blocking

True Experiments The Gold Standard for Causal Inference The Gold Standard for Causal Inference Random assignment to treatment and control conditions Random assignment to treatment and control conditions A treatment is imposed on the subjects, the treatment is under the control of the researcher A treatment is imposed on the subjects, the treatment is under the control of the researcher

Treatment is Imposed Presence or Absence of Treatment Presence or Absence of Treatment Dosage Level Controlled By Researcher Dosage Level Controlled By Researcher Multiple Types of Treatment Conditions Multiple Types of Treatment Conditions

True Experiments The Gold Standard for Causal Inference The Gold Standard for Causal Inference Why is it so hard to do in education? Why is it so hard to do in education? Is it the only “way of knowing” in educational research? Is it the only “way of knowing” in educational research?

Causal and Effect The IV precedes the DV in time The IV precedes the DV in time The IV and DV are related The IV and DV are related There are no plausible additional variables that could reasonably explain the relationship There are no plausible additional variables that could reasonably explain the relationship

Causal and Effect When we say the IV and DV are related, think of a dose – response relationship in a drug study. When we say the IV and DV are related, think of a dose – response relationship in a drug study. The amount of treatment exposure is related to a corresponding amount of some effect. The amount of treatment exposure is related to a corresponding amount of some effect.

The Essential Characteristics Random Selection Random Selection Random Assignment Random Assignment Treatment is Imposed Treatment is Imposed Control Condition Control Condition

Additional Factors to Consider Placebo or Comparison Group(s) Placebo or Comparison Group(s) Multiple Measurements Over Time Multiple Measurements Over Time Control Over Confounding Variables Control Over Confounding Variables

Controlling Confounds The Experimental Environment The Experimental Environment Admissibility criteria Admissibility criteria Blocking Blocking

Blocking Creates homogeneous subsets Creates homogeneous subsets Builds potential confounds into the design Builds potential confounds into the design Reduces error term Reduces error term Makes a more sensitive and therefore powerful experiment Makes a more sensitive and therefore powerful experiment

Blocking When using blocking, be sure to: When using blocking, be sure to: Divide the sample into homogeneous subsets Divide the sample into homogeneous subsets Randomly assign subjects to treatments within blocks Randomly assign subjects to treatments within blocks

Experimental Validity Internal Validity Internal Validity External Validity External Validity Construct Validity Construct Validity Statistical Conclusion Validity Statistical Conclusion Validity

Released Questions Work on 1999 Free Response #3 with your partner Work on 1999 Free Response #3 with your partner

Released Questions – 1999 #3 3. The dentists in a dental clinic would like to determine if there is a difference between the number of new cavities in people who eat an apple a day and in people who eat less than one apple a week. They are going to conduct a study with 50 people in each group. Fifty clinic patients who report that they routinely eat an apple a day and 50 clinic patients who report that they eat less than one apple a week will be identified. The dentists will examine the patients and their records to determine the number of new cavities the patients have had over the past two years. They will then compare the number of new cavities in the two groups. Why is this an observational study and not an experiment? Explain the concept of confounding in the context of this study. Include an example of a possible confounding variable. If the mean number of new cavities for those who ate an apple a day was statistically smaller than the mean number of new cavities for those who ate less than one apple a week, could one conclude that the lower number of new cavities can be attributed to eating an apple a day? Explain.

The Essential Characteristics Random Selection Random Selection Random Assignment Random Assignment Treatment is imposed Treatment is imposed Control Condition Control Condition

Confounding Variables Have to be related to the DV Have to be related to the DV Are mixed up with, or inseparable from Group membership Are mixed up with, or inseparable from Group membership Create non-equivalence of treatment groups Create non-equivalence of treatment groups Disrupt your ability to conclude that the treatment and only the treatment caused the outcomes Disrupt your ability to conclude that the treatment and only the treatment caused the outcomes

Released Questions Work on 2003 Free Response #4 with your partner Work on 2003 Free Response #4 with your partner 2003 Free Response # Free Response #4

2003 #4

Rubric for Part A Identify a plausible example of a problem Identify a plausible example of a problem –“Because a deadline has been moved back...” Relate the identified problem to the change in stress level Relate the identified problem to the change in stress level –“...the stress levels of those working in the department have been lowered...” State that the problem effects can not be distinguished from the treatment effects State that the problem effects can not be distinguished from the treatment effects –“...which could be mistakenly attributed to the treatment.”

Rubric for Part A Give a reason for the necessity of random assignment. Give a reason for the necessity of random assignment. State that randomization is relied upon to create comparable groups. State that randomization is relied upon to create comparable groups. State that randomization helps reduce the influence of potential confounding variables. State that randomization helps reduce the influence of potential confounding variables.

Rubric for Part A “Without random assignment of volunteers to the two programs, it is possible that the two treatment groups could differ in some way that affects the outcome of the experiment. Randomization “evens out” the possible effects of potentially confounding variables.” “Without random assignment of volunteers to the two programs, it is possible that the two treatment groups could differ in some way that affects the outcome of the experiment. Randomization “evens out” the possible effects of potentially confounding variables.”

Rubric for Part B Indicate that a control group does provide additional information Indicate that a control group does provide additional information Explain that the control group allows the company to determine if either or both treatments are effective in reducing stress Explain that the control group allows the company to determine if either or both treatments are effective in reducing stress Explain that the control group provides a baseline for comparison, an indication of what might have happened anyway, even without the treatment Explain that the control group provides a baseline for comparison, an indication of what might have happened anyway, even without the treatment

Rubric for Part B “Without the control group, the company could compare the two treatments, but would not be able to say whether the observed reduction in stress was attributable to participation in the programs. For example, a change in the work environment during this period might have reduced the stress level of all employees. The addition of a control group would enable the company to assess the magnitude of the mean reduction attributable to each treatment, as opposed to just determining if the two programs differ.” “Without the control group, the company could compare the two treatments, but would not be able to say whether the observed reduction in stress was attributable to participation in the programs. For example, a change in the work environment during this period might have reduced the stress level of all employees. The addition of a control group would enable the company to assess the magnitude of the mean reduction attributable to each treatment, as opposed to just determining if the two programs differ.”

Rubric for Part C Indicate that one cannot generalize, and give a plausible reason, such as... Indicate that one cannot generalize, and give a plausible reason, such as... The participants were volunteers and volunteers my not be representative of the population The participants were volunteers and volunteers my not be representative of the population The participants were not randomly selected from the population The participants were not randomly selected from the population

Rubric for Part C “No it is not, for this experiment we took volunteers but the problem with it is that the people who volunteered are very likely the ones who needed the stress reduction the most...Therefore, it is not reasonable to generalize because most likely the people who volunteered are not representative of the population.” “No it is not, for this experiment we took volunteers but the problem with it is that the people who volunteered are very likely the ones who needed the stress reduction the most...Therefore, it is not reasonable to generalize because most likely the people who volunteered are not representative of the population.”

Common Student Errors Did not understand the difference between random allocation of subjects and random sampling. Did not understand the difference between random allocation of subjects and random sampling. Often used the word "confounding" in part (a), but did not explain how the treatment results were mixed up with some other variable. Often used the word "confounding" in part (a), but did not explain how the treatment results were mixed up with some other variable.

Common Student Errors Seemed to think that a larger sample size would fix any problem in the experiment, rather than recognizing that the major problem of the experiment was that there was no random sampling of employees. Seemed to think that a larger sample size would fix any problem in the experiment, rather than recognizing that the major problem of the experiment was that there was no random sampling of employees. Incorrectly stated that random allocation "eliminates" bias. Incorrectly stated that random allocation "eliminates" bias.

Released Questions Work on 2001 Free Response #4 with your partner Work on 2001 Free Response #4 with your partner 2001 Free Response # Free Response #4

2001 #4

Rubric for Part A Blocking Scheme A is preferable Blocking Scheme A is preferable Creates homogeneous blocks with respect to forest exposure Creates homogeneous blocks with respect to forest exposure Plots will have similar forest exposure Plots will have similar forest exposure

Rubric for Part B Randomization within blocks should reduce bias due to the influence of confounding variables Randomization within blocks should reduce bias due to the influence of confounding variables (fertility of soil, moisture, etc.) (fertility of soil, moisture, etc.) on the productivity of the trees. on the productivity of the trees FR #4 Rubric 2001 FR #4 Rubric

Extension Questions How would you randomize trees within blocks? How would you randomize trees within blocks? What other confounding variables might impact the results? What other confounding variables might impact the results?