Validity of Quantitative Research Conclusions

Internal Validity External Validity Issues of Cause and Effect Issues of Generalizability Validity of Quantitative Research Conclusions

Issues of Cause and Effect Issues of Generalizability Validity of Quantitative Research Conclusions Internal Validity External Validity

Issues of Cause and Effect Issues of Generalizability Validity of Quantitative Research Conclusions Statistical Conclusion Validity Internal Validity Construct Validity External Validity

Statistical Conclusion Validity Is there a relationship between A & B? ThreatSolutions

Statistical Conclusion Validity Is there a relationship between A & B? ThreatSolutions Low Statistical Power Violated Assumptions of Statistics Fishing (Multiple Comparisons) Low Reliability of Measures Treatment Differs Across Occasions Random Confounds in the Setting Random Differences in Subjects

Internal Validity Is there a cause and effect relationship between A & B? ThreatSolutions

Internal Validity Is there a cause and effect relationship between A & B? ThreatSolutions History Maturation Testing Instrumentation Statistical Regression (to the Mean) Selection (and Interaction with Selection) Mortality Unknown Direction of Causation Diffusion of Treatment Compensatory Equalization of Treatment Compensatory Rivalry by Subjects Resentful Demoralization of Subjects

Construct Validity of Cause and Effect Assumption Is the cause and effect relationship between A & B? ThreatSolutions

Construct Validity of Cause and Effect Assumption Is the cause and effect relationship between A & B? ThreatSolutions Poor Representation of Constructs Single Operation Bias Single Method Bias Hypothesis Guessing by Subjects Evaluation Manipulation Experimenter Expectations Confounding Constructs and Measurement Level Multiple Treatment Confound Measurement and Treatment Interaction Restricted Generalization Across Possible Dependent Variables

External Validity Is the relationship between A and B generalizable? ThreatSolutions

External Validity Is the relationship between A and B generalizable? ThreatSolutions Sample does not represent the population of interest. The setting of a study does not represent all settings of interest. The time during which a study takes place does not represent all future times. (History and Treatment Interaction)

Research Design Choices Use large sample size. Choose powerful statistics tests. Meet assumptions of statistics. Choose robust statistics tests. Use a conservative post-hoc comparison correction procedure. Use internally reliable tests. Use group means, not individual scores. Correct for attenuation (low reliability). Precisely define & standardize treatment. Choose settings free of confounds. Use more than one dependent variable. Select subjects randomly from a well- defined population. Vary settings and analyze across settings. Replicate the same experiment at different times. Conduct a literature review to see if prior evidence is consistent with hypotheses. Measure potential confounds. Use a within-subjects (repeated Measures design. Randomly assign to treatment group. Share study rationale with subjects. Treat all subjects justly. Hide true hypotheses from subjects. Define constructs precisely. Use measures with high validity. Measure dependent variable with multiple measures. Manipulate independent variable in many ways. Use double-blind study design. Use continuous level of measurement. Give a single treatment to subjects. Measure effect of pretests on dependent variable.

Statistical Conclusion Validity Is there a relationship between A & B? ThreatSolutions Low Statistical Power Violated Assumptions of Statistics Fishing (Multiple Comparisons) Low Reliability of Measures Treatment Differs Across Occasions Random Confounds in the Setting Random Differences in Subjects

Statistical Conclusion Validity Is there a relationship between A & B? ThreatSolutions Low Statistical Power Use large sample size. Choose powerful statistics tests. Violated Assumptions of Statistics Fishing (Multiple Comparisons) Low Reliability of Measures Treatment Differs Across Occasions Random Confounds in the Setting Random Differences in Subjects

Statistical Conclusion Validity Is there a relationship between A & B? ThreatSolutions Low Statistical Power Use large sample size. Choose powerful statistics tests. Violated Assumptions of Statistics Meet assumptions of statistics. Choose robust statistics tests. Fishing (Multiple Comparisons) Low Reliability of Measures Treatment Differs Across Occasions Random Confounds in the Setting Random Differences in Subjects

Statistical Conclusion Validity Is there a relationship between A & B? ThreatSolutions Low Statistical Power Use large sample size. Choose powerful statistics tests. Violated Assumptions of Statistics Meet assumptions of statistics. Choose robust statistics tests. Fishing (Multiple Comparisons) Use a conservative post-hoc comparison correction procedure. Low Reliability of Measures Treatment Differs Across Occasions Random Confounds in the Setting Random Differences in Subjects

Statistical Conclusion Validity Is there a relationship between A & B? ThreatSolutions Low Statistical Power Use large sample size. Choose powerful statistics tests. Violated Assumptions of Statistics Meet assumptions of statistics. Choose robust statistics tests. Fishing (Multiple Comparisons) Use a conservative post-hoc comparison correction procedure. Low Reliability of Measures Use internally reliable tests. Use group means, not individual scores. Correct for attenuation (low reliability). Treatment Differs Across Occasions Random Confounds in the Setting Random Differences in Subjects

Statistical Conclusion Validity Is there a relationship between A & B? ThreatSolutions Low Statistical Power Use large sample size. Choose powerful statistics tests. Violated Assumptions of Statistics Meet assumptions of statistics. Choose robust statistics tests. Fishing (Multiple Comparisons) Use a conservative post-hoc comparison correction procedure. Low Reliability of Measures Use internally reliable tests. Use group means, not individual scores. Correct for attenuation (low reliability). Treatment Differs Across Occasions Precisely define & standardize treatment. Random Confounds in the Setting Random Differences in Subjects

Statistical Conclusion Validity Is there a relationship between A & B? ThreatSolutions Low Statistical Power Use large sample size. Choose powerful statistics tests. Violated Assumptions of Statistics Meet assumptions of statistics. Choose robust statistics tests. Fishing (Multiple Comparisons) Use a conservative post-hoc comparison correction procedure. Low Reliability of Measures Use internally reliable tests. Use group means, not individual scores. Correct for attenuation (low reliability). Treatment Differs Across Occasions Precisely define & standardize treatment. Random Confounds in the Setting Choose settings free of confounds. Measure potential confounds. Random Differences in Subjects

Statistical Conclusion Validity Is there a relationship between A & B? ThreatSolutions Low Statistical Power Use large sample size. Choose powerful statistics tests. Violated Assumptions of Statistics Meet assumptions of statistics. Choose robust statistics tests. Fishing (Multiple Comparisons) Use a conservative post-hoc comparison correction procedure. Low Reliability of Measures Use internally reliable tests. Use group means, not individual scores. Correct for attenuation (low reliability). Treatment Differs Across Occasions Precisely define & standardize treatment. Random Confounds in the Setting Choose settings free of confounds. Measure potential confounds. Random Differences in Subjects Use a within-subjects (repeated measures) design.

Internal Validity Is there a cause and effect relationship between A & B? ThreatSolutions History Maturation Testing Instrumentation Statistical Regression (to the Mean) Selection (and Interaction with Selection) Mortality Unknown Direction of Causation Diffusion of Treatment Compensatory Equalization of Treatment Compensatory Rivalry by Subjects Resentful Demoralization of Subjects

Internal Validity Is there a cause and effect relationship between A & B? ThreatSolutions History Randomly Assign to Treatment Group Maturation Randomly Assign to Treatment Group Testing Randomly Assign to Treatment Group Instrumentation Randomly Assign to Treatment Group Statistical Regression (to the Mean) Randomly Assign to Treatment Group Selection (and Interaction with Selection) Randomly Assign to Treatment Group Mortality Randomly Assign to Treatment Group Unknown Direction of Causation Randomly Assign to Treatment Group Diffusion of Treatment Compensatory Equalization of Treatment Compensatory Rivalry by Subjects Resentful Demoralization of Subjects

Internal Validity Is there a cause and effect relationship between A & B? ThreatSolutions History Randomly Assign to Treatment Group Maturation Randomly Assign to Treatment Group Testing Randomly Assign to Treatment Group Instrumentation Randomly Assign to Treatment Group Statistical Regression (to the Mean) Randomly Assign to Treatment Group Selection (and Interaction with Selection) Randomly Assign to Treatment Group Mortality Randomly Assign to Treatment Group Unknown Direction of Causation Randomly Assign to Treatment Group Diffusion of Treatment Share Study Rationale with Subjects Compensatory Equalization of Treatment Compensatory Rivalry by Subjects Resentful Demoralization of Subjects

Internal Validity Is there a cause and effect relationship between A & B? ThreatSolutions History Randomly Assign to Treatment Group Maturation Randomly Assign to Treatment Group Testing Randomly Assign to Treatment Group Instrumentation Randomly Assign to Treatment Group Statistical Regression (to the Mean) Randomly Assign to Treatment Group Selection (and Interaction with Selection) Randomly Assign to Treatment Group Mortality Randomly Assign to Treatment Group Unknown Direction of Causation Randomly Assign to Treatment Group Diffusion of Treatment Share Study Rationale with Subjects Compensatory Equalization of Treatment Treat All Subjects Justly Compensatory Rivalry by Subjects Resentful Demoralization of Subjects

Internal Validity Is there a cause and effect relationship between A & B? ThreatSolutions History Randomly Assign to Treatment Group Maturation Randomly Assign to Treatment Group Testing Randomly Assign to Treatment Group Instrumentation Randomly Assign to Treatment Group Statistical Regression (to the Mean) Randomly Assign to Treatment Group Selection (and Interaction with Selection) Randomly Assign to Treatment Group Mortality Randomly Assign to Treatment Group Unknown Direction of Causation Randomly Assign to Treatment Group Diffusion of Treatment Share Study Rationale with Subjects Compensatory Equalization of Treatment Treat All Subjects Justly Compensatory Rivalry by Subjects Resentful Demoralization of Subjects Hide True Hypotheses from Subjects

Internal Validity Is there a cause and effect relationship between A & B? ThreatSolutions History Randomly Assign to Treatment Group Maturation Randomly Assign to Treatment Group Testing Randomly Assign to Treatment Group Instrumentation Randomly Assign to Treatment Group Statistical Regression (to the Mean) Randomly Assign to Treatment Group Selection (and Interaction with Selection) Randomly Assign to Treatment Group Mortality Randomly Assign to Treatment Group Unknown Direction of Causation Randomly Assign to Treatment Group Diffusion of Treatment Share Study Rationale with Subjects Compensatory Equalization of Treatment Treat All Subjects Justly Compensatory Rivalry by Subjects Resentful Demoralization of Subjects Hide True Hypotheses from Subjects Treat All Subjects Justly

Construct Validity of Cause and Effect Assumption Is the cause and effect relationship between A & B? ThreatSolutions Poor Representation of Constructs Single Operation Bias Single Method Bias Hypothesis Guessing by Subjects Evaluation Manipulation Experimenter Expectations Confounding Constructs and Measurement Level Multiple Treatment Confound Measurement and Treatment Interaction Restricted Generalization Across Possible Dependent Variables

Construct Validity of Cause and Effect Assumption Is the cause and effect relationship between A & B? ThreatSolutions Poor Representation of ConstructsDefine constructs precisely. Use measures with high validity. Single Operation Bias Single Method Bias Hypothesis Guessing by Subjects Evaluation Manipulation Experimenter Expectations Confounding Constructs and Measurement Level Multiple Treatment Confound Measurement and Treatment Interaction Restricted Generalization Across Possible Dependent Variables

Construct Validity of Cause and Effect Assumption Is the cause and effect relationship between A & B? ThreatSolutions Poor Representation of ConstructsDefine constructs precisely. Use measures with high validity. Single Operation BiasMeasure dependent variable with multiple measures. Use measures with high validity. Manipulate independent variable in many ways. Single Method Bias Hypothesis Guessing by Subjects Evaluation Manipulation Experimenter Expectations Confounding Constructs and Measurement Level Multiple Treatment Confound Measurement and Treatment Interaction Restricted Generalization Across Possible Dependent Variables

Construct Validity of Cause and Effect Assumption Is the cause and effect relationship between A & B? ThreatSolutions Poor Representation of ConstructsDefine constructs precisely. Use measures with high validity. Single Operation BiasMeasure dependent variable with multiple measures. Use measures with high validity. Manipulate independent variable in many ways. Single Method BiasMeasure dependent variable with multiple methods. Hypothesis Guessing by Subjects Evaluation Manipulation Experimenter Expectations Confounding Constructs and Measurement Level Multiple Treatment Confound Measurement and Treatment Interaction Restricted Generalization Across Possible Dependent Variables

Construct Validity of Cause and Effect Assumption Is the cause and effect relationship between A & B? ThreatSolutions Poor Representation of ConstructsDefine constructs precisely. Use measures with high validity. Single Operation BiasMeasure dependent variable with multiple measures. Use measures with high validity. Manipulate independent variable in many ways. Single Method BiasMeasure dependent variable with multiple methods. Hypothesis Guessing by SubjectsHide True Hypotheses from Subjects Use double-blind study design. Evaluation Manipulation Experimenter Expectations Confounding Constructs and Measurement Level Multiple Treatment Confound Measurement and Treatment Interaction Restricted Generalization Across Possible Dependent Variables

Construct Validity of Cause and Effect Assumption Is the cause and effect relationship between A & B? ThreatSolutions Poor Representation of ConstructsDefine constructs precisely. Use measures with high validity. Single Operation BiasMeasure dependent variable with multiple measures. Use measures with high validity. Manipulate independent variable in many ways. Single Method BiasMeasure dependent variable with multiple methods. Hypothesis Guessing by SubjectsHide True Hypotheses from Subjects Use double-blind study design. Evaluation ManipulationUse measures with high validity. Experimenter Expectations Confounding Constructs and Measurement Level Multiple Treatment Confound Measurement and Treatment Interaction Restricted Generalization Across Possible Dependent Variables

Construct Validity of Cause and Effect Assumption Is the cause and effect relationship between A & B? ThreatSolutions Poor Representation of ConstructsDefine constructs precisely. Use measures with high validity. Single Operation BiasMeasure dependent variable with multiple measures. Use measures with high validity. Manipulate independent variable in many ways. Single Method BiasMeasure dependent variable with multiple methods. Hypothesis Guessing by SubjectsHide True Hypotheses from Subjects Use double-blind study design. Evaluation ManipulationUse measures with high validity. Experimenter ExpectationsUse double-blind study design. Confounding Constructs and Measurement Level Multiple Treatment Confound Measurement and Treatment Interaction Restricted Generalization Across Possible Dependent Variables

Construct Validity of Cause and Effect Assumption Is the cause and effect relationship between A & B? ThreatSolutions Poor Representation of ConstructsDefine constructs precisely. Use measures with high validity. Single Operation BiasMeasure dependent variable with multiple measures. Use measures with high validity. Manipulate independent variable in many ways. Single Method BiasMeasure dependent variable with multiple methods. Hypothesis Guessing by SubjectsHide True Hypotheses from Subjects Use double-blind study design. Evaluation ManipulationUse measures with high validity. Experimenter ExpectationsUse double-blind study design. Confounding Constructs and Measurement Level Use continuous (interval/ratio) level of measurement. Multiple Treatment Confound Measurement and Treatment Interaction Restricted Generalization Across Possible Dependent Variables

Construct Validity of Cause and Effect Assumption Is the cause and effect relationship between A & B? ThreatSolutions Poor Representation of ConstructsDefine constructs precisely. Use measures with high validity. Single Operation BiasMeasure dependent variable with multiple measures. Use measures with high validity. Manipulate independent variable in many ways. Single Method BiasMeasure dependent variable with multiple methods. Hypothesis Guessing by SubjectsHide True Hypotheses from Subjects Use double-blind study design. Evaluation ManipulationUse measures with high validity. Experimenter ExpectationsUse double-blind study design. Confounding Constructs and Measurement Level Use continuous (interval/ratio) level of measurement. Multiple Treatment ConfoundGive a single treatment to subjects, not a mix. Measurement and Treatment Interaction Restricted Generalization Across Possible Dependent Variables

Construct Validity of Cause and Effect Assumption Is the cause and effect relationship between A & B? ThreatSolutions Poor Representation of ConstructsDefine constructs precisely. Use measures with high validity. Single Operation BiasMeasure dependent variable with multiple measures. Use measures with high validity. Manipulate independent variable in many ways. Single Method BiasMeasure dependent variable with multiple methods. Hypothesis Guessing by SubjectsHide True Hypotheses from Subjects Use double-blind study design. Evaluation ManipulationUse measures with high validity. Experimenter ExpectationsUse double-blind study design. Confounding Constructs and Measurement Level Use continuous (interval/ratio) level of measurement. Multiple Treatment ConfoundGive a single treatment to subjects, not a mix. Measurement and Treatment Interaction Measure effects of pretests on dependent variable. Restricted Generalization Across Possible Dependent Variables

Construct Validity of Cause and Effect Assumption Is the cause and effect relationship between A & B? ThreatSolutions Poor Representation of ConstructsDefine constructs precisely. Use measures with high validity. Single Operation BiasMeasure dependent variable with multiple measures. Use measures with high validity. Manipulate independent variable in many ways. Single Method BiasMeasure dependent variable with multiple methods. Hypothesis Guessing by SubjectsHide True Hypotheses from Subjects Use double-blind study design. Evaluation ManipulationUse measures with high validity. Experimenter ExpectationsUse double-blind study design. Confounding Constructs and Measurement Level Use continuous (interval/ratio) level of measurement. Multiple Treatment ConfoundGive a single treatment to subjects, not a mix. Measurement and Treatment Interaction Measure effects of pretests on dependent variable. Restricted Generalization Across Possible Dependent Variables Use more than one dependent variable.

External Validity Is the relationship between A and B generalizable? ThreatSolutions Sample does not represent the population of interest. The setting of a study does not represent all settings of interest. The time during which a study takes place does not represent all future times. (History and Treatment Interaction)

External Validity Is the relationship between A and B generalizable? ThreatSolutions Sample does not represent the population of interest. Select subjects randomly from a well-defined population. The setting of a study does not represent all settings of interest. The time during which a study takes place does not represent all future times. (History and Treatment Interaction)

External Validity Is the relationship between A and B generalizable? ThreatSolutions Sample does not represent the population of interest. Select subjects randomly from a well-defined population. The setting of a study does not represent all settings of interest. Vary settings and analyze for relationships in each setting. The time during which a study takes place does not represent all future times. (History and Treatment Interaction)

External Validity Is the relationship between A and B generalizable? ThreatSolutions Sample does not represent the population of interest. Select subjects randomly from a well-defined population. The setting of a study does not represent all settings of interest. Vary settings and analyze for relationships in each setting. The time during which a study takes place does not represent all future times. (History and Treatment Interaction) Replicate the same experiment at different times. Conduct a literature review to see if prior evidence is consistent with hypotheses.