Statistical Analyses & Threats to Validity Lab 9—Last Lab Statistical Analyses & Threats to Validity

Background Recall from last lab that there are two major types of designs. If you are interested in the association between continuous variables, you should use a correlational design. If you are interested in comparing two (or more) groups, it is likely you will use an experimental design. The design you use will determine: statistical analysis you will perform threats to validity you will most likely need to address conclusions you can make from your research.

Correlational/Observational Design In this design, you are not comparing groups. Rather, you are examining the association between two continuous variables. You can use either correlation or regression analysis to analyze your data.

Correlation Used when interested in assessing the strength of association between two variables Used with continuous variables Example research hypothesis: are higher levels of self-esteem associated with lower levels of depression (what direction correlation would be expected?)

Regression Simple Linear Regression is a statistical method concerned with the amount of variance that one variable accounts for in another variable (that is, the amount of shared variance between variables). You also want to use regression when you are predicting one variable from the other. Used when you have continuous variables (e.g., motivation measured on a Likert scale, body weight).

Regression

Correlation vs. Regression Both correlation and regression are concerned with the amount of variance that variables share The major difference is that with correlation we are only interested in an association between variables, but with regression we assign one variable to be the independent variable (x; the predictor) and the other to be the dependent variable (y; the criterion) Note: While we typically are concerned with predictive relationships when using regression, the certainty with which we can say that two variables are causally related depends upon the design (true experiments are best!...why?) and quality of the study

Comparing Groups If you have an experimental design (manipulation of IV), you should use the following analyses to compare groups (non-continuous variables). In some cases, you will want to compare groups even when you don’t manipulate anything. Comparing male versus female on performance. Examining differences in opinion of various political or religious groups.

Independent samples (between-groups) t-test Used to determine dependent variable mean differences between two different groups Used with categorical or dichotomous variables (e.g., gender, democrat or republican, yes/no responses) Only one independent variable with only two levels/groups Example research hypothesis: do women have higher levels of depression on average than do men?

Analysis of Variance (ANOVA) Like t-test, used to determine dependent variable mean differences between groups/levels, but can use when there are more than two groups (which is the same as more than two levels of your IV) Remember, though, you’re still only using one IV, just there are more than two groups/levels (e.g., you might have the IV of political party, with the groups Democrat, Republican, and Independent)

Analysis of Variance (ANOVA) Can use for your final project if you have just one IV with more than two groups (i.e., one-way ANOVA), but can be used for more than one IV (i.e. two-way ANOVA, where say you have political party [3 groups] that may differ by gender [2 groups] for a 2x3 two-way ANOVA) Independent variable(s) have two or more levels; if you only have one IV with two levels, though, use a t-test Example research question: Do psychology graduate students have higher IQs than medical students, physics graduate students, and English graduate students?

Repeated measures (within-groups) t-test or ANOVA Used to test mean differences in the DV for the group measured at two or more times (e.g., the same group of clients is exposed to every type of counseling [the IV] and symptoms [the DV] are measured) Remember, the t-test involves an independent variable with two levels; ANOVA involves independent variable(s) with more than two levels. Population parameters and sample statistics the same as between-group t-test and ANOVA above.

Repeated measures (within-groups) t-test or ANOVA Example research hypothesis for repeated-measures t- test: Do people like the taste of Pepsi more than Coke (same people would taste both types of pop)? Example research hypothesis for repeated-measures ANOVA: Does exposure to death metal music lead to better cognitive performance than exposure to jazz, classical, or country music (same people would listen to each type of music [IV] and perform a cognitive task [DV])?

Threats to validity

Threats to Internal & External Validity Keep in mind that in science it is always important to balance multiple concerns. Ethics: Harm versus Benefit Sampling: Representativeness versus Practicality When choosing a research design, it is important to address whether: You can address or eliminate alternative explanations for your results (Internal Validity). You can generalize your results (External Validity). Basically, you want to 1) make your arguments as strong as possible and as widely applicable as possible while also 2) not overstating your conclusions or misleading consumers of your research.

Research Designs & Validity In general, experiments with random assignment are high on internal validity. Why? (What does random assignment do?) Correlational studies are often higher than experiments on external validity but lower on internal validity. Ideally, you would want studies to both be high internal and external validities. No single method does this. Multiple methods important.

An example You are interested in studying why individuals help each other. You believe that sadness leads to helping because it makes individuals feel better. Let’s say you conduct your study on September 12, 2001. How do you think this would influence your results? To the extent that other explanations (outside of sadness) can account for the associations that you find, your internal validity is threatened.

Internal Validity Threats Given that the IV is related to the DV, is it plausibly causal from one operational variable to another? History – is there something about the experience & history of one group that could lead to differences? Maturation – would natural development influence the associations? Testing – does actual participation in the study change behavior or responses? Instrumentation – what happens if the instruments you use change during the course of a study or if different groups of participants interpret your questions differently? Regression towards the mean – if you start with extreme groups, they are likely to be less extreme at another time point. Selection – are participants in your study somehow different than those who are not? Mortality – those who drop out of your study could be qualitatively different than those who remain.

Internal Validity Threats Given that the IV is related to the DV, is it plausibly causal from one operational variable to another? Diffusion or imitation – what would happen if participants in the different conditions communicated? Interaction with Selection – what if any of the above happen differentially for one group? Direction of causality – can the association happen in the opposite direction than what you predicted? Compensatory equalization of treatments – what if experimenters know what conditions participants are in? Compensatory rivalry – what would happen if participants in the different conditions started competing? Resentful demoralization – is it possible that participants in one condition give up?

External Validity Threats Given that there is probably a causal relationship from construct A to construct B, how generalizable is this relationship across persons, settings, and times? Interaction of selection and treatment (sample to population) Interaction of setting and treatment Interaction of history and treatment (time)

For your paper You will need to discuss the internal and external threats to validity in the discussion of your paper. Keep in mind that no study is perfect. However, when you bring up threats to validity, you MUST explain: How it is a threat to validity How your results would change if the threat were addressed What you could do to address the threat In other words, DO NOT simply state the threat.

Homework 9 For each of the three research hypotheses described below, list what type of statistical test you would use to test it (independent (between-groups) t- test, between-groups ANOVA, repeated measures t-test, repeated measures ANOVA, correlation, regression), and whether external or internal validity is most threatened and why (6 points; 2 for each hypothesis). Research Hypotheses: Women who experience a great deal of stress during pregnancy are more likely to give birth to low birth weight or premature babies than pregnant women who do not experience a great deal of stress. The degree to which someone believes in God relates to his/her psychological well-being. Further, the more someone believes in God the higher his/her well-being will be.

Homework 9 Research Hypotheses: A new manualized (standardized) treatment for depression will decrease depressive symptoms in a group of people seeking help at a mental health clinic. They will be assessed when they come into the clinic, immediately after the treatment is completed, and 6 months after treatment is complete. Lastly, describe your hypothesis, what design you will use for your project (from previous homework), what statistical analysis you plan to conduct, and identify 2 threats to validity (any combination of external or internal) and how they could influence the interpretation of your results. (4 points)