Research Methods in Psychology Complex Designs

 Experiments that involve two or more independent variables studies simultaneously at least one dependent variable  Simplest complex design one independent variable with two levels one dependent variable

Complex Designs, continued  Factorial combination combine independent variables in an experiment pair each level of one IV with each level of the other IV(s)‏ example:  Closer examination of the Dittmar, Halliwell, and Ive (2006) study: Barbie and young girls’ body image  They also examined the IV of grade in school (kindergarten, 1st, 2nd) as a natural groups variable

Complex Designs, continued  Factorial combination of 2 IVs version of picture book: Barbie, Emme, neutral grade: kindergarten, 1st, 2nd factorial combination: 9 conditions referred to as a “3 x 3”—2 IVs, each with 3 levels 9872nd 6541st 321 kindergarten Grade NeutralEmmeBarbie Version of Picture Book

Complex Designs, continued  Factorial combination allowed Dittmar et al. (2006) to examine overall effect of Version of Picture book  Barbie images caused greater body dissatisfaction than Emme and neutral images overall effect of Grade  body dissatisfaction increased as grade in school increased combined effect of both IVs together  results indicated interesting effects for combinations of grade and exposure to the images

Complex Designs, continued  Guidelines for identifying complex designs at least two IVs IVs can be independent groups designs  random groups, natural groups, matched groups IVs can be repeated measures designs  when independent groups and repeated measures designs are combined, it’s called a mixed design

Complex Designs, continued  Main effects overall effect of an IV in a complex design  effect on DV as if only that IV was studied  Interaction effects combined effect of IVs considered simultaneously An interaction effect occurs when the effect of an independent variable differs depending on the level of a 2nd independent variable.

Research Example  Kassin, Goldstein, and Savitsky (2003)‏ pp. 275–283 in text Research questions  Do interrogators’ expectations about a suspect’s guilt or innocence influence the interrogation tactics they use?  Do interrogators have a confirmation bias in which their initial beliefs about a suspect’s guilt cause them to interrogate more aggressively?

Research Example, continued  Research design complex design with 2 levels (a 2 x 2 design)‏  Interrogator Expectation (random groups)‏ guilty expectation innocent expectation  Suspect Status (random groups)‏ actual guilt actual innocence students participated as interrogators or suspects in a laboratory “mock crime”

Research Example, continued Dependent Variables  They measured many, we will focus on:  Number of guilt-presumptive questions the interrogator selects for the interview with suspect  Number of persuasive interrogation techniques used during the interview with the suspect  Ratings of the amount of effort the interrogator used to obtain a confession

Research Example, continued factorial combination of 2 x 2 design → 4 conditions Interrogators were led to believe the suspect was innocent and the suspect did not commit the crime Interrogators were led to believe the suspect was guilty and the suspect did not commit the crime Actual Innocence Interrogators were led to believe the suspect was innocent and the suspect actually committed the crime Interrogators were led to believe the suspect was guilty and the suspect actually committed the crime Actual Guilt Suspect Status InnocentGuilty Interrogator Expectation

Research Example, continued Hypothesis Based on behavioral confirmation theory, interrogators were predicted to behave toward the suspect in ways that were consistent with their belief of guilt or innocence. In turn, suspects were predicted to respond in ways that support the interrogator’s belief.

Research Example, continued  Kassin et al.’s (2003) findings Main effects A main effect is the effect of one IV, ignoring (or collapsing across) the effect of the other IV Two main effects are possible for each DV  Interrogator Expectation  Suspect Status

Research Example, continued Main effect of Interrogator Expectation  compare Guilty Expectation and Innocent Expectation  DV: number of guilt-presumptive questions Actual Innocence Suspect Status Actual Guilt Means for Interrogator Expectation InnocentGuilty Interrogator Expectation

Research Example, continued Means for Interrogator Expectation  Guilty: M = 3.62 ← ( ) ÷ 2  Innocent: M = 2.60← ( ) ÷ 2 A test of statistical significance revealed that these two means are statistically different.  Interrogators who suspected their suspect to be guilty chose more guilt-presumptive questions (M = 3.62) than interrogators who expected an innocent suspect (M = 2.60)‏

Research Example, continued Main effect of Suspect Status  compare Actual Guilt and Actual Innocence  DV: Number of Persuasive Techniques Means for Suspect Status Actual Innocence Suspect Status Actual Guilt InnocentGuilty Interrogator Expectation

Research Example, continued Means for Suspect Status  Actual Guilt: M = 7.15 ← ( ) ÷ 2  Actual Innocence: M = ← ( ) ÷ 2 A test of statistical significance revealed that these two means are statistically different  Interrogators who interviewed a suspect who was actually innocent used more persuasive techniques (M = 11.42) than interrogators who interviewed a suspect who was actually guilty (M = 7.15)‏

Research Example, continued  Interaction effects occurs when the effect of one independent variable differs depending on the level of a second independent variable Kassin et al.’s (2003) experiment  look at the effect of suspect status (actual guilt, innocence) at each level of interrogator expectation variable An initial approach to examine interaction effects is the subtraction method

Research Example, continued Interaction effect of Interrogator Expectation X Suspect Status  DV: Ratings for effort to obtain a confession Actual Innocence Suspect Status –0.29–1.53Difference Between Means Actual Guilt InnocentGuilty Interrogator Expectation

Research Example, continued Because the outcome of the subtraction method yielded very different values –1.53and – 0.29 an interaction effect between the IVs is likely. A test of statistical significance would be needed to confirm this.

Research Example, continued  Examine the means to understand the interaction effect When suspects were actually guilty, the effort to obtain a confession was not affected by whether the interrogator expected the suspect to be guilty (M = 5.64) or innocent (M = 5.56)‏  Not a statistically significant difference However, when the suspects were actually innocent, the effort to obtain a confession was greater when the interrogator expected a guilty suspect (M = 7.17) compared to when the interrogator expected an innocent suspect (M = 5.85)‏  A statistically significant difference

Research Example, continued Interrogators differed in their effort to obtain a confession depending on their expectations and whether the suspect was actually guilty or innocent = an interaction effect between Suspect Status and Interrogator Expectation independent variables The effect of one IV differed depending on the level of the 2nd IV  this is the definition of an interaction effect

Research Example, continued  Graphs (“Figures”) can be used to detect interaction effects easily An interaction effect is likely when lines in the graph that display the means  are not parallel  that is, the lines either intersect, converge, or diverge However, a statistical test is always used to determine whether an interaction is statistically significant

Research Example, continued DV: Ratings of effort to obtain a confession (means)‏

Research Example, continued  ANOVA summary table Statistical significance  p <.05 Information in summary table is only useful in conjunction with descriptive statistics (e.g., means) for each condition of the experiment. Example  DV: effort to obtain a confession  Statistically significant effects: Interaction effect of Interrogator Expectation X Suspect Status main effect of Interrogator Expectation main effect of Suspect Status

Research Example, continued ANOVA Summary Table for DV: Effort to obtain a confession SourcedfSSMSFpeta2 Interrogator Expectation Suspect Status Interrogator Expect. X Suspect Status Error _______________________________________ Total

Analysis of Complex Designs  Steps for Data Analysis Check the data for errors and outliers Summarize the results using descriptive statistics  Factorial design tells you who many means you need to analyze e.g., a 2 x 2 → 4 conditions (4 means)‏  Graph the means

Analysis of Complex Designs, continued Confirm what the data reveal.  The means in an experiment will not all be the same  There will be some variability  Key question: Is the variability greater than chance (error variation)?  Variability greater than chance is attributed to the effect of the independent variable(s)‏  Null hypothesis testing is used to decide whether the IVs produced an effect on the DVs  In complex designs, Analysis of Variance is used

Analysis of Complex Designs, continued  Analysis of Variance (ANOVA)‏ tells us whether main effects and interaction effects are statistically significant when an effect is statistically significant  we say IV caused the effect of DV  assuming experiment is internally valid results are presented in an ANOVA Summary Table  statistics in ANOVA Summary Table are only useful when descriptive statistics (e.g., mean) are also considered

Analysis of Complex Designs, continued  How are these effects reported? Main effect of Interrogator Expectation On average, interrogators who expected a guilty suspect worked harder to obtain a confession than interrogators who expected an innocent suspect (Ms = 6.40 and 5.71, respectively), F(1, 294) = 4.96, p =.027, d =.26 Main effect of Suspect Status On average, interrogators exerted more effort to obtain a confession when the suspect was innocent (M = 6.51) than when the suspect was guilty (M = 5.60), F(1, 294) = 8.33, p =.004, d =.34

Analysis of Complex Designs, continued Interrogator x Suspect Status interaction effect: The interrogator Expectation x Suspect Status interaction was statistically significant, F(1, 294) = 3.93, p =.048, η 2 =.029. Interrogators worked hardest to obtain a confession when they expected a guilty suspect and interviewed a suspect who was actually innocent (M = 7.17). The mean rating for this cell was statistically greater than the other three cells, which did not differ significantly from each other.

Analysis of Complex Designs, continued Omnibus ANOVA  initial test of main effects and interaction effects If interaction effect is statistically significant,  conduct follow-up or “post-hoc” tests of statistical significance, such as  simple main effects  comparisons of two means

Analysis of Complex Designs, continued  Guidelines for the analysis of a complex design experiment Step 1: determine whether interaction effects are statistically significant  in a 2-factor experiment, only 1 interaction effect is possible Step 2: If interaction effect is statistically significant, identify source of interaction  simple main effects and comparisons of two means Then examine whether the main effects of each independent variable are statistically significant

Analysis of Complex Designs, continued  Interaction effects Definition  Effect of one independent variable differs depending on the level of a second independent variable Analyze the simple main effects to determine source of interaction

Analysis of Complex Designs, continued Simple main effects  the effect of one independent variable at one level of a 2nd IV  for example, the effect of the suspect status IV in the Expect-Guilty condition or the Expect-Innocent condition

Analysis of Complex Designs, continued Simple main effect of Suspect Status for the Guilty-Expectation condition Statistically significant

Analysis of Complex Designs, continued Simple main effect of Suspect Status for the Innocent-Expectation condition Not statistically significant

Analysis of Complex Designs, continued  Two other simple main effects in Kassin et al.’s experiment the simple main effect of Interrogator Expectation for Guilty Suspects the simple main effect of Interrogator Expectation for Innocent Suspects Which is statistically significant?

Interaction Effects and Theory Testing  Kassin et al. (2003) showed support for behavioral confirmation theory “Interrogator expectations [triggered] a range of behavioral confirmation effects, ultimately biasing perceptions of guilt … leading them to exert the most pressure on innocent suspects” (Kassin et al., 2003, p. 199)‏

Interaction Effects and External Validity  Interaction effect is not statistically significant generalize findings across conditions of experiment example  Kassin et al.’s findings for number of persuasive techniques used by interrogators  Interrogator Expectation x Suspect Status interaction not statistically significant

Interaction Effects and External Validity, continued Interrogators used more persuasive tactics when the suspect was actually innocent than when the suspect was actually guilty  This was true in the guilty-expectation condition and the innocent-expectation condition Effect of suspect status generalized across the levels of interrogator expectation

Interaction Effects and External Validity, continued The presence of a statistically significant interaction effect sets limits on the external validity of a finding  example: Suspect Status x Interrogator Expectation interaction for number of persuasive tactics Not all interrogators who expected a suspect to be guilty exerted a high degree of effort to obtain a confession We can’t generalize findings for effort across guilty and innocent suspects, or across interrogator expectation  the number of persuasive tactics depends on suspect status and interrogator expectations

Interaction Effects and Ceiling/Floor Effects  Floor and ceiling effects Sometimes an interaction effect can be statistically significant “by mistake”  This occurs when the means for one or more condition(s) reach, on average, near the highest possible score (ceiling effect)‏ the lowest possible score (floor effect)‏ When floor or ceiling effects occur, an interaction effect is uninterpretable

Ceiling Effect, example interaction effect between Task Difficulty (easy, hard) and Study Hours (10, 15)‏  hours of study had an effect only in the hard- test condition, not in the easy-test condition  How do we interpret this interaction when we know the highest possible test score is 50?

Ceiling Effect, example If we have enough “room” in our DV to assess the effect of the IV, the interaction effect disappears This graph shows two main effects: Study Hours and Test Difficulty

Interaction Effects and Natural Groups Design  With complex designs, researchers can test causal inferences for natural groups variables  but wait … isn’t it impossible to make causal inferences for natural groups variables?  natural groups variables are correlational  so, how does one make causal inferences? Test a theory for why the natural groups differ

Interaction Effects and Natural Groups Design, continued  Steps for making causal inferences about natural groups variables using complex designs State your theory.  Why do the groups differ? What is the theoretical process? Identify a relevant independent variable.  This IV should influence the likelihood that the theorized process will occur

Interaction Effects and Natural Groups Design, continued Look for an interaction effect.  The natural groups variable and manipulated IV should produce a statistically significant interaction effect in the predicted direction  This interaction effect allows a causal inference about why individuals differ