1. Programmatic Research
Re-introduction to Programmatic research
Re-introduction to Multivariate research
Factorial designs  “It Depends”
The importance of interactions
Application of Factorial Designs in Programmatic research
Multiple regression  “Unique Contributions”
Importance of collinearity & suppressor effects
Application of Multiple regression in programmatic research
Path Analysis  “Direct & Indirect Effects
Meta Analysis – Organizing & Summarizing Programmatic Research

2. Library Research
Learning “what is known” about the target behavior
Hypothesis Formation
Based on Lib. Rsh., propose some “new knowledge”
Research Design
Determine how to obtain the data to test the RH:
the “Research Loop”
Novel RH:
Replication
Convergence
Data Collection
Carrying out the research design and getting the data.
Draw Conclusions
Decide how your “new knowledge” changes “what is known” about the target behavior
Does a bivariate finding “hold up” within a multivariate examination ???
Data Analysis
Data collation and statistical analysis
Hypothesis Testing
Based on design properties and statistical results

3. Why multivariate research designs?  Multicausality
During the first lecture of this section we talked about the importance of going beyond bivariate research questions and statistical analyses to multivariate questions and analyses.
Why multivariate research designs?  Multicausality
Multicausality is the idea that behavior is complicated – that any behavior has multiple causes, and so, can be better studied using multivariate research designs with multiple IVs than with bivariate research designs with only a single IV!!!
So, multivariate research can be used to ….
involve multiple IVs in a single study  to get a more complete picture of the interrelationships among the behaviors we are studying
“check up” on previous results from bivariate research  to see if the results we got “hold up” within a multivariate context
is “the effect” we found with the bivariate analysis what we “thought it was”?
Want to design that little “it depends” warm up to this lecture??????

4. Studied using Factorial Designs Unique contributions
As we’ve discussed, there are two fundamental questions about multicausality that are asked in multivariate research…
Interactions
does the effect of an IV upon the DV depend upon the value of a 2nd IV?
Studied using Factorial Designs
Unique contributions
Does an IV tell us something about a DV that other IVs don’t?
Studied using Multiple Regression
Causal Structures
Is a DV an IV for another DV?
Behaviors are effects of some things and causes of others
Structural Modeling & Path Analysis
Want to design that little “it depends” warm up to this lecture??????
Again, all of these are used to because bivariate “snapshots” of complex behaviors can be incomplete & inaccurate!

5. Using Factorial Designs in Programmatic Research I
Adding a 2nd Treatment
Perhaps the most common application of factorial designs it so look at the separate (main) and combined (interaction) effects of two IVs
Often our research starts with a simple RH: that requires only a simple 2-group BG research design.
Keep in mind that to run this study, we made sure that none of the participants had any other treatments !
Computer Lecture

6. Factorial Designs – Separate (Main) and combined (interaction)
Factorial Designs – Separate (Main) and combined (interaction) effects of two treatments
At some point we are likely use Factorial designs to ask ourselves about how a 2nd IV also relates to the DV
Gets Lect & Hw
Gets both Comp & Hw
Comp Lect
Homework
Exam Prep
Gets Comp & Prep
Gets neither Lect & Prep

7. Using Factorial Designs in Programmatic Research II
“Correcting” Bivariate Studies
High% Low%
Our well sampled, carefully measured, properly analyzed study showed …
… nothing ! 40 40
Male Female
Our well sampled, carefully measured, properly analyzed study showed …
… nothing ! 40 40
Looks like neither IV is related to the DV !!!

8. Using Factorial Designs in Programmatic Research I
High% Low%
Male
Female 40 60 20
However, when we analyzed the same data including both variables as IVs … 20 60 40 40
There are High-Low effects both for Females & for Males – the marginal means are an “aggregation error”
There are Gender effects both High% & Low% – the marginal means are an “aggregation error”
So, instead of the “neither variable matters” bivariate results, the multivariate result shows that both variables related to the DV t -- they interact !!!!!

9. Using Factorial Designs in Programmatic Research III
Between Groups Factorial Designs:
Generalizability across Populations, Settings & Tasks
Often our research starts with a simple RH: that requires only a simple 2-group BG research design.
Keep in mind that to run this study, we had to make some choices/selections:
For example:
population  College Students
setting  Lecture setting
stim/task  teach Psychology
Computer Lecture

10. Using Factorial Designs in Programmatic Research III
When we’ve found and replicated an effect, making certain selections, it is important to check whether changing those selections changes the results.
Computer Lecture 60 40
If there is no interaction – if the results “don’t depend upon” the population, task/stimulus, setting, etc – we need to know that, so we can apply the results of the study to our theory or practice, confident in their generalizability
If there is an interaction – if the results “depend upon” the population, task/stimulus, setting, etc – we need to know that, so we can apply the “correct version” of the study to our theory or practice

11. Between Groups Factorial Designs – Do effects “depend upon”
Between Groups Factorial Designs – Do effects “depend upon” Populations, Settings or Tasks ?
At some point we are likely use BG Factorial designs to ask ourselves how well the results will generalize to:
Tx Control
Col HS
other populations – college vs. high school
other settings – lecture vs. laboratory
other tasks/stimuli – psyc vs. philosophy
Tx Control
Tx Control
Lecture
On-line
Psyc
Phil

12. Tx Control 60 40 60 40 ?? ?? Tx Control 60 40 ?? ?? Tx Control 60 40
Notice that each factorial design includes a replication of the earlier design, which used the TX instructional methods to :
teach Psychology
to College Students
in a Lecture setting
Tx Control
Psyc
Phil 60 40
Tx Control 60 40 ?? ??
Tx Control
Col HS 60 40 ?? ??
Tx Control
Each factorial design also provides a test of the generalizability of the original findings:
w/ Philosophy vs. Psychology
to High School vs. College Students
in an On-line vs. Lecture setting
Lecture
On-line 60 40 ?? ?? ?? ??

13. Using Factorial Designs in Programmatic Research IV
Mixed Groups Factorial Designs:
Time-course research
Tx Tx2
As before, often our research starts with a simple RH: that requires only a simple 2-group BG research design. 20 40
Time Course Investigations
In order to run this study we had to select ONE treatment duration (say 16 weeks):
we assign participants to each condition
begin treatment of the Tx groups
treat for 16 weeks and then measured the DV

14. Mixed Groups Factorial Designs for Time Course Investigations
Tx Tx2
Using this simple BG design we can “not notice” some important things. A MG Factorial can help explore the time course of the Tx effects. 20 40
16 wks wks
By using a MG design, with different lengths of Tx as the 2nd IV, we might find different patterns of data that we would give very different interpretations
Tx1
Tx2 20 20 40 40
16 wks wks
16 wks wks
16 wks wks
Tx1
Tx2
Tx1
Tx2
Tx1
Tx2 20 40 20 60 20 40 40 40 40 40 60

15. Evaluating initial equivalence if RA is not possible
Using Factorial Designs in Programmatic Research V
Mixed Groups Factorial Designs:
Evaluating initial equivalence if RA is not possible
Tx Tx2
As before, often our research starts with a simple RH: that requires only a simple 2-group BG research design.
Initial Equivalence Investigations
In order to causally interpret the results of this study, we’d have to have initial equivalence
but we can’t always RA & manipulate the IV
So what can we do to help interpret the post-treatment differences of the two treatments?
Answer – compare the groups before treatment too!

16. Mixed Groups Factorial Designs to evaluate Initial Equivalence
By using a MG design, we can compare the groups pre-treatment and use that information to better evaluate post-treatment group differences (but can’t really infer cause). For which of these would you be more comfortable conclusing that Tx1> Tx2 ??
Pre Post
Tx1
Tx2 40 40 20 20
Nah – Post dif = pre dif !
Pre Post
Pre Post
Pre Post
Tx1
Tx2
Tx1
Tx2 20 40 30 60
Tx1
Tx2 60 40 20 20 20 20 20 40
As good as it gets!
Nah – Tx1 lowered score
Maybe – more in- crease by Tx1

17. Multiple Regression in Programmatic Research …..
We often perform both bivariate (correlation) and multivariate (multiple regression) analyses – because they tell us different things about the relationship between the predictors and the criterion…
Correlations (and bivariate regression weights) tell us about the “separate” relationships of each predictor with the criterion (ignoring the other predictors)
Multiple regression weights tell us about the relationship between each predictor and the criterion that is unique or independent from the other predictors in the model.
What does this predictor tell us about the criterion that no other predictors tells us?
Is “the predictor” we studied with the bivariate analysis “what we thought it was” ???

18. There are 5 patterns of bivariate/multivariate relationship
Simple correlation with the criterion
Bivariate relationship and multivariate contribution (to this model) have same sign
“Suppressor effect” – bivariate relationship & multivariate contribution (to this model) have different signs
“Suppressor effect” – no bivariate relationship but contributes (to this model)
Multiple regression weight
Non-contributing – probably because colinearity with one or more other predictors
Non-contributing – probably because of weak relationship with the criterion
Non-contributing – probably because colinearity with one or more other predictors
Can we give a general definition of a suppressor variable somewhere in here???
Bivariate relationship and multivariate contribution (to this model) have same sign
“Suppressor effect” – bivariate relationship & multivariate contribution (to this model) have different signs
“Suppressor effect” – no bivariate relationship but contributes (to this model)

19. Among these 9 outcomes are 5 “kinds” of results
Bivariate relationship and multivariate contribution (to this model) have same sign -- bivariate results “still look ok”
Non-contributing – probably because of weak relationship with the criterion -- bivariate results “still look ok”
Non-contributing – probably because collinearity with one or more other predictors -- variables has no unique information that is unique to the others in the model
“Suppressor variable” – no bivariate relationship but contributes (to this model) -- bivariate results “missed” an important variable
“Suppressor variable” – bivariate relationship & multivariate contribution (to this model) have different signs -- bivariate results “got the relationship wrong”
Look at “Work hours”. We’re saying it’s a suppressor b/c no bivariate rel but contributes to the model --- BUT the p-value we give is .04
See Work hours – p-value for r shows it is a contributor – also sign is opposite b – are you trying to throw a nice slow pitch? If so, make r and b have the same sign and change p-value on r so it’s >.05.

20. predictor ice cream temp UGPA sales r(p) .32 (<.01) .55 (<.001)
An Example ….
We discussed the “curious” bivariate finding of a substantial relationship between ice cream sales & violent crimes, say r(363) = .32, p < .01
Violent crimes
Ice cream sales
Our hypothesis was that this finding is a result of a third variable “temperature” that is causing both violent crimes and ice cream sales. To test this we might put both temperature and ice cream sales into a multiple regression model to predict violent crimes.
Ice cream sales has a nice correlation; temperature has a stronger correlation
predictor ice cream temp UGPA sales
r(p) (<.01) (<.001)
However, when we put them both into the model, only temperature has a unique contribution. Why??
b(p) (.82) (.01)

21. Here’s the Venn diagram for this multivariate model = violent crimes
= ice cream sales
= temperature
Both ice cream sales and temperature are correlated with violent crimes. But notice that they are highly correlated with each other (collinear)!
This collinearity means that part of what each predictor shares with the criterion they also with the other predictor.
The result of this collinearity is…
-- most of what ice cream sales shares with violent crimes it shares with temperature, resulting in a small b value for ice cream sales
Look at “Work hours”. We’re saying it’s a suppressor b/c no bivariate rel but contributes to the model --- BUT the p-value we give is .04
See Work hours – p-value for r shows it is a contributor – also sign is opposite b – are you trying to throw a nice slow pitch? If so, make r and b have the same sign and change p-value on r so it’s >.05.
-- however, much of what temperature shares with violent crimes is independent of ice cream sales, resulting in a large b value for temperature

22. Another Example ….
Nearly everybody who looks for it finds a relationship between “practice” and “performance”.
For example, in a recent semester the correlation between % EDU homeworks completed and Exam 1% grade was r(127) = .33, p<.01. This would be interpreted as, “Those who completed more of the EDU homeworks tended to have higher grades on Exam 1.
However, this is not an experimental study, so the “EDU effect” may be confounded by lots of other variables.
While we can’t measure or even think all of the possible confounding variables, we can consider what are things that might be related to both % EDUs completed and Exam scores ????
We chose 3 for study: motivation, exam study time, GPA

23. Bivariate & Multivariate contributions – DV = Exam 1% grade
predictor Motiv St. Time GPA % Pink
r(p) (<.01) (<.01) (<.01) (<.01)
All of these predictors have substantial correlations with Exam grades!!
β(p) .32(.02) (.04) (.51) (.01)
GPA does not have a significant regression weights – after taking the other variables into account, it has no unique contribution!
Exam study time has a significant regression weight, however, notice that it is part of a suppressor effect! After taking the other variables into account, those who study more for the test actually tend to do poorer on the exam.
%Pink does have a significant regression weight. Even after taking the other variables into account, those who do more EDU exercises tend to do better on the exam.
Look at “Work hours”. We’re saying it’s a suppressor b/c no bivariate rel but contributes to the model --- BUT the p-value we give is .04
See Work hours – p-value for r shows it is a contributor – also sign is opposite b – are you trying to throw a nice slow pitch? If so, make r and b have the same sign and change p-value on r so it’s >.05.
Motivation does have a significant regression weight. Even after taking the other variables into account, those who are more motivated tend to do better on the exam.
Notice that only two of the 4 predictors had the same “story” from the bivariate and multivariate analysis!!!!

24. GPA  no direct effect – but indirect effects thru %pink & St Time
Path Analysis – allows us to look at how multiple predictors relate to the criterion – considering both “direct” and “indirect” relationships!!
Direct effects
(same as MReg βs)
Motiv
-.31
St Time
-.25
.33
.32
Indirect effects
Exam 1%
%Pink
.58
GPA
.21
GPA  no direct effect – but indirect effects thru %pink & St Time
Motiv  direct effect – also indirect effects thru %pink & St Time
%Pink  direct effect – also indirect effect thru St Time
-β for St Time? Less %Pink predicts more St Time, suggesting that those who study more were those who did less work before they started to study for the exam, and they also did poorer on the exam!

25. Literature reviews & meta analyses …
The Introduction to any research paper includes 3 basic parts:
The purpose of the research
A review of the related literature
A statement of the research hypotheses about the relationships among characteristics and/or behaviors
Most literature reviews address the reliability, size and/or importance of the effects under consideration, using a combination of significance testing and effect sizes.
Sometimes, even after considerable research in an area, there is dispute or debate about “whether or not there is an effect”…
Sometimes there is dispute or debate about “the right way” to run a study – with the implication that different methods produce different results…
These are the two questions meta analysis can help answer!

26. Combining studies to answer the questions. “How large is the effect
Combining studies to answer the questions “How large is the effect?” & “Is the effect significant ?”
As you know, the size of the effect and the size of the sample combine to lead to the statistical significance of the result!
The study was designed to test the RH: that, “Students with faster reading speeds get higher test grades.” Eighty 4th grade students from a local elementary school were recruited for the study. Each student was tested using the Elementary Reading Speed Measure (ERSM) and an aggregate academic score (AAS) was composed from their math and social studies grades. The correlation between these measures was found to be r(78) = .38, p = .001.
This is a “medium” effect size, and it is statistically significant (only 1/1000 chance of a Type I error.
So far, so good ….

27. None have p < .05… But look at the r-values!!!
Upon hearing the results of this study, several teachers decided to replicate the study in their own classes. Each administered the ERMS and correlated it with exam scores. The results are listed below:
The author of the original research was disheartened at these findings!
Nine replication studies & none gave confirmation of the results of the original study !?!?
Carter Elementary r(28) = .36, p = .063
Fartner Elementary r(16) = .46, p = .082
Cressewell Elementary r(31) = .38, p = .057
Lettrennth Elementary r(17) = .36, p = .107
Kostplen Elementary r(23) = .37, p = .062
Kostplen Elementary r(12) = .45, p = .101
Planary Elementary r(20) = .38, p = .121
Madison Elementary r(16) = .37, p = .213
Bellemiso Elementary r(15) = .47, p = .059
Let’s not be too hasty!
None have p < .05… But look at the r-values!!!
Each of these effect size is comparable to the original study, or larger (r=.38) ! If so, why were none of these effect significant?

28. One of the important uses of meta analysis is to ask, what does a set of studies tell us about the likely effect size and significance of the relationship being studied?
It does this by combining the results from multiple studies into a single quantitative estimate of the represented effect size!
Then, you can perform a significance test of this estimate, taking advantage of the sample size from the combined studies!!!
In 1952 Hans J. Eysenck reviewed the available literature and concluded that there were no favorable effects of psychotherapy, starting a raging debate. Hundreds of studies run over the next 20 years failed to resolve the debate!!!
In 1978 Gene V. Glass statistically aggregated the findings of 375 psychotherapy outcome studies (building on earlier statistical models developed by Fisher and Cochran) and concluded that psychotherapy did indeed work !!!
Glass called his method “meta-analysis”

29. With N=196, no surprise that r=.39 is significant !!!
How can we apply this to the reading speed – exam performance data?
We put in the effect sizes and sample sizes from each study.
There’s some math…
We get an estimate of the population effect size aggregated from the studies.
And a significance test based on the total sample size.
With N=196, no surprise that r=.39 is significant !!!

30. Even though I’ve worked hard to sell the importance of replication, it doesn’t always work that way…
Glass concluded..
“A scientific study should be designed and reported in such a way that it can be replicated by other researchers. However, researchers seldom attempt to replicate previous findings. Instead, they pursue funding for the new, the novel… The result can be an overwhelming number of studies on a given topic, with no two studies exactly alike.” (
Extensions of meta analyses allow us to compare studies, trying to find what study attributes are related to the size and significance of the effect

31. Remember that we pointed out that, rejecting the idea of a single “critical experiment,” there were many, many different ways to run a particular study of how an IV and a DV are related within a given population…
different types of research design
different manipulations of the IV
different measurements of the DV
different setting
different task/stimulus
The simplest type of meta analytic comparison is much like an ANOVA, you have 2 (or more) ways of running the studies, and want to know if that difference ways of running the study produce different effect sizes.
In these comparative analyses:
the “case” is the individual study
the “DV” is the effect size
the “IV” is how the studies differ!!!

32. Applying this to the reading speed studies…
Some of the studies used Social Studies tests, and others used Math tests – could the type of test change the resulting effect size?
We put in the effect sizes from the studies that use Social Studies exams and those that used Math exams.
There’s some math…
We get an estimate of the population effect size from each type of study.
And a significance test the effect size difference.

33. The results of these studies help researchers:
Meta analytic studies of what leads different studies to find different effect sizes can involve hundreds of studies, several study-difference variables, and sophisticated multivariate models!
Population
Study Effect Size
DV oper.
Task
IV oper.
Setting
Stimulus
Design
When ?
(Soc Temp)
The results of these studies help researchers:
understand the rich research literature of an area of study
decide the best ways to conduct future research studies!!!