Experimental Design

Check in Exams? – Will be returned on Weds Proposal assignment – Posted on the wiki, due 11/9 Experimental design – Internal validity and threats – In class exercise

Experimental Design What is an experiment? – When the researcher manipulates the independent variable to view change in the dependent variable Why do we do experiments? – To establish and study cause and effect relationships

Famous Experiments Alexander Fleming-Discovery of Penicillen – Initially an accident – But he had to experimentally test its influence to be sure How would you test if a “mold” killed bacteria? How could you make sure something else didn’t kill the bacteria?

Experimental Design-Example Experimental Group Zombie Virus Control Group Bacteria Control Group Zombie Virus

Experimental Design-Example Experimental Group Zombie Virus Control Group Bacteria Control Group Zombie Virus Anti Zombie Serum

Experimental Design-Example Experimental Group Zombie Virus Dies Control Group Bacteria Control Group Zombie Virus LIVES!!!

How can we be sure of cause and effect? The bacteria died How do we know it was the serum? – Not environment (cold, hot, wind) – Not some other contaminant – Not time/age – Not the shape of the dish – Not a preexisting problem with our virus

How can we be sure? We control the circumstances We have a “control group” that is exposed to everything in exactly the same way…except our intervention A true experiment is when the researcher has complete control over who gets the intervention and who does not

Internal Validity When we have high control over an experiment, and can confidently dismiss alternate influences, we say that it has “good internal validity” Internal validity is the degree to which we can be sure the IV influenced the DV – And we ‘know’ that nothing else caused our outcome

Internal Validity Can be hard to achieve in psychology experiments There are many specific “threats” to internal validity There are ways to deal with some of these threats

Experimental Design-Example The researcher Blue Schools Green Schools The researcher wants to know if a program will Prevent early onset sexual activity

Experimental Design-Example The researcher Blue Schools Green Schools The researcher wants to know if a program will Prevent early onset sexual activity SEX ED PROGRAM

How will I know? If green schools have lower rates of sexual activity, how will I know that its my program? – Can I be sure its not something else? What are some threats to knowing that my program is the cause of changes in sexual activity?

This was a real study (is) 24 schools middle schools in the greater Boston area Randomly assigned by school to – treatment (special sex ed program each year for three years) – Control (whatever they would normally do) Surveyed in 6 th, 7 th, 8 th and 9 th grade

Threats to Internal Validity-History History – External events or circumstances that influence outcome – If kids in one or more blue schools had a “pact” to get pregnant? More blue kids might get pregnant, and make my green schools look “better” – If some green schools gave kids even more sex ed than they are getting from my program? This actually happened

How to “control” history? Can be quite difficult Ask participants to avoid other treatments… – Though this won’t control all exposures – Schools in our study violated this Try to collect good, comprehensive data and be aware of the circumstances of your participants as much as possible – At least then you can statistically control for some influences in your data

Threats…Maturation Maturation – When growth may account for our effect – This could certainly happen with a sex program over a 3 year period

How to control maturation? You can’t control it But you have a control group Both the experiment and control group should mature at the same rate – If random assignment has “worked” – And factors related to maturation have been balanced across conditions So any difference would reflect your intervention

Maturation? Factors related to sex behavior maturation in teens? – Poverty – Violent neighborhood – Lack of parental monitoring Single parenting Parents who have to work “swing shift” So you better hope that random assignment fixed this…

Threats--Selection When treatment groups are unequal in a way that influences outcome – If blue schools kids are poorer than green schools? – Poverty increases early onset sexual behavior So poorer kids might be harder to “treat” – Poor kids may have higher rates of sex behavior even before my study starts – These things could make my treatment look “better”

How to control for selection? Randomization – If many poor and wealthy schools, and randomly assign, groups should be pretty equal But sample size is crucial here Stratification may help Controlling for Baseline – Measure sexual onset before intervention – Adjust for differences between blue and green schools – Remaining difference should be due to my program

Threats-Participant expectations When participants believe they are getting a treatment So they improve Also known as placebo effect E.g. kids know they are getting an education program This makes them behave differently – Not the program itself

Controlling placebo? When possible, have a “placebo control” – Control group gets a “treatment” – Gives them attention like intervention group – That should not influence the outcome – That way both treatment and control get “attention” Groups “blind” to treatment type – If possible

Threats—researcher expectations When the researcher treats treatment groups differently – Has a different expectation of them – Or actually gives them something extra – Or actually evaluates them differently This actually changes the outcome

Researcher expectations Can be very subtle Positive reinforcement to tx group for being “special” Giving tx people special information or extra attention or help

Avoiding researcher expectations? Separate the researcher from the treatment Researcher should be blind to who is in what treatment – If possible Or researcher should be external to treatment process – As in our study

Instrumentation/Measurement If your way of measuring changes over time, it could alter how your outcome is measured – E.g. I ask the question “did you have sex” differently, responses may vary – E.g. If grading papers, and my view of what a “good answer” changes as I read answers, this could change how I grade later versus earlier responses

Instrumentation? Avoid this by piloting your measure first and then sticking with it If you must change your measure then collect new data Keep track of changes in questions