1. BHS 204-01 Methods in Behavioral Sciences I
May 21, 2003
Chapter 10 (Ray)
The Ecology of the Experiment

2. Ecology of the Experiment
Ecological validity – what is the impact of important relationships among the scientist, participant and context of the experiment.
Can be divided into:
Experimenter factors
Subject factors
Cultural and social bias

3. Experimenter Factors
Experimenter effects – results that are due to the attitude or behavior of the experimenter, not the manipulation.
Two kinds:
Biased data collection
Influence exerted by the experimenter on the performance of the subjects.

4. Biased Data Collection
Personal equation – constant error in observation related to individual differences.
Use two or more raters
Compare inter-rater agreement (Cohen’s kappa).
Unconscious or unintentional bias in the direction of the experimenter’s hypothesis.
Errors tend to go in the expected direction.
Acceptance or discarding of data.
Interpretation of results

5. Biased Interactions with Subjects
Participants in experiments react differently to different experimenters.
Gender, attractiveness, dress, race, age.
Experimenters can communicate to subjects that they want them to respond differently.
Rosenthal effect
Cultural expectations
Attributions about animal behavior

6. Ways to Avoid Bias
Expose subjects to different experimenters and different contexts and stimuli.
Keep the person interacting with subjects blind to the experimental hypothesis and to which group is the control or treatment group.
Try to see your experiment from different perspectives

7. Kuhn’s Recommendations
Be aware of underlying assumptions and state them explicitly in papers.
The person who plans the study should not be the person who analyzes the data.
The person who plans the study should be the person who conducts it (deals with subjects).
Researchers should be pilot subjects in their own studies, to understand the experience.

8. More Recommendations
Pilot studies should develop and rehearse standardized procedures before the actual experiment starts.
Recognize that different analysis methods may lead to different conclusions.
Good research methods are more important than finding statistically significant results.

9. More Recommendations
Research methods should be followed to the letter and results reported honestly.
Investigators must make sure their procedures are being followed correctly during expt.
Use multiple experimenters who vary in their personal characteristics.
Keep experimenters blind whenever possible.
Seek and reward replication.