1. Chapter 13 Experiments & observational studies

2. Observational studies
researchers don’t assign choices, they observe them
a study in which no manipulation of factors has been employed
helpful for discovering trends and possible relationships
although an observational study may identify important variables related to an outcome. there is no guarantee that we have found the right or most important related variable

3. Observational studies
Retrospective Study
Prospective Study
subjects are selected then previous conditions or behaviors are determined
not based on random samples, they focus on estimating differences between groups or associations between variables
have a restricted view of the world because they are usually restricted to a small part of the population
because they are based on historical data there could be errors
identifying subjects in advance and collecting data as the events unfold

4. Observational studies that try to discover variables related to rare outcomes are often retrospective
Example: specific disease
identify people with the disease
look at their history and heritage
something that could be related to their condition

5. Who gets good grades? or why?
In 1981 a study was conducted at a high school in California. Researchers compared the scholastic performance of music students with that of non-music students.
The music students had a higher GPA of 3.59 compared to 2.91 that non-music students had
16% of music students earned all A’s where only 5% of non-music students earned all A’s
Does this study tell us that music will improve students GPAs?
students that study music might still differ from the others in some important way that we failed to observe.

6. Can we prove cause and effect?!?
Experiment:
studies the relationships between two or more variables
manipulates factor levels to create treatments, randomly assigns subjects to these treatment levels, and then compares the responses of the subject groups across treatment levels.
Random assignment:
an experiment must assign experimental units to treatment groups at random for the experiment to be valid.

7. Experiment Vocabulary
Factor: a variable whose levels are controlled by the experimenter. Experiments attempt to discover the effects that differences in factor levels may have on the responses of the experimental units
Response variable: a variable whose values are compared across different treatments. In a randomized experiment, large response differences can be attributed to the effect of differences in treatment levels.
Subject/experimental unit: individuals on whom an experiment is performed. (participants for humans too)
Level of the factor: the specific values that the experimenter chooses for a factor
Treatment: the process, intervention, or other controlled circumstance applied to randomly assigned experimental units. Treatments are the different levels of a single factor or are made up of combinations of levels of two or more factors

8. Experiment example Once all this is set up you need to
Design an experiment to see whether the amount of sleep and exercise you get affects your performance
Subjects: the people in the sleep study
Factors: sleep and exercise
Factor levels:
sleep: 4, 6, or 8 hours
exercise: 0 min or 30 mins on a treadmill
Treatment: 6 total
Once all this is set up you need to
RANDOMLY assign each subject to a treatment
4 hours
0 min
30 mins
6 hours
30 min
8 hours

9. Music & GPA example
only an experiment can justify a claim like “music lessons cause higher grades”
Music and GPA experiment
take a group of 3rd graders (subjects)
½ study music and ½ never take music classes (factor and levels)
collect data about their GPA (response variable)

10. 4 Principles Of Experimental Design
1) Control
make conditions as similar as possible for all treatments groups.
reduces the variability of the responses, making it easier to detect differences among the treatment groups
Risky!! we are testing laundry detergents. we control the water temperature at this would reduce variation in our results due to water temperature. BUT now we can’t say anything about the detergent in cold water
we control a factor by assigning subjects to different levels because we want to see how the response will change at different levels
we control other sources of variation to prevent them changing and affecting the response variable

11. 2) Randomize
allows us to equalize the effects of unknown or uncontrollable sources of variation
it does not eliminate the effects of these sources, it spreads them out over all the treatments so we can see past them
protects us from things we didn’t even know about
“control what you can, and randomize the rest”

12. 3) Replicate
we should repeat the experiment, applying the treatments to a number of subjects
only this can we estimate the variability of responses
an experiment is only complete once you have assessed the variation
the outcome of an experiment on a single subject is an anecdote, not data
when the subjects are not a representative sample of the population of interest; repeat the experiment with people from different ages and different time of year
Replication of an entire experiment with controlled sources of variation at different levels is an essential step in science

13. (Not Required)
4) Block
to reduce the effects of identifiable attributes of the subjects that cannot be controlled.
example: 10 people – 2 math teachers and 8 math students
we want to break them up into two groups for a math competition. if we pick people randomly the math teachers could be on the same team. (unfair) so we put 1 math teacher on each team then we randomly assign 4 students to each team.
we are “blocking” the occupation variable
allowing us to remove the variability due to the differences among the blocks.

14. Diagrams
an experiment is carried out over time with specific actions occurring in a specified order
a diagram of the procedure can help think about experiments
random allocation
group #1
treatment # 1
group # 2
treatment #2
compare

15. Designing An Experiment: Step-by-step
An ad for OptiGro plan fertilizer claims that with this product you will grow “juicer, tastier” tomatoes. You would like to test this claim, and wonder whether you might be able to get by with half the specified dose.
Basically we need to buy some tomato plants and use OptiGro on some of them.
We will set up a completely randomized experiment in one factor

16. Think Plan: state what you want to know Response variable
I want to now whether tomato plants grown with OptiGro yield juicer, tastier tomatoes than plants raised in otherwise similar circumstances but without the fertilizer.
Response variable
I’ll evaluate the juiciness and taste of the tomatoes by asking a panel of judges to rate them on a scale from 1 to 7 in juiciness and in taste.
Treatments: specify the factor levels and the treatments
The factor is fertilizer, specifically OptiGro. I’ll grow tomatoes at three different factor levels; some with no fertilizer, some with half the specified amount, and some with the full dose of OptiGro. These are the three treatments.

17. Experimental Design: observe the principles of design
Experimental Units:
I’ll obtain 24 tomato plants of the same variety from a local garden store.
Experimental Design: observe the principles of design
Control: any source of variability you know of and can control
Randomly assign: experimental units to treatments, to equalize the effects of unknown or uncontrollable sources of variation
Replicate: results by placing more than one plant in each treatment group
I’ll locate the farm plots near each other so that the plants get similar amounts of sun and rain and experience similar temperatures. I will weed the plots equally and otherwise treat the plants alike. I’ll randomly divide the plants into three groups. I will use random numbers from a table to determine the assignment. There will be 8 plants in each treatment group.
Draw a picture
24 tomato plants from a garden store
8 plants
Group 1
control (no fertilizer)
Treatment 1
Group 2
Treatment 2
½ dose
Group 3
Treatment 3
full fertilizer
compare juiciness and tastiness

18. Specify any other experiment details
Specify any other experiment details. You must give enough details so that another experimenter could exactly replicate your experiment. It’s generally better to include details that might seem irrelevant than to leave out matter that could turn out to make a difference. Specify how to measure the response.
I will grow the plants until the tomatoes are mature, as judged by reaching a standard color. I will then harvest the tomatoes when ripe and store then for evaluation. I will set up a numerical scale of juiciness and one for tastiness for the taste testers. Several people will taste slices of tomato and rate them.

19. Show
Once you collect the data, you’ll need to display them and compare the results for the three treatment groups.
I will display the results with side-by-side box-plots to compare the three treatment groups. I will compare the means of the groups.

20. Tell
To answer the initial question, we ask whether the difference we observe in the means of the three groups are meaningful.

21. Control Treatment
another level of the factor in order to compare the treatment results to a situation in which “nothing happens”
example: the group of tomatoes that received no fertilized
Different from “control” (one of the 4 principles of experiments).
controlling extraneous sources of variations by keeping them constant
example: buying the plants from the same nursery, weeding all the plots the same way

22. Blinding To avoid bias, we disguise the levels of the factors
There are two main classes of individuals who can affect the outcome of the experiment:
those who could influence the results
the subjects, treatment administrators, or technicians
those who evaluate the results (judges, treating physicians, etc)
Single Blind: when one of the groups is blinded
Double Blind: when both of the groups are blinded
Tomato Experiment: we don’t want our tasters to be bias so we will not tell them plants received OptiGro

23. Placebo a control treatment that mimics the real treatment
the best way to blind subjects
placebo effect: it is not unusual for 20% or more of subjects given a placebo to report reduction or pain, improved movement, or greater alertness
highlights both the importance of effective blinding and the importance of comparing treatments with a control

24. The best experiments are usually
randomized
comparative
double-blind
placebo-controlled

25. Blocking Tomato plants: we want 18 plants
we get 12 from store A and 6 from store B
The store the plants came from could effect our results
So we break the 18 plants into two BLOCKS
one for each store
Then we randomly assign the plants in each block to one of the three treatments
This is called randomized block design
blocking is the same idea for experiments as stratifying for samples
we use blocks to reduce variability so we can see the effects of the factors, we are nut usually interested in studying the effects of the blocks themselves

26. Diagram 12 tomato plants from store A and 6 plants from store B
Block A: 12 tomato plants
4 plants
group 1
treatment 1
control
compare juiciness and tastiness
group 2
½ dose
treatment 2
group 3
full dose
treatment 3
Block B: 6 tomato plants
2 plants
group 4
group 5
group 6

27. Matching
Matching: when subjects are paired because they are similar in ways NOT being studied in retrospective and prospective studies
Music and grades: we might match each student who studies an instrument with someone of the same sex and similar family income who doesn’t play an instrument

28. Example Problems
Describe a strategy to randomly split the 24 tomato plants into the three groups for the 3 treatments in the OptiGro experiment.

29. A running shoe manufacturer wants to test the speed of its new sprinting shoe on the 100-meter dash times. The company sponsors 5 athletes who are running the 100-meter dash in the 2004 Summer Olympic games. To test the show, they have all 5 runners run the 100- meter dash with a competitor's shoe and then again with their new shoe. They use the difference in times as the response variable.
Suggest some improvements to the design.
Why might the shoe manufacturer not be able to generalize the results they find to all runners?

30. Athletes who had suffered hamstring injuries were randomly assigned to one of two exercise programs. Those who engages in static stretching returned to sports activity in a mean of 37.4 days (SD = 27.6 days). Those assigned to a program of agility and trunk stabilization exercises returned to sports in a mean of 22.2 days (SD = 8.3 days).
Explain why it was important to assign the athletes to the two different treatments randomly.
There was no control group of athletes who did not participate in a special exercise program. Explain the advantage of including such a group in this experiment.
How might blinding have been used in this experiment?
Do you think the difference in times is statistically significant?

31. Adding More Factors OptiGro Experiment
We want to add a factor about watering
Levels: only natural watering (rain) and watering by hand daily
Treatments: 6
Completely randomized two-factor experiment
No fertilizer
½ fertilizer
full fertilizer
No watering 1 2 3
Daily watering 4 5 6

32. 12 tomato plants from a garden store Compare juiciness and tastiness
Group 1
2 plants
Treatment 1
control/ no water
Group 2
Treatment 2
½ /no water
Compare juiciness and tastiness
Group 3
Treatment 3
full/no water
Group 4
Treatment 4
control/water
Group 5
Treatment 5
½ / water
Group 6
Treatment 6
full/water

33. Confounding
When the levels of one factor are associated with the levels of another factor, the factors are said to be confounding
Example: A credit card bank wanted to test the sensitivity of the market by two factors: the annual fee charged for a card and the annual percentage rate charged. The bank selected 100,000 people at random from a mailing list. It sent out 50,000 offers with a low rate and no fee and 50,000 offers with a higher rate and a $50 annual fee.
They found out people signed up (more than twice the rate) for the card with low rate and no annual fee.
Problem: The question that the bank was trying to answer was “how much of the change is due to the rate and how much was due to the fee?”
How could they have avoided this?