Chapter 13 Experiments & observational studies

Observational studies researchers don’t assign choices, they observe them a study in which no manipulation of factors has been employed helpful for discovering trends and possible relationships although an observational study may identify important variables related to an outcome. there is no guarantee that we have found the right or most important related variable

Observational studies Retrospective Study Prospective Study subjects are selected then previous conditions or behaviors are determined not based on random samples, they focus on estimating differences between groups or associations between variables have a restricted view of the world because they are usually restricted to a small part of the population because they are based on historical data there could be errors identifying subjects in advance and collecting data as the events unfold

Observational studies that try to discover variables related to rare outcomes are often retrospective Example: specific disease identify people with the disease look at their history and heritage something that could be related to their condition

Who gets good grades? or why? In 1981 a study was conducted at a high school in California. Researchers compared the scholastic performance of music students with that of non-music students. The music students had a higher GPA of 3.59 compared to 2.91 that non-music students had 16% of music students earned all A’s where only 5% of non-music students earned all A’s Does this study tell us that music will improve students GPAs? students that study music might still differ from the others in some important way that we failed to observe.

Can we prove cause and effect?!? Experiment: studies the relationships between two or more variables manipulates factor levels to create treatments, randomly assigns subjects to these treatment levels, and then compares the responses of the subject groups across treatment levels. Random assignment: an experiment must assign experimental units to treatment groups at random for the experiment to be valid.

Experiment Vocabulary Factor: a variable whose levels are controlled by the experimenter. Experiments attempt to discover the effects that differences in factor levels may have on the responses of the experimental units Response variable: a variable whose values are compared across different treatments. In a randomized experiment, large response differences can be attributed to the effect of differences in treatment levels. Subject/experimental unit: individuals on whom an experiment is performed. (participants for humans too) Level of the factor: the specific values that the experimenter chooses for a factor Treatment: the process, intervention, or other controlled circumstance applied to randomly assigned experimental units. Treatments are the different levels of a single factor or are made up of combinations of levels of two or more factors

Experiment example Once all this is set up you need to Design an experiment to see whether the amount of sleep and exercise you get affects your performance Subjects: the people in the sleep study Factors: sleep and exercise Factor levels: sleep: 4, 6, or 8 hours exercise: 0 min or 30 mins on a treadmill Treatment: 6 total Once all this is set up you need to RANDOMLY assign each subject to a treatment 4 hours 0 min 30 mins 6 hours 30 min 8 hours

Music & GPA example only an experiment can justify a claim like “music lessons cause higher grades” Music and GPA experiment take a group of 3rd graders (subjects) ½ study music and ½ never take music classes (factor and levels) collect data about their GPA (response variable)

4 Principles Of Experimental Design 1) Control make conditions as similar as possible for all treatments groups. reduces the variability of the responses, making it easier to detect differences among the treatment groups Risky!! we are testing laundry detergents. we control the water temperature at 1800. this would reduce variation in our results due to water temperature. BUT now we can’t say anything about the detergent in cold water we control a factor by assigning subjects to different levels because we want to see how the response will change at different levels we control other sources of variation to prevent them changing and affecting the response variable

2) Randomize allows us to equalize the effects of unknown or uncontrollable sources of variation it does not eliminate the effects of these sources, it spreads them out over all the treatments so we can see past them protects us from things we didn’t even know about “control what you can, and randomize the rest”

3) Replicate we should repeat the experiment, applying the treatments to a number of subjects only this can we estimate the variability of responses an experiment is only complete once you have assessed the variation the outcome of an experiment on a single subject is an anecdote, not data when the subjects are not a representative sample of the population of interest; repeat the experiment with people from different ages and different time of year Replication of an entire experiment with controlled sources of variation at different levels is an essential step in science

(Not Required) 4) Block to reduce the effects of identifiable attributes of the subjects that cannot be controlled. example: 10 people – 2 math teachers and 8 math students we want to break them up into two groups for a math competition. if we pick people randomly the math teachers could be on the same team. (unfair) so we put 1 math teacher on each team then we randomly assign 4 students to each team. we are “blocking” the occupation variable allowing us to remove the variability due to the differences among the blocks.

Diagrams an experiment is carried out over time with specific actions occurring in a specified order a diagram of the procedure can help think about experiments random allocation group #1 treatment # 1 group # 2 treatment #2 compare

Designing An Experiment: Step-by-step An ad for OptiGro plan fertilizer claims that with this product you will grow “juicer, tastier” tomatoes. You would like to test this claim, and wonder whether you might be able to get by with half the specified dose. Basically we need to buy some tomato plants and use OptiGro on some of them. We will set up a completely randomized experiment in one factor

Think Plan: state what you want to know Response variable I want to now whether tomato plants grown with OptiGro yield juicer, tastier tomatoes than plants raised in otherwise similar circumstances but without the fertilizer. Response variable I’ll evaluate the juiciness and taste of the tomatoes by asking a panel of judges to rate them on a scale from 1 to 7 in juiciness and in taste. Treatments: specify the factor levels and the treatments The factor is fertilizer, specifically OptiGro. I’ll grow tomatoes at three different factor levels; some with no fertilizer, some with half the specified amount, and some with the full dose of OptiGro. These are the three treatments.

Experimental Design: observe the principles of design Experimental Units: I’ll obtain 24 tomato plants of the same variety from a local garden store. Experimental Design: observe the principles of design Control: any source of variability you know of and can control Randomly assign: experimental units to treatments, to equalize the effects of unknown or uncontrollable sources of variation Replicate: results by placing more than one plant in each treatment group I’ll locate the farm plots near each other so that the plants get similar amounts of sun and rain and experience similar temperatures. I will weed the plots equally and otherwise treat the plants alike. I’ll randomly divide the plants into three groups. I will use random numbers from a table to determine the assignment. There will be 8 plants in each treatment group. Draw a picture 24 tomato plants from a garden store 8 plants Group 1 control (no fertilizer) Treatment 1 Group 2 Treatment 2 ½ dose Group 3 Treatment 3 full fertilizer compare juiciness and tastiness

Specify any other experiment details Specify any other experiment details. You must give enough details so that another experimenter could exactly replicate your experiment. It’s generally better to include details that might seem irrelevant than to leave out matter that could turn out to make a difference. Specify how to measure the response. I will grow the plants until the tomatoes are mature, as judged by reaching a standard color. I will then harvest the tomatoes when ripe and store then for evaluation. I will set up a numerical scale of juiciness and one for tastiness for the taste testers. Several people will taste slices of tomato and rate them.

Show Once you collect the data, you’ll need to display them and compare the results for the three treatment groups. I will display the results with side-by-side box-plots to compare the three treatment groups. I will compare the means of the groups.

Tell To answer the initial question, we ask whether the difference we observe in the means of the three groups are meaningful.

Control Treatment another level of the factor in order to compare the treatment results to a situation in which “nothing happens” example: the group of tomatoes that received no fertilized Different from “control” (one of the 4 principles of experiments). controlling extraneous sources of variations by keeping them constant example: buying the plants from the same nursery, weeding all the plots the same way

Blinding To avoid bias, we disguise the levels of the factors There are two main classes of individuals who can affect the outcome of the experiment: those who could influence the results the subjects, treatment administrators, or technicians those who evaluate the results (judges, treating physicians, etc) Single Blind: when one of the groups is blinded Double Blind: when both of the groups are blinded Tomato Experiment: we don’t want our tasters to be bias so we will not tell them plants received OptiGro

Placebo a control treatment that mimics the real treatment the best way to blind subjects placebo effect: it is not unusual for 20% or more of subjects given a placebo to report reduction or pain, improved movement, or greater alertness highlights both the importance of effective blinding and the importance of comparing treatments with a control

The best experiments are usually randomized comparative double-blind placebo-controlled

Blocking Tomato plants: we want 18 plants we get 12 from store A and 6 from store B The store the plants came from could effect our results So we break the 18 plants into two BLOCKS one for each store Then we randomly assign the plants in each block to one of the three treatments This is called randomized block design blocking is the same idea for experiments as stratifying for samples we use blocks to reduce variability so we can see the effects of the factors, we are nut usually interested in studying the effects of the blocks themselves

Diagram 12 tomato plants from store A and 6 plants from store B Block A: 12 tomato plants 4 plants group 1 treatment 1 control compare juiciness and tastiness group 2 ½ dose treatment 2 group 3 full dose treatment 3 Block B: 6 tomato plants 2 plants group 4 group 5 group 6

Matching Matching: when subjects are paired because they are similar in ways NOT being studied in retrospective and prospective studies Music and grades: we might match each student who studies an instrument with someone of the same sex and similar family income who doesn’t play an instrument

Example Problems Describe a strategy to randomly split the 24 tomato plants into the three groups for the 3 treatments in the OptiGro experiment.

A running shoe manufacturer wants to test the speed of its new sprinting shoe on the 100-meter dash times. The company sponsors 5 athletes who are running the 100-meter dash in the 2004 Summer Olympic games. To test the show, they have all 5 runners run the 100- meter dash with a competitor's shoe and then again with their new shoe. They use the difference in times as the response variable. Suggest some improvements to the design. Why might the shoe manufacturer not be able to generalize the results they find to all runners?

Athletes who had suffered hamstring injuries were randomly assigned to one of two exercise programs. Those who engages in static stretching returned to sports activity in a mean of 37.4 days (SD = 27.6 days). Those assigned to a program of agility and trunk stabilization exercises returned to sports in a mean of 22.2 days (SD = 8.3 days). Explain why it was important to assign the athletes to the two different treatments randomly. There was no control group of athletes who did not participate in a special exercise program. Explain the advantage of including such a group in this experiment. How might blinding have been used in this experiment? Do you think the difference in times is statistically significant?

Adding More Factors OptiGro Experiment We want to add a factor about watering Levels: only natural watering (rain) and watering by hand daily Treatments: 6 Completely randomized two-factor experiment No fertilizer ½ fertilizer full fertilizer No watering 1 2 3 Daily watering 4 5 6

12 tomato plants from a garden store Compare juiciness and tastiness Group 1 2 plants Treatment 1 control/ no water Group 2 Treatment 2 ½ /no water Compare juiciness and tastiness Group 3 Treatment 3 full/no water Group 4 Treatment 4 control/water Group 5 Treatment 5 ½ / water Group 6 Treatment 6 full/water

Confounding When the levels of one factor are associated with the levels of another factor, the factors are said to be confounding Example: A credit card bank wanted to test the sensitivity of the market by two factors: the annual fee charged for a card and the annual percentage rate charged. The bank selected 100,000 people at random from a mailing list. It sent out 50,000 offers with a low rate and no fee and 50,000 offers with a higher rate and a $50 annual fee. They found out people signed up (more than twice the rate) for the card with low rate and no annual fee. Problem: The question that the bank was trying to answer was “how much of the change is due to the rate and how much was due to the fee?” How could they have avoided this?