Presentation on theme: "Designing Experiments"— Presentation transcript:
1 Designing Experiments 4/15/2017Designing ExperimentsSuppose we wanted to design an experiment to see ifcaffeine affects pulse rate (in this class).What is the explanatory variable (factor)?caffeineWhat is the response variable?pulse rateWhat will be the experimental units?the students in this class2.3.ppt
2 Designing Experiments Here is an initial plan:measure initial pulse rategive each student some caffeinewait for a specified timemeasure final pulse ratecompare final and initial rates
3 Designing Experiments What are some problems with this plan?If there is a change in pulse rates, we do not know if caffeine was the cause. For example, suppose I told a joke while we were waiting, and every one laughed so hard that their pulse rates went up. Or, suppose we took notes and everybody’s pulse rate slowed down to sleeplike levels.
4 Designing Experiments Some problems can be easily solved by including a ___________ which does not receive caffeine. In our experiment, we can accomplish this by using 2 ________ of caffeine: no caffeine and some caffeine. For example, we could assign each member one of two ___________: Regular Coke or Caffeine Free Coke.
5 Designing Experiments Why don’t we give Coke to one group and nothing to the other group?Often times applying any treatment can create a change in the response variable. For example, when a child gets hurt, they feel better when their wound is kissed or covered with a band-aid, even though neither of those treatments actually take away the pain.
6 Designing Experiments In our study, if only one group got a treatment, the fact that they were chosen to receive free soda might make their pulse increase before the caffeine even hits their bloodstream!The ________________ occurs when subjects in an experiment know they are receiving a treatment. This knowledge may cause a change in the response variable which ___________ the effect of the treatment. In other words, we will not know which caused the change in the response variable: the explanatory variable or the placebo effect.
7 Designing Experiments Having every subject receive a treatment ensures that the placebo effect will treat both groups the same. Then, any difference between their pulse rates can be attributed to the _______________________ and not the excitement of being in an experiment.Of course, it is essential that the subjects do not know which treatment they are receiving! When a person doesn’t know who is receiving which treatment, that person is ________.
8 Designing Experiments There are two classes of individuals who can influence the results of an experiment:those who directly participate in the experiment (subjects, treatment administrators, etc.)those who evaluate the resultsWhen every individual in one of these classes is blinded, the experiment is called ___________. If every individual in both classes is blinded, then the experiment is ________________.
9 Designing Experiments Can our experiment be run in a D________ B_______manner?Yes, if the subjects, the people handing out the treatments, and the people measuring the pulse rates all do not know which treatment is which.But doesn’t someone need to know which is which? Yes, but that person should not have any interaction with the subjects at any point of the experiment.
10 Designing Experiments Four Key Principles of a Good Experiment:THE BIG IDEA: Our goal when designing an experiment is to make the treatment groups as similar as possible, with the exception of the treatments. Then, if there is a change in the response, it can be attributed to the explanatory variable and not any other extraneous variables.
11 Designing Experiments An _______________ is one that is not of interest in the current study but is thought to affect the response variable.For example, sugar is an extraneous variable since it may affect pulse rates. If one treatment group was given regular Coke and the other treatment group was given caffeine free Diet Coke, then sugar and caffeine would be confounded. If there was a difference in the average pulse rates of the two groups after receiving the treatments, we wouldn’t know which variable caused the change, and to what extent.
12 Designing Experiments To prevent sugar from becoming a __________ variable, we need to make sure that both treatment groups get the same amount of sugar.
13 Designing Experiments Principle #1: _________________ means holding extraneous variables constant for all treatment groups so that their effects are not confounded with the explanatory variable.temperature of drinkamount of drinkdrinking rateamount of sugar (no diet)waiting time between treatment and pulse readingpulse rate before experiment (need to be resting)room temperaturecarbonationhow we measure pulse
14 Designing Experiments If we do not control these extraneous variables by making them the same for all treatment groups, they could confound the effects of the caffeine on pulse rates.For example, we may not be able to tell if it was the caffeine or the temperature that causes the higher pulse rate.
15 Designing Experiments Principle #2: _________ is when subjects are divided into groups (blocks) based on some extraneous variable and then separated into different treatment groups.What if men react to caffeine differently than women?If more men end up in the experimental group and more women end up in the control group, then gender and caffeine will be confounded. We will not know which variable caused the change in pulse rates, gender or caffeine.
16 Designing Experiments How can we eliminate this potential confounding variable?Eliminate one gender from the study, but then we could only draw conclusions about one gender.Make sure there is a representative number of men and women in each treatment. For example, if there are 20 women and 30 men in the experiment, then the experimental group should have10 women and 15 men and the control group should have the same.In this example, we have formed 2 blocks: men and women. Then, we assigned treatments to the subjects within each block.
17 Designing Experiments Blocking in experiments is similar to stratification in sampling.Blocking reduces the variability of the results, just like stratifying.Blocks should be chosen like strata: the units within the block should be similar, but different than the units in the other blocks. You should only block when you expect that the subjects in one block will have a different response than subjects in other blocks.
18 Designing Experiments What are some other extraneous factors that we can block for?age, weight, initial pulse rate, etc.You should try to make the blocks as small as possible. Ideally, the size of the block should be the same as the number of treatments. For example, if there are 3 treatments, then there should be 3 subjects in each block.If each block has only 2 subjects, then the subjects are called a ______________.
19 Designing Experiments Principle #3: _______________ is random assignment of subjects to treatments to ensure that the experiment doesn’t systematically favor one treatment over the other.What about all of the other extraneous variables we do not think of? What about the variables we cannot directly control or block for?amount of food eaten before experimentcaffeine tolerance
20 Designing Experiments If we randomly assign subjects to treatments, this should _______________ (but not eliminate) the effects of these variables since their effects should be spread equally between the treatment groups.Note: We must ALWAYS randomize since there will always be extraneous variables we do not consider.
21 Designing Experiments How do we randomize?Draw names from a hat. The first half chosen are in one group, the remaining names in the other.Number the class from Then, generate random numbers without replacement until half are chosen for one group. The remaining names go in the other group.For matched pairs, we can flip a coin to determine which subjects go into which group. If its heads, the first person in the pair goes to A and the other to B. If its tails, it’s the opposite.
22 Designing Experiments If you do not use blocking when dividing the subjects, the result is a _______________________________.If you incorporate blocking in your design, it is called a _________________________ (every subject is assigned to a block based on some characteristics and the members of the block are randomly assigned to the different treatments).
23 Designing Experiments Principle #4: ____________ means ensuring that there is an adequate number of observations in each treatment group.If each treatment group only had one experimental unit, then we would not be able to conclude that any changes in the response are due to the treatments. It is also possible that some characteristic of the unit was the cause of the change.Ex: our coke experiment
24 Designing Experiments Increasing the __________________ makes randomization more effective. The more subjects we have, the more balanced our treatment groups will be. For example, if we have 10 subjects and only 2 have a certain unknown characteristic, it is quite likely that both of those subjects will end up in the same treatment group simply by chance.
25 Designing Experiments However, if we have 100 subjects and 20 have the characteristic, it is very unlikely for all 20 to end up in the same group. There is a much better chance that the groups will be close to balanced (10/10, 9/11, 11/9, etc.) when the sample size is larger.If you were flipping a coin and wanted to get as close as possible to 50%, you would decide to flip the coin more than a few times!
26 Designing Experiments Note: Replication can also refer to repeating the experiment with different subjects. This can help us be more confident applying the results of our experiment to a ________________________.SUMMARY: With control, blocking, randomization, and replication, each treatment group should be nearly identical, and the effects of extraneous variables should be the same in each group. Now, if changes in the explanatory variable are associated with changes in the response variable, we can conclude that it is a cause-and-effect relationship.
30 More Designing Experiments Not all experiments have CONTROL GROUPS or use a PLACEBO, as long as there is comparison. For example, if you are testing a new drug, it is usually compared to the currently used drug, not a placebo. Or, you can do an experiment to compare four brands of paint without using a placebo.
31 More Designing Experiments There are ethical issues to consider when doingexperiments:smoking and lung cancer: we cannot force people to smoke, but that would be the best way to prove smoking causes lung cancer.many medical experiments are ended early if the experimenters discover that one treatment is much more effective: ARTICLE: Angioplasty Protocol Questioned: LA Times,“The Tuskegee Study of Untreated Syphilis in the Negro Male”
32 More Designing Experiments The ethical rules that govern medical experiments are a relatively recent development.“The Tuskegee Study of Untreated Syphilis in the Negro Male” (1932 – 1972):- Before 1945, there was no effective treatment. The ones attempted were dangerous. Initially, the study addressed the question whether untreated syphilis is preferable to these remedies.- In 1945, a cure for syphilis was discovered. However, the study was continued and treatment was withheld.- By the end of the study, 128 out of 399 died of syphilis or its complications.
33 More Designing Experiments Even if caffeine has no effect on pulse rates, the average pulse rate of the two groups will probably be slightly different. Thus, whenever we do an experiment and find a difference between two groups, we need to determine if this difference occurred because of RANDOMIZATION VARIABILITY or because there really is a difference in the treatments.The results of an experiment are called STATISTICALLY SIGNIFICANT if they are unlikely to occur by random chance.
34 More Designing Experiments AP Test, 1999, Question 3:
37 The Scope of InferenceNot all experiments and their resultant hypothesis tests use random samples.For example, in a study at UNC hospital, the researchers tried to determine the minimum daily requirement of choline for adults. They advertised at homeless shelters and natural food stores for volunteers. The subjects will spend 71 consecutive days in a hospital room and be paid $4,500 for their time and blood.
38 The Scope of InferenceThese subjects are just the first 80 people who are willing to participate and who pass whatever general health requirements are necessary. They are not a random sample from any population. Many medical studies proceed along these lines. The randomization lies in the random assignment of treatment to the experimental unit (subject), not in the random selection of subjects from a population. With a random assignment of treatment to subject, the researchers can make causal inferences from the results of their study. They cannot make inferences to a larger population. It is the inference to the population from which the sample was drawn that is lost when techniques of random sampling are not used.
39 The Scope of InferenceOn the other hand, in observational studies, there are generally random samples from distinct populations, so inferences can be made to those populations. However, causality cannot be attributed, since there was no random assignment to treatment groups.
40 The Scope of InferenceThe scope of inference refers to the type of inferences (conclusions) that can be drawn from a study. The types of inferences we can make (inferences about the population and inferences about cause-and-effect) are determined by two factors in the design of the study:how the subjects were selected from the populationhow the subjects were assigned to groups.
41 The Scope of Inference Allocation of Subjects to Groups Randomized Not randomizedSelection of subjects from the populationRandomInferences about pop. – YESInferences about cause and effect – YESSome experiments are here.Inferences about cause and effect – NOMost obs. studies are here.Not randomInferences about pop. – NOMost experiments are here.Some obs. studies are here.
42 The Scope of InferenceSuppose a dentist wants to know if a daily dose of 500 mg of vitamin C will result in fewer canker sores in the mouth than taking no vitamin C.
43 The Scope of InferenceCase 1) The dentist, working through the local dental society, convinces all of the dental patients in town with appointments the first two weeks in December to be subjects in an experiment. He divides them into two groups, those who take at least 500 mg of vitamin C each day and those who don't. He then asks them how often they have canker sores in their mouth and checks their patients records to see who has complained about canker sores. He compares the proportion of those who take vitamin C daily and complain of canker sores with the proportion of those who don't take vitamin C and complain of canker sores. There is a significant difference in the two proportions, with a significantly smaller proportion of those taking vitamin C having canker sores.
44 The Scope of Inference What can we conclude? Since the patients do not represent a random sample from any population, it is not possible to make any inference about a larger population.Since the study was observational, with subjects not randomly assigned to treatments, no causal inference can be made.We just know that for these patients, those who take vitamin C have fewer canker sores than those who don't. We don't know why, and we don't know if this result would be consistent with another group.
45 The Scope of InferenceCase 2) A dentist, working through the local dental society, convinces all of the dental patients in town with appointments the first two weeks in December to be subjects in an experiment. He randomly assigns half of them to take 500 mg of vitamin C each day and the other half to abstain from taking vitamin C for three months. At the end of this time he determines the proportion of each group that has suffered from canker sores during those three months. There is a significant difference in the two proportions, with a significantly smaller proportion of those taking vitamin C having canker sores.
46 The Scope of Inference What can we conclude? Since the patients do not represent a random sample from any population, it is not possible to make any inference about a larger population.However, the treatments were randomly assigned to the subjects, so (assuming other factors were controlled or randomized) the difference in proportions having canker sores can be attributed to the vitamin C.We don't know if this result would be consistent with another group, but we believe we know why, for this group, the proportions differ.
47 The Scope of InferenceCase 3) The dentist, working through the local dental society, selects a random sample of dental patients in town and convinces them to be subjects in an experiment. He divides them into two groups, those who take at least 500 mg of vitamin C each day and those who don't. He then asks them how often they have canker sores in their mouth and checks their patients records to see who hascomplained about canker sores. He compares the proportion of those who take vitamin C daily and complain of canker sores with the proportion of those who don't take vitamin C and complain of canker sores. There is a significant difference in the two proportions, with a significantly smaller proportion of those taking vitamin C having canker sores.
48 The Scope of Inference What can we conclude? Since the patients selected were a random sample of dental patients in town, we can infer that the results observed in this experiment would be consistent with results from the whole population of dental patients in this town.However, since the study was observational, with subjects not being randomly assigned to treatments, no causal inference can be made.We believe that, for the population of dental patients in this town, those taking vitamin C have fewer canker sores than those who didn't. We don't know if it is the vitamin C that causes this reduction or some other confounding variable. We cannot conclude that for the general population, those taking vitamin C have fewer canker sores, since the sample was only of dental patients.
49 The Scope of InferenceCase 4) The dentist, working through the local dental society, selects a random sample of dental patients in town and convinces them to be subjects in an experiment. He randomly assigns half of them to take 500 mg of vitamin C each day and the other half to abstain from taking vitamin C for three months. At the end of this time he determines the proportion of each group that has suffered from canker sores during those three months. There is a significant difference in the two proportions, with a significantly smaller proportion of those taking vitamin C having canker sores.
50 The Scope of Inference What can we conclude? Since the patients selected were a random sample of dental patients in town, we can infer than the results observed in this experiment would be consistent with results from the whole population of dental patients in this town.Moreover, the treatments were randomly assigned to the subjects, so (assuming other factors were controlled or randomized) the difference in proportions having canker sores can be attributed to the vitamin C.We believe that for the population of dental patients in this town, taking vitamin C results in fewer canker sores.
54 The Caffeine Experiment Initial pulse rate: ____________________Drink you were assigned to: ___________________
55 The Caffeine Experiment Experiment Summary:Response variable:pulse rateExplanatory variable:caffeineExperimental units:students in this AP Statistics classTreatments:caffeine (the quantity in a cup of Regular Coke)very little caffeine (the quantity in a cup of Caffeine-Free Coke)
56 The Caffeine Experiment Variables that were directly controlled:amount of sugardrink temperatureenvironmental conditions while drinkingVariables that were blocked:genderinitial pulse rate.
57 The Caffeine Experiment What other extraneous factors can you think of?genetic factorstolerance to caffeinewhat or if students ate in the morning …What was done about these other extraneous factors?Randomly allocating students in each matched pair should balance the groups with respect to any other factor.
58 The Caffeine Experiment How was the replication principle applied in the experiment?There were several matched pairs (more would have been better). Class size large lots of matched pairs.Unfortunately, the experiment won’t be repeated with two classes, as there is only 1 AP Stat class this year.