1. Designing Experiments
4/15/2017
Designing Experiments
Suppose we wanted to design an experiment to see if
caffeine affects pulse rate (in this class).
What is the explanatory variable (factor)?
caffeine
What is the response variable?
pulse rate
What will be the experimental units?
the students in this class
2.3.ppt

2. Designing Experiments
Here is an initial plan:
measure initial pulse rate
give each student some caffeine
wait for a specified time
measure final pulse rate
compare final and initial rates

3. Designing Experiments
What are some problems with this plan?
If there is a change in pulse rates, we do not know if caffeine was the cause. For example, suppose I told a joke while we were waiting, and every one laughed so hard that their pulse rates went up. Or, suppose we took notes and everybody’s pulse rate slowed down to sleeplike levels.

4. Designing Experiments
Some problems can be easily solved by including a ___________ which does not receive caffeine. In our experiment, we can accomplish this by using 2 ________ of caffeine: no caffeine and some caffeine. For example, we could assign each member one of two ___________: Regular Coke or Caffeine Free Coke.

5. Designing Experiments
Why don’t we give Coke to one group and nothing to the other group?
Often times applying any treatment can create a change in the response variable. For example, when a child gets hurt, they feel better when their wound is kissed or covered with a band-aid, even though neither of those treatments actually take away the pain.

6. Designing Experiments
In our study, if only one group got a treatment, the fact that they were chosen to receive free soda might make their pulse increase before the caffeine even hits their bloodstream!
The ________________ occurs when subjects in an experiment know they are receiving a treatment. This knowledge may cause a change in the response variable which ___________ the effect of the treatment. In other words, we will not know which caused the change in the response variable: the explanatory variable or the placebo effect.

7. Designing Experiments
Having every subject receive a treatment ensures that the placebo effect will treat both groups the same. Then, any difference between their pulse rates can be attributed to the _______________________ and not the excitement of being in an experiment.
Of course, it is essential that the subjects do not know which treatment they are receiving! When a person doesn’t know who is receiving which treatment, that person is ________.

8. Designing Experiments
There are two classes of individuals who can influence the results of an experiment:
those who directly participate in the experiment (subjects, treatment administrators, etc.)
those who evaluate the results
When every individual in one of these classes is blinded, the experiment is called ___________. If every individual in both classes is blinded, then the experiment is ________________.

9. Designing Experiments
Can our experiment be run in a D________ B_______
manner?
Yes, if the subjects, the people handing out the treatments, and the people measuring the pulse rates all do not know which treatment is which.
But doesn’t someone need to know which is which? Yes, but that person should not have any interaction with the subjects at any point of the experiment.

10. Designing Experiments
Four Key Principles of a Good Experiment:
THE BIG IDEA: Our goal when designing an experiment is to make the treatment groups as similar as possible, with the exception of the treatments. Then, if there is a change in the response, it can be attributed to the explanatory variable and not any other extraneous variables.

11. Designing Experiments
An _______________ is one that is not of interest in the current study but is thought to affect the response variable.
For example, sugar is an extraneous variable since it may affect pulse rates. If one treatment group was given regular Coke and the other treatment group was given caffeine free Diet Coke, then sugar and caffeine would be confounded. If there was a difference in the average pulse rates of the two groups after receiving the treatments, we wouldn’t know which variable caused the change, and to what extent.

12. Designing Experiments
To prevent sugar from becoming a __________ variable, we need to make sure that both treatment groups get the same amount of sugar.

13. Designing Experiments
Principle #1: _________________ means holding extraneous variables constant for all treatment groups so that their effects are not confounded with the explanatory variable.
temperature of drink
amount of drink
drinking rate
amount of sugar (no diet)
waiting time between treatment and pulse reading
pulse rate before experiment (need to be resting)
room temperature
carbonation
how we measure pulse

14. Designing Experiments
If we do not control these extraneous variables by making them the same for all treatment groups, they could confound the effects of the caffeine on pulse rates.
For example, we may not be able to tell if it was the caffeine or the temperature that causes the higher pulse rate.

15. Designing Experiments
Principle #2: _________ is when subjects are divided into groups (blocks) based on some extraneous variable and then separated into different treatment groups.
What if men react to caffeine differently than women?
If more men end up in the experimental group and more women end up in the control group, then gender and caffeine will be confounded. We will not know which variable caused the change in pulse rates, gender or caffeine.

16. Designing Experiments
How can we eliminate this potential confounding variable?
Eliminate one gender from the study, but then we could only draw conclusions about one gender.
Make sure there is a representative number of men and women in each treatment. For example, if there are 20 women and 30 men in the experiment, then the experimental group should have10 women and 15 men and the control group should have the same.
In this example, we have formed 2 blocks: men and women. Then, we assigned treatments to the subjects within each block.

17. Designing Experiments
Blocking in experiments is similar to stratification in sampling.
Blocking reduces the variability of the results, just like stratifying.
Blocks should be chosen like strata: the units within the block should be similar, but different than the units in the other blocks. You should only block when you expect that the subjects in one block will have a different response than subjects in other blocks.

18. Designing Experiments
What are some other extraneous factors that we can block for?
age, weight, initial pulse rate, etc.
You should try to make the blocks as small as possible. Ideally, the size of the block should be the same as the number of treatments. For example, if there are 3 treatments, then there should be 3 subjects in each block.
If each block has only 2 subjects, then the subjects are called a ______________.

19. Designing Experiments
Principle #3: _______________ is random assignment of subjects to treatments to ensure that the experiment doesn’t systematically favor one treatment over the other.
What about all of the other extraneous variables we do not think of? What about the variables we cannot directly control or block for?
amount of food eaten before experiment
caffeine tolerance

20. Designing Experiments
If we randomly assign subjects to treatments, this should _______________ (but not eliminate) the effects of these variables since their effects should be spread equally between the treatment groups.
Note: We must ALWAYS randomize since there will always be extraneous variables we do not consider.

21. Designing Experiments
How do we randomize?
Draw names from a hat. The first half chosen are in one group, the remaining names in the other.
Number the class from Then, generate random numbers without replacement until half are chosen for one group. The remaining names go in the other group.
For matched pairs, we can flip a coin to determine which subjects go into which group. If its heads, the first person in the pair goes to A and the other to B. If its tails, it’s the opposite.

22. Designing Experiments
If you do not use blocking when dividing the subjects, the result is a _______________________________.
If you incorporate blocking in your design, it is called a _________________________ (every subject is assigned to a block based on some characteristics and the members of the block are randomly assigned to the different treatments).

23. Designing Experiments
Principle #4: ____________ means ensuring that there is an adequate number of observations in each treatment group.
If each treatment group only had one experimental unit, then we would not be able to conclude that any changes in the response are due to the treatments. It is also possible that some characteristic of the unit was the cause of the change.
Ex: our coke experiment

24. Designing Experiments
Increasing the __________________ makes randomization more effective. The more subjects we have, the more balanced our treatment groups will be. For example, if we have 10 subjects and only 2 have a certain unknown characteristic, it is quite likely that both of those subjects will end up in the same treatment group simply by chance.

25. Designing Experiments
However, if we have 100 subjects and 20 have the characteristic, it is very unlikely for all 20 to end up in the same group. There is a much better chance that the groups will be close to balanced (10/10, 9/11, 11/9, etc.) when the sample size is larger.
If you were flipping a coin and wanted to get as close as possible to 50%, you would decide to flip the coin more than a few times!

26. Designing Experiments
Note: Replication can also refer to repeating the experiment with different subjects. This can help us be more confident applying the results of our experiment to a ________________________.
SUMMARY: With control, blocking, randomization, and replication, each treatment group should be nearly identical, and the effects of extraneous variables should be the same in each group. Now, if changes in the explanatory variable are associated with changes in the response variable, we can conclude that it is a cause-and-effect relationship.

27. Caffeine Experiment Design

29. End of Designing Experiments

30. More Designing Experiments
Not all experiments have CONTROL GROUPS or use a PLACEBO, as long as there is comparison. For example, if you are testing a new drug, it is usually compared to the currently used drug, not a placebo. Or, you can do an experiment to compare four brands of paint without using a placebo.

31. More Designing Experiments
There are ethical issues to consider when doing
experiments:
smoking and lung cancer: we cannot force people to smoke, but that would be the best way to prove smoking causes lung cancer.
many medical experiments are ended early if the experimenters discover that one treatment is much more effective: ARTICLE: Angioplasty Protocol Questioned: LA Times,
“The Tuskegee Study of Untreated Syphilis in the Negro Male”

32. More Designing Experiments
The ethical rules that govern medical experiments are a relatively recent development.
“The Tuskegee Study of Untreated Syphilis in the Negro Male” (1932 – 1972):
- Before 1945, there was no effective treatment. The ones attempted were dangerous. Initially, the study addressed the question whether untreated syphilis is preferable to these remedies.
- In 1945, a cure for syphilis was discovered. However, the study was continued and treatment was withheld.
- By the end of the study, 128 out of 399 died of syphilis or its complications.

33. More Designing Experiments
Even if caffeine has no effect on pulse rates, the average pulse rate of the two groups will probably be slightly different. Thus, whenever we do an experiment and find a difference between two groups, we need to determine if this difference occurred because of RANDOMIZATION VARIABILITY or because there really is a difference in the treatments.
The results of an experiment are called STATISTICALLY SIGNIFICANT if they are unlikely to occur by random chance.

34. More Designing Experiments
AP Test, 1999, Question 3:

35. More Designing Experiments

36. More Designing Experiments

37. The Scope of Inference
Not all experiments and their resultant hypothesis tests use random samples.
For example, in a study at UNC hospital, the researchers tried to determine the minimum daily requirement of choline for adults. They advertised at homeless shelters and natural food stores for volunteers. The subjects will spend 71 consecutive days in a hospital room and be paid $4,500 for their time and blood.

38. The Scope of Inference
These subjects are just the first 80 people who are willing to participate and who pass whatever general health requirements are necessary. They are not a random sample from any population. Many medical studies proceed along these lines. The randomization lies in the random assignment of treatment to the experimental unit (subject), not in the random selection of subjects from a population. With a random assignment of treatment to subject, the researchers can make causal inferences from the results of their study. They cannot make inferences to a larger population. It is the inference to the population from which the sample was drawn that is lost when techniques of random sampling are not used.

39. The Scope of Inference
On the other hand, in observational studies, there are generally random samples from distinct populations, so inferences can be made to those populations. However, causality cannot be attributed, since there was no random assignment to treatment groups.

40. The Scope of Inference
The scope of inference refers to the type of inferences (conclusions) that can be drawn from a study. The types of inferences we can make (inferences about the population and inferences about cause-and-effect) are determined by two factors in the design of the study:
how the subjects were selected from the population
how the subjects were assigned to groups.

41. The Scope of Inference Allocation of Subjects to Groups Randomized
Not randomized
Selection of subjects from the population
Random
Inferences about pop. – YES
Inferences about cause and effect – YES
Some experiments are here.
Inferences about cause and effect – NO
Most obs. studies are here.
Not random
Inferences about pop. – NO
Most experiments are here.
Some obs. studies are here.

42. The Scope of Inference
Suppose a dentist wants to know if a daily dose of 500 mg of vitamin C will result in fewer canker sores in the mouth than taking no vitamin C.

43. The Scope of Inference
Case 1) The dentist, working through the local dental society, convinces all of the dental patients in town with appointments the first two weeks in December to be subjects in an experiment. He divides them into two groups, those who take at least 500 mg of vitamin C each day and those who don't. He then asks them how often they have canker sores in their mouth and checks their patients records to see who has complained about canker sores. He compares the proportion of those who take vitamin C daily and complain of canker sores with the proportion of those who don't take vitamin C and complain of canker sores. There is a significant difference in the two proportions, with a significantly smaller proportion of those taking vitamin C having canker sores.

44. The Scope of Inference What can we conclude?
Since the patients do not represent a random sample from any population, it is not possible to make any inference about a larger population.
Since the study was observational, with subjects not randomly assigned to treatments, no causal inference can be made.
We just know that for these patients, those who take vitamin C have fewer canker sores than those who don't. We don't know why, and we don't know if this result would be consistent with another group.

45. The Scope of Inference
Case 2) A dentist, working through the local dental society, convinces all of the dental patients in town with appointments the first two weeks in December to be subjects in an experiment. He randomly assigns half of them to take 500 mg of vitamin C each day and the other half to abstain from taking vitamin C for three months. At the end of this time he determines the proportion of each group that has suffered from canker sores during those three months. There is a significant difference in the two proportions, with a significantly smaller proportion of those taking vitamin C having canker sores.

46. The Scope of Inference What can we conclude?
Since the patients do not represent a random sample from any population, it is not possible to make any inference about a larger population.
However, the treatments were randomly assigned to the subjects, so (assuming other factors were controlled or randomized) the difference in proportions having canker sores can be attributed to the vitamin C.
We don't know if this result would be consistent with another group, but we believe we know why, for this group, the proportions differ.

47. The Scope of Inference
Case 3) The dentist, working through the local dental society, selects a random sample of dental patients in town and convinces them to be subjects in an experiment. He divides them into two groups, those who take at least 500 mg of vitamin C each day and those who don't. He then asks them how often they have canker sores in their mouth and checks their patients records to see who has
complained about canker sores. He compares the proportion of those who take vitamin C daily and complain of canker sores with the proportion of those who don't take vitamin C and complain of canker sores. There is a significant difference in the two proportions, with a significantly smaller proportion of those taking vitamin C having canker sores.

48. The Scope of Inference What can we conclude?
Since the patients selected were a random sample of dental patients in town, we can infer that the results observed in this experiment would be consistent with results from the whole population of dental patients in this town.
However, since the study was observational, with subjects not being randomly assigned to treatments, no causal inference can be made.
We believe that, for the population of dental patients in this town, those taking vitamin C have fewer canker sores than those who didn't. We don't know if it is the vitamin C that causes this reduction or some other confounding variable. We cannot conclude that for the general population, those taking vitamin C have fewer canker sores, since the sample was only of dental patients.

49. The Scope of Inference
Case 4) The dentist, working through the local dental society, selects a random sample of dental patients in town and convinces them to be subjects in an experiment. He randomly assigns half of them to take 500 mg of vitamin C each day and the other half to abstain from taking vitamin C for three months. At the end of this time he determines the proportion of each group that has suffered from canker sores during those three months. There is a significant difference in the two proportions, with a significantly smaller proportion of those taking vitamin C having canker sores.

50. The Scope of Inference What can we conclude?
Since the patients selected were a random sample of dental patients in town, we can infer than the results observed in this experiment would be consistent with results from the whole population of dental patients in this town.
Moreover, the treatments were randomly assigned to the subjects, so (assuming other factors were controlled or randomized) the difference in proportions having canker sores can be attributed to the vitamin C.
We believe that for the population of dental patients in this town, taking vitamin C results in fewer canker sores.

51. Our Experiment

52. Does Caffeine Affect Pulse Rate?
An AP Stats Experiment
With Blocking by Gender

54. The Caffeine Experiment
Initial pulse rate: ____________________
Drink you were assigned to: ___________________

55. The Caffeine Experiment
Experiment Summary:
Response variable:
pulse rate
Explanatory variable:
caffeine
Experimental units:
students in this AP Statistics class
Treatments:
caffeine (the quantity in a cup of Regular Coke)
very little caffeine (the quantity in a cup of Caffeine-Free Coke)

56. The Caffeine Experiment
Variables that were directly controlled:
amount of sugar
drink temperature
environmental conditions while drinking
Variables that were blocked:
gender
initial pulse rate.

57. The Caffeine Experiment
What other extraneous factors can you think of?
genetic factors
tolerance to caffeine
what or if students ate in the morning …
What was done about these other extraneous factors?
Randomly allocating students in each matched pair should balance the groups with respect to any other factor.

58. The Caffeine Experiment
How was the replication principle applied in the experiment?
There were several matched pairs (more would have been better). Class size large lots of matched pairs.
Unfortunately, the experiment won’t be repeated with two classes, as there is only 1 AP Stat class this year.

60. The Caffeine Experiment
Describe: Shape, Center, Bias, Variability (Spread)

61. The Caffeine Experiment
AP Test, 2000, Question 5:

62. The Caffeine Experiment

63. The Caffeine Experiment

64. The Caffeine Experiment