1. Experimental Design (Part 1)
Statistical Reasoning
Unit 5: Producing Data: Designing Experiments
Spring 2017

2. Classify each method as a survey, an experiment, or observational study, and explain which would be most reliable.
Example

3. How to Experiment Badly
Many laboratory experiments use a design like this one:
Experiment
Experimental Units
Treatment
Measure Response
In the lab environment, simple designs often work well.
Field experiments and experiments with animals or people deal with more variable conditions.
Outside the lab, badly designed experiments often yield worthless results because of confounding.
What is this design missing?
A Control Group

4. Principles of Experimental Design
Control – make conditions as similar as possible for all treatment groups (aside from the actual treatments).
If we observe a difference between groups, we want to know that it is a result of the treatment(s)!
Randomization – the use of chance to assign subjects/units to treatments
This helps create roughly equivalent groups of experimental units by balancing the effects of lurking variables that aren’t controlled on the treatment groups.
Replication of the experiment on many subjects/in different locations/etc.

5. How to Experiment Well Definition:
The remedy for confounding is to perform a comparative experiment in which some units receive one treatment and similar units receive another. Most well designed experiments compare two or more treatments.
Comparison alone isn’t enough, if the treatments are given to groups that differ greatly, bias will result. The solution to the problem of bias is random assignment.
Experiments
Definition:
In an experiment, random assignment means that experimental units are assigned to treatments at random, that is, using some sort of chance process.

6. why is random assignment important?
In theory, randomization EQUALIZES all other variables (including the things we can’t predict).
BUT…
…VARIATION WILL OCCUR. Sometimes we end up with “unfair” groups… That is why we replicate the experiment (perform the experiment a number of times, with different groups of subjects, in different locations, etc)

7. How to Experiment Well
Definition:
In a completely randomized design, the treatments are assigned to all the experimental units completely by chance.
Some experiments may include a control group that receives an inactive treatment or an existing baseline treatment.
Often, to demonstrate a cause and effect hypothesis, an experiment must show two things. First, that a phenomenon occurs with the treatment; and second, that the phenomenon does not occur in the absence of the treatment.

8. In a randomized comparative experiment, the individuals are assigned to the control group or the treatment group at random, in order to minimize bias. An experiment that is not a randomized comparative experiment may be subject to bias, and any conclusions drawn from the experiment may not be valid.
Group 1
Group 2
Treatment 1
Treatment 2
Compare Results
Experimental Units
Random Assignment

9. Experimental Design
In 2007, deaths of a large number of pet dogs and cats were ultimately traced to contamination of some brands of pet food. The manufacturer NOW claims that the food is safe, but before it can be released, an experiment to test whether the food is now safe for dogs and cats to eat must be conducted. What would the treatments be? What would the response variable be?
New food from the company, and another that I am certain is safe
Health of the pets, assessed by a veterinarian
*It may disturb you (as it does us) to think of deliberately putting dogs at risk in this experiment, but in fact that is what is done. 

10. Experimental Design
A group of 32 dog owners have volunteered their pets for this experimental study. Explain how you would carry out a completely* randomized experiment to see if the new food is safe for dogs to eat. The dogs will eat the assigned food for a period of 6 weeks.
*completely randomized experiment means NO BLOCKING ALLOWED!!!

11. COMPLETELY RANDOMIZED DESIGN
Treatment 1:
Dogs eat new food for 6 weeks
Group 1:
16 dogs
RANDOM ASSIGNMENT
Compare health of dogs, to be evaluated by veterinarian
Group of 32 dog volunteers
Group 2:
16 dogs
Treatment 2:
Dogs eat “safe” food for 6 weeks
*completely randomized experiment means NO BLOCKING ALLOWED!!!

12. (explaining the randomization procedure…)
Using a RNG on a calculator/computer, RANDOMLY assign each dog a unique number from 1 to 32.
The dogs with numbers 1 – 16 are placed in treatment group 1. The rest of the dogs will be placed in treatment group 2 (or the control group). OR
We will number the dogs from 1 to 32. Take 32 slips of paper, number them from 1 to 32, and put them in a hat. Stir the slips in the hat, then draw 16 of the slips without looking. The dogs with those numbers are placed in treatment group 1…

13. Determine whether the study described in the health bulletin below is a randomized comparative experiment. If it is, describe the treatment, the treatment group, and the control group. If it is not, explain why not and discuss whether the conclusions drawn from the study are valid.

14. SOLUTION
The study is not a randomized comparative experiment because the individuals were not randomly assigned to a control group and a treatment group. (In fact, the study is an observational study, not an experiment, since no treatment is imposed.)
The study’s conclusion that milk fights cavities may or may not be valid. There may be other reasons why students who chose milk had fewer cavities. For example, students who voluntarily choose milk at lunch may be more likely to have other healthy eating or dental care habits that could affect the number of cavities they have.

15. Example Treatment? Bathing in Epsom Salt Treatment Group?
The study described in the report is a randomized comparative experiment. Describe the treatment, the treatment group, and the control group.
Example
One hundred arthritis sufferers reported the severity of their symptoms daily for a month. Fifty of the subjects were given Epsom salt to bathe in at least every other day. At the end of the month, 30% of the subjects who used Epsom salt reported a decrease in severity of their symptoms, compared to 5% in the other group.
Treatment?
Bathing in Epsom Salt
Treatment Group?
50 subjects given Epsom Salt
Control Group?
50 not given Epsom salt

16. RECALL: The Randomized Comparative Experiment
Experiments
In a randomized comparative experiment, the individuals are assigned to the control group or the treatment group(s) at random, in order to minimize bias.
Group 1
Group 2
Treatment 1
Treatment 2
Compare Results
Experimental Units
Random Assignment

17. Principles of Experimental Design
RECALL:
What are the principles of experimental design?
Control
Randomization
Replication
Principles of Experimental Design

18. Experiments: What Can Go Wrong?
The logic of a randomized comparative experiment depends on our ability to treat all the subjects the same in every way except for the actual treatments being compared.
Good experiments, therefore, require careful attention to details to ensure that all subjects really are treated identically.
Experiments
A response to a dummy treatment is called a placebo effect. The strength of the placebo effect is a strong argument for randomized comparative experiments.
Whenever possible, experiments with human subjects should be double-blind.
BLINDING - units do not know which treatment they are getting

19. BLINDING
units do not know which treatment they are getting
DOUBLE BLINDING
neither the subjects nor the evaluator* know which treatment a subject received *(nor anybody who administers the treatments and/or cares for the subjects… basically ANYBODY who interacts directly with the experimental units)

20. Randomized Block Design
If an experimenter is aware of specific differences among groups of subjects or objects within an experimental group, he or she may prefer a randomized block design to a completely randomized design. In a block design, experimental subjects are first divided into homogeneous blocks before they are randomly assigned to a treatment group.

21. Example
If, for instance, an experimenter had reason to believe that age might be a significant factor in the effect of a given medication, he might choose to first divide the experimental subjects into age groups, such as under 30 years old, years old, and over 60 years old. Then, within each age level, individuals would be assigned to treatment groups using a completely randomized design. In a block design, both control and randomization are considered.

22. Blocking (a form of control)
Of the 32 dogs, 16 are poodles and 16 are German shepherds. Explain the changes you would make to your previous design by incorporating blocking.
It may disturb you (as it does us) to think of deliberately putting dogs at risk in this experiment… but in fact that is what is done. 

23. Blocking (a form of control)
Of the 32 dogs, 16 are poodles and 16 are German Shepherds. Explain the changes you would make to your previous design by incorporating blocking.
We will block by breed of dog (separate poodles from German shepherds) because….
(be sure to explain WHY!)

24. RANDOMIZED BLOCK DESIGN
Group 1:
8 dogs
Treatment 1
Dogs eat new food for 6 weeks
RANDOM
Block A:
16 poodles
Compare health of dogs
Treatment 2
Dogs eat “safe” food for 6 weeks
Group 2:
8 dogs
BLOCK BY
BREED
32 dogs
Group 3:
8 dogs
Treatment 1
Dogs eat new food for 6 weeks
RANDOM
Block B:
16 German shepherds
Compare health of dogs
Group 4:
8 dogs
Treatment 2
Dogs eat “safe” food for 6 weeks

25. Blocking is to experiments as ________ is to sampling.
stratifying

26. Matched-Pairs Design
A common type of randomized block design for comparing two treatments is a matched pairs design. The idea is to create blocks by matching pairs of similar experimental units.
Experiments
Definition
A matched-pairs design is a randomized blocked experiment in which each block consists of a matching pair of similar experimental units.
Chance is used to determine which unit in each pair gets each treatment.
Sometimes, a “pair” in a matched-pairs design consists of a single unit that receives both treatments. Since the order of the treatments can influence the response, chance is used to determine with treatment is applied first for each unit.

27. MATCHED PAIRS (a special type of BLOCKING)
EITHER…
Create blocks of size 2 (similar in some aspect)
Randomly assign each member of the pair to one of the two treatments OR
“Before-and-after” testing
Every subject receives BOTH treatments (order of two treatments is usually randomized)
Then measure difference between two results…

28. Pair experimental units according to specific characteristics.
Treatment A
Treatment B
Next, randomly assign one unit from a pair to Treatment A. The other unit gets Treatment B.
Pair experimental units according to specific characteristics.
This is one way to do a matched pairs design – another way is to have each individual unit do both treatments (as in a taste test).

29. Treatment A
Do not write:
If we flip “heads”, then ALL of the #1’s get treatment A (and ALL of the #2’s get treatment B)…
You should give each #1 (and #2) a fair chance of going either way.
Treatment B
In each pair, assign one unit the number “1” and the other the # “2”.
In each block (pair), we will flip a fair coin such that if the side of the coin facing up is…
“heads”, #1 will get treatment A (and #2 will get treatment B)
“tails”, #2 will get treatment A (and #1 will get treatment B)
Just make sure you flip a coin for EACH pair!

30. Matched-Pairs Example
A study to compare two new methods for teaching reading uses a matched subject research program. The researchers want to compare two methods, the current method and a modern method. They select two groups of children and match pairs of children across the two groups according to ability, using the results of their last reading comprehension test. If the researchers wanted to test another method, they would have to find three comparable children to compare between the three groups
Martyn Shuttleworth (Feb 27, 2009). Matched Subjects Designs. Retrieved Jan 08, 2017 from Explorable.com: 

31. Examples
A researcher is carrying out a study of the effectiveness of four different skin creams for the treatment of a certain skin disease. He has eighty subjects and plans to divide them into 4 treatment groups of twenty subjects each. Using a randomized block design, the subjects are assessed and put in blocks of four according to how severe their skin condition is; the four most severe cases are the first block, the next four most severe cases are the second block, and so on to the twentieth block. The four members of each block are then randomly assigned, one to each of the four treatment groups. 
(Example taken from Valerie J. Easton and John H. McColl's Statistics Glossary v1.1)

32. Is the difference in our results statistically significant?
(what does that even mean???)

33. Inference for Experiments
In an experiment, researchers usually hope to see a difference in the responses so large that it is unlikely to happen just because of chance variation.
Experiments
Definition:
An observed effect so large that it would rarely occur by chance is called statistically significant.
A statistically significant association in data from a well-designed experiment does imply causation.

34. SAT PREP CLASSES! (blegh!)
Higher SAT scores in only 6 weeks!
top-secret far-eastern study strategies!
3 sessions a week, only 5 hours per session!
all the other kids are being forced into it by their parents, so why not!?!
Cost: only $30,000!!! What a bargain!
Scores will improve (guaranteed!) or your money back!!!

35. SAT PREP CLASSES! 18 points!!! IT WORKED!!! …take this SAT class…
Group of 20 students
…mean score improvement of…
18 points!!!
IT WORKED!!!

36. SAT PREP CLASSES! IT WORKED!!! Group A scored WAYYY higher!!!
These 10 kids CARE about their grades… and choose to take this SAT class…
Group A scored WAYYY higher!!!
IT WORKED!!!
Group of 20 students
GROUP B:
These 10 kids DON’T CARE about their grades… and DON’T take this SAT class…

37. ::halftime::