1. Chapter 9 Experiments1 Chapter 9 Producing Data: Experiments

2. Chapter 9 Experiments2 Experimentation u An experiment is the process of subjecting experimental units to treatments and observing their response. u Only properly designed and executed experiments can reliably demonstrate causation.

3. Chapter 9 Experiments3 Common Language u Subjects The individuals in an experiment are the experimental units. If they are human, we call them subjects. u Response variable –what is measured as the outcome or result of a study. u Explanatory variable (factors) –what we think explains or causes changes in the response variable –often determines how subjects are split into groups

4. Chapter 9 Experiments4 Common Language u Treatments –specific experimental conditions (related to the explanatory variable) applied to the subjects –If an experiment has several factors, a treatment is a combination of specific values of each factor.  The factor may be the administration of a drug.  One group of people may be placed on a diet/exercise program for 6 months (treatment), and their blood pressure (response variable) would be compared with that of people who did not diet or exercise

5. Chapter 9 Experiments5 Thought Question In an experiment to determine whether the drug memantine improves cognition of patients with moderate to severe Alzheimer’s disease, whether or not the patient received memantine is one variable, and cognitive score is the other. Which is the explanatory variable and which is the response variable?

6. Common Language Chapter 9 Experiments6 u If the experiment involves giving two different doses of a drug, we say that we are testing two levels of the factor. In a study of sickle cell anemia, 150 patients were given the drug hydroxyurea, and 150 were given a placebo (dummy pill). The researchers counted the episodes of pain in each subject. Identify: The subjects : The factors/treatments: And the response variable: (patients, all 300) (hydroxyurea and placebo) (episodes of pain)

7. How to Experiment Badly Chapter 9 Experiments7 Subjects Treatment Measure response In a controlled environment of a laboratory (especially if human subjects are not being used), a simple design like this one, where all subjects receive the same treatment, can work well.  Field experiments and experiments with human subjects are exposed to more variable conditions and deal with more variable subjects.  Designs that do not take lurking variables into account may produce misleading results due to confounding.

8. Chapter 9 Experiments8 Confounding (Lurking) Variables u The problem: –in addition to the explanatory variable of interest, there may be other variables that make the groups being studied different from each other –the impact of these variables cannot be separated from the impact of the explanatory variable on the response

9. Chapter 9 Experiments9 Confounding u The effects from two or more variables cannot be distinguished from each other. Example: Smoking and Lung Cancer - Suppose most smokers are adults - Is Cancer due to age or smoking?

10. Chapter 9 Experiments10

11. Chapter 9 Experiments11 Confounding (Lurking) Variables u The solution: –Experiment: randomize experimental units to receive different treatments (possible confounding variables should “even out” across groups) –Observational Study: measure potential confounding variables and determine if they have an impact on the response (may then adjust for these variables in the statistical analysis)

12. 3 Principles of Experimental Design 1. Control the effects of lurking variables on the response, most simply by comparing two or more treatments. 2. Randomize—use chance to assign subjects to treatments. 3. Replicate—use enough subjects in each group to reduce chance variation in the results. Chapter 9 Experiments12

13. Principles of Comparative Experiments Experiments are comparative in nature: We compare the response to a treatment versus to:  another treatment  no treatment (a control)  a placebo  or any combination of the above A control is a situation in which no treatment is administered. It serves as a reference mark for an actual treatment (e.g., a group of subjects does not receive any drug or pill of any kind). A placebo is a fake treatment, such as a sugar pill. It is used to test the hypothesis that the response to the treatment is due to the actual treatment and not to how the subject is being taken care of. Chapter 9 Experiments13

14. About the Placebo Effect The “placebo effect” is an improvement in health due not to any treatment but only to the patient’s belief that he or she will improve. –The placebo effect is not understood, but it is believed to have therapeutic results on up to a whopping 35% of patients. –It can sometimes ease the symptoms of a variety of ills, from asthma to pain to high blood pressure and even to heart attacks. –An opposite, or “negative placebo effect,” has been observed when patients believe their health will get worse. Chapter 9 Experiments14 The most famous and perhaps most powerful placebo is the “kiss,” blow, or hug—whatever your technique. Unfortunately, the effect gradually disappears once children figure out that they sometimes get better without help, and vice versa.

15. Chapter 9 Experiments15 Randomization in Expts. u Randomized Comparative Experiments The experimental units are divided into different groups through a process of random selection.

16. Chapter 9 Experiments16 The Role of Randomization Well designed statistical studies employ randomization to avoid subjective and other biases. u Experiments should use random assignment of experimental subjects to treatment groups –to ensure comparisons are fair –i.e., treatment groups are as similar as possible in every way except for the treatment being used.

17. Chapter 9 Experiments17 Completely Randomized Design In a completely randomized experimental design, individuals are randomly assigned to groups, then the groups are randomly assigned to treatments.

18. Chapter 9 Experiments18 Thought Question In an observational study, researchers observe what individuals do (or have done) naturally, while in an experiment, they randomly assign the individuals to groups to receive one of several “treatments.” Give an example of a situation where an experiment would not be feasible and thus an observational study would be needed.

19. Chapter 9 Experiments19 Thought Question In testing the effect of memantine on the cognition of Alzheimer’s disease patients (from TQ #1), how would you go about randomizing 100 patients to the two treatment groups (memantine group & placebo group)? Why is it necessary to randomly assign the subjects, rather than having the experimenter decide which patients should get which treatment?

20. Chapter 9 Experiments20 Why Not Always Use a Randomized Experiment? u Sometimes it is unethical or impossible to assign people to receive a specific treatment. –-Smoking/Lung cancer Experiment. u Certain explanatory variables, such as handedness or gender, are inherent traits and cannot be randomly assigned.

21. Chapter 9 Experiments21 Statistical Significance u If an experiment or observational study finds a difference in two (or more) groups, is this difference really important? u If the observed difference is larger than what would be expected just by chance, then it is labeled statistically significant. u Rather than relying solely on the label of statistical significance, also look at the actual results to determine if they are practically important.

22. Chapter 9 Experiments22 Example: A large study used records from Canada’s national health care system to compare effectiveness of two ways to treat prostrate disease. The two treatments are traditional surgery and a new method that does not require surgery. The records described many patients whose doctors had chosen the other method. The study found that patients treated by the new method were significantly more likely to die within 8 years. (a) Further study of the data showed that this conclusion was wrong. The extra death among patients treated with the new method could be explained by lurking variables. What lurking variables might be confounded with a doctor’s choice of surgical or nonsurgical treatment?

23. Chapter 9 Experiments23 Example Cont. Patients who have little chances of survival would probably be referred to the new treatment – no need going through painful surgery, save money and hospital admission. (b) You have 300 prostrate patients who are willing to serve as subjects in an experiment to compare the two methods. Use a diagram to outline the design of a randomized comparative experiment.

24. Chapter 9 Experiments24 Clinical Trials u Experiments that study the effectiveness of medical treatments on actual patients.

25. Chapter 9 Experiments25 Thought Question Suppose you are interested in determining if drinking a glass of red wine each day helps prevent heartburn. You recruit 40 adults age 50 and older to participate in an experiment. You want half of them to drink a glass of red wine each day and the other half to not do so. You ask them which they would prefer, and 20 say they would like to drink the red wine and the other 20 say they would not. You ask each of them to record how many cases of heartburn they have in the next six months. At the end of that time period, you compare the results reported from the two groups. Give three reasons why this is not a good experiment.

26. Chapter 9 Experiments26 Blocking u Refers to the idea of only making comparisons within relatively similar groups of subjects In the smoking lung cancer example, we could “block” age, subjects in a block will have similar ages. u Completely Randomized Block Design

27. Block Designs u In a block design, subjects are divided into groups, or blocks, prior to the experiment to test hypotheses about differences between the groups. In a block design, each block is treated as a completely randomized design. Chapter 9 Experiments27 The blocking here is by gender. Completely randomized designs

28. Chapter 9 Experiments28 Example: The progress of a type of cancer differs in men and women. A clinical experiment to compare four therapies for this cancer therefore treats sex as a blocking variable. (a) You have 500 male and 300 female patients who are willing to serve as subjects. Use a diagram to outline a block design for this experiment.

29. Chapter 9 Experiments29 Example Cont. (b) What are the advantages of a block design over a completely randomized design using these 800 subjects? A block design allows the researchers to control for differences between men and women. If, for example, the treatment is effective for men but not women (or vice versa), the treatment might be found to be ineffective overall if both genders are mixed together.

30. Chapter 9 Experiments30 Blocking and Randomization in Experiments u Block to ensure fair comparisons with respect to factors known to be important u Randomize to try to obtain comparability with respect to unknown factors u Randomization also allows the calculation of how much the estimates made from the study data are likely to be in error

31. Chapter 9 Experiments31 Blocking and randomization in expts. u Block to ensure fair comparisons with respect to factors known to be important. “Block what you can and randomize what you cannot.”

32. Chapter 9 Experiments32 “Blocking” vs “stratification” “Blocking” u word used in describing an experimental design “Stratification” u used in describing a survey or observational study u Both refer to idea of only making comparisons within relatively similar groups of subjects

33. Chapter 9 Experiments33 Blinding: –Preventing people involved in an experiment from knowing which experimental subjects have received which treatment –One may be able to blind v subjects themselves v people administering the treatments v people measuring the results u Double Blind: Both the subjects and those administering the treatments have been blinded.

34. Chapter 9 Experiments34 Hawthorne, Placebo, and Experimenter Effects u The problem: –people may respond differently when they know they are part of an experiment. u The solution: –use placebos, control groups, and double- blind studies when possible.

35. Chapter 9 Experiments35 Hawthorne, Placebo, and Experimenter Effects : Case Study I 1920’s Experiment by Hawthorne Works of the Western Electric Company u What changes in working conditions improve productivity of workers? –More lighting? –Less lighting? –Other changes? u All changes improved productivity!

36. Chapter 9 Experiments36 Hawthorne, Placebo, and Experimenter Effects : Case Study II Experimenter Effects in Behavioral Research (Rosenthal, 1976, Irvington Pub., p. 410) u Teachers given a list of student names –told these were students “who would show unusual academic development.” u IQ was measured at end of year –first graders on list: 15 points higher –second graders on list: 9.5 points higher –older: no striking difference u Great expectations = self-fulfilling prophecy –students were randomly selected (did not have high IQ)

37. Chapter 9 Experiments37 Experiments: Some Techniques u Matched Pairs: Choose pairs of subjects that are closely matched— e.g., same sex, height, weight, age, and race. Within each pair, randomly assign who will receive which treatment. –to reduce a source of variability in responses –the same or similar subjects receive each treatment

38. Matched Pairs Designs u Matched pair designs can also be implemented on individuals: For example weight change before and after a weight-loss diet, or change of property values for a group of houses before and after a shopping mall is constructed in their neighborhood. u It is also possible to just use a single person and give the two treatments to this person over time in random order. In this case, the “matched pair” is just the same person at different points in time. Chapter 9 Experiments38 The most closely matched pair studies use identical twins.

39. Chapter 9 Experiments39 Double-Blinded: Case Study Quitting Smoking with Nicotine Patches (JAMA, Feb. 23, 1994, pp. 595-600) u Variables: –Explanatory: Treatment assignment –Response: Cessation of smoking (yes/no) u Double-blinded –Participants do not know which patch they received –Nor do those measuring smoking behavior

40. Chapter 9 Experiments40 Experiments: Difficulties and Disasters u Extraneous variables –Confounding variables –Interacting variables u Hawthorne, placebo, and experimenter effects u Refusals, non-adherers, dropouts u Extending the results (generalizing)

41. Chapter 9 Experiments41 Interacting Variables u The problem: –effect of explanatory variable on response variable may vary over levels of other variables. u The solution: –measure and study potential interacting variables. v does the relationship between explanatory and response variables change for different levels of these interacting variables? v if so, report results for different groups defined by the levels of the interacting variables.

42. Chapter 9 Experiments42 Interacting Variables: Case Study Quitting Smoking with Nicotine Patches (JAMA, Feb. 23, 1994, pp. 595-600) u Researchers considered: –smoker at home v found this to be an interacting variable: Percent quitting Nicotine Placebo Smoker at home 31% 20% No smoker at home 58% 20% – other variables: age, weight, depression v no interactions found

43. Chapter 9 Experiments43 Extending the Results ( Can We Generalize? ) u The problem: –lack of generalizability due to: v unrealistic treatments v unnatural settings v sample that is not representative of population u The solution: –Researchers should use natural settings with a properly chosen sample.

44. Chapter 9 Experiments44 Extending the Results : Case Study Does Aspirin Prevent Heart Attacks? (NEJM, Jan. 28, 1988, pp. 262-264) u Participants were measured in their natural setting (at home) u Only healthy male physicians were participants –Results may not apply to: v male physical labourers v women

45. Chapter 9 Experiments45 Example: Page 234 #9.46 An herb for depression? Does the herb Saint-John’s wort relieve major depression? Here are some excerpts from the report of a study of this issue. The study concluded that the herb is no more effective than a placebo. a) “Design: Randomized, double-blind, placebo-controlled clinical trial…” A clinical trial is a medical experiment using actual patients as subjects. Explain the meaning of each of the other terms in this description. b) “Participants… were randomly assigned to receive either Saint-John’s wort extract (n=98) or placebo (n=102)…The primary outcome measure was the rate of change in the Hamilton Rating Scale for Depression over the treatment period.” Based on this information, use a diagram to outline the design of this clinical trial.

46. Chapter 9 Experiments46 Quiz u In what sense does random allocation make for comparisons that are fair or unbiased? u What is a completely randomized design? u What is blocking? u Why do we use blocking in designing experiments? u What is a control group and why are control groups used? u What is a placebo effect?

47. Chapter 9 Experiments47 Key Concepts u Critical evaluation of an experiment or observational study u Common terms –explanatory vs. response variables –treatments, randomization u Randomized experiments –basic principles and terminology –problem with confounding variables u Double-Blind Experiment

48. Chapter 9 Experiments48 Key Concepts u Difficulties and Disasters u Experimental Designs –Completely Randomized Design –Matched Pairs Design –Block Design