2ResearchIn your “Testing Claims” project you will be investigating research studies that back up a claim.You use the C.R.A.P. test to evaluate sources, but what do you do if you find a research study from what appears to be a reliable source, but may have other problems.How do you analyze the problems that may exist in a research study?
4Example:Based on results from a recent experiment, Dr. Lite claims that she has found a way to prevent 95% of hair loss in men and women (aged 25-80), with no adverse side effects. You are interested in investing in Dr. Lite’s product, but first you need to determine the quality of her study. You would like to make sure that Dr. Lite’s research findings would replicate if the study was repeated with a different sample. You know that research findings from studies with rigorous methods are more likely to replicate, so you decide to ask Dr. Lite a few questions.
5Clicker Question 1Which of the following questions about the study’s research methodology would you ask Dr. Lite first?Did the participants think the treatment was easy to use?How did you select your sample?Did participants report hair regrowth?Was there a significant difference between treatment and control groups?
6Clicker Question 2After learning that Dr. Lite randomly selected participants (50% female, aged 25-80) from the population of interest, what would you ask Dr. Lite next?How many participants were in your sample?Was this a convenience sample?Were subjects paid to participate?Did family members notice a change in the participants’ hair following the treatments?
7Clicker Question 3Dr. Lite informs you that there were approximately 1000 participants in the study. Half of the participants were in the experimental group (used the treatment) and half of the participants were in the control group (used a placebo treatment). What would you want to ask Dr. Lite next?Was there a significant difference in hair loss between the treatment and control group?How many weeks did the treatment effects last?How were subjects assigned to conditions?Were there any confounds in this study?
8Clicker Question 4You learn that the 1000 participants were randomly assigned to the experimental and control groups.Were the investigators and participants blind to the study conditions?How much does the treatment cost?Does the treatment effectiveness change depending on initial hair thickness?Who funded this study?
9Clicker Question 5Dr. Lite informs you that the researchers and participants were blind to the conditions of the study (i.e., double blind). What might you ask next?How many researchers helped with this study?How many weeks did the treatment effects last?Was there a significant difference in hair loss between the treatment and control group?Were the procedures for the experimental and control group the same, with the exception of the actual treatment?
10Clicker Question 6After learning that Dr. Lite used strict experimental control, and that the conditions for each group were exactly the same, how would you rate the strength of the evidence provided by this study’s findings?very weakmoderately weakmoderately strongvery strongWeak StrongSmall sample sizeLarge sample sizeNon-random sampleRandom sampleSignificant ConfoundsInsignificant ConfoundsResearcher Bias PresentResearcher Bias Absent
11Your Project:Use one of the references from your project with the scale below to evaluate the strength of the evidence reported in the study.For each of the 4 categories, explain why you thought the study earned it’s +1 point rating.Then provide an overall rating out of 4.Weak StrongSmall sample sizeLarge sample sizeNon-random sampleRandom sampleSignificant ConfoundsInsignificant ConfoundsResearcher Bias PresentResearcher Bias Absent