1. 1 Chapter 4 The Designing Research Consumer

2. 2 High Quality Research: Evaluating Research Design High quality evaluation research uses the scientific method to investigate the effectiveness of programs and practices. Some evaluation studies are higher quality than others, and the research consumer must learn to distinguish among them. One important index of quality is the rigor of the studys research design.

3. 3 Research Design: The Structure of a Study An evaluations research design is its structure. At its most stingy, the structure consists of the new program study participants timing of outcome measures The research consumer must be able to distinguish between experimental and observational designs and what makes them internally and externally valid.

4. 4 The Randomized Controlled Trial (RCT): Going for the Gold An RCT is an experimental study in which eligible individuals or groups of individuals (e.g., schools, communities) are assigned at random to receive one of several programs or interventions. The group in an experiment that receives the specified program is called the experimental group. The term control group refers to another group assigned to the experiment, but not for the purpose of being exposed to the program. The performance of the control group usually serves as a standard against which to measure the effect of the program on the experimental group.

5. 5 The Randomized Controlled Trial: Going for the Gold (Continued) The control program may be typical practice (usual care), an alternative practice, or a placebo (a treatment or program believed to be inert or innocuous). Random assignment means that people end up in the experimental or in the control group by chance rather than by choice. In some randomized studies, the participants and investigators do not know which participants are in the experimental or the control groups: This is the double-blind experiment. When participants do not know, but investigators do, this is called the blinded trial.

6. 6 The Randomized Controlled Trial: Going for the Gold (Continued) Randomized controlled trials are sometimes called true experiments because, at their best, they can demonstrate causality. That means that, in theory, the researcher can assume that if participants in an RCT achieve desirable outcomes, the program caused them.

7. 7 Randomized Controlled Trials in Action Two commonly used randomized control designs are: 1. Concurrent controls in which two (or more) groups are randomly constituted and they are studied at the same time (concurrently). 2. Wait-list controls in which one group receives the program first; if the program appears to be effective, participants on the wait list receive it. Participants are randomly assigned to the experimental and wait-list groups.

8. 8 Non Randomized Controlled Trials (Quasi- Experiments) and Observational Studies True experiments (randomized controlled trials) are contrasted with non randomized controlled trials and observational studies. In non randomized controlled trials, the control group is predetermined (without random assignment) to be comparable to the program group Non randomized controlled trials are also called quasi experiments. In observational designs, the evaluator does not intervene but studies the effects of already existing programs Observational designs are sometimes referred to as descriptive.

9. 9 Non Randomized Controlled Trials Non randomized controlled trials rely on participants who -- 1) volunteer to join the study OR 2) are geographically close to the study site OR 3) conveniently turn up (at a clinic, school) while the study is being conducted Because the study groups are opportunistically rather than randomly composed, study group characteristics (age, sex) may not be balanced before (at baseline) the study begins. Baseline differences between groups may confound the studys results.

10. 10 Quasi Experimental Designs or Non Randomized Controlled Trials (Continued) Typical confounding variables include age, educational level, motivation, severity of illness, social structure, and income. Evaluation researchers worry that study groups in non randomized trials will differ from one another at baseline, and the studys findings will be compromised. They aim to create study groups that are as similar to one another as possible (equivalent) at baseline or before treatment. Among the strategies commonly used to ensure equivalence is one called matching.

11. 11 Quasi Experimental Designs or Non Randomized Controlled Trials (Continued) Matching requires selecting pairs of participants or clusters of individuals who are comparable to one another on important variables. A researcher who is interested in comparing the acuity of vision among smokers and non smokers can try to balance the two groups by selecting pairs of smokers and non smokers who are same age, sex, and have the same medical history Statistical methods such as analysis of covariance and propensity score analysis are sometimes used to deal with the problem of confounding after the data are collected for the study.

12. 12 More Quasi Experimental Designs Time-series Designs Time-series designs are longitudinal studies that enable the researcher to monitor change from one time to the next. They are sometimes called repeated measures analyses. Interrupted or Single Time-series The interrupted or single time-series design without a control group involves repeated measurement of a variable (e.g., reported crime) before and after implementation of a program. The goal is to evaluate whether the program has "interrupted" or changed a pattern established before the program's implementation.

13. 13 Even More Quasi Experimental Designs Self-Controlled or Pretest-Post Test Designs Each participant is measured on some important program variable and serves as his or her own control. Participants are usually measured twice (at baseline and after program participation), but they many be measured multiple times afterward as well. Historical Controls Investigators compare outcomes among participants who receive a new program with outcomes among a previous group of participants who received the standard program.

14. 14 Observational Designs Cohort Designs A cohort is a group of people who have something in common and who remain part of a study group over an extended period of time. In public health research, cohort studies are used to describe and predict the risk factors for a disease and the disease's cause, incidence, natural history and prognosis. They tend to be extremely large studies. Cohort studies may be prospective or retrospective. With a prospective design, the direction of inquiry is forward in time, while with a retrospective design, the direction is backward in time.

15. 15 More Observational Designs Case Control Designs Case-control designs are generally retrospective. They are used to explain why a phenomenon currently exists by comparing the histories of two different groups, one of which is involved in the phenomenon. For example, a case control design might be used to help understand the social, demographic, and attitudinal variables that distinguish people who at the present time have been identified with frequent headaches from those who do not at the present time have frequent headaches.

16. 16 More Observational Designs Cross-Sectional Designs Cross-sectional designs result in a portrait of one or many groups at one period of time. They are sometimes called descriptive, pre experimental, or survey designs.

17. 17 Internal and External Validity: The Route to Quality Internal Validity is the ability to make accurate inferences about a programs outcomes and effectiveness. (Program A caused Outcome A.) External validity refers to the extent to which the results are applicable to other programs, populations, and settings. Another term for external validity is generalizability.

18. 18 Internal Validity: How it is Jeopardized The research consumer wants to be sure that Program A which was conducted in Setting A 1.caused Outcome A (internal validity) and 2.will be effective in Settings B and C, etc. (external validity) Study design flaws can threaten internal and external validity.

19. 19 Internal Validity and Its Threats Threat to ValidityExplanation SelectionGroup characteristics (e.g., age, gender) are not evenly balanced HistoryUnanticipated outside events occur while study is in progress MaturationParticipants change or mature TestingTaking one test effects performance on the second

20. 20 Internal Validity and Its Threats (Continued) Threat to ValidityExplanation InstrumentationChanges occur in measures or observers Statistical regressionParticipants who are selected because of extremely high or low scores return to the average over time AttritionParticipants drop out of the study in a non random manner ExpectancyParticipants or researchers have expectations of the program and study activities

21. 21 External Validity and Its Threats Threat to External ValidityExplanation Interaction between selection and experiment A unique mixture is created that is unlikely to occur again Reactive effects of testingCompleting the baseline measures influences program participation uniquely Reactive effects of experimentation (Hawthorne Effect) Study participants behave uncharacteristically because they are being observed Multiple program interferenceComplementary activities may influence participants behavior and study outcomes