1. UIC University of Illinois at Chicago Health Research and Policy Centers “Phases of Research and Development for Health Promotion Programs” AAHB Annual Conference, March 17 2003 PHASES OF RESEARCH AND DEVELOPMENT FOR HEALTH PROMOTION PROGRAMS Brian R. Flay, D.Phil., FAAHB Health Research and Policy Centers University of Illinois at Chicago Presented at the Annual Conference of the American Academy of Health Behavior St. Augustine, Florida, March 16-19, 2003

2. UIC University of Illinois at Chicago Health Research and Policy Centers “Phases of Research and Development for Health Promotion Programs” AAHB Annual Conference, March 17 2003 Phases of Research: An Historical View FDA (19??) OTA (1978) Medical Innovations (McKinley, 1981) NHLBI (1983) NCI (Greenwald & Cullen, 1984) Smoking Prevention (Flay & Best, 1982; Flay, 1986) Expedited FDA (for AIDS) (1990) Alcohol Prevention (Holder, Boyd, et al, 1995; Holder, Flay, et al, 1999)

3. UIC University of Illinois at Chicago Health Research and Policy Centers “Phases of Research and Development for Health Promotion Programs” AAHB Annual Conference, March 17 2003 Phases of Research in Intervention Development (Flay, 1986) 1.Basic Research 2.Hypothesis Development 3.Component Development and Pilot Studies 4.Prototype Studies of Complete Programs 5.Efficacy Trials of Refined Programs –Well controlled randomized trials 6. Treatment Effectiveness Trials –Generalizability of effects under standardized delivery 7. Implementation Effectiveness Trials –Effectiveness with real-world variations in implementation 8. Demonstration Studies –Implementation and evaluation in multiple whole systems

4. UIC University of Illinois at Chicago Health Research and Policy Centers “Phases of Research and Development for Health Promotion Programs” AAHB Annual Conference, March 17 2003 Limited Use of Phases Few researchers or research groups have followed the phases sequence systematically Sussman comes closest at the earliest phases –See Sussman et al., 1995 monograph –Used theory-driven program development –Conducted small-scale pilot studies of program components –Altered or added new activities as a result –Piloted full-scale intervention Botvin comes closest at the efficacy-effectiveness phases –Has conducted multiple efficacy trials –Has conducted multiple treatment effectiveness trials –Has conducted multiple implementation effectiveness trials To my knowledge, no one research group has done it all -- the substance abuse prevention field almost has.

5. UIC University of Illinois at Chicago Health Research and Policy Centers “Phases of Research and Development for Health Promotion Programs” AAHB Annual Conference, March 17 2003 Reasons for limited use of Phases Interventions are becoming more complex –Multi-component, involving multiple stake-holders Higher stages require large, rigorous studies – Often too costly for standard research grant funding Different research skills and interests are needed at the different phases Many interventions are developed outside the research environment –Developed by professionals in the field, communities or legislators –Holder and colleagues have suggested modifications to incorporate these types of interventions

6. UIC University of Illinois at Chicago Health Research and Policy Centers “Phases of Research and Development for Health Promotion Programs” AAHB Annual Conference, March 17 2003 Complex Interventions Always thought of as curricula, or whole programs, not separate components –Few field-based tests of efficacy of separate components to date –But curricula/programs based on basic and hypothesis-driven research Programs have grown more complex over the years Multiple outcomes are becoming the norm –E.g., multiple behaviors + Character + Achievement Also multiple ecologies are involved –School-wide –Involvement of parents/families –Involvement of community Therefore, multiple mediators, both distal and proximal –Distal: Family patterns, school climate, community involvement –Proximal: Attitudes, normative beliefs, self-efficacy, intentions

7. UIC University of Illinois at Chicago Health Research and Policy Centers “Phases of Research and Development for Health Promotion Programs” AAHB Annual Conference, March 17 2003 School-based Prevention/Promotion Studies are Large and Complex Large randomized trials –With multiple schools per condition Comparisons with “treatment as usual” Measurement of implementation process and program integrity Assessment of effects on presumed mediators –Helps test theories Multiple measures/sources of data –Surveys of students, parents, teachers, staff, community –Teacher and parent reports of behavior –School records for behavior and achievement Multiple, independent trials of promising programs –At both efficacy and effectiveness levels Cost-effectiveness analyses

8. UIC University of Illinois at Chicago Health Research and Policy Centers “Phases of Research and Development for Health Promotion Programs” AAHB Annual Conference, March 17 2003 Holder et al (1999) modifications For field-developed programs and policy interventions –Often skip efficacy trials and use “natural experiments” and quasi- experimental effectiveness trials. To distinguish between research-driven and program-driven interventions –Program-driven programs are not researcher-initiated –They are program- or policy-driven To allow for natural experiments or other quasi- experiments –Time-series analyses with natural variations –Comparison of states that adopt a policy with those that don’t

9. UIC University of Illinois at Chicago Health Research and Policy Centers “Phases of Research and Development for Health Promotion Programs” AAHB Annual Conference, March 17 2003 Phases of research for program-driven prevention interventions (Holder et al. 1999) 1. Foundational Research –Epidemiology, etiology and behavior change models 2. Developmental (Preliminary Effectiveness) Studies –Documentation and preliminary evaluation of an existing intervention –Should address safety and cost issues 3. Efficacy Studies –Quasi-experimental test of an optimally implemented intervention –Acceptance (compliance/readines) is also maximized when possible –Experimental or quasi-experimental test of well implemented variations of the intervention 4. Effectiveness Studies –Evaluation of the quality and levels of acceptance, implementation and overall effectiveness under real-world conditions 5. Diffusion Studies –Acceptance Evaluation -- of levels of acceptance and effects of –Implementation Evaluation -- of levels of implementation and effects of –Monitoring -- Continued monitoring of program maintenance and diffusion

10. UIC University of Illinois at Chicago Health Research and Policy Centers “Phases of Research and Development for Health Promotion Programs” AAHB Annual Conference, March 17 2003 Examples from Alcohol See Holder et al. (1999) for more examples and references DUI Enforcement Ross evaluated the effects of DUI enforcement with time-series analysis of archival data. Short-term results approach an estimate of maximum effects (I.e., efficacy). Long-term results (decay) represent effectiveness under on- going real-world conditions. Minimum drinking age Natural variation in time of legislation and implementation allowed for evaluation of the effects of changes in the minimum drinking age on youth alcohol use, DUI, crashes and other outcomes. Studies of high versus low implementation/enforcement states. Pricing Policies Early cross-sectional studies led to time-series econometric analyses. Studies in “early change” States where enforcement was high Studies comparing high and low enforcement states Other national-level studies

11. UIC University of Illinois at Chicago Health Research and Policy Centers “Phases of Research and Development for Health Promotion Programs” AAHB Annual Conference, March 17 2003 Much of Prevention is STUCK … We’re Spinning our Wheels At the Efficacy Trial phase –How can we get more programs into effectiveness trials At the Effectiveness Trial phase –How can we get more proven programs adopted At the “Model Program” phase –How can we ensure the ongoing effectiveness of model programs

12. UIC University of Illinois at Chicago Health Research and Policy Centers “Phases of Research and Development for Health Promotion Programs” AAHB Annual Conference, March 17 2003 What does the future hold? Where are the Dissemination Trials? –Deliberate studies of planned variations in implementation/delivery, training, etc. Where are the Demonstration Studies? –Large scale studies of effects in the real world –On-going assessments of sustained effectiveness Alternatives: Where do we go next? Conclusions

13. UIC University of Illinois at Chicago Health Research and Policy Centers “Phases of Research and Development for Health Promotion Programs” AAHB Annual Conference, March 17 2003 Selected References Flay BR (1986) Efficacy and effectiveness trials (and other phases of research) in the development of health promotion programs. Prev Med, 15, 451-474, 1986. Flay BR, Best JA (1982) Overcoming design problems in evaluating health behavior problems. Evaluation and Health Prof., 5, 43-69. Food and Drug Administration (1990) From test tube to patient: New drug development in the United States. DHHS Pub. No. (FDA) 90-3168. Greenwald P, Cullen JW (1984) The scientific approach to cancer control. CA Cancer J. Clin. 34, 328-332. Holder H, Boyd G, Howard J, Flay B, Voas R, Grossman M (1995) Alcohol-related prevention research policy: The need for a phases research model. J of Public Health Policy, 16(3), 324-346. Holder H, Flay B, Howard J, Boyd G, Voas R, & Grossman M (1999) Phases of alcohol problem prevention research. Alcoholism: Clinical and Experimental Research, 23(1), 183-194. McKinlay JB (1981) From “promising report” to “standard procedure”: Seven stages in the career of a medical innovation. Milbank Memorial Fund Quarterly/Health Sot. 59, 374-411. National Heart, Lung, and Blood Institute (1983) Guidelines for Demonstration and Education Research Grants. NHLBI, Washington, D.C. Office of Technology Assessment (1978) Assessing the Efficacy and Safety of Medical Technologies. U.S. Govt. Printing Office, Washington, D.C.