Presentation on theme: "1 From Efficacy to Effectiveness: the Safe in the City Model 2009 Meeting of the International Society for STD Research June 28 – July 1, 2009 London 2009."— Presentation transcript:
1 From Efficacy to Effectiveness: the Safe in the City Model 2009 Meeting of the International Society for STD Research June 28 – July 1, 2009 London 2009 Meeting of the International Society for STD Research June 28 – July 1, 2009 London
2 Efficacy vs. Effectiveness Efficacy: How well an intervention works in optimal settings –Well-trained and paid research staff –Controlled Setting –Carefully recruited study subjects –Carefully constructed and executed research protocol
3 Efficacy vs. Effectiveness Effectiveness: How well an intervention works in the real world –Staff not specifically trained for intervention; and need to meet competing demands –Targets all patients/clients, not selected subjects –Adaptation of intervention protocol often necessary to make it practical for the setting (threat to fidelity)
4 Effectiveness Research Implicitly takes into account the acceptability, feasibility, implementation, and impact of an intervention in the settings for which the intervention was designed
5 Effectiveness Research 3 Main Components Participatory research model Efficacy/Effectiveness evaluation Scale-up process and evaluation
6 Participatory Research Model Involves all stake holders that determine the ultimate success of the intervention: –Academics / intervention developers –Target population representatives –Representatives from the organizations responsible for implementation
7 Efficacy/Effectiveness Evaluation Evaluation of efficacy/effectiveness should take place in an environment that closely mimics the environment for which the intervention is ultimately intended
8 Scale – Up Process and evaluation Marketing strategies and implementation Evaluation of intervention up-take by target audience Short to long term assessment of implementation and sustainability
10 Safe in the City Project Investigators Centers for Disease Control and Prevention Lee Warner, Andrew Margolis, Jocelyn Patterson, Craig Borkowf Denver Public Health Cornelis Rietmeijer, John Douglas, Doug Richardson Education Development Center, Inc. Lydia O’Donnell, Athi Myint-U, Carl O’Donnell Long Beach, California State University and Department of Health and Human Services Kevin Malotte, Shelley Vrungos, Nettie DeAugustine San Francisco Department of Public Health Jeffrey Klausner, Gregory Greenwood, Carolyn Hunt
11 Safe in the City: Focus on Effectiveness Collaborative, participatory research process involving all stakeholders Formative process Intervention outcome study that involved entire clinic populations Use of STI markers of effectiveness Closely-guided and evaluated post- study dissemination phase
12 Safe in the City Development of the Intervention Cornelis A. Rietmeijer, MD, PhD Denver Public Health Department Denver, Colorado, USA
13 Study Rationale 340,000,000 incident STDs worldwide annually STD clinics provide access to men and women likely to be infected and to acquire new infections over time Yet behavioral interventions with counseling or multiple sessions are difficult to implement in busy medical settings Recent interest in simple, easy to use, and low cost interventions for waiting rooms
14 Rationale continued Previous research suggests benefits of video- based approaches, but subject to limitations: Controlled research settings Tailored videos Single site Inclusion of group counseling Effectiveness of stand-alone video in ‘real-world’ setting is unknown
15 Safe in the City Project Overview 5-year CDC-funded multi-site study Develop a brief video-based STD clinic waiting room intervention to reduce (or eliminate) STI and risky sexual behavior Evaluate effectiveness in 3 publicly funded STD clinics in Denver, San Francisco, and Long Beach, CA.
16 Intervention Development Considerations Waiting rooms in medical settings provide an underused opportunity to reach patients who are thinking about their health. Yet to be feasible and sustainable, interventions must: Be easy and inexpensive to administer Result in minimal interruption of patient flow Require few clinic resources, especially staff time Be acceptable to diverse clients
17 Formative Process Identification of intervention medium, theoretical framework, and key messages by research team Collaboration with award-winning film maker to integrate framework in an appealing product Multi-step participatory process involving target audience, clinic staff, and community advisors Intervention research study in 3 STD clinics
18 Intervention Development: Integrated Theoretical Framework Core constructs grouped into interconnected elements → HIV/STD risk, knowledge, perception → Positive attitudes toward condom use → Self-efficacy/skills for condom negotiation, acquisition, use → Modeling of appropriate behaviors Theory of Planned Behavior Social Cognitive Theory Information Motivation Behavior Model
19 Focus Groups 3 sites held 12 focus groups with 176 participants 3 different stages of video development: Story line development Script development Post-production editing
20 What Is the Intervention? 23-minute video 3 story lines 2 cartoon animations Condom variety and selection Instructions for use Posters in waiting and exam rooms
21 Story Line – Paul and Jasmine Things are getting more serious between Paul and Jasmine, but Paul “slips” and has a sexual encounter with Teresa. Teresa gets an STD and tells Paul. Now Paul has to tell Jasmine.
22 Story Line – Rubén, Tim and Christina Rubén’s girlfriend Christina doesn’t know about his interest in men. Rubén and Tim have a casual sex encounter after meeting in a bar. Days later, Christina suspects something is wrong. She insists on a visit to the STD clinic.
23 Story Line – Teresa and Luis Teresa has recently met Luis. After her STD scare with Paul, Teresa is serious about wanting to use condoms. Now she has to convince Luis.
26 Assessing Intervention Effectiveness in an STD Clinic Population: the Safe in the City Model Lee Warner, PhD Centers for Disease Control and Prevention Atlanta, Georgia, USA
27 Examples of Challenges in Study Design Examining “real-world” effectiveness prohibits active patient enrollment *still need to deliver intervention, include entire clinic population Large sample size required, given effectiveness of brief intervention likely modest Biologic markers (STI) preferred effectiveness measure to self-reported behavior Unable to randomize in waiting room setting, but need balance between study conditions
28 Maximizing Intervention Delivery and Exposure Identify environmental characteristics of waiting rooms Observe waiting room flow Determine appropriate playback frequency Identify factors to increase viewership (goal: 80%) Assess and adjust to clinic staff acceptance of video
32 Video Viewership Viewership as defined by watched most or all of the video + identified a main message
33 Study Objective To determine whether this brief, structural intervention reduced incident laboratory- confirmed infection among typical visitors to an STD clinic
34 Overview of Study Design Population: = all patients attending 3 STD clinics from December 2003 – August 2005 (N = ~40,000) Study design: 2 arm non-randomized controlled trial Arm assignment: alternating 4-week control & intervention periods Data collection: retrospective review of clinic data & external surveillance records to ascertain new STI diagnoses, conducted under waiver of informed consent
35 Study Flow Diagram Patient Presents at Clinic N=38,635 Intervention: 23 Minute Video Shown in Clinic Waiting Room Control: Standard Waiting Room Experience Clinical Evaluation Review of Medical Records & Surveillance Registry Data for STD outcomes Assignment Based on Clinic Visit Date
36 Characteristics of Clinic Populations, by Condition Control (n = 19,562) Intervention (n = 19,073) Sex Male 70% 69% Female 30% 31% Race/ethnicity Black, non-Hispanic 19% 18% White, non-Hispanic 46% Hispanic Other 25% 10% 25% 11% Age <25 >25 31% 69% 31% 69% Sexual orientation: MSM STI at index visit 22% 15% 21% 16%
37 Approach to Statistical Analysis Outcome: Incident laboratory-confirmed infection –gonorrhea, chlamydia, trichomoniasis, syphilis, and HIV Analysis: Survival analyses estimating time to infection –Kaplan-Meier analyses comparing survival at specific points in time by condition* –Cox proportional hazards regression comparing hazard ratios by condition*
38 Control N (%) Intervention N (%) GC539 (2.8) 409 (2.1) CT666 (3.4) 573 (3.0) Trich 71 (0.4) Syphilis 40 (0.2) 35 (0.2) HIV 10 (0.0) 4 (0.0) Any infection 1,113 (5.7)929 (4.9) First New Laboratory-Confirmed Infection, By Condition
39 Overall Effect of Intervention on Laboratory-Confirmed Infection* Hazard Ratio (95% CI) All patients0.91 (0.84-0.99) * = 9% reduction in STI incidence
40 Intervention Effect, by Characteristic Hazard Ratio (95% CI) STI at index visit Yes0.86 (0.75-0.99) No0.93 (0.84-1.04) Sex Males0.87 (0.78-0.96) Females Sexual orientation Heterosexual MSM Age <25 >25 1.06 (0.89-1.25) 0.84 (0.71-0.98) 0.90 (0.79-1.03) 1.02 (0.88-1.17) 0.85 (0.77-0.95)
41 ControlIntervention STD clinic76.5%77.3% Other facilities23.5%22.7% Source of STI Diagnosis, By Condition p = 0.67
42 ControlIntervention Any visit40.0%40.1% Mean no. visits0.850.80 Return Visits to Clinic, By Condition p = 0.86 p = 0.39 p = 0.86 p = 0.39
43 Factors Significantly Associated with Incident STI, by Strength of Effect Measure MSM Baseline STI diagnosis Minority race / ethnicity Age < 25 yrs Highest Lowest
44 Meeting Challenges in Study Design Evaluation of intervention effectiveness in “real-world” setting with “passive enrollment” of patient population Efficient examination of laboratory-confirmed STI using existing medical / surveillance records Inclusion of entire patient population, ensuring generalizability Non-randomized design with systematic allocation balanced all measured characteristics
45 Factors Critical to Success of Evaluation Ability to examine study population in advance: –Patient flow (e.g., number of patients, waiting room time) –Reaction of patient / staff to video –STI prevalence and incidence of population known Access to electronic medical records and surveillance registries to identify incident STI Waiver of informed consent from IRB critical
46 Conclusions “Safe in the City” associated with reduction in incident STI *** important -- estimate = effectiveness, not efficacy. This reduction can have significant public health benefit, but requires sufficient availability of intervention and wide adoption by clinics. Relatively* easy-to-implement, low cost interventions can reach large numbers of high-risk populations with minimal effort. Type of design used to examine effectiveness of “Safe in City” promising for studies of STD prevention interventions.
53 CDC’s Diffusion Goals Get the word out Make it available Provide support Coordinated by the Diffusion of Effective Behavioral Interventions (DEBI) CDC & Academy of Education Development Expertise in disseminating STD/HIV trainings and interventions www.EffectiveInterventions.org
54 CDC/DEBI’s Action Plan SITC homepage and order form launched on DEBI website the same day the PLoS article is released. SafeInTheCity.org goes live – view and download the video. CDC issues a “Dear Colleague” letter encouraging STD clinics to consider implementing SitC. Podcasts and CDC press release
57 Safe in the City Kit Materials DVD with video and pdf posters User’s Guide Now Showing poster CDC documents
58 Safe in the City Kit Requests June 24, 2008 – May 28, 2009 Agency type# of kit requests STD Clinics444 Health Departments266 Community-Based Organizations266 Health Services Clinics258 Family Planning Clinics190 Others (e.g., Hospitals, Univ. Research, Univ. Student Health Centers) 421 Total1,845
59 We built it, they came, but… How many clinics were actually using the intervention? What were the obstacles to implementing it? Could we help?
60 Follow-up on the Initial Launch Call clinics to do a brief survey and to offer technical assistance. –Clinics requesting technical assistance –Random sample STD clinics Two net meetings with intervention developers
61 Survey Details 128 clinics sampled in Nov 2008 52 clinics who indicated a possible need for tech assistance 76 additional randomly selected STD clinics 87 completed a baseline survey 81 completed a follow-up survey 4-5 months later
62 Site characteristics Clinic Type (self reported) n % STD 7080 Health Services 1315 Family Planning 22 Other 22 Median Patient Annual Volume 2500
63 How did you learn about SitC? Colleague/word of mouth26 CDC website16 CDC’s Dear Colleague Letter6 EffectiveInterventions.org5 SafeInTheCity.org2 STDPreventionOnline.org2 STD Conference/Meeting2 PLoS article0 Don’t know11 Other32
64 Sites showing the video Baseline – 35 of 87 sites, 40% Mean Days From Video Receipt to Baseline Survey – 117 days Follow-up – 46 of 81 sites, 57% Mean Days From Video Receipt to Follow-up survey – 250 days
65 Among sites with follow-up data (n=81) Of the 34 sites showing the video at baseline 27 (79%) were showing it at follow-up. Of the 47 sites NOT showing it at baseline, 19 (40%) were showing at the follow-up.
66 Reasons for not showing the video (Baseline) 33% TV/DVD player purchase 15% Not appropriate for their clients 12% Pending approval 40% Other reasons –Too busy, staffing, lost DVD, using it outside of the clinic, developing augmenting materials
67 Factors Critical to Successful Diffusion Creating a coordinated diffusion action plan Proactively assess intervention uptake Identify barriers to implementation
68 Additional Resources Order the video and view past informative net meetings on the Safe in the City webpage at www.effectiveinterventions.org www.effectiveinterventions.org Project website (includes video links) www.safeinthecity.org www.safeinthecity.org