1. How Much Evidence is Needed
How Much Evidence is Needed? Dissemination & Implementation Science Workshop Edward Ellerbeck, MD, MPH Kansas University Medical Center

5. PAR-18-007: Participating Organizations

6. Objectives
Provide a framework for addressing evidence in implementation science research
Identify key features of Hybrid implementation-effectiveness designs

7. Traditional Translation Framework
Efficacy
Effectiveness
20 + years
Implementation
Dissemination & Diffusion

8. Conceptual Model of implementation research
Proctor; Adm Policy Ment Health January ; 36(1):

9. Translating Evidence into Practice
Implementation
Strategy(s)
Intervention
Tool
Processes of
Care
Health
Outcomes

10. Translating Evidence into Practice: Smoking Cessation
Facilitation
5 As for Smoking
Cessation
Cessation
Counseling
& PTX
Smoking
Cessation

12. Scientific Premise

13. Evidence for the Intervention
Practice
Facilitation
5 As for Smoking
Cessation
Cessation
Counseling
& PTX
Smoking
Cessation

14. Pure Implementation study
Rationale:
Effectiveness of intervention clearly established
Gaps in uptake of the intervention
Justify the costly efforts to support implementation (proposed implementation is feasible)
Curran et al. Med Care. 2012; 50(3):217-26

15. Pure Implementation study
Practice
Facilitation
5 As for Smoking
Cessation
Ask
Assess
Advise
Assist
Arrange
VERSUS
Audit &
Feedback

16. Evidence for the Implementation Strategy
Practice
Facilitation
5 As for Smoking
Cessation
Cessation
Counseling
& PTX
Smoking
Cessation

17. 73 Implementation Strategy(s)
Powell et al. Implementation Science (2015) 10:21

18. Implementation research: Evidence & Scientific Premise
Evidence (for intervention)
RCTs linking intervention to outcome
Practice guidelines
Pilot studies
Models and theories
Evidence (for implementation strategy)
Impact for other interventions
Theoretical framework

19. Limitations of Implementation-only studies
Interventions are generally not implemented by within the same context or by the same actors that were used in the original efficacy studies
Interventions may not have the same outcomes in new contexts or with new actors
Critical features of interventions may be lost in translation (fidelity)
Cost-effectiveness can’t be assessed in implementation-only studies

20. Hybrid Designs

21. Hybrid trial designs Effectiveness Research Implementation Research
Hybrid Type 1:
-test intervention
-collect data on implementation
Hybrid Type 2:
-test intervention and implementation simultaneously
Hybrid Type 3:
-test implementation
-observe intervention outcomes

22. Hybrid designs: rationale
Speed/improve translation of evidence into practice
How to implement
Barriers to implementation
Understand effectiveness in real clinical settings
Bring fidelity & ‘critical components’ to the forefront
Identify the ‘costs’ of translation into practice

23. Dissemination and Implementation Research in Health (R01s)
Initial issue in 2002: PA
Latest issue: PAR
PAR : 48 funded R01s; 6 (12%) ‘hybrid’
PAR : 25 funded R01s; 8 (32%) ‘hybrid’
Type I: 4
Type II: 1
Type III: 3

24. Hybrid Type 1 Design
Test the effectiveness of the intervention while gathering information on implementation
Rationale:
Good preliminary data, but further effectiveness studies still needed
Applying the intervention to a new population, setting, or delivery method
Adapting the intervention to a new setting/context
Intervention is of minimal risk

25. Hybrid Type 1 implementation questions
What were the facilitators/barriers to delivering the intervention?
What were the facilitators/barriers to sustaining the intervention after the study was completed?
How might the intervention be changed to improve adoption and sustainability?

26. Implementation Barriers & Facilitators
Hybrid Type 1 Design
Clinic or Pts
Randomized
Intervention
Control
Implementation Barriers & Facilitators
Clinical
Outcome

27. Hybrid Type I: PAR-16-238 examples
M-health intervention to increase adherence to triage of HPV+ women who have performed self-collection
SURVIVORLINK: Scalability of an electronic personal health record for cancer survivors and caregivers at pediatric cancer centers
Implementing a virtual tobacco treatment intervention in community oncology practices
Translating an efficacious illness management intervention for African American youth with poorly controlled asthma to real world settings

28. Hybrid Type I: Translating an efficacious intervention for AA youth with poorly controlled asthma to a ‘real world’ setting*
Clinical Intervention:
‘Reach for Control’: Home-based, multi-component program delivered by CHWs
Control: Agency’s ‘standard program’
Design:
Patient-level RCT in one hospital emergency department/community agency
Implementation strategy: Training
Implementation evaluation
Reach (identification, screening, and referral in the ED)
Barriers (from perspective of ED staff and CHWs)
Fidelity of the intervention
*Deborah Ellis – Wayne State

29. Children with asthma in ED
CHW-delivered ‘Reach for Control’
Standard care
Implementation Reach, Barriers, and Fidelity
Asthma Exacerbations

30. Hybrid Type I: Translating an efficacious intervention for AA youth with poorly controlled asthma to a ‘real world’ setting*
Evidence base: Demonstrated effectiveness of a similar intervention delivered in a different setting by trained professionals*
Primary Aim: Evaluation of the Reach for Control Intervention
(RCT trial design)
Secondary Aims:
Identify barriers and facilitators to the intervention
Examine intervention fidelity
Conduct a cost analysis

31. Hybrid Type II Design
Simultaneously test intervention and implementation strategy
Intervention effectiveness study within:
a controlled implementation study (factorial design)
non-randomized study of implementation strategy (e.g. step-wedge)
Rationale:
Enough evidence to support study of both intervention and implementation
Evidence not sufficient for the specific context or population
System/policy demands for evidence on intervention and implementation
Reasonable evidence that the implementation strategy could be supported/sustained within the given context

32. Implementation strategy 1 Implementation strategy 2
Clinics
Randomized
Implementation strategy 1
Implementation strategy 2
Control
Intervention
Implementation Barriers & Facilitators
Clinical
Outcome

33. Hybrid Type II: PAR-16-238 example
A multicenter trial of a shared decision support intervention for patients offered implantable cardioverter-defibrillators: DECIDE-ICD Trial

34. Breakout #4: Paul Estabrooks
Operationalizing Type II Hybrid Effectiveness Implementation Designs

35. Hybrid Type III Design
Tests the implementation strategy while simultaneously gathering data on clinical outcomes
Conditions:
Strong ‘demand’ for implementation despite limited data of how the intervention influences outcomes
Strong face-validity for the clinical intervention and/or implementation strategy
At least indirect evidence to support the clinical intervention and/or implementation strategy
Effectiveness of intervention may be vulnerable to implementation fidelity or different effectiveness in the new context

36. Implementation Strategy 1 Implementation Strategy 1
Clinics
Randomized
Implementation Strategy 1
Implementation Strategy 1
Implementation Barriers & Facilitators
Clinical
Intervention
Clinical
Outcome

37. Hybrid Type III: PAR-16-238 examples
IAMSBIRT: Implementing alcohol misuse SBIRT in a national cohort of pediatric trauma centers
Increasing implementation of evidence-based interventions at low-wage worksites
Effectiveness and implementation of MPATH-CRC: A mobile health system for colorectal cancer screening

38. Hybrid III – MPATH-CRC A mobile health system for CRC screening
Clinical Intervention:
MPATH-CRC: iPAD delivered CRC decision-support, self-ordering, with text message follow-up
Design:
Primary care clinics (n=28) cluster randomized to implementation strategy
Implementation strategies:
‘High-touch’: Clinic facilitation, clinic champions, data feedback, follow-up training and adaptation
‘Low-touch’
Implementation framework:
Technology Acceptance Model & Dynamic Sustainability Framework

39. Hybrid study designs and Implementation Frameworks
Nilsen Implementation Science (2015) 10:53

40. Implementation Barriers & Facilitators
Clinics
Randomized
N=28
High Touch
Clinic Facilitation
Low Touch
Clinic Facilitation
Implementation Barriers & Facilitators
M-Path: iPad CRC decision support*
Clinical
Outcome
*Evidence-based
intervention

41. Hybrid III – MPATH-CRC A mobile health system for CRC screening
Evidence base: R01-funded RCT in 450 pts MPATH doubled the rate of CRC screening completion* Aim 1: Evaluation of implementation - Screening rates in ‘high’ vs ‘low’ touch clinics (RCT trial design) Aim 2: Evaluation of intervention - CRC completion at 16 weeks post-intervention (Nested, pre-post design) Aim 3: Maintenance of intervention; barrier and facilitators to implementation (based on provider/staff interviews)
*Ann Intern Med. 2018 Apr 17;168(8):

42. Hybrid III - issues
Is there ever a time when we wouldn’t want to collect ‘effectiveness’ outcomes as part of an implementation study?
How much ‘rigor’ is required for the ‘effectiveness’ component of a type III study?
Can you use pragmatic or secondary sources for effectiveness data?

43. Hybrid designs & RE-AIM
RE-AIM: Very common evaluation framework for hybrid trial
Must be accompanied by clear, measurable elements for each component
RE-AIM does not provide a:
Theoretical implementation model
Framework for determinants of implementation
Model for change

44. Hybrid designs and scientific rigor
Greater demand for rigor with the primary aim
Consider an RCT
Greater demand for pragmatism with implementation strategies

45. How much ‘evidence’ of effectiveness is required?
Evidence needed for both:
Clinical intervention
Implementation strategies
‘Evidence’ supported by:
Prior studies in different context/population
Pilot work
Theoretical framework
Need for evidence may justify a hybrid design

46. How much ‘evidence’ of effectiveness is required?
At the conclusion of the trial, there must be strong empiric evidence that the clinical intervention (as implemented) works
Based on prior studies (or)
Based on effectiveness data from your study (hybrid I or II design)

47. How much ‘evidence’ of effectiveness is required?
At the outset of the trial, there must be strong empiric evidence and/or theoretical support that the implementation strategy will work
Effectiveness studies: tightly control the implementation (e.g. employ and train the actors)
Hybrid I or II: Can’t study effectiveness of the clinical intervention if the intervention is not implemented (Hybrid I and II)
Hybrid III or implementation only: Must have a theoretical framework to justify testing the implementation strategy

48. Questions?