1. Hannah E. Frank, Lisa Saldana, Philip C. Kendall, Holle Schaper
Bringing evidence-based interventions into the schools: The impact of organizational factors on implementation success
Hannah E. Frank, Lisa Saldana, Philip C. Kendall, Holle Schaper

2. Background
Children with mental health problems often do not receive mental health services (Farrell & Barrett, 2007)
When they do, it is most often through their schools (Canino et al. 2004)
Several barriers to implementing interventions (Paulus et al. 2016)
Planning required
Staff time
Financial Constraints
Confidentiality concerns
Organizational support for the program

3. Computerized and Internet-Based Intervention Programs
Alternative to traditional manualized treatments
Cost effective, standardized, improve feasibility of widespread dissemination
Camp Cope-A-Lot (Kendall & Khanna, 2008)
Computer-assisted CBT based on Coping Cat (Kendall & Hedtke, 2006)
Requires minimal staff training, staff time, and financial burden
Dissemination and implementation of Camp Cope-A-Lot in schools is an appropriate model to examine factors affecting the likelihood of school-based intervention adoption

4. Implementation Framework to Guide Success
Consolidated Framework for Implementation Research (CFIR; Damschroder et al., 2009)
Five major domains
Inner setting: characteristics of organization implementing the intervention
Minimal research has examined how organizational factors affect the implementation of evidence-based interventions in schools

5. Current Study
Aim: Examine organizational factors that relate to Camp Cope-a-Lot implementation outcomes as measured by the Stages of Implementation Completion (SIC)
Part of larger study examining sustainability of computer-assisted Camp Cope- A-Lot intervention to treat anxious youth in schools in the United States (U.S.) and Canada (NIMH R01MH086438)
Hypotheses:
Higher scores on organizational measures will be associated with more successful implementation outcomes (i.e., program start-up, competency)
Schools with better organizational scores will be more likely to successfully implement Camp Cope-A-Lot with at least one child

6. Methods: Participants
Schools:
New Jersey/Pennsylvania (n= 7)
Canada (n = 13)
Providers:
Participants (N = 38) were school staff members
Did not need to have prior mental health training (24% did not)
Received Camp Cope-A-Lot training

7. Methods: Measures
Organizational Social Context (OSC: Glisson et al. 2008)
105 items administered directly to school staff
Yields T-scores
Culture domain: Proficiency, rigidity, resistance
Climate domain: Engagement, functionality, stress
Stages of Implementation Completion (SIC; Saldana, 2014)
Examines implementation process and milestones across 3 phases and 8 stages specific to this intervention
Scoring: Duration and proportion scores

8. Methods: Measures Knowledge Test (Beidas et al. 2009): Role Plays:
20 items that assess providers’ knowledge about CBT for youth anxiety
Three versions
Role Plays:
Providers read a vignette about an anxious child and were asked to conduct an exposure with that child
Had up to 8-minutes to demonstrate how they would prepare the “child” (a trained graduate student using a script) for the task
Role plays were transcribed and coded for adherence and skill

9. Methods: Treatment Program
Camp Cope-A-Lot
Computer-assisted treatment
Target Audience
Schools
Community Mental Health Centers
Private Practices
Medical Centers
Training Clinics
Social Service Agencies
12 “Levels” (6 child only; 6 with coach/therapist)

10. Methods: Procedures Provider Training One-day workshop
Completed measures before and after workshop
Received weekly consultation calls by phone
Youth Referral
Youth with elevated BASC anxiety scales or teacher nomination were eligible to receive Camp Cope-A-Lot
Scheduled to meet with providers weekly

11. Methods: Procedures Organizational Data Collection
Teachers and school staff members at each school were asked to complete organizational measures prior to use of the Camp Cope-A-Lot program
Implementation Behavior
Measured by the SIC
Competency measured by:
Knowledge Tests: before training, after training, end of year 1
Role Plays: after training, end of year 1

12. Results: Correlations
Duration of Scored Activities (Phase 1)
Duration of Scored Activities (Phase 2)
Proportion of Scored Activities (Phase 1)1
Proportion of Scored Activities (Phase 2)
OSC Proficiency
OSC
Rigidity
OSC Resistance
OSC Engagement
OSC Functionality
OSC Stress —
-.55* --
.23
.18
.65**
.37
.03
OSC Rigidity
-.03
.02
.07
.43┼
-.33
.04
-.15
.13
.61**
.39┼
-.24
.17
.25
-.05
-.04
.41┼
-.30
-.01
.34
-.22
-.28
.68**
-.16
-.003
-.13
.24
.36
.20
-.57**
-.43┼
┼p <.10 *p <.05 **p <.01; 1Cells are blank for proportion in phase 1 because there was no variability due to all participating schools completing all activities in phase 1

13. Results: Comparisons between U.S. and Canada
Provider training dates:
Varied for US schools
Only two dates for Canadian schools
T-tests compared organizational factors and implementation behaviors:
Organizational Factors: Proficiency was significantly higher at domestic schools
Pre-Implementation (Phase 1): Significantly longer duration for domestic sites; No variability in proportion scores (all schools completed all activities)
Implementation (Phase 2): No significant differences in duration or proportion
Program Start-Up: No significant differences between domestic and international sites regarding whether a child was treated, c2 (2, N = 20) = 2.69, p = .10

14. Results: Predictors of Pre-Implementation SIC duration
Variable
Model 1
Model 2 B SE p
SE B P
OSC Proficiency
3.76
1.04
.002*
1.90
0.96
.07+
International
(0 = domestic; 1 = international)
-55.51
15.58 R2
.42
.67
∆R2
.25
Fchange (df1, df2)
12.69 (1, 17)*
* p < .05; + p < .10

15. Results: Predictors of Implementation Duration and Proportion
Longer pre-implementation durations predicted shorter implementation durations, b = -0.38, t = -2.78, p = 0.01, R2 = .30
Site location and OSC proficiency were not significant predictors of implementation duration.
Predictors of implementation proportion scores (i.e., OSC proficiency, pre-implementation duration, site location) were not significant

16. Results: Predictors of Camp Cope-A-Lot Start Up
Camp Cope-A-Lot start-up: defined as a site treating at least one child
Logistic regression with OSC proficiency, site location, and pre- implementation duration as predictors was not significant
Number of children treated with Camp Cope-A-Lot
Same predictors (OSC proficiency, site location, and pre- implementation duration)
Site location was the only significant predictor, such that domestic sites treated significantly more children, b = -6.03, t = , p < 0.001, R2 = .93.

17. Results: Predictors of competency
Measures of competency:
Knowledge test – 3 total
Role plays – 2 total
Predictors of competency:
Pre-implementation duration
Implementation duration
Site location
OSC proficiency
Proficiency emerged as the only significant predictor of:
Number of knowledge tests passed, b = 0.06, t = 2.23, p = 0.04, R2 = .40
Number of role plays passed: b = 0.06, t = 2.33, p = 0.03, R2 = .29

18. Discussion: Overview of Findings
Organizational Factors
Proficiency was the only organizational factor associated with implementation outcomes
Trend for higher proficiency scores being associated with longer pre-implementation duration
Longer pre-implementation duration predicted shorter implementation duration
Higher proficiency scores predicted passing of more knowledge tests and role plays (i.e., competency)

19. Discussion: Overview of Findings
Comparisons between Domestic and International Sites
Proficiency and pre-implementation duration were higher at domestic sites
No differences in duration of implementation activities
Differences in organizational structure (i.e., Canadian sites governed by district administrator)
Predictors of Successful Start Up
16 out of 20 programs achieved program start-up
Domestic sites treated a significantly higher number of children than international sites, likely because of number of providers per school

20. Implications for School Settings
Importance of proficiency
Proficient schools may spend longer on pre-implementation activities in order to ensure that they have the necessary resources to implement the intervention
Implementation may proceed more quickly when more time is spent on pre-implementation
Differences in organizational structure affects outcomes
Size of the workforce may affect the degree to which use of the program is widespread throughout the school

21. Implications Beyond School Settings
Importance of creating proficient organizational cultures to improve implementation outcomes
Has been demonstrated in previous studies as well (e.g., Williams & Glisson, 2013)
Careful planning for implementation activities during pre- implementation may reduce the amount of time needed during implementation

22. Limitations and Future Directions
Small sample size
Only incorporated one element of larger contextual framework (CFIR)
Future directions
With larger sample size, could examine a mediational model:
Pre-implementation phase duration  Organizational culture  Provider behavior
Examine in middle school and high school contexts

23. Acknowledgments The Oregon Social Learning Center, especially:
This research was supported by NIMH under award number R01MH086438
The Oregon Social Learning Center, especially:
Lisa Saldana
Holle Schaper
Jason Chapman
The Child and Adolescent Anxiety Disorders Clinic, especially:
Philip Kendall
Matthew Carper
The children and school staff who participated
Thank you!!!