1. Enhancing Sustainability at the District and State Levels
Kent McIntosh, University of Oregon
Bridget Drobac, Bethel School District
Eric Kloos, Minnesota Dept. of Ed.
2014 PBIS Implementers Forum
Handouts:

2. Who are we? Who are we? Who are you? Kent Bridget Eric Levels?
School, district, region, state, nation?
PBIS implementation experience?
How many have attended previous sessions in this strand?
Who are you? – school personnel, school admin, district admin, district support personnel, state admin

3. Support for these projects:
IES: NCSER (R324A120278)
OSEP: TA Center on PBIS (H326S03002)
Social Sciences and Humanities Council of Canada (SRG F )
Hampton Endowment Fund (J )

4. Thanks and Acknowledgments
Participants in these studies
State Networks
Jerry Bloom, Susan Barrett and PBIS Maryland
Cristy Clouse, Barbara Kelley and CalTAC
Eric Kloos and Minnesota DOE
Mike Lombardo, Celeste Rossetto Dickey, and Placer COE
Nanci Johnson and MO SW-PBS
Justyn Poulos, Wisconsin PBIS
Cayce McCamish, NC DOE
Co-authors

5. Handouts: http://www.pbis.org
Session Overview
Research on strategies to support implementation and sustainability of PBIS
Kent
Systems for sustainability at the district level
Bridget, Bethel School District (OR)
Systems for sustainability at the state level
Eric, Minnesota Department of Education
Handouts:

6. Perceived Importance of Contextual Features for Sustainability of PBIS
McIntosh, K., Predy, L., Upreti, G., Hume, A. E. & Mathews, S. (2014). Perceptions of contextual features related to implementation and sustainability of School-wide Positive Behavior Support. Journal of Positive Behavior Interventions, 16,

7. Perceived Factors Related to Sustainability of PBIS
Sample: 257 schools from 14 US states
49% Elementary
16% Middle
5% High School
Average implementation: 6 years (1 to 15)
Open-ended question:
“What is the most important enabler of sustainability of PBIS?”

8. Enablers of Sustainability
How would you answer?

9. District and state systems are the keel in the school’s boat (McIntosh & Goodman, in press)

10. Most Important Single Perceived Factor in Sustainability?
School Administrator Support
Can districts play a role in increasing school administrator support?

11. Sustaining PBIS through Administrator Turnover (Strickland-Cohen, McIntosh, & Horner, 2014)
School Team
Maintain the PBIS handbook
Document support among staff and stakeholders
Collect and share outcomes data
Meet with the new administrator
District Team
Build PBIS into written policy
Build PBIS competencies into hiring criteria
Develop district coaching capacity

12. How Do Principals Go From Skeptics to PBIS Champions?
McIntosh, K., Kelm, J. L., & Canizal Delabra, A. (2014). In search of how principals change: Critical incidents in enhancing administrator support for school-wide prevention. Manuscript submitted for publication.

13. Research into Enhancing Principal Support
Qualitative interviews with 10 principals initially opposed or lukewarm to PBIS but now champions
Interview questions:
What helped your active support for PBIS?
What hindered your active support for PBIS?
What would have made you support PBIS from the onset?

14. Strategies for Enhancing Principal Support
District Training and Support
Provide “Principal PBIS Academies” for new administrators
Basics of PBIS
Role of administrators
Provide coaching to schools
District Networking
Arrange informal conversations with other principals supportive of PBIS
Arrange site visits at nearby PBIS schools
At the School
Help school staff demonstrate support

15. What is the strongest predictor of PBIS sustainability?
McIntosh, K., Mercer, S. H., Hume, A. E., Frank, J. L., Turri, M. G., & Mathews, S. (2013). Factors related to sustained implementation of School-wide Positive Behavior Support. Exceptional Children, 79,

16. Results: Predictive Model
Model fit indices acceptable (except χ2)
χ2 (731) = , p < .001, CFI = .96, TLI = .96, RMSEA = .03
R 2 = .45
Factors
Priority (B = .14, SE = .39, p > .05)
Team Use of Data (B = .61, SE = .24, p < .05)
District Priority (B = -1.14, SE = .66, p > .05)
Capacity Building (B = .98, SE = .43, p < .05)
45% of the variance explained

17. 5.38** .07 .47 -.34 .41 School Priority Sustained PBIS Fidelity
Team Use of Data 8
.47 8 8 8 8 8 8
Priority (20 items, reliability = .94)
Staff support, administrator support, perceived effectiveness, perceived efficiency, integration into new initiatives
Implementation (11 items, reliability = .94)
School team/staff skill, functioning, regular meetings, data collection, use of data for decision making, presenting data to staff and community 8 8
District Priority 8
-.34 8 8 8
Sustained PBIS Fidelity 8
Capacity Building 8 8
.41 8

18. Four Factors School Priority (20 items) Team Use of Data (11 items)
Administrator support, staff support, perceived effectiveness, perceived efficiency, integration into new initiatives
Team Use of Data (11 items)
School team/staff skill, functioning, regular meetings, data collection, use of data for decision making, presenting data to staff and community
District Priority (5 items)
District support, state support, funding, district policy, promoted to external organizations
Capacity Building (3 items)
Access to district coaching, yearly professional development, connection to a community of practice
Below the close fit hypothesis – close yet failing model
13.52
2.50
2 items loaded on both factors (effective for a large proportion of students; all school personnel have a basic understanding of PBIS) largest residual correlation -.25
nested

19. Takeaways
School teams can benefit from training in running meetings and using data
District coaching, professional development, and connection to a community of practice were effective district supports
No significant independent contribution of active support, general funding, policy
Note that they are correlated at .78 and .58
Simplistic notion to stop doing non-significant factors

20. What predicts sustained PBIS implementation at 3 and 5 years after training?
McIntosh, K., Mercer, S. H., Nese, R. N. T., Strickland-Cohen, M. K., & Hoselton, R. (2014). Predictors of sustained implementation of School-wide PBIS at 3 and 5 years after initial implementation. Manuscript in preparation.
Sneak peek into what we’re up to now…

21. Sample
1242 schools submitting fidelity data in PBIS Assessment (starting from to 12-13)
530 districts
29 states
Grade levels:
70% elementary schools
21% middle schools
9% high schools
Perceived impact

22. Variance in Fidelity Across Years

23. What were successful states (>50% at criterion) doing?
State PBIS leadership teams
State-level trainers, trainings, and training curricula
State recognition systems (for schools with strong implementation and/or outcomes)
Regular use of the SWPBIS Implementer’s Blueprint (Sugai et al., 2010) to assess and inform implementation

24. Bethel School District PBIS Team Approach
Be the guiding/driving force for Bethel’s Systems Approach to improving behavior and achievement for all the students in the district.
None
Adapted from Horner, Sugai and Bethel Staff

25. Bethel School District Demographics
Bethel is a school district in Eugene, Oregon that is home to approximately 5,700 students.
60% of students qualify for free & reduced lunches.
17% of students identified as having a special education eligibility.
5% of students are ELD
Five Elementary Schools
Two K-8 Schools
Two traditional middle schools
One alternative high school (grades 10-12)
One traditional high school
None

26. Bethel Behavior Data : 10 year span
I’m proud to share some of Bethel’s data accrued over time. This graph shows overall improvement in behavior data (referrals per 100 students) over the last ten years of PBIS implementation. While there are a variety of ways to implement effective PBIS, I will share more about Bethel’s process

27. Bethel PBS Structure
Monthly District PBS Leadership Team reviews data, process, plans training and provides support for successful implementation. This team made up of a variety of building and district level staff help guide the direction for District PBIS.
School Teams drive a building’s PBIS program. Their focus is to maintain the integrity of the school-wide PBIS systems, look at classroom PBIS and the individual student PBIS system (Tiers 1-3).
Data-based decision making plays a big role. Schools use SWIS-School-wide Information System to make timely adjustments or preventative interventions on a monthly basis or more often with Individual student system.
Of course, keeping everyone trained in the model is key to successful implementation. As new PBIS changes come forward Bethel has a high level of training involvement to ensure student success.
Durable, and adaptable school-wide PBIS in a school requires systemic support that extends beyond an individual school. It is important to organize multiple schools (e.g., cluster, complex, district, county, state) so that a common vision, language, and experience are established. This approach allows districts and states to improve the efficiency of resource use, implementation efforts, and organizational management. An expanded infrastructure also enhances the district and state level support (e.g., policy, resources, competence) and provides a supportive context for implementation at the local level.

28. Practices for Student Success: Linking of Academic and Behavioral Interventions
Social Behavior
Individual
Individually Design Instruction
Individual
Behavior Support Plan (BSP)
Safety Plan
1-5%
1-5%
Targeted: Some Students (at-risk)
Core Plus
Pre-teach, Re-teach
Supplemental Programs
Alter Group Size
5-10%
5-10%
Targeted: Some Students (at-risk)
Advance CICO
Social Skills
Lunch Buddies
Boy/Girls Groups
Check-in, Check-out (CICO)
This version of the triangle demonstrates how we work to tie in behavior & academics. These efforts come from Bethel’s district policy and our Tiered interventions managed by our school teams.
Universal Screening:
All Students
Core Program
80-90%
Universal Screening:
All Students
Schoolwide expectations
taught explicitly
reinforced frequently
80-90%
Adapted from: Horner & Sugai 28

29. What is involved in being the “guiding force” behind Bethel’s Systems Approach to improving behavior and achievement for all the students in the district?
District Coordination of PBIS Systems for the past 13 years
PBIS in hiring practices
PBIS grades K-8 across the district
Uniformed behavior referrals and minors
PBIS incorporation with transportation
PBIS District-wide eases transitions
District Coordination of PBIS systems present and future
District “On Track” reports
PBIS practices and SWIS at the high school
Uniformity in behavior form for transportation
District PBIS team blog.
Coordination past-
PBIS has become district policy and plays a role in hiring for licensed staff
PBIS is implemented across the district in grades K-8 and has recently been implemented at the high school
PBIS lead team action prompted consistent expectations and uniformed referrals and minor K-8
PBIS incorporation on Bethel and First Student Transportation
PBIS District-wide and these efforts ease transition for students and staff with movement between buildings or for families with students in multiple buildings
Coordination future
District “On Track” reports reviewing academics, attendance and behavior in progression toward graduation. Look at whole student
Implementation of PBIS practices and use of Student Wide Information Systems data collection at the high school level
Include transportation in use of uniform behavior forms for data continuity and clarity
Development of PBIS team blog. Create a forum for building level teams to share information and successes.

30. Bethel Triangle Data
Comparison of behavior data with PBIS implementation in Elementary 2000/2001 compared to

31. Bethel Triangle Data
Comparison of behavior data with PBIS implementation in Middle 2003/2004 compared to

32. Comparison of behavior data with PBIS implementation K-8
Bethel Triangle Data
Comparison of behavior data with PBIS implementation K-8
to present
From our second year of full implementation, Bethel School District’s behavior data has improved in all Tiers, but most notably, Tier I (Gains of 5-7%) The gains indicated are across grades K-8 and over time from

33. Coordination of PBIS : the team approach
Ensuring coordination and implementation is a District Lead Team member responsibility
The DLT includes:
PBIS Coordinator
Behavior Specialist
Building Administrators
Director of Special Services
School Counselor/PBIS Facilitators
District Equity Coordinator
First Student transportation representative
Building administrators from all levels- K-5, K-8, Middle and now HS

34. District Leadership Team Purpose
To support, improve and sustain PBIS in the Bethel School District. The DLT’s toolbox includes:
District-wide data Stable funding to support schools
PBIS as a district focus Coordination of PBIS across schools
Professional development Building team evaluation measures
Capacity for all students Coordinate systems (EBISS, On-Track)
Effective practices Partner (U of O, ORI, First Student)
Analyze district-wide data and maintain PBIS as a district focus. Provide stable funding to support schools including determination of need and provision of resources for professional development
Ensure coordination of PBIS across schools and levels and build capacity to meet the needs of all students. Measures to ensure coordination across systems including (EBISS, On-Track)
Review research for effective practices and partner with organizations (U of O, First Student, ORI, etc.)
Review team and building evaluation measures to insure fidelity and effective support

35. Schools Implementing PBIS in Bethel
This chart indicates the level of implementation district wide. Last year, we added the WHS to the schools implementing PBIS in Bethel. 10 of Bethel’s 11 schools currently implement and sustain PBIS practice and use SWIS data for decision making

36. Review of Team Assessment Data
Buildings complete annual assessments. Recent assessments include:
Team Implementation Checklist (TIC)
Benchmarks of Quality (BOQ)
Tiered Fidelity Inventory (TFI)
Results are indicator of possible growth for building team
Results provide information to the PBIS District Lead Team
District and building level teams review the results of assessment surveys. Building teams use the survey results to develop actions plans for the next year, as well as, long-term goals.
The District Lead team reviews survey data to maintain district focus on PBIS practices. DLT provides training and support to buildings based on building fidelity, commitment, implementation and need.

37. Intensive Interventions Specialized Individualized
Systems for Students with High-Risk Behavior
~5%
Targeted Interventions
Specialized Group
Systems for Students with At-Risk Behavior
Universal Interventions
School-/Classroom-
Wide Systems for
All Students,
Staff, & Settings
~15%
Many people place all their emphasis on interventions—what will we do, what kind of targeted interventions do we need?
These factors are important BUT we have learned from SWPBS that it is every bit as important to focus on key features to guide intervention implementation. Important aspects can be contextual fit for student population and staff (“buy in”), feasibility of intervention and fidelity
~80% of Students

38. BETHEL PBS IMPLEMENTATION PROCESS
Administrative Support
District PBIS Team
School PBS Team
Representative Staff
BETHEL PBS IMPLEMENTATION PROCESS
Agreements
Establish PBS Goals
Develop Data-based
Action Plan
In general, the implementation of a school-wide PBIS approach at the school level is built around five main implementation steps. Teams use data and information gleaned from surveys including the TIC and BOQ to draft action plan. Action plans are submitted to the district level team. Action plans guide the building team efforts throughout the year.
Evaluation
Implementation

39. Snapshot of District Team Survey Data 2013-2014
A snapshot of the Team Implementation Checklist shows Bethel Team scores on indicators for team sustainability. At all levels K-8, teams identify over 90% commitment from building administration and staff. Teams show at least 80% representation of all stake-holders including administration, certified/classified staff, students and families. Finally, Teams relate 100% coordination between all three Tiers of Intervention. This number translates to clear communication and transition between PBIS and IPBS/Data-teams.

40. Recent Barriers and Challenges
Counseling FTE reduced by .5
Administrative changes/reduction
Staff changes/new staff
Maintaining contextual fit within district-wide system
Competing foci
Partial implementation or “drift” from best practice
Reduction of counseling created a challenge in that many district counselors participated on the building PBIS team (Tier I) and facilitated IPBS/Data-teams (Tiers II & III).
In conjunction with reduction of administration, changes in staff can disrupt continuity and require more intensive training.
Changes in curriculum, testing and information sharing with families has provided district and building level competition for attention.
Most important nugget of Bethel’s presentation: Awareness of potential for drift from best practice. Avoid complacency (this is how we’ve always done it…does it still fit?)

41. Provision of support for Bethel’s District PBIS
Attend building level PBIS meetings
Monitor transitions between PBIS Tier I, II, III teams
Meet as a team with building level Tier I & Tier II/III facilitators each trimester
Review building level data
Provide district, local, state and national training information
Maintain connection and engagement with Bethel transportation
Attend building level PBIS meetings for coordination with District Lead Team
Monitor transitions between PBIS Tier I, II, III teams to ensure clear communication between building PBIS team and IPBS/Data Teams
Meet as a team with building level Tier I & Tier II/III facilitators each trimester to maintain momentum, review outcomes, celebrate success and respond to needs
Review building level data to note gains and identify areas that may require more intensive focus or intervention
Provide district training, information about and access to(funding when possible) local, state and national trainings/conferences. Bethel, thus far, sends at least one representative per building to the state PBIS conference.
Visit Bethel transportation offices and ride different bus routes, when possible and provide training and ongoing support

42. From power-point to practice
District-wide data indicates increase in percent of ODRs for physical aggression.
DLT response: PBIS coach meets with building level teams to review data and increase school wide interventions to address percentage change
Revitalize focus on safety and Stop/Walk/Talk programs
Increase pro-social lessons
Use of peer mediation programs
Provide retraining, as needed
While I am happy to identify the many strengths of Bethel’s PBIS program, I also want to be realistic about some of the challenges Bethel faces. In reviewing year end SWIS data, we noted an increase in the category of physical aggressions across the district. The DLT developed a plan to address this concerning trend
Trainings implemented thus far include: Focus on safety, Second Steps, Steps to Respect, Peer Mediation (on-going) and Active Supervision (for adults)

43. District Level Implementation: What to do to be effective?
Investment and Commitment to School-Wide Prevention
Provision of District Training for Personnel
Use of Data Systems
Team-Based Decision Making Model
Investment and Commitment to School-Wide Prevention including coordination of positive intervention and support
District Training for Personnel-follow up and on-site capacity building and district and school based coaching
Data Systems provide access to effective, clear and “user friendly” data systems. Data is reviewed at building and district level. Data is used to track Tier I, II & III for intervention outcomes and trends.
Team-Based Decision Making Model: teams engage in regularly scheduled meetings with representative teams (stake-holders). Data is used as a tool in decision making

44. Bridget Drobac- PBIS Coordinator
Contact Information
Bridget Drobac- PBIS Coordinator
Bethel School District

45. Enhancing Sustainability at the State Level
Eric Kloos
Minnesota Department of Education
education.state.mn.us

46. Building Capacity of Effective Implementation of SW-PBIS
Team-based training
9 training days over two years
Distributed, team-based implementation of PBIS
Intended to build capacity, skills, competency and beliefs to sustain implementation beyond initial training
education.state.mn.us

47. Creating Implementation Informed Expectations at a School Level
In Minnesota, baselines are rising (average baseline SET = 69), but there are still predictable differences between schools starting training and sustaining schools (average SET = 90, BoQ = 84).
What features are similar between baseline schools and sustaining schools?
Administrator is an active PBIS team member (96% baseline schools/97% sustaining schools)
Administrator reports that team meetings occur (98%/98%)
What features are different between baseline schools and sustaining schools?
Documented system of teaching expectations (46%/83%)
Teaching expectations has occurred this year (74%/94%)
SW behavior program has been taught/reviewed with staff this year (78%/97%)
Team provides discipline data summary to staff at least 3 times per year (50%/91%)
90% of team members report that discipline data is used for decision-making (57%/97%)
Why do we measure implementation across time in a school?
Because it varies!
education.state.mn.us

48. 2014 School SET Profile “A Snapshot over Time”
Illustrates normal implementation variation over time within in a MN school that has been implementing PBIS 7 years
education.state.mn.us

49. Creating Implementation Informed Expectations at a District-Level
At a district-level, it is often a challenge to accurately track which schools:
have been trained,
are in training, and
have yet to participate in training.
Differentiate outcome expectations for schools by what we know about their implementation.
Get the right information to the right people at the right time to inform district decisions.
Support patience and focus to get to results.
Why do we measure implementation across a District?
- Because it varies!
education.state.mn.us

50. 2014 District SET Profile “A Snapshot in Time”
Illustrates normal implementation variation over time within in a MN district that has been implementing PBIS 6 years
education.state.mn.us

51. Example of a District Data Dashboard: Effort, Fidelity and Outcome Data
education.state.mn.us

52. District Capacity Assessment (DCA)
education.state.mn.us

53. District Calendar for Implementation
education.state.mn.us

54. Implementation Informed Expectations for States
When do we expect to see state-level outcome changes?
How many schools and districts need to be implementing?
At what standard?
As many variables change, can we continue to produce good outcomes
Change in team members
New trainers
New coaches
New evaluators
Why do we measure implementation across the state?
It varies across schools, districts, regions and over time
education.state.mn.us

55. Cohort 8 SET Results Fall 12-Spring 14
education.state.mn.us

56. Sharing Data and Outcomes: Disciplinary Reductions for District and State

57. Closing Thoughts
We are learning a lot by studying schools and districts that have been doing PBIS well over time.
Tools and processes that synthesize data for school and district teams support sustained implementation.
Watch for larger data sets and outcomes to change when at least 25% are effectively implementing.
education.state.mn.us

58. The Minnesota Department of Education: The Active Implementation Hub:
Resources
Minnesota PBIS:
The Minnesota Department of Education:
The Active Implementation Hub:
education.state.mn.us

59. Contact Information
Kent McIntosh
Special Education Program
1235 University of Oregon
Eugene, OR 97403
Cannon Beach, Oregon
© GoPictures, 2010
Handouts:

60. Selected References
McIntosh, K., & Goodman, S. (in press). Multi-tiered systems of support: Integrating academic RTI and school-wide PBIS. New York: Guilford Press. McIntosh, K., Kelm, J. L., & Canizal Delabra, A. (2014). In search of how principals change: Critical incidents in enhancing administrator support for school-wide prevention. Manuscript submitted for publication. McIntosh, K., Mercer, S. H., Hume, A. E., Frank, J. L., Turri, M. G., & Mathews, S. (2013). Factors related to sustained implementation of school-wide positive behaviour support. Exceptional Children, 79, McIntosh, K., Mercer, S. H., Nese, R. N. T., Strickland-Cohen, M. K., & Hoselton, R. (2014). Predictors of sustained implementation of School-wide PBIS at 3 and 5 years after initial implementation. Manuscript in preparation. McIntosh, K., Predy, L. K., Upreti, G., Hume, A. E., Turri, M. G., & Mathews, S. (2014). Perceptions of contextual features related to implementation and sustainability of school-wide positive behaviour support. Journal of Positive Behavior Interventions, 16, Strickland-Cohen, M. K., McIntosh, K., & Horner, R. H. (2014). Sustaining effective practices in the face of principal turnover. Teaching Exceptional Children, 46(3), Sugai, G., Horner, R. H., Algozzine, R., Barrett, S., Lewis, T., Anderson, C., Simonsen, B. (2010). School-wide positive behavior support: Implementation blueprint and self-assessment (2nd ed.). Eugene, OR: University of Oregon. Available at