1. Establishing a District Evaluation Plan
Susan Barrett
Director, Mid-Atlantic PBIS Network
Sheppard Pratt Health System

2. Outcomes District Team Clear function/purpose (aligned with SPP)
Members with clear role and function
Communication/feedback loops
Implementation/Evaluation Plan

3. Professional Development
Training Curriculum
Training Capacity
Site Selection
Readiness
Coaching
Implementation and Evaluation Plan
What data point should change? How will we track progress and show impact?
Selection of the Tools
Data Base
PBS Surveys,
PBS Eval
Local data base
Start here
Marketing and Dissemination
Annual Reports
Presentations
Newsletters

4. If something is worth doing once, it's worth building a tool to do it.
Evaluation Template- What will the impact be?
Identify Progress Monitoring Tools
Tier 1
Tiered Fidelity Inventory
School-wide Evaluation Tool (annual for 20%)
Tier 2/3
I-SET

5. Implementation Tools How can we organize the work?
Can we work smarter to provide coaching and technical assistance by tracking progress across phases?
Can district and state use Phases of Implementation (POI) to track progress and assign resources?

6. Exploration Stage Need for change identified, possible solutions are explored, learning about what it takes to implement the innovation effectively, stakeholders are identified and developed, and decision is made to move forward Installation Resources needed to implement innovation with fidelity and desired outcomes are in place Initial Implementation Innovation is in place in schools, implementation largely guided by external TA providers Full Implementation- Innovation is implemented and sustained by local stakeholders and is well-integrated into policy/written documentation Innovation and Sustainability Innovation is adapted to fit local context, innovation becomes more efficient and is integrated with other initiatives
POI

7. POI Tools-Coach POI- Illinois- All Tiers PIC- Florida- All Tiers
Implementation Phase Inventory- MD Tier 1- strong reliability

8. Lessons Learned Training sequence and materials are developed
Refinement of implementation and deepening content expertise in the State, District and schools continuous and areas of improvement are identified by data
A unified, expanded cadre of trainers and implementation coaches prepared to deliver professional development
Trainers understand content and implementation science

9. Lessons Learned
An aligned training and support system for both trainers and implementation coaches.
Standardized training and coaching materials.
Standardized data collection system, tools and processes with practice-level feedback loops at each level of the system for planning and continuous improvement.

10. Readiness Structures 4-6 months of “Pre-Work”
Tier 1
Leadership Forum-
Planning Phase Checklist
SWIS Readiness
Leaders, Coaches, New Teams walk in prepared with basic knowledge, data, SIP
Advanced Tiers
Readiness- completed by Local Coordinator and Team
CICO Readiness, CICO SWIS Readiness
Readiness T2/T3

11. Training Content- Example
“Anchored on the tool”
Overview
Critical Feature Modules
Trainer Presentation
Workbook (self assess, activity , action step)
Implementation Snapshot
Team Workbook

12. Professional Development Blueprint…
Lists and describe minimum core readiness, skill, and performance competencies for SWPBS trainers.
Lists and describe minimum outcome competencies for SWPBS implementation by (a) school staff, (b) school leadership teams, and (c) coaching personnel.
Provides self-assessment guides for monitoring training and coaching progress.
Provides evaluation procedures for assessing training and coaching fidelity and integrity.
Recommends schedules and sequence for training and coaching activities.
Assumes that trainers have implementation experience and fluency with SWPBS practices and systems.
Focuses on the school, district/region, and state as the context for sustained training implementation and support.

13. Rethinking Technical Assistance
Moving from a case by case expert model to building expertise in the school
Focus of all TA is on teaching the school team to solve problems or address challenges for themselves
Shift from providing answers to asking questions
Shift from developing plans to prompting plan development
Shift from being viewed as the expert to being viewed as a facilitator
Will not replace need for specialist, re-focus all to building capacity

14. Build parallel systemic processes
Provide school/district teams with a process to address the presenting challenge (SWPBS)
Develop a parallel process for districts/states to support school implementation and continue to expand with integrity (District /State Leadership Team)

15. CSBS Training Model Creating the Conditions for Skill Development
Intro to Administrators
Readiness/Planning Phase
Team and Coach Training Event
Facilitated Action Planning- Team Time Workbook
Track Fidelity/Progress
Access to Community of Practice-knowledge sharing
Track Outcomes
Building Training Capacity- Scope and Sequence
Training Event
Implementation Team Workbook

16. PBIS Building Capacity and Sustainability
Project Leadership Team
State Implementation Team
Regional Implementation Team
District Coordinators
Coaches
We will be working through exploration to build this system of support
Team Leaders
Problem-Solving Teams
School Staff, Families, Transportation, Communities

17. State Example March 30 July 18-22 Feb- June
Spring Forum Administrator attends
July 18-22
New Team Training
Feb- June
Planning Phase for New Teams Checklist Completed
Sept. 1st – Feb. 28th
New SWIS and CICO-SWIS subscriptions setup and invoiced
Sept
Oct
Nov
Dec
Jan
Feb
Mar
Apr
May
Jun
Jul
Aug
Oct 21 State Coaches Meeting
Nov 10
IPI Due
Dec 6
State Coaches Meeting
April 7
State Coaches Meeting
April 10
IPI Due
August
Team Planning
SWIS Readiness Completed
Establishing Annual Training/Meeting Cycles
** Regional Returning Team Events held during Summer- Dates TBA
**CICO Training completed Regional by request

18. Training Capacity Fidelity
Delivering the training content as it is intended to be delivered is essential
Participants who attend a PBIS training regardless of what part of the state they reside, should have the same training experience
This is a challenge as scaling occurs
TOT developmental process
Learner, Content Fluent, Co-trainer, Trainer

19. Implementation Snapshot
Used during training to clearly define each critical feature and the research that supports use and potential benefit to overall school climate.
The trainer facilitator will provide training slides for each feature. Teams will have an opportunity to use their guidebook and items brought to the training to use during action planning sessions. Roles of each participant are clearly described each snapshot to ensure clear expectations.
Administrators are expected to play an active role. Coaches are expected to guide process, communicate with the district coordinator and share implementation products and examples as needed.
Teams will walk away from the training with an annual action plan that will help guide the effort.

20. Systems Practices/Skills Data Building Coaching Capacity State
Occurs at ALL Levels
Systems
Conditions that support skill development
for staff
Policy and Procedures alignment
Budget Re-allocation
Recruitment and Selection of Coaches
Supervision of Coaching within Organization
Training Curriculum and Scope and Sequence
Access to certification
Facilitative Administrator Supports
Practices/Skills
The technical skill set required to achieve fidelity
Problem Solving (Team, Classroom, Staff, Student)
Team Building/Collaboration
Delivering Feedback
Behavioral Consultation
State
District
Building
Classroom
Staff
Student
Family
Data
Information required to guide skill development process
Action Plan with short/long term measurable goals
Self Assessment
Process Measures/Fidelity Checks
Performance Feedback Measure
Progress Monitoring Tools
Evaluation Tools
Student Outcomes
Data used for continuous regeneration (PEP/PIP) 20

21. Breaking it down … Stages of Implementation
State and District Level Implementation Guide

22. Stages of Implementation
Exploration/Adoption
Installation
Initial Implementation
Development Commitment
Full Implementation
Establish Leadership Teams, Set Up Data Systems
Provide Significant Support to Implementers
Innovation and Sustainability
Embedding within Standard Practice
Should we do it?
Improvements: Increase Efficiency and Effectiveness
Doing it right
Doing it better

23. Exploration Stage Need for change identified, possible solutions are explored, learning about what it takes to implement the innovation effectively, stakeholders are identified and developed, and decision is made to move forward Installation Resources needed to implement innovation with fidelity and desired outcomes are in place Initial Implementation Innovation is in place in schools, implementation largely guided by external TA providers Full Implementation- Innovation is implemented and sustained by local stakeholders and is well-integrated into policy/written documentation Innovation and Sustainability Innovation is adapted to fit local context, innovation becomes more efficient and is integrated with other initiatives

24. State District Implementation Guide
Workbook organized by POI
Specific outcomes for each Phase
Guiding Questions to build case/consensus
Facilitated Activities that guide similar TA process
Never meant to be a “paper product”

25. Professional Development/ Technical Assistance Guiding Questions
2/20/2018
Exploration
Professional Development/ Technical Assistance
Guiding Questions
Hyperlink to guided questions, then facilitated exploration activities, and then professional development/ta.
After the professional development/ta slide, just advance forward to complete ppt presentation.
Facilitated Exploration Activities
Outcomes

26. Exploration Phase
Need for change identified, possible solutions are explored, learning about what it takes to implement the innovation effectively, stakeholders are identified and developed, and decision is made to move forward

27. Before you add one more thing….
We plan for systems to support top two tiers .Creating foundations to support the maintenance of those behaviors.

28. Exploration Phase
Major emphasis is on long term commitment, innovation and implementation are discussed so that stakeholders understand system-level features necessary for initial and sustained implementation of the innovation

29. GOAL: 100% of students achieve
TIER I: Core, Universal
GOAL: 100% of students achieve
at high levels
Tier I: Implementing well researched programs and practices demonstrated to produce good outcomes for the majority of students.
Tier I: Effective if at least 80% are meeting benchmarks with access to Core/Universal Instruction.
Tier I: Begins with clear goals:
What exactly do we expect all students to learn ?
How will we know if and when they’ve learned it?
How you we respond when some students don’t learn?
How will we respond when some students have already learned?
Questions 1 and 2 help us ensure a guaranteed and viable core curriculum
Use Guiding Questions to get folks thinking in an integrated way
District facilitates discussions among schools on curriculum standards preschool through12th grade.
Systematic process for monitoring, evaluating, reviewing curriculum.
Florida PBS 29 29 29

30. For approx. 20% of students
TIER II: Supplemental, Targeted
Tier II
For approx. 20% of students
Core +
Supplemental
…to achieve benchmarks
Tier II Effective if at least 70-80% of students improve performance (i.e., gap is closing towards benchmark and/or progress monitoring standards).
Where are the students performing now?
Where do we want them to be?
How long do we have to get them there?
How much do they have to grow per year/monthly to get there?
What resources will move them at that rate?
District facilitates discussions among schools on curriculum standards preschool through12th grade.
Systematic process for monitoring, evaluating, reviewing curriculum. 30 30 30

31. Intensive, Individualized
Tier III
For Approx 5% of Students
Core +
Supplemental
Intensive Individual Instruction
…to achieve benchmarks
Where is the students performing now?
Where do we want him to be?
How long do we have to get him there?
What supports has he received?
What resources will move him at that rate?
Tier III Effective if there is progress (i.e., gap closing) towards benchmark and/or progress monitoring goals.
TIER III:
Intensive, Individualized
District facilitates discussions among schools on curriculum standards preschool through12th grade.
Systematic process for monitoring, evaluating, reviewing curriculum. 31 31 31

32. Installation
Resources needed to implement innovation with fidelity and desired outcomes are in place
Identify:
Coordinator
State Team
Implementation Team
Trainers, Coaches
Roles and Functions must be clear
Use the Blueprint and Team Workbook

33. Initial Implementation
Commitment to Capacity Building
Demonstrated High Fidelity/High Impact
Demand Increases
State Team won’t be able to keep up with demand

34. Features Point of Contact and Coaches become Local Coordinators
Transfer role to local person
Use phase of implementation to guide decision points
Meet with local team to build action plan- model after state team

35. District Level Phases Phase One: Commitment to Collaboration
Role of District
Role of State Leadership Team
ID Point of Contact
Small number of schools trained in PBIS
Small number of coaches identified and trained
Conduct Site visit to school two times/school year (completion of “implementation phase checklist”)
Provide technical assistance to local coaches
Provide SWIS training to schools as needed
Maintain communication with Point of Contact
Nominate Exemplar
Nag for Forms
Attend State Leadership Team Meetings monthly
Complete SET and SET Report as needed
Coordinate Regional Meetings for team leaders and coaches
Phase Two: Commitment to Capacity Building
More than 5% of schools trained in PBIS within district/region
Coach facilitator or lead coach identified (% of time allocated for PBIS activities)
Informal District/Regional team identified
SWIS facilitators identified and trained
SET Assessors identified and trained
Provide support and technical assistance to coach facilitator (coach facilitator coordinates site visits, “implementation phase checklist”, SWIS facilitation, SET assessment and support to local coaches)
Conduct site visits to schools by special request only
Assist with district self assessment and action plan
Assist with dissemination activities such as presentation to LMB, Community, Businesses, Superintendent
Conduct capacity building awareness activities using PBIS Blueprint with key stakeholders
Submit formal reports to PBIS management team to include progress notes, updates, training needs
Conduct needs assessment
Conduct SWIS facilitator training, SET assessor training
Phase Three: Commitment to Large Scale Implementation
Identification of PBIS Coordinator
Working Budget
District/Regional Action Plan in place to address capacity building around training, coaching, evaluation, coordination
District/Regional PBIS functioning team
Development of database
Progress Report in development
Development of newsletters, brochure and other marketing material
Alignment with LSS Master Plan, Charter, Code of Conduct
Formal report cycle to key stakeholders (superintendent, board, LMB, community/business, Parent organizations)
Active communication and coordination with PBIS Coordinator and PBIS District/Regional Team
Attendance to district/regional monthly meetings
Identify resources such as grant opportunities
Promote visibility to State
Assist with marketing/dissemination/policy

36. Full Implementation
Innovation is implemented and sustained by local stakeholders and is well-integrated into policy/written documentation
“New” is now standard practice

37. Innovation and Sustainability
Innovation and Integration
Demonstrated impact throughout
Change/Adapt to fit culture every year
Renew Commitment
Easier, More Efficient, Cost Reduces
Organizational Framework allows for integration
Educators as better consumers

38. Hybrid Training: Face to Face combined with Use of Technology
Training – always an evolving process….
Continuous opportunities to deepen both content and implementation knowledge and skills.
Establish training cycle that includes:
Face to face training and planning
Storage space
Adobe connect

39. Using a Google Drive, Dropbox
“Fast” access to content/tools/resources
Targets group needs
Build with group
Interactive or Storage Space