1. Overview: Evaluation Tools, On-Line Systems and Action Planning
Cohort 12
August 2016

2. Outcomes of Session Introduction of Tier 1 Evaluation Tools
Understand purpose of Tools
Understand process of Tool completion
Understand basic action planning process used with PBIS
Know where to complete surveys and generate reports on
Dirty Data

3. Use of Data
“Data speaks to the staff, ‘Look what we’ve accomplished; look where we need to go.’ It keeps it impersonal and really clarifies decision making for the future and how to celebrate the present and past.”
Andreou, T. E., McIntosh, K., Ross, S. W., & Kahn, J. D. (under review)

4. Predictors of Sustainability
School Priority:
Administrator support, staff support, perceived effectiveness, perceived efficiency, integration into new initiatives
.07
Team Use of Data:
School team/staff skill, functioning, regular meetings, data collection, use of data for decision making, presenting data to staff and community
.47
District Priority:
District support, state support, funding, district policy, promoted to external organizations
-.34
Capacity Building:
Access to district coaching, yearly professional development, connection to a community of practice
.41
McIntosh, K., Mercer, S. H., Hume, A. E., Frank, J. L., Turri, M. G., & Mathews, S. (2013). Factors related to sustained implementation of School-wide Positive Behavior Support. Exceptional Children, 79,

5. Classes of Data Effort data (Are we working the plan?)
Tiered Fidelity Inventory (TFI) - Used for progress monitoring
Self-Assessment Survey (SAS)
Outcome data (Is it having an effect?)
Core Reports - Graphs
Triangle Data
Fidelity data (What is the “fit” between the plan and the practice?)
Tiered Fidelity Inventory (TFI)
Used with walkthrough as an annual assessment

6. Effort data (Are we working the plan?)
“The plan” – Your action plan based on the 8 Steps in Chapter 2 and data that is collected from the tools described here
Collecting/gathering data to evaluate how you are doing. Where are you in getting the action done and working in your school?
TFI – Tiered Fidelity Inventory – for progress monitoring
SAS – Self-Assessment Survey

7. Self Assessment Survey (SAS)
SAS assesses effective behavior support systems in your school and guides team action planning
What do the 8 steps look like right now in your school?
Data is collected across four systems
School-wide
Non-classroom
Classroom
Individual Student
SAS provides entire staff (licensed and non-licensed) perspective

8. Self-Assessment Survey
WHO: Completed by all teachers, staff and administrators in your school building
WHAT: Examines the status and need for improvement of four behavior support systems (school-wide, classroom, non classroom, individual students)
WHERE: At your school
WHEN: Twice in – Sept/Oct and April/May
HOW: All school staff take the survey online at PBIS Assessment:

9. How?! How do you get the entire staff to take a survey?!
Make it a priority and explain why it’s important and how the data will be used. Reassure people that it’s OK to not really know how to respond (particularly this fall when you are just beginning) but to make an educated guess.
Make it a competition/game
During back-to-school workshop, faculty meeting, training for non- licensed?
Candy!

10. SAS Total Score Report
School-wide
Non-classroom
Classroom
Individual

11. SAS Subscale Report TRAINER NOTES:
This report shows the Current Status for each of the elements below, plus the implementation average:
Expectations Defined, Expectations Taught, Reward System, Violations System, Monitoring, Management, and District Support.

12. SAS Individual Items Report
TRAINER NOTES:
Especially useful for action planning, this report shows the Current Status (in place, partially in place, and not in place) and Improvement Priority (high, medium, and low) percents for each item within each system. 
It also displays the number of responses for each occasion the survey was conducted. 
You can select individual years and specific systems to assist with your action planning.

13. Data Based Decision Making Action Planning from the SAS
Review your results
Determine Action Steps
Your review of the SAS reports will lead you to action steps your team will take to improve your implementation of PBIS.
Look for items that are closest to 80% under current status/in place.
Look for items that may be well below 80% but have a very high priority for improvement.
If you have high ‘In place’ results, look for items match between high priority and partially in place.
Determine two or three actions your team will take based on SAS data.
Develop a plan for putting those items in place at your school.
Share results and action step proposals with staff to increase staff knowledge and willingness to participate in the plan.

14. Match Partially In Place and Improvement Priority
TRAINER NOTES:
Especially useful for action planning, this report shows the Current Status (in place, partially in place, and not in place) and Improvement Priority (high, medium, and low) percents for each item within each system. 
It also displays the number of responses for each occasion the survey was conducted. 
You can select individual years and specific systems to assist with your action planning.

15. School-wide PBIS Tiered Fidelity Inventory
Version 2.1
TFI is a tool to collect both Effort and Fidelity data T

16. Outcome data (Is it having an effect?)
“It” – Implementation of the 8 steps in Chapter 2 at your school
Effect on what? – Discipline, attendance, school climate
Office Discipline Referrals
Attendance
Student Survey, Parent Survey, Climate surveys

17. Average Referrals per Day per Month

18. Referrals by Location

19. Referrals by Behavior

20. Referrals by Time of Day

21. Referrals by Day of the Week

22. Referrals by Grade

23. Triangle Data

24. Fidelity data (What is the “fit” between the plan and the practice?)
Are we doing what we’re saying we’re doing?
Are there documented changes in “how we do things around here” or does the plan just say there are changes?
Tiered Fidelity Inventory (TFI)
Used with walkthrough as an annual assessment

25. School-wide PBIS Tiered Fidelity Inventory
Version 2.1 T

26. Purpose of the School-wide PBIS Tiered Fidelity Inventory
The purpose of the School-wide PBIS Tiered Fidelity Inventory is to provide an efficient and valid index of the extent to which PBIS core features are in place within a school.
Tier I (Universal PBIS)
Whole School Universal Prevention
Tier II (Targeted PBIS)
Secondary, Small Group Prevention
Tier III (Intensive PBIS)
Tertiary, Individual Support Prevention

27. SWPBIS Tiered Fidelity Inventory is a Self-Assessment
Primary purpose of the instrument is to help school teams improve
Primary audience for instrument results is the team, faculty, families and administrators of the school.
Effective use of the instrument requires multiple administrations (progress monitoring)

28. Uses of the SWPBIS Tiered Fidelity Inventory
Formative Assessment
Determine current PBIS practices in place and needed prior to launching implementation
Progress monitoring
Self-assess PBIS practices by tier to guide implementation efforts, and assess progress by tier
Build action plan to focus implementation efforts
Annual Self-Assessment
Self-assess annually to facilitate sustained implementation of PBIS; includes walk- through
State Recognition
Determine schools warranting recognition for their fidelity of PBIS implementation.

29. Tiered Fidelity Inventory
WHO: Completed by the PBIS Team
WHAT: Assesses the extent to which PBIS core features are in place within a school – Think 8 Steps from Chapter 2
WHERE: Duluth, St. Cloud, Bemidji
WHEN: At August, November and March training in and May which will include the walk-through both years
HOW: The PBIS team takes the inventory online at PBIS Assessment:

30. TFI Summary Data and Reports
Total Score
Subscale
Individual Items
70% is the benchmark at each tier to indicate “implementation at criterion”

31. Subscale Reports
This example shows a school that started only using Tier I for at least the first year. Once Tier I scored above 70 (almost 80) they decided to assess Tier 2. Once Tier 2 scored above benchmark, they added Tier 3.

32. Reviewing Scoring & Action Planning
This is a an sample report. The sub subscale report

33. Data-Based Decision Making
Action Planning
•Focus on Tier 1
•Choose 1-3 items to take action on
•Develop a plan to share with staff to increase their knowledge and willingness to participate
•Your External Facilitator will help guide your action planning

34. Pbisapps.org SWIS – School-Wide Information System PBIS Assessments
Tiered Fidelity Inventory (TFI)
Self-Assessment Survey (SAS)

35. PBIS APPS

36. School-Wide Information System
Brief introduction
SWIS –
Login as demo user
Click on the link to share a demo login for SWIS - it is ebs, ebs. The Core Reports are Average Referrals Per Day Per Month, Referrals by Time, Referrals by Location, Referrals by Day of the Week, Referrals by Problem Behavior, Referrals by Grade, and Referrals by Student. You could also show them the Drill Down feature.

37. A Brief Overview of PBIS Assessment:
You can again go to the PBISApps site and go to PBIS Assessment - type in Demo and you can show teams what the Dashboard looks like.

38. PBIS Assessment Account
Tiered Fidelity Inventory
Self Assessment Survey

39. Your Dashboard

40. PBIS Assessment Reports
Three reports are available for the Tools:
Total Score
Subscale Score
Individual Item

41. Online View of Self-Assessment Survey (SAS)

42. Online View of Self-Assessment Survey (SAS)

43. Resources http://bit.ly/TFIResources
TFI Manual
TFI Crosswalk with PBIS Assessment Tools
Targeted Interventions Reference Guide Map
TFI Action Planning Guide
TFI Tier III Support Plan Worksheet
TFI with Teams Presentation

44. TFI Resources http://bit.ly/TFIResources
Pre-Administration & Preparation
Supports for Tier II/Tier III and Action Planning
TFI Manual
Pre-Administration Documentation
TFI Crosswalk with PBIS Assessment Tools
Administering the TFI with Teams Slide Presentation
Targeted Interventions Reference Guide Map
Tier III Support Plan Worksheet
Action Planning Guide
There are additional resources available to help with preparing for the TFI and action planning following the TFI

45. Plan for Taking the TFI Today
Group 1
Group 2
12:00-1:30
Take the TFI with External Coach
1:30-3:00
Work on Implementation Steps 3 & 4
12:00-1:30
Work on Implementation Steps 3 & 4
1:30-3:00
Take the TFI with External Coach