Evaluation Tools, On-Line Systems, and Data-Based Decision Making Version 3.0, Rev  This is a presentation of the Illinois PBIS Network. All rights reserved. Adapted from the Illinois PBIS Network

Agenda Evaluation Tools:  Team Implementation Checklist (TIC)  Benchmarks of Quality (BoQ)  Self Assessment Survey (SAS) PBIS

Outcomes of Session  Introduction of Tier 1 Evaluation Tools  Understand purpose of Tools  Understand process of Tool completion  Practice using the Tools  Know where to complete surveys and generate reports on

Tier 1 Evaluation Tools…  Team Implementation Checklist (TIC)  Benchmarks of Quality (BoQ)  Self-Assessment Survey (SAS)

 Effort data (Are we working the plan?)  Team Implementation Checklist (TIC)  Self-Assessment Survey (SAS)  Outcome data (Is it having an effect?)  “Big 5” Graphs  Triangle %  Fidelity data (Are we following the plan?)  School Evaluation Tool (SET)  Benchmarks of Quality (BoQ)

Purpose of PBIS Evaluation Tools  Getting Started Self Assessment Survey (SAS) Team Implementation Checklist (TIC)  Getting to Implementation Fidelity Team Implementation Checklist (TIC) Benchmarks of Quality (BoQ)  Sustaining Improvement SAS, BoQ, SET

PBIS Assessment Account  Team Implementation Checklist 3.1  Benchmarks of Quality  Self Assessment Survey

Your PBIS Assessment Account Your PBIS Technical Assistance Coordinator or External Coach will:  Give each school a unique account number  Provide written directions on how to complete the surveys  Set open/close time periods convenient for school(s) to complete the survey  When a survey period has been closed the survey results and graphs are generated automatically  Closed surveys can be reopened  You can take a survey more than once

A Brief Overview of PBIS Assessment:

School Account and Login

Select “Surveys” Access your open survey by clicking on “Launch”

PBIS Assessment Reports Three reports are available for the Tools: Total Score Subscale Score Individual Item

Team Implementation Checklist (TIC)

 Self-assessment tool for monitoring the implementation of school-wide PBS  Team self-assessment in six core areas Establish commitment Establish and maintain team Self-assessment Establish school-wide expectations Establish information system Build capacity for function-based support

Team Implementation Checklist (TIC)  WHO: Completed by the Universal PBIS Team  WHAT: Guides the development, implementation, monitoring and revision process for building a positive school culture  WHERE: At your school during a Universal PBIS Team meeting  WHEN: 3x per year  HOW: One person, usually the Internal PBIS Coach, enters the data online at

Online View of Team Checklist Enter date the survey was completed

Enter the TIC Data

TIC Total Score Report

TIC Subscale Report

TIC Individual Items Report

TIC Activity In a Small Group…  Review the results of the TIC Total Score, Subscale & Individual Item Reports  What do the survey data results say about this Universal team’s functioning?  What next steps would you recommend this team include in their Action Plan?  Why?

School-Wide Evaluation Tool SET

 The SET is a research-validated instrument that is designed to assess and evaluate the critical features of school-wide effective behavior support across an academic school year. School-Wide Evaluation Tool (SET)

 To determine the extent to which schools are already using School-wide Positive Behavior Support (PBIS)  To determine if training and technical assistance efforts result in change when using PBIS  To determine if use of PBIS procedures is related to valued change in the safety, social culture, and violent behavior in schools The Set was Designed To:

 Assess features that are in place  Determine annual goals  Evaluate on-going efforts  Design and revise procedures  Compare year to year efforts in the area of PBIS Information Gathered Can Be Used To:

 Expectations defined  Behavioral expectations taught  Acknowledgement procedures  Correction procedures  Monitoring and evaluation  Management  District-level support SET Evaluates Questions Across Seven Featured Areas:

SWIS

What are our “major” and “minor” behavior referral forms indicating? What data options are available when a student needs additional social interventions by a teacher in the classroom (minor) or when other school personnel need to me informed (major), how data gets collected. SWIS – Major & Minor Other district based system

Report Options using SWIS:  Average Referrals Per Day Per Month  Referrals By Problem Behavior  Referrals By Location  By Student  Referrals By Time

How are we doing? Look for trends Plan time to re-teach

Are there target behaviors? Behavior of the week or Staff training/development By Problem Behavior

Look for hot spots Review the rules for the locations Review the lessons for teaching the rules By Location

By Time

By Student

How are we doing? Look for trends Plan time to re-teach

Do we have a solid green zone? 80%?

Self Assessment Survey (SAS)

 SAS assesses the extent to which PBIS practices and systems (essential components) are in place within a school School-wide Non-classroom Classroom Individual Student  SAS provides entire staff (certified and non-certified) perspective

Self-Assessment Survey (SAS )  WHO: Completed by all teachers, staff and administrators in your school building  WHAT: Examines the status and need for improvement of four behavior support systems (school-wide, classroom, non classroom, individual students)  WHERE: At your school  WHEN: Annually, usually at the beginning or towards the end of each school year  HOW: All school staff take the survey online at PBIS Assessment:

Online View of Self-Assessment Survey (SAS)

School-wide Non-classroom Classroom Individual SAS Total Score Report

SAS Subscale Report

SAS Individual Items Report

Self-Assessment Survey Activity with a Small Group…  Use the SAS Individual Items Report as a reference.  Discuss how you might approach problem solving with your team if the staff identified an item or items as HIGH for Not In Place but LOW for Improvement Priority.

Data-Based Decision Making

Data-Based Decision-Making SWIS or SWIS Like Systems  Developing Precise Statements  Solution Development Prevention, Teaching, Reward, Extinction, Corrective Consequence, and Data Collection  Action Plan for Results

Using Data for On-Going Problem-Solving  Start with the perceptions not the data  Use data in “decision layers” Is there a problem? (overall rate of ODR) Localize the problem  (location, problem behavior, students, time of day) Get specific  Don’t drown in the data  It’s “OK” to be doing well  Be efficient

Six Things to Avoid  Define a solution before defining the problem  Build solutions from broadly defined, or fuzzy problem statements  Failure to use data to confirm/define problem  Agree on a solution without building a plan for how to implement or evaluate the solution  Agree on a solution but never assess if the solution was implemented  Serial problem solving without decisions

Uses of Data  Review current status and identify problems early. Use data on a regular basis (every two weeks) to monitor key indicators, and identify problems before they become difficult.  Refine a problem statement to a level of precision that will allow functional solutions.  Use data to test possible solutions.  Use data to assess if solutions are working If many students are making the same mistake it typically is the system that needs to change not the students.

Benchmarks of Quality (BoQ)

 The Benchmarks of Quality measures the degree to which a school is implementing the universal level of school-wide positive behavior support.  The results provide a mechanism for school teams to identify strengths and weakness based on fidelity criteria for use in establishing future action plans.

Benchmarks of Quality (BoQ)  WHO: The PBIS Universal Team  WHAT: Lists 10 critical elements of quality for Universal School-wide PBIS implementation. This tool identifies strengths and opportunities for change and growth  WHERE: At your school  WHEN: The spring of each school year Mar/Apr/May  HOW: Online directions at PoI Narrated Instructions (PPT 2003 & 2007)

Technical Assistance Coordinator Enters BoQ Online

BoQ Total Score Report

BoQ Subscale Report

BoQ Individual Items Report

BoQ Activity In a Small Group…  Review the results of the BoQ Subscale and BoQ Individual Item Reports.  What action items would you recommend this team consider for improving their Faculty Commitment and Discipline Procedures?

Collect and Use and UseData Review Status and Identify Problems Develop and Refine Hypotheses Discuss and Select Solutions Develop and Implement Action Plan Evaluate and Revise Action Plan Problem Solving Foundations Team Initiated Problem Solving (TIPS) Model

Identifying Problems/Issues  What data to monitor ODR per day per month OSS, ISS, Attendance, Teacher report Team Checklist/ SET (are we doing what we planned to do?)  What question to answer Do we have a problem?  What questions to ask of Level, Trend, Peaks How do our data compare with last year? How do our data compare with national/regional norms? How do our data compare with our preferred/expected status?  If a problem is identified, then ask What are the data we need to make a good decision?

Using Data to Refine Problem Statement  The statement of a problem is important for team-based problem solving. Everyone must be working on the same problem with the same assumptions.  Problems often are framed in a “Primary” form, that creates concern, but that is not useful for problem-solving. Frame primary problems based on initial review of data Use more detailed review of data to build “Solvable Problem Statements.”

Precise Problem Statements (What are the data we need for a decision?)  Precise problem statements include information about the Big Five questions: What is problem, and how often is it happening Where is it happening Who is engaged in the behavior When the problem is most likely Why the problem is sustaining

Primary vs. Precision Statements  Primary Statements Too many referrals September has more suspensions than last year Gang behavior is increasing The cafeteria is out of control Student disrespect is out of control  Precision Statements There are more ODRs for aggression on the playground than last year. These are most likely to occur during first recess, with a large number of students, and the aggression is related to getting access to the new playground equipment.

Primary vs. Precision Statements  Primary Statements Too many referrals September has more suspensions than last year Gang behavior is increasing The cafeteria is out of control Student disrespect is out of control  Precision Statements There are more ODRs for aggression on the playground than last year. These are most likely to occur during first recess, with a large number of students, and the aggression is related to getting access to the new playground equipment.

Precise or Primary Statement?  Children are using inappropriate language with a high frequency in the presence of both adults and other children. This is creating a sense of disrespect and incivility in the school.  James D. is hitting others in the cafeteria during lunch, and his hitting is maintained by peer attention.

Precise or Primary Statement?  ODRs during December are higher than in any other month.  Minor disrespect and disruption are increasing over time, and are most likely during the last 15 minutes of our block periods when students are engaged in independent seat work. This pattern is most common in 7 th and 8 th grades, involves many students, and appears to be maintained by escape from work (but may also be maintained by peer attention… we are not sure).

Precise or Primary Statement?  Three 5 th grade boys are name calling and touching girls inappropriately during recess in an apparent attempt to obtain attention.  Boys are engaging in sexual harassment.