Version 2.1 School-wide Evaluation Tool (SET)

 Define  Purpose of the School-wide Evaluation Tool (SET)  Process  Who participates?  Duration and Data-collection protocol  What permanent products will you need to see/collect?  Duration and Data-collection protocol  Item Analysis  Considerations for each of the items  Scoring Practice Training Goals

 The purpose of the SET is to provide an efficient and valid index of the extent to which PBIS core features are in place within a school.  Tier I (Universal PBIS)  Whole School Universal Prevention  Conducted annually by external coach/evaluator Purpose of the SET

 Formative Assessment  Determine current PBIS practices in place and needed prior to launching implementation  Annual Self-Assessment  Self-assess annually to facilitate sustained implementation of PBIS  Progress monitoring  Build action plan to focus implementation efforts  District/State Recognition  Determine schools warranting recognition for their fidelity of PBIS implementation  Research Studies on Effectiveness of SW-PBIS Uses of the SET

 The SET is a research-validated tool that has been shown to be:  Reliable  Test-Retest (M = 97.3%)  IOA (M = 99%)  Valid  Construct Validity (Pearson r =.71)  Internal Consistency  27 of 28 items are highly correlated  “Crisis plan readily available”  kept in SET for safety and legal reasons SET Related Research

School-wide Evaluation Tool How to Use the SET

 Interviews  Administrator  randomly selected staff  A minimum of 3 PBIS team members  15 Randomly selected students  Observations/Walk-through  School rules posted in 10 locations  Crisis procedure posted in 7 locations  Permanent products Administering the SET

 Let’s take a minute to familiarize ourselves with the tool…

Feature Evaluation Questions Data Source Score: 0-2 Essential Features of Universal Support Collection of Evidence to Document the Implementation of Tier I Essential Features P = Product I = Interview O = Observation 0 = descriptors of the feature not being implemented 1 = descriptors of the feature being partially implemented 2 = descriptors of the feature being fully implemented Features of the Scoring Tool

 Discipline Handbook  School Improvement Plan Goals  Annual Action Plan  Social Skills Instructional Materials/Lesson Plans  Behavior Incident Summaries or Reports  ODR Forms Permanent Product Review

School-wide Evaluation Tool Essential Features of Universal Support

 Expectations Defined (2)  Expectations Taught (5)  Acknowledgement Procedures (3)  Corrective Procedures (4)  Monitoring & Evaluation (4)  Management (8)  District-level Support (2) SET Features (# of evaluation questions)

A & B: Expectations Defined and Taught

 Is there documentation that staff has agreed to 5 or fewer positively stated school rules/ behavioral expectations?  (0=no; 1= too many/negatively focused; 2 = yes)  Is there agreement between information from administrator and staff interviews and publicly posted rules?  Is there a documented SYSTEM for teaching behavioral expectations to students on an annual basis?  (0= no; 1 = states that teaching will occur; 2= yes)  Let’s look at some examples… Behavioral Expectations: Permanent Products

Example: Score of 2

Example: Score of1

School-wide Evaluation Tool C: Rewarding Behavioral Expectations

 The key feature is a Documented SYSTEM for acknowledging appropriate student behaviors C: Rewarding Behavioral Expectations

Example: Score of 2

Example: Score of1

D: Responding to Behavioral Violations

Example: Score of 2

E: Monitoring & Decision Making

 Does the discipline referral form list: (1) student/grade, (2) date, (3) time, (4) referring staff, (5) problem behavior, (6) location, (7) persons involved, (8) probable motivation, & (9) administrative decision?  0= 0-3 items 1= 4-6 items 2= 7-9 items  Using the ODR forms provided, work with a partner to identify any missing information and score the forms as:  0=0-3 items  1= 4-6 items  2= 7-9 items Activity

F & G: Management and District Level Support

 F1: Does the school improvement plan list improving behavior support systems as one of the top 3 school improvement plan goals?  Note: Academic Goals can be combined/collapsed  F8: Does the team have an action plan with specific goals that is less than one year old?  Let’s look at an example… Management: Permanent Products

Example: Annual Action Plan

Interviews & Observations

 Schedule date and define expectations—time, people, materials  Interviews/Observations  Administrator interview should be conducted first  Allow for min for administrator interview and document review  Best to schedule 30 to 45 minutes before a natural break in the schedule (e.g., lunch, recess)  Walk-through Observations, Student/Staff Interviews  Can be conducted simultaneously and/or divided up among a pair of evaluators  Should be conducted when academic instruction is not occurring  Action Plan Development  May be done at same time or at a later meeting Typical Sequence of Administration

 School/Discipline Handbook  Documentation of systems for:  Teaching behavioral expectations  Rewarding appropriate student behaviors  Addressing and reporting specific behavioral violations  Office Discipline Referral Form  School Improvement Plan  School/PBIS Team Action Plan Possible Pre-Administration Documentation

 Discipline System 1.Do you collect and summarize office discipline referral information? Yes No (If no, skip to #4) 2.What system do you use for collecting and summarizing office referrals? (E2) a.What data do you collect? b.Who collects and enters the data? 3.What do you do with the ODR information? (E3) a.Who looks at the data? b.How often do you share it with other staff? 4.What type of problems do you expect teachers to refer to the office rather than handling in the classroom? (D2) 5.What is the procedure for handling extreme emergencies in the building (i.e., stranger with a gun)? (D4) Administrator Interview

 School-wide Expectations 6. Do you have school rules or a motto? Yes No (If no, skip to 10) 7. How many are there? 8. What are the rules/motto? (B4, B5) 9. What are they called? (B4, B5) 10. Do you acknowledge students for doing well socially? Yes No 11. If yes, what are the social acknowledgements/activities/ routines called (student of the month, positive referral, letter home, stickers, high 5’s)? (C2, C3) Administrator Questions, cont’d

 Data Systems and Team 12.Has the team taught/reviewed the school-wide program with staff this year? Yes No (B3) 13.Is your school-wide team representative of your school staff? Yes No (F3) 14.Are you on the team? Yes No (F5) 15.How often does the team meet? (F6) 16.Do you attend team meetings consistently? Yes No (F5) 17.Who is your team leader/facilitator? (F4) 18.Does the team provide updates to faculty on activities & data summaries? Yes No If Yes, how often? (E3, F7) 19.Do you have an out-of-school liaison in the state or district to support you on positive behavior support systems development? Yes No (G2) If yes, who? 20.What are your top 3 school improvement goals? (F1) 21.Does the School budget contain an allocated amount of money for building and maintaining school-wide behavior support? Yes No (G1)

Staff and Student Interviews

Staff/Faculty (at least 10) Questions 1.What are the __________________ (school rules, high 5's, 3 bee’s)? (B5) a.(Define what the acronym means) 2.Have you taught the school rules/behavioral expectations this year? (B2) 3.Have you given out any (rewards for appropriate behavior) since (2 month ago)? (C3) 4.What types of student problems do you or would you refer to the office? (D2) 5.What is the procedure for dealing with a stranger with a gun? (D4) 6.Is there a school-wide team that addresses behavioral support in your building? 7.Are you on the team?

Team Member (3-4) Questions 1.Does your team use discipline data to make decisions? (E4) 2.Has your team taught/reviewed the school-wide program with staff this year? (B3) 3.Who is the team leader/facilitator? (F4)

Student (15) Questions *Important to interview a representative sample of students 1. What are the _________________ (school rules, high 5's, 3 bee’s, define what the acronym means)? (B4) 2. Have you received a _____________ since (2 months ago)? (C2)  Hint: When talking to young students, select a recent holiday or break to refer back to (e.g., Have you received a High-Five Card from your teacher since Spring Break/Halloween/etc.)

Tip: Scoring Behavioral Expectations Taught  B4: Can at least 70% of 15 or more students state 67% of the school rules?  (0= 0-50%; 1= 51-69%; 2= %)  B5: Can 90% or more of the staff asked list 67% of the school rules?  (0= 0-50%; 1= 51-89%; 2=90%-100%)  Translation:  Can at least 11 of 15 students and 14 of 15 staff interviewed state:  2 out of 3;  3 out of 4; or  4 out of 5 school-wide expectations

Let’s Practice…

Your Turn! Small Group Activity

SET Overall Mean Score  When entering SET data into PBIS Assessment, there is no need to calculate Mean scores by hand…  Understanding how scores are determined:  Calculate percentage pts for each of seven subscales  Total the percentage pts  Divide total by 7 to get overall mean

Calculating SET Overall Mean Scores  A = 3 /4 = 75%  B = 5/10 = 50%  C = 3/6 = 50%  D = 4/8 = 50%  E = 4/8 = 50%  F = 4/16 = 25%  G = 2/4 = 50%  Total = 350  Mean = 350/7 = 50%  A = 4 /4 = 100%  B = 6/10 = 60%  C = 4/6 = 67%  D = 8/8 = 100%  E = 4/8 = 50%  F = 9/16 = 56%  G = 3/4 = 75%  Total = 508  Mean = 508/7 = 72.5%

What is Meant by 80/80?  A = 3 /4 = 75%  B = 5/10 = 50%  C = 3/6 = 50%  D = 4/8 = 50%  E = 4/8 = 50%  F = 4/16 = 25%  G = 2/4 = 50%  Total = 350  Mean = 350/7 = 50%  A = 4 /4 = 100%  B = 6/10 = 60%  C = 4/6 = 67%  D = 8/8 = 100%  E = 4/8 = 50%  F = 9/16 = 56%  G = 3/4 = 75%  Total = 508  Mean = 508/7 = 72.5%

For more information: Horner, R. H., Todd, A. W., Lewis-Palmer, T., Irvin, L. K., Sugai, G., & Boland, J. B. (2004). The school-wide evaluation tool (SET): A research instrument for assessing school-wide positive behavior support. Journal of Positive Behavior Interventions, 6(1),

Kathleen Strickland-Cohen, Ph.D., BCBA-D Thank You!