1. Data Based Decision Making: Evaluating the Impact of SWPBIS
Idaho SWPBIS Training Institute

2. Purpose Define the outcomes for SWPBIS
Is SWPBIS related to reduction in problem behavior?
Is SWPBIS related to improved school safety?
Is SWPBIS related to improved academic performance?
Define tools for measuring SWPBIS outcomes
Examine a problem-solving approach for using ODR data for decision-making
Provide strategies for using data for decision-making and action planning

3. Idaho’s Tiered Instructional and Positive Behavioral Interventions and Support (PBIS) Framework
Academic Systems
Behavioral Systems
Intensive, Individual Interventions
Individual Students
Assessment-based
High Intensity
Intensive, Individual Interventions
Individual Students
Assessment-based
Intense, durable procedures
1-5%
1-5%
Targeted Group Interventions
Some students (at-risk)
High efficiency
Rapid response
5-10%
5-10%
Targeted Group Interventions
Some students (at-risk)
High efficiency
Rapid response
Transition 1 tier at time, giving participants time to discuss each one.
One of the most important organizing components of PBIS is the establishment of a continuum of behavior support that considers all students and emphasizes prevention. This logic of this 3-tiered approach is derived from the public health approach to disease prevention.
All students and staff should be exposed formally and in an on-going manner to primary prevention interventions. Primary prevention is provided to all students and focuses on giving students the necessary pro-social skills that prevent the established and occurrences of problem behavior. If done systematically and comprehensively, a majority of students are likely to be affected.
Some students will be unresponsive or unsupported by primary prevention, and more specialized interventions will be required. One form of assistance is called secondary prevention, and is characterized by instruction that is more specific and more engaging. These interventions can be standardized to be applied similarily and efficiently across a small number of students. The goal of secondary prevention is to reduce/prevent the likelihood of problem behavior occurrences, and to enable these students to be supported by the schoolwide PBIS effort.
If primary prevention is in place, a small proportion of students will require highly individualized and intensive interventions. The goal or tertiary level interventions is to reduce the intensity, complexity, and impact of the problem behaviors displayed by these students by providing supports that are:
Function-based
Contextually appropriate and person-centered
Strength-based and instructionally oriented
Continuously evaluated and enhanced
Linked to the schoolwide PBIS approach
Tier 2 & 3: Handled by a small group of educators who are intensely trained.
Universal Interventions
All students
Preventive, proactive
80-90%
Universal Interventions
All settings, all students
Preventive, proactive
80-90%

4. To Improve Schools for Children
Use evidence-based practices
Always look for data of effectiveness
Never stop doing what is working
Implement the smallest change that will result in the largest improvement
Measure
Compare
Improvement

5. Model of Continuous Improvement
Plan Do
Check
Act
Plan: Identify an opportunity and plan for change.
Do: Implement the change on a small scale.
Check: Use data to analyze the results of the change and determine whether it made a difference.
Act: If the change was successful, implement it on a wider scale and continuously assess your results. If the change did not work, begin the cycle again.
Other widely used methods of continuous improvement — such as Six

6. Improving Decision-Making
Problem
Solution
Problem
Problem-solving Information
Solution
Moving from the first model (problem →solution) to new model (problem→problem-solving information→solution)

7. Problem-solving Steps
Define the problem(s)
Analyze the data
Define the outcomes and data sources for measuring the outcomes
Consider 2-3 options that might work

8. Problem-solving Steps
Evaluate each option
Is it safe?
Is it doable?
Will it work?
5. Choose an option to try

9. Problem-solving Steps
Determine the timeframe to evaluate effectiveness
Evaluate the effectiveness by using the data
Is it worth continuing?
Try a different option?
Redefine the problem?

10. Key Features of Effective Data Systems
Data are accurate
Data are very easy to collect
Data are used for decision-making
Data are available when decision need to be made
Data collectors must see the information used for decision making

11. Key Features of Effective Data Systems
Activity: Reflect on your data.
What data do you have? Is it:
Accurate?
Very easy to collect?
Used for decision-making?
Available when decisions need to be made?
Collected to be used?
Team activity – share out

12. Guiding Considerations
Use accessible data
Handle data as few times as possible
Build data collection into daily routines
Establish and use data collection as a conditioned positive reinforcer
Share data summaries with those who collect it

13. Questions to Consider When Collecting Data
Is the current approach achieving the intended outcomes?
Is the plan working as well as or better than it did last year?
Is a change in the plan needed?
Do students have the skills to do what is expected?
Are the behavioral needs of all students being adequately met?
What areas need improvement?
Which grade level(s ) need additional skills training?
What physical areas of the school are perceived as less safe?
Which classroom routines do students need to be retaught?
Which students need additional support?
Which students received two or more ODRs in the first month of school?
Which students consistently show signs of emotional distress (e.g., anxiety, depression)?
Young et al., 2012
See Handout – as a team, discuss each question

14. Collecting data without having a definite question is like going shopping without a list. Teams may spend a great deal of effort and time collecting data and creating systems that do not give them the answers they really need.
Young et al., 2012

15. Types of Questions Initial Assessment Questions:
What type or which program do we need?
Where should we focus our efforts?
Ongoing Evaluation Questions:
Is the program working?
If no:
Can it be changed?
Should we end the program?
If yes:
Do we need this program anymore?
What do we need to do to sustain success?
Initial questions should have been discussed/answered during Day 1-2 of training.
Ask teams to discuss the “Ongoing Evaluation Questions”; share out

16. What Data Should Be Collected?
Always start with the questions you want to answer.
Make data that will answer your question.
Balance between reliability and accessibility.
Systems approach
Consider logistics.
Who?
When?
Where?
How?
Two levels
What is readily accessible?
What requires extra resources?
Reliability – are you truly measuring what you set out to measure?
EASY, EFFECTIVE, & EFFICIENT!

17. When & By Whom Should Data Decisions Be Made?
Natural cycles, meeting times
Weekly, monthly, quarterly, annually
Level of system addressed
Individual: daily, weekly
Schoolwide: daily, weekly
District/Region
State-level
Teacher, coach, support personnel, paras, secretary

18. Basic Evaluation Questions by School or Program
What does “it” look like now?
How would we know if we are successful?
Are we satisfied with how “it” looks?
YES:
Celebrate
NO:
What do we want “it” to look like?
What do we need to do to make “it” look like that?
What can we do to keep “it” like that?
Team Activity: discuss questions, document on poster and share out

19. Basic SWPBIS Evaluation Questions
Are our efforts making a difference?
Is our school adopting SWPBIS to criterion?
Is our school perceived as safe?
Are teachers delivering instructional lessons with fidelity as planned?
Is SWPBIS improving student outcomes?
Discuss questions within teams.

20. Is SWPBIS Having a Positive Influence on School Culture?
Using Office Discipline Referral Data

21. Office Discipline Referrals
Examine office discipline referral rates and patterns
Major Problem Events
Minor Problem Events

22. Office Discipline Referrals
Ask the “BIG 5” questions:
How Often are problem behavior events occurring?
Where are they happening?
What types of problem behaviors?
When are the problems occurring?
Who is contributing?
Team Activity: Look at school ODR data answer the “Big 5”
Use “fake” data – if team didn’t bring any of their own

23. Office Discipline Referral Caution
Data reflects 3 factors:
Students
Staff members
Office personnel
Data reflects overt rule violators
Data is useful when implementation is consistent.
Do staff and administration agree on office-managed problem behavior versus classroom-managed behavior?
Ask teams, “Do staff and administration agree on office-managed problem behavior versus classroom-managed behavior?

24. Defining MINOR, MAJOR, & CRISIS events in your school.
Complete Handout: “Defining MINOR, MAJOR, & CRISIS events in your school”

25. Staff Managed (minors) Office Managed (majors)
Tardy
Unprepared; no homework/materials
Violation of classroom expectations
Inappropriate language
Classroom disruption
Minor safety violation
Lying/cheating
Consequences are determined by staff.
Repeated minor behaviors
Insubordination
Blatant disrespect
Abusive/inappropriate language
Harassment/intimidation
Fighting/physical aggression
Safety violations that are potentially harmful to self, others and/or property
Vandalism/property destruction
Plagiarism
Theft
Skipping classes
Illegal behaviors: arson, weapons, tobacco, alcohol/drugs
Additional example in Handout: “Staff Managed Behaviors. Office Managed Behaviors”

26. General Procedure for Dealing with Problem Behaviors
Observe problem
behavior
Find a place to talk with student(s) NO
Is behavior major?
YES
Ensure safety
Problem solve
Write referral and
Escort student to office
Determine
consequence
Problem solve
Determine
consequence
Follow procedure
documented
Follow
documented
procedure
Does
student
have 3? NO
YES
Follow
through with
consequences
Diagram – make into handout
File necessary
documentation
Send
referral to
office
File necessary
documentation
Follow up
with student
within a
week

27. Compatibility Question
SWIS™ Compatibility Checklist
Procedure for Documenting Office Discipline Referrals
School ___________________________ Date ____________________
Compatibility Question
Date
1. Does a clear distinction exist between problem behaviors that are staff managed versus office managed exist and is it available for staff reference?
Yes No
2. Does a form exist that is SWIS™ compatible for SWIS™ data entry that includes the following categories?
a. Student name?
b. Date?
c. Time of incident?
d. Student’s grade level?
e. Referring staff member?
f. Location of incident?
g. Problem behavior?
h. Possible motivation?
i. Others involved?
j. Administrative decision?
k. Other comments?
l. No more than 3 extra info.
3. Does a set of definitions exist that clearly defines all categories on the office discipline referral form?
Ask teams to complete the compatibility checklist referencing their ODR.
Next review date: _______________
Redesign your form until answers to all questions are “Yes.”
Readiness requirements 4 and 5 are complete when you have all “Yes” responses.

28. Tables versus Graphs

29. Year
Month
Number of Days
Number of Referrals
Average Referrals
Per Day
2001
Aug
0.00
Sep 19 5
0.26
Oct 21 18
0.86
Nov 17
0.94
Dec 14
1.50
2002
Jan 22
0.82
Feb 15
0.88
Mar 26
1.37
Apr
0.67
May 13
0.72
Jun 11 2
0.18
Jul
Totals:
180
149
0.83

30. Number of ODR per Day and Month

31. Total versus Rate

32. Total Number of ODRs per Month

33. Number of ODRs per Day and Month

34. Priorities & Rational
Graphs
Rate

35. Interpreting Office Referral Data: Is there a problem?
Absolute level (depending on size of school)
Middle Schools (>5 per day)
Elementary Schools (>1.5-2 per day)
Compare levels to last year
Improvement?
Trends
Peaks before breaks?
Gradually increasing trend across year?

36. Are Schools Adopting SWPBIS to Criterion?
Use the:
Team Implementation Checklist (TIC 3.1)
Schoolwide Evalutation Tool (SET)
EBS Self-Assessment Survey (SAS – “Schoolwide” section)
Measure and analyze annually
Today we will focus on the TIC 3.1
*Remember: SET must be done by an “outside” evaluator

37. Team Implementation Checklist (TIC 3.1)
Characterizes the evolution of SWPBIS implementation:
“Achieved,” “In progress,” or “Not started”
Assists in:
Initial assessment
Getting started on action plan
Measuring progress of SWPBIS implementation
Assesses team-based response
Quarterly or monthly
Facilitators: distribute copies of teams’ completed TIC from the August training (Day 3-4)

38. TIC 3.1 Feature Areas Establish commitment Establish and maintain team
Conduct self-assessment
Define expectations
Teach expectations
Establish reward system
Establish violations system
Establish information system
Build capacity for function-based support
Ongoing activities

39. Scoring the TIC 3.1 Implementation Points
Achieved = 2
In progress = 1
Not started = 0
Percentage of items implemented
Total
Number of items scored as “Achieved” divided by 17 (items)
Subscale scores
Number of items in each subscale area scored as “Achieved” divided by the number of items in that subscale area
Percentage of points implemented
Total number of points divided by 34
Total number of points in each subscale divided by total number of items multiplied by 2
Not sure teams correctly scored their TIC during August training – may need to go through and correctly score them to save time and provide a model????
Ask teams to complete TIC 3.1 and hand in one copy to training facilitator.

40. SWPBIS Main Messages Invest in prevention
Create an effective environment
Leadership, teams; hosts for effective practices
Use different systems for different problems
Individual student level alone will be insufficient
Collaboration with Mental Health Professionals
Build a culture of competence
Define, teach, monitor, and reward appropriate behavior
Build sustainable systems
Resist person-dependent interventions
Invest in gathering and using information for decision-making and problem-solving

41. Action Planning for SWPBIS
Use your self-assessment information
Rally schoolwide commitment
Establish a PBIS team
Focus on prevention (define, teach, monitor, and reward appropriate behavior)
Ask kids tomorrow if they know the expectations
Ask kids if they are being acknowledged for appropriate behavior
Use information system to guide implementation efforts
Build Action Plan
When will the team meet?
What will be reported to faculty?
What will be reported to families?

42. Action Planning for SWPBIS
Which system are you going to work on?
What are the specific outcomes?
When will they be completed?
What short-term activities are needed?
Who will be responsible?
Reported Schedule
What information will be gathered and by whom?
When will information be reported?

43. Dirty Data Video: Dirty Data (“School Data – A Comedy”) – 2:57 min