1. Drilling Down in SWIS Data
Using a Problem-Solving Model to Address Discipline Disproportionality
Jennifer Rollenhagen, Eben McKnight, & Susannah Everett
NEPBIS Leadership Forum 2017

2. Acknowledgements: Educational and Community Supports
ECS is a research unit within the University of Oregon’s College of Education. ECS focuses on the development and implementation of practices that result in positive, durable, and scientifically substantiated change in the lives of individuals. ECS engages in federal and state funded projects focused on PBIS to support research, teaching, dissemination, and technical assistance.
Thank you Bert Eliason & Katie Conley!

3. Organizing and Analyzing the Data Putting It Into Practice
Session Intentions
Goal
Use a problem-solving model focused on the use of data to address discipline disproportionality
Resources
Data Guide for School Teams (pbis.org, 2014)
SWIS Ethnicity Report (pbisapps.org)
Four-step problem-solving model
Organizing and Analyzing the Data
Data Sources
Common Metrics
Drill Down Process
Putting It Into Practice
Simulated School Example

4. Intentions vs. Realities
We truly believe that…
Schools want good outcomes for all their students
By and large, the profession is not purposely leaving anyone behind
But unfortunately, the outcomes are not equal for everyone

5. Disproportionality:
Disproportionality in discipline referral data is the overrepresentation or underrepresentation of a subgroup of students based on their race or gender in comparison to the total student population.
PBIS.org (September 2014)

6. Does Disproportionality Really Exist?
Is it everywhere?
Who does it affect?
What can we do about it?

7. Disproportionality in School Discipline (Losen et al., 2015)

10. How do we measure this?
What is necessary?
What else is recommended?
How do we get this information?

11. U.S. Department of Education Office for Civil Rights 3 Civil Rights Data Collection: Data Snapshot (School Discipline) March 21, 2014

12. Black students (16.4%) are 3 x more likely to be suspended or expelled than white students (4.6%).
Black boys and girls have higher suspension rates than any of their peers. Twenty percent (20%) of black boys and more than 12% of black girls receive an out-of-school suspension.
U.S. Department of Education Office for Civil Rights 3 Civil Rights Data Collection: Data Snapshot (School Discipline) March 21, 2014

13. U.S. Department of Education Office for Civil Rights 3 Civil Rights Data Collection: Data Snapshot (School Discipline) March 21, 2014

14. U.S. Department of Education Office for Civil Rights 3 Civil Rights Data Collection: Data Snapshot (School Discipline) March 21, 2014

16. States? State Districts Above Threshold Districts in the State
% of Districts in State that would be Significantly Disproportionate CT
122
197
61.9% DE 19 43
44.2% MA
257
418
61.5% ME 59
259
22.8% NH 62
178
34.8% NJ
410
691
59.3% NY
491
1008
48.7% PA
338
738
45.8% RI 33 57
57.9%
Nat’l.
8,148
17,371
46.9%
Number and percent of districts with a Risk Ratio in at least one subgroup in at least one disability, ed. environment or discipline category that Exceeds Two MADs above the Nat’l median with a minimum cell size of 10 by state for all Three years ( , , )
Excludes districts in CO for any identification metrics, and all but one district in VT for Disciple removal metrics.

19. The Data Guide Identified:
Revision coming soon!
The Data Guide Identified:
Data Needed for Investigating Disproportionality
Four-step Problem Solving Model
Metrics to Use to Monitor Disproportionality
Bias in Decision Making – Explicit vs Implicit
Vulnerable Decision Points

20. 1. Problem Identification
From the Data Guide
Problem Solving Model
2. Problem Analysis
3. Plan Implementation
4. Plan Evaluation
1. Problem Identification
Is there a problem?
Why is it happening?
You also need for your school’s PBIS team to have a problem-solving model… it doesn’t need to be overly complicated, this one has 4 basic steps:
Is the plan working?
What should be done?

21. Step 1: Problem Identification
Use valid & reliable metrics.
Quantify the difference between current outcomes and goals.
This is the performance gap!
1. Problem Identification
Is there a problem?

22. 1. Problem Identification
Step 1: Problem Identification
For disproportionality
Quantify outcomes across racial/ethnic subgroups
Compare differences
Specified group vs. Comparator group
Subgroup vs. White subgroup
Subgroup vs. All Other students
Native vs. All Non-Native students
1. Problem Identification
Is there a problem?
Multiple metrics are recommended!
IDEA Data Center, 2014

23. Risk Ratios A metric commonly used to measure disproportionality
Represents the likelihood of discipline referrals for one group in relation to a comparison group
Calculated by using the Risk Index of one group divided by the Risk Index of the comparison group
Example threshold: If at or above 1.5, there is a risk for one group to be over represented for discipline referrals compared to another group

24. Automatically calculates the risk ratio to compare one group with all other groups

25. SWIS Ethnicity Reports -
Automatically calculates the proportionality between subgroup’s percent of population and subgroup’s percent of students with referrals.
overrepresentation
underrepresentation

26. SWIS Ethnicity Reports -
Automatically calculates the proportionality between subgroup’s percent of population and subgroup’s percent of total referrals.
overrepresentation
underrepresentation

27. Review Step 1: Problem Identification Select multiple metrics to use
Relative measure – Use Risk Indices to calculate Risk Ratio
Absolute measure – ODRs per 100 Students per Day
Composite measurements – What is composition of the problem?
Calculate metrics and compare to reasonable goals
Previous years from same school
Local, state or national norms
Logical criteria – “4/5ths Rule”
State’s reasonable threshold for “significant disproportionality”
Monitor metrics throughout the year
Monthly or quarterly
Be careful of small “Ns”
Be careful of risk indices

28. Step 2: Problem Analysis
1. Problem Identification
2. Problem Analysis
Is there a problem?
What is happening and why?
Once you have identified that there is a problem you can then analyze the problem to get more precise about the context and why it keeps occurring.

29. Step 2: Problem Analysis
Purpose: Identify context and underlying cause(s)
Focus: Systems & practices that can be changed
Evaluate: Tier 1 (universal systems)
Check fidelity of PBIS implementation
TFI - CR Companion
Disparities other than discipline
Academic Placement
Attendance
School climate
Graduation
Discipline data for patterns of bias
 Explicit bias  Implicit bias
Research indicates that oftentimes bias is implicit rather than explicit. Implicit bias tends to be localized to specific situations (e.g., behaviors, locations).

31. What is implicit bias? Unconscious, automatic
Generally not an indication of our beliefs and values
We all have it (even those affected by it)
Based on stereotypes
More likely to influence:
Snap decisions
Decisions that are ambiguous

32. What is a Vulnerable Decision Point?
Step 2: Problem Analysis
What is a Vulnerable Decision Point?
A VDP is a specific decision that is more vulnerable to effects of implicit bias.
Two parts:
Elements of the situation
External or environmental context
The person’s decision state
Internal state or personal context

33. Two Systems for Decision Making
Step 2: Problem Analysis
Two Systems for Decision Making
System 1: Fast Decisions
Automatic, snap judgments
Intuitive, unconscious
System 2: Slow Decisions
Deliberate decisions
Allows for conscious attention
Thinking, fast and slow – have you read it? I have not, but I heard it on NPR, so I think I’ve read it.
1: auto-pilot, gut reaction
2: simmer, sink in
What are some decisions:
ODR?
Suspension?
And 2 kinds of bias
Kahneman, 2011

34. VDPs from national ODR data
Step 2: Problem Analysis
VDPs from national ODR data
Subjective problem behavior
Defiance, Disrespect, Disruption
Vague discipline system
Major vs. Minor
Non-classroom areas
Hallways
Playgrounds
Afternoons
AMBIGUITY
“Ambiguity is disproportionality’s best friend”
LACK OF CONTACT
FATIGUE / HUNGER

35. What can we do about these VDPs?
Step 2: Problem Analysis
What can we do about these VDPs?
Teach two-step Neutralizing Routine for Staff:
When you see problem behavior, stop and ask:
Is this possibly a VDP?
Consider the context of the situation
Consider your personal decision state
If so, use an agreed-upon alternative response
Slow down, consider options

36. Neutralizing Routines for Reducing Effects of Implicit Bias
Step 2: Problem Analysis
Neutralizing Routines for Reducing Effects of Implicit Bias
Self-assessment
“Is this a vulnerable decision point?”
Setting event
Antecedent
Behavior
Consequence
Lack of positive interactions with student
Fatigue
Loud complaints about work (subjective behavior)
Send student to office (ODR)
Student leaves class (Escape social interaction)
Alternative Response
“See me after class.”

37. Step 2: Problem Analysis
Defining Problems with Precision
Who is involved?
What are the problem behaviors?
Where is it happening?
When is it happening?
Why are these things happening?
Perceived function of problem behavior

38. When we believe we have a problem related to disproportionality….
Step 2: Problem Analysis
When we believe we have a problem related to disproportionality….
Assess
PBIS implementation fidelity
TFI-CR Companion
Performance gaps
Achievement gap
Academic placement
Attendance
School climate
Identified Subgroup
Location
Time of Day
Problem Behavior
Motivation
Many seventh grade Black students are receiving referrals from the classroom in the afternoon for inappropriate language. Referrals are perceived to be task avoidance and getting adult attention.

39. Precise Problem Statement
SWIS Drill Down
When we have a problem with disproportionate discipline…
Precise Problem Statement

40. Subgroup: African American Students
Step 2: Problem Analysis
SWIS Drill Down
Subgroup: African American Students
Who?
When?
What?
Where?
Why?
3rd grade
11:30 - Noon
Physical Aggression
Playground
Peer Attention
4th grade
8:00 AM– 9:30 AM
M-Defiance
Classroom
Avoid task
7th grade
After 12:00 PM
Inappropriate Language
Classroom Hallway
Obtain Peer Attention

41. Precise Problem Statements
Step 2: Problem Analysis
Precise Problem Statements
African American students in the 3rd grade are receiving referrals for physical aggression during noon recess. Referrals seem to be related to gaining peer attention.
African American students in the 4th grade are more likely to receive referrals for minor defiance in the classroom during the morning instructional block. Referrals seem to be related to task avoidance.
African American students in the 7th grade are receiving afternoon referrals in the classroom and hallways for inappropriate language. Referrals are related to avoiding tasks and gaining peer attention.

42. Drill down

43. Step 3: Plan Implementation
1. Problem Identification
2. Problem Analysis
Is there a problem?
3. Plan Implementation
What is happening and why?
What should be done?

44. Step 3: Plan Implementation
Information from Step 2 is used to select tasks and strategies to address the problem.
An action plan is created to ensure adequate implementation of the tasks and strategies.
Action plans show everyone –
WHO will do WHAT by WHEN.
Action plans that are published –
Help create accountability

45. Step 3: Plan Implementation
Examples
All issues
Calculate and share disproportionality data regularly
Inadequate PBIS implementation
Implement core features of PBIS to establish a foundation of support
Consider using TFI Culturally Responsive Companion
Misunderstandings regarding school-wide expectations
Enhance culturally responsive PBIS with input from the students/families
Academic achievement gap
Implement effective academic instruction

46. What is happening and why?
Step 4: Plan Evaluation
1. Problem Identification
2. Problem Analysis
Is there a problem?
4. Plan Evaluation
3. Plan Implementation
What is happening and why?
Is the plan working?
What should be done?

47. Step 4: Plan Evaluation Evaluation Regularly assess
Progress and fidelity of plan implementation
Fidelity of PBIS implementation
Calculate metrics from Step 1
Compare to the goal determined in Step 1
Share results with relevant stakeholders
Plan for what is next

48. Step 4: Plan Evaluation Evaluation Time Frame:
Identify time periods for regularly evaluating and analyzing disproportionality data.
Caution: Disproportionality metrics may not be sensitive to rapid change.
Consider monthly assessment of plan implementation & quarterly assessment of disproportionality metrics.
Avoid relying on risk indices as they will increase throughout the year.
Use multiple measures to ensure that you are tracking the correct thing.

49. 1. Problem Identification
Problem Solving Model
2. Problem Analysis
3. Plan Implementation
4. Plan Evaluation
1. Problem Identification
Is there a problem?
Why is it happening?
Is the plan working?
What should be done?

50. PBIS.org - National Resource Center
is the national resource center for materials and information related to the implementation of PBIS with fidelity.
School

51. PBIS.org - National Resource Center
is the national resource center for materials and information related to the implementation of PBIS with fidelity.
School
Equity & PBIS

53. 5-point Multicomponent Approach to Reduce Disproportionality in Schools
Collect, Use, and Report Disaggregated Discipline Data
Effective Instruction
School-wide PBIS
Policies with Accountability for Disciplinary Equity
Teach Neutralizing Routines for Vulnerable Decision Points

54. Recommendations for Addressing Discipline Disproportionality
5-Point Intervention Approach to Enhance Equity in School Discipline
Collect, use, and report disaggregated discipline data
Implement a behavior framework that is preventative, multi-tiered, and culturally responsive
Use engaging academic instruction to reduce the opportunity (achievement) gap
Develop policies with accountability for disciplinary equity
Teach strategies to neutra

55. Point #2: Implement a Behavior Framework
Why start with a foundation of SWPBIS?
Proactive, instructional approach may prevent unwanted behavior and exposure to biased responses to unwanted behavior
Increasing positive student-teacher interactions may enhance relationships to prevent challenges
More objective referral and discipline procedures may reduce subjectivity and influence of cultural bias
Professional development may provide teachers with more instructional responses

57. Effects of PBIS on Discipline
Vincent et al., 2011: Statistically significant lower black- white ODR disproportionality in 72 schools implementing SWPBIS than in 81 schools not implementing SWPBIS
Vincent et al., 2009: Decreases in ODRs seen across racial/ethnic groups in 69 schools implementing SWPBIS
Scott, 2001: Larger decreases in suspensions for Black students when SWPBIS implemented
McIntosh et al., 2014: Sustained decrease in suspensions over eight years of SWPBIS implementation in an Indigenous school