Drilling Down in SWIS Data Using a Problem-Solving Model to Address Discipline Disproportionality Jennifer Rollenhagen, Eben McKnight, & Susannah Everett NEPBIS Leadership Forum 2017

Acknowledgements: Educational and Community Supports ECS is a research unit within the University of Oregon’s College of Education. ECS focuses on the development and implementation of practices that result in positive, durable, and scientifically substantiated change in the lives of individuals. ECS engages in federal and state funded projects focused on PBIS to support research, teaching, dissemination, and technical assistance. Thank you Bert Eliason & Katie Conley!

Organizing and Analyzing the Data Putting It Into Practice Session Intentions Goal Use a problem-solving model focused on the use of data to address discipline disproportionality Resources Data Guide for School Teams (pbis.org, 2014) SWIS Ethnicity Report (pbisapps.org) Four-step problem-solving model Organizing and Analyzing the Data Data Sources Common Metrics Drill Down Process Putting It Into Practice Simulated School Example

Intentions vs. Realities We truly believe that… Schools want good outcomes for all their students By and large, the profession is not purposely leaving anyone behind But unfortunately, the outcomes are not equal for everyone

Disproportionality:   Disproportionality in discipline referral data is the overrepresentation or underrepresentation of a subgroup of students based on their race or gender in comparison to the total student population. PBIS.org (September 2014)

Does Disproportionality Really Exist? Is it everywhere? Who does it affect? What can we do about it?

Disproportionality in School Discipline (Losen et al., 2015) http://civilrightsproject.ucla.edu/resources/projects/center-for-civil-rights-remedies/school-to-prison-folder/federal-reports/are-we-closing-the-school-discipline-gap

How do we measure this? What is necessary? What else is recommended? How do we get this information?

U.S. Department of Education Office for Civil Rights 3 Civil Rights Data Collection: Data Snapshot (School Discipline) March 21, 2014

Black students (16.4%) are 3 x more likely to be suspended or expelled than white students (4.6%). Black boys and girls have higher suspension rates than any of their peers. Twenty percent (20%) of black boys and more than 12% of black girls receive an out-of-school suspension. U.S. Department of Education Office for Civil Rights 3 Civil Rights Data Collection: Data Snapshot (School Discipline) March 21, 2014

U.S. Department of Education Office for Civil Rights 3 Civil Rights Data Collection: Data Snapshot (School Discipline) March 21, 2014

U.S. Department of Education Office for Civil Rights 3 Civil Rights Data Collection: Data Snapshot (School Discipline) March 21, 2014

States? State Districts Above Threshold Districts in the State % of Districts in State that would be Significantly Disproportionate CT 122 197 61.9% DE 19 43 44.2% MA 257 418 61.5% ME 59 259 22.8% NH 62 178 34.8% NJ 410 691 59.3% NY 491 1008 48.7% PA 338 738 45.8% RI 33 57 57.9% Nat’l. 8,148 17,371 46.9% Number and percent of districts with a Risk Ratio in at least one subgroup in at least one disability, ed. environment or discipline category that Exceeds Two MADs above the Nat’l median with a minimum cell size of 10 by state for all Three years (2011-12, 2012-13, 2013-14) Excludes districts in CO for any identification metrics, and all but one district in VT for Disciple removal metrics.

The Data Guide Identified: Revision coming soon! The Data Guide Identified: Data Needed for Investigating Disproportionality Four-step Problem Solving Model Metrics to Use to Monitor Disproportionality Bias in Decision Making – Explicit vs Implicit Vulnerable Decision Points

1. Problem Identification From the Data Guide Problem Solving Model 2. Problem Analysis 3. Plan Implementation 4. Plan Evaluation 1. Problem Identification Is there a problem? Why is it happening? You also need for your school’s PBIS team to have a problem-solving model… it doesn’t need to be overly complicated, this one has 4 basic steps: Is the plan working? What should be done?

Step 1: Problem Identification Use valid & reliable metrics. Quantify the difference between current outcomes and goals. This is the performance gap! 1. Problem Identification Is there a problem?

1. Problem Identification Step 1: Problem Identification For disproportionality Quantify outcomes across racial/ethnic subgroups Compare differences Specified group vs. Comparator group Subgroup vs. White subgroup Subgroup vs. All Other students Native vs. All Non-Native students 1. Problem Identification Is there a problem? Multiple metrics are recommended! IDEA Data Center, 2014

Risk Ratios A metric commonly used to measure disproportionality Represents the likelihood of discipline referrals for one group in relation to a comparison group Calculated by using the Risk Index of one group divided by the Risk Index of the comparison group Example threshold: If at or above 1.5, there is a risk for one group to be over represented for discipline referrals compared to another group

Automatically calculates the risk ratio to compare one group with all other groups

SWIS Ethnicity Reports - Automatically calculates the proportionality between subgroup’s percent of population and subgroup’s percent of students with referrals. overrepresentation underrepresentation

SWIS Ethnicity Reports - Automatically calculates the proportionality between subgroup’s percent of population and subgroup’s percent of total referrals. overrepresentation underrepresentation

Review Step 1: Problem Identification Select multiple metrics to use Relative measure – Use Risk Indices to calculate Risk Ratio Absolute measure – ODRs per 100 Students per Day Composite measurements – What is composition of the problem? Calculate metrics and compare to reasonable goals Previous years from same school Local, state or national norms Logical criteria – “4/5ths Rule” 0.80 - 1.25 State’s reasonable threshold for “significant disproportionality” Monitor metrics throughout the year Monthly or quarterly Be careful of small “Ns” Be careful of risk indices

Step 2: Problem Analysis 1. Problem Identification 2. Problem Analysis Is there a problem? What is happening and why? Once you have identified that there is a problem you can then analyze the problem to get more precise about the context and why it keeps occurring.

Step 2: Problem Analysis Purpose: Identify context and underlying cause(s) Focus: Systems & practices that can be changed Evaluate: Tier 1 (universal systems) Check fidelity of PBIS implementation TFI - CR Companion Disparities other than discipline Academic Placement Attendance School climate Graduation Discipline data for patterns of bias  Explicit bias  Implicit bias Research indicates that oftentimes bias is implicit rather than explicit. Implicit bias tends to be localized to specific situations (e.g., behaviors, locations).

What is implicit bias? Unconscious, automatic Generally not an indication of our beliefs and values We all have it (even those affected by it) Based on stereotypes More likely to influence: Snap decisions Decisions that are ambiguous

What is a Vulnerable Decision Point? Step 2: Problem Analysis What is a Vulnerable Decision Point? A VDP is a specific decision that is more vulnerable to effects of implicit bias. Two parts: Elements of the situation External or environmental context The person’s decision state Internal state or personal context

Two Systems for Decision Making Step 2: Problem Analysis Two Systems for Decision Making System 1: Fast Decisions Automatic, snap judgments Intuitive, unconscious System 2: Slow Decisions Deliberate decisions Allows for conscious attention Thinking, fast and slow – have you read it? I have not, but I heard it on NPR, so I think I’ve read it. 1: auto-pilot, gut reaction 2: simmer, sink in What are some decisions: ODR? Suspension? And 2 kinds of bias Kahneman, 2011

VDPs from national ODR data Step 2: Problem Analysis VDPs from national ODR data Subjective problem behavior Defiance, Disrespect, Disruption Vague discipline system Major vs. Minor Non-classroom areas Hallways Playgrounds Afternoons AMBIGUITY “Ambiguity is disproportionality’s best friend” LACK OF CONTACT FATIGUE / HUNGER

What can we do about these VDPs? Step 2: Problem Analysis What can we do about these VDPs? Teach two-step Neutralizing Routine for Staff: When you see problem behavior, stop and ask: Is this possibly a VDP? Consider the context of the situation Consider your personal decision state If so, use an agreed-upon alternative response Slow down, consider options

Neutralizing Routines for Reducing Effects of Implicit Bias Step 2: Problem Analysis Neutralizing Routines for Reducing Effects of Implicit Bias Self-assessment “Is this a vulnerable decision point?” Setting event Antecedent Behavior Consequence Lack of positive interactions with student Fatigue Loud complaints about work (subjective behavior) Send student to office (ODR) Student leaves class (Escape social interaction) Alternative Response “See me after class.”

Step 2: Problem Analysis Defining Problems with Precision Who is involved? What are the problem behaviors? Where is it happening? When is it happening? Why are these things happening? Perceived function of problem behavior

When we believe we have a problem related to disproportionality…. Step 2: Problem Analysis When we believe we have a problem related to disproportionality…. Assess PBIS implementation fidelity TFI-CR Companion Performance gaps Achievement gap Academic placement Attendance School climate Identified Subgroup Location Time of Day Problem Behavior Motivation Many seventh grade Black students are receiving referrals from the classroom in the afternoon for inappropriate language. Referrals are perceived to be task avoidance and getting adult attention.

Precise Problem Statement SWIS Drill Down When we have a problem with disproportionate discipline… Precise Problem Statement

Subgroup: African American Students Step 2: Problem Analysis SWIS Drill Down Subgroup: African American Students Who? When? What? Where? Why? 3rd grade 11:30 - Noon Physical Aggression Playground Peer Attention 4th grade 8:00 AM– 9:30 AM M-Defiance Classroom Avoid task 7th grade After 12:00 PM Inappropriate Language Classroom Hallway Obtain Peer Attention

Precise Problem Statements Step 2: Problem Analysis Precise Problem Statements African American students in the 3rd grade are receiving referrals for physical aggression during noon recess. Referrals seem to be related to gaining peer attention. African American students in the 4th grade are more likely to receive referrals for minor defiance in the classroom during the morning instructional block. Referrals seem to be related to task avoidance. African American students in the 7th grade are receiving afternoon referrals in the classroom and hallways for inappropriate language. Referrals are related to avoiding tasks and gaining peer attention.

Drill down

Step 3: Plan Implementation 1. Problem Identification 2. Problem Analysis Is there a problem? 3. Plan Implementation What is happening and why? What should be done?

Step 3: Plan Implementation Information from Step 2 is used to select tasks and strategies to address the problem. An action plan is created to ensure adequate implementation of the tasks and strategies. Action plans show everyone – WHO will do WHAT by WHEN. Action plans that are published – Help create accountability

Step 3: Plan Implementation Examples All issues Calculate and share disproportionality data regularly Inadequate PBIS implementation Implement core features of PBIS to establish a foundation of support Consider using TFI Culturally Responsive Companion Misunderstandings regarding school-wide expectations Enhance culturally responsive PBIS with input from the students/families Academic achievement gap Implement effective academic instruction

What is happening and why? Step 4: Plan Evaluation 1. Problem Identification 2. Problem Analysis Is there a problem? 4. Plan Evaluation 3. Plan Implementation What is happening and why? Is the plan working? What should be done?

Step 4: Plan Evaluation Evaluation Regularly assess Progress and fidelity of plan implementation Fidelity of PBIS implementation Calculate metrics from Step 1 Compare to the goal determined in Step 1 Share results with relevant stakeholders Plan for what is next

Step 4: Plan Evaluation Evaluation Time Frame: Identify time periods for regularly evaluating and analyzing disproportionality data. Caution: Disproportionality metrics may not be sensitive to rapid change. Consider monthly assessment of plan implementation & quarterly assessment of disproportionality metrics. Avoid relying on risk indices as they will increase throughout the year. Use multiple measures to ensure that you are tracking the correct thing.

1. Problem Identification Problem Solving Model 2. Problem Analysis 3. Plan Implementation 4. Plan Evaluation 1. Problem Identification Is there a problem? Why is it happening? Is the plan working? What should be done?

PBIS.org - National Resource Center is the national resource center for materials and information related to the implementation of PBIS with fidelity. School

PBIS.org - National Resource Center is the national resource center for materials and information related to the implementation of PBIS with fidelity. School Equity & PBIS

5-point Multicomponent Approach to Reduce Disproportionality in Schools Collect, Use, and Report Disaggregated Discipline Data Effective Instruction School-wide PBIS Policies with Accountability for Disciplinary Equity Teach Neutralizing Routines for Vulnerable Decision Points

Recommendations for Addressing Discipline Disproportionality 5-Point Intervention Approach to Enhance Equity in School Discipline Collect, use, and report disaggregated discipline data Implement a behavior framework that is preventative, multi-tiered, and culturally responsive Use engaging academic instruction to reduce the opportunity (achievement) gap Develop policies with accountability for disciplinary equity Teach strategies to neutra

Point #2: Implement a Behavior Framework Why start with a foundation of SWPBIS? Proactive, instructional approach may prevent unwanted behavior and exposure to biased responses to unwanted behavior Increasing positive student-teacher interactions may enhance relationships to prevent challenges More objective referral and discipline procedures may reduce subjectivity and influence of cultural bias Professional development may provide teachers with more instructional responses

Effects of PBIS on Discipline Vincent et al., 2011: Statistically significant lower black- white ODR disproportionality in 72 schools implementing SWPBIS than in 81 schools not implementing SWPBIS Vincent et al., 2009: Decreases in ODRs seen across racial/ethnic groups in 69 schools implementing SWPBIS Scott, 2001: Larger decreases in suspensions for Black students when SWPBIS implemented McIntosh et al., 2014: Sustained decrease in suspensions over eight years of SWPBIS implementation in an Indigenous school