1. Measuring Performance within School Climate Transformation Grants
Rob Horner
OSEP TA-Center on PBIS

2. Goal
Define resources for measuring performance within SCTG grants that may improve efficiency, consistency and effectiveness of evaluation efforts.
Provide a brief summary of schools using PBIS in the U.S.

3. Why a Framework for Multi-tiered Behavior Support?
The fundamental purpose of a Multi-tiered Behavior Framework is to make schools more effective and equitable learning environments.
Predictable
Positive
Consistent
Safe

4. All the parts Logic Model Intervention Strategies/Programs
Why the strategies work
Logic Model
Multi-tiered
Intervention Strategies/Programs
Build Capacity to Scale and Sustain
Implementation Process
Are we doing the strategy/program
Fidelity of Implementation
Student/ Family Outcomes
Valued outcomes/ Academic, Behavior, Social Emotional

5. Four ways the OSEP TA-Center may be helpful
Aligning multiple initiatives begins by defining common fidelity and outcome measures
Coordination and Communication
Webinars and information dissemination
Intervention Content
Tier I
Tier II
Tier III
Measuring Fidelity of Multi-tiered Behavior Support
Measuring Student Outcomes

6. Resources Grantees are held accountable for what they submitted.
But some consistency across projects may help all evaluation efforts
Resources that may be helpful:
Evaluation Blueprint (
Evaluation Annual Reports (
State and District Capacity Assessments
Multi-tiered Fidelity Assessment (Tier I, Tier II, Tier III)
Multi-tiered student discipline data systems
SWIS, CICO-SWIS, ISIS-SWIS

8. Performance Indicators: SEA
The number of training and technical assistance events provided by the SEA School Climate Transformation Grant Program to assist LEAs in implementing a multi-tiered behavioral framework.
The number and percentage of LEAs provided training or technical assistance by the SEA School Climate Transformation Grant Program that report an improvement in knowledge and understanding of the implementation of a multi-tiered behavioral framework.
The number and percentage of schools in LEAs provided training or technical assistance by the SEA School Climate Transformation Grant Program that implement a multi-tiered behavioral framework.

9. Performance Indicators: LEA
Number and percentage of schools that report an annual decrease in office disciplinary referrals.
Number and percentage of schools that report an annual improvement in the attendance rate.
Number and percentage of schools that report an annual decrease in suspensions and expulsions, including those related to possession or use of drugs or alcohol.
Number and percentage of schools annually that are implementing the multi-tiered behavioral framework with fidelity.

10. Performance Measure Considerations
Use a Formal Evaluation Model
Measure Fidelity at All Three Tiers
Consider Measuring Improved State/ District Capacity
Measure student outcomes within a formal “decision system”

11. Multi-Tiered Behavior Support
Evaluation Model: Evaluation Blueprint/
Multi-Tiered Behavior Support

12. Context Who is participating How selected Initial Assessment
Process for support plan development

13. Input Type and amount of technical assistance Training Coaching
Systems for Performance Feedback
Development of evaluation capacity
Development of Leadership Team Capacity

14. Fidelity
Are schools using Tier I practices and systems of behavior support?
Are schools using Tier II practices and systems of behavior support?
Are schools using Tier III practices and systems of behavior support?

15. Impact State: Capacity of State
Has the state established the capacity to (a) improve district capacity, (b) implement evidence-based, multi-tiered behavior support in schools, (c) monitor the fidelity of school-level implementation and (d) benefits for children and families?
District/Region: Capacity of LEA or Region
Has the district/region established the capacity to (a) implement evidence-based, multi-tiered behavior support in schools, and (b) monitor the fidelity of implementation in schools and (c) benefits for children and families?
Schools: Fidelity of Imp and Impact on Students
Is multi-tiered behavior support being implemented with fidelity in target schools?
Is there an effect of multi-tiered behavior support on student behavior as measured by office discipline referral (ODR) patterns?
Is there an effect of multi-tiered behavior support on student attendance?
Is there an effect of multi-tiered behavior support on student suspension and expulsion?
Is there an effect of multi-tiered behavior support on student bullying and violence?
Is there an effect of multi-tiered behavior support on juvenile arrests?
Is there an effect of multi-tiered behavior support on student academic performance?

16. Are Schools Implementing Multi-tiered Systems of Behavior Support?

17. Possible Measures of Multi-Tiered Behavior Support Fidelity
Measures (
Tiered Fidelity Inventory (TFI) Tier III
Individual Student System Evaluation Tool (ISSET)
Benchmark of Advanced Tiers (BAT)
Measure of Advanced Tiers Tool ( MATT)
Tiered Fidelity Inventory (TFI) Tier II
Tiered Fidelity Inventory (TFI) Tier I
School-wide Evaluation Tool (SET)
Team Implementation Checklist (TIC)
Benchmarks of Quality (BoQ)
Regardless of the Intervention approach used, TA-Center Fidelity measures are available and appropriate

18. TFI- Technical Adequacy (McIntosh, Massar et al. 2014)
Available from OSEP TA-Center
No Cost
Assessors Training PowerPoint available
Assessors Training Video in development
TFI- Technical Adequacy
(McIntosh, Massar et al. 2014)
Content Validity (.95; .93; .91)
Usability (12 of 14 > 80%) (15 min per Tier)
Inter-rater Agreement (.95; .96; .89)
Test-retest reliability (.98; .99; .99)
Factor Analysis (in process)

20. TFI Data Tier I: 2014

21. Tier II
Tier III

23. Action Planning

24. State/Region Capacity
Does a state have the capacity to implement multi-tiered systems of behavior support?
Is Technical Assistance improving the capacity of a state/region to implement multi-tiered systems of behavior support?

25. Possible Measures of SEA/LEA Capacity www.sisep.org
State Capacity Assessment (SCA)
Assessment of extent to which state department has capacity to implement evidence-based practices at scales of social significance?
No Cost, from OSEP’s SISEP TA-Center
Scores: Total, Sub-scale, Item
Used for initial assessment, action planning, progress monitoring
District Capacity Assessment (DCA)
Assessment of extent to which a school district (or region) has capacity to implement evidence-based practices as scales of social significance?
No Cost, from OSEP’s SISEP TA-Center
Implementers Training Video

26. SCA/ DCA Format (Team with Coach/ Assessor)
Item Data Sources Scoring Criteria Score

27. Minnesota
State

28. Minnesota
State

29. Minnesota
State

30. State SCA Item Analysis: Action Planning

32. District Capacity
Is Technical Assistance improving the capacity of a district to implement multi-tiered systems of support?

33. District Capacity

34. District Capacity Assessment

35. District Capacity Assessment
Data from Duda and Horner data system
District Capacity Assessment: Duda, et al. 2012
(c) Dean Fixsen and Karen Blase, 2013

36. District Capacity Assessment
Data from Duda and Horner data system
District Capacity Assessment: Duda, et al. 2012
(c) Dean Fixsen and Karen Blase, 2013

37. District Capacity Assessment

38. District Capacity Assessment

39. Measures of Student Benefit
Office Discipline Referrals
Tier I = SWIS
Tier II = CICO-SWIS
Tier III = ISIS-SWIS
School Climate
Under development
School Safety Survey
Risk Factor/ Protective Factor

40. Performance Monitoring Summary
Consider including measures of MTBSF fidelity at all three tiers.
Many Tier I options
Several Tier II options
One new measure that does all three (Tiered Fidelity Inventory (TFI)
Consider assessing SEA/ LEA “Capacity to implement evidence-based interventions”
State Capacity Assessment
District Capacity Assessment
Outcome measures
Office discipline referrals (SWIS, CICO-SWIS, ISIS-SWIS)… later
School Climate
School Safety Survey
Excellent Examples of State and District Reports at