1. Evaluation Planning & Reporting for School Climate Transformation Grant (SCTG) Sites Bob Algozzine University of North Carolina at Charlotte Steve GoodmanMichigan's Integrated Behavior and Learning Support Initiative Toni CloverDetroit Public Schools Sheila Williams-WhiteFrederick Douglass Academy PBIS National Leadership Forum Breakout Session A1 October 22, 2015 10:00 - 11:15 Donald E. Stephens Convention Center Rosemont, Illinois

2. Goals/Assumptions  Goals: Participants will be able to…  define core features of a program evaluation.  define core evaluation questions.  define the core features of an evaluation report.  Assumptions: Participants already…  can define core features of SCTG project.  are implementing core features of SCTG.  are responsible for documenting progress and products to support decision making.

3. Take Home Questions  Where are we in our implementation?  What did I hope to learn?  What did I learn?  What will I do with what I learned?

4. Take Home Message Consistent with the project’s goals, objectives, and logic model, the evaluation is focused on documenting what was planned to bring about change (inputs and outputs) as well as what was accomplished (formative outcomes) and the difference it made (summative impact). Core Features of a Program Evaluation

5. Identify Questions to be Answered Select Measures and Gather Data to Answer Questions Analyze Data and Answer Questions Plan/Define/Do/Document Identify Purpose, Team, and Timeline Core Features of a Program Evaluation

6. Core Evaluation Questions [Inputs and Outputs] Context  What are/were goals and objectives for the project?  State/district capacity/school adoption?  Student, family, and community outcomes?  Who are primary and secondary stakeholders? Input  What technical assistance was part of implementation?  Was projected level of TA capacity provided (training/coaching)?  Who delivered the training and technical assistance?  Who received technical assistance (schools/ cohorts)?  How many schools/districts/regional centers?

7. Formative …documenting the technical assistance needed to move a school from “ready to implement” core features to “actively implementing” evidence-based interventions and practices to improve outcomes for students… Key Evaluation Concepts Summative …documenting the quantifiable results of our work and the difference we made for students, schools, districts, and states… What difference we made. What we did to bring about change. Intervention Fidelity Was intervention technical assistance provided as intended? Were interventionists properly trained? What they did with what we did. Implementation Fidelity Was intervention implemented as intended? Were interventionists properly practicing? What we accomplished.

8. Core Evaluation Questions [Formative Outcomes] Fidelity  To what extent are we providing technical assistance with fidelity?  To what extent are schools implementing interventions with fidelity?  To what extent are participants satisfied?

9. Core Evaluation Questions [Summative Impact] Impact  To what extent is leadership and policy structure established?  To what extent is project associated with changes in academic outcomes?  To what extent is project associated with changes in behavior outcomes?  To what extent is project associated with changes in other outcomes?

10. Core Evaluation Questions Replication, Sustainability, and Improvement  To what extent is district/state capacity (local training, coaching, evaluation, behavioral expertise) established?  To what extent do outcomes sustain across time?  To what extent does initial implementation affect implementation with later cohorts?  To what extent did implementation change educational/behavioral capacity/policy?  To what extent did implementation affect systemic educational practice?  To what extent are modifications needed?

11. Core Features of an Evaluation Report  Global Outline – Context, Input, Fidelity, Impact, Future Directions  Local Adaptation – Cohorts – Focused Questions Disproportionality Bully Prevention Cost – Program Implementation Changes Increased emphasis on Tier II and Tier III Increased emphasis on High Schools  Examples – www.pbis.org/evaluation www.pbis.org/evaluation – http://www.dpi.state.nc.us/positivebehavior/data/evaluation/ http://www.dpi.state.nc.us/positivebehavior/data/evaluation/ – http://pbismissouri.org http://pbismissouri.org

12. Context [What We Are] Think Globally The School Climate Transformation Grant Program provides local education agencies with assistance to develop, enhance, or expand systems of support for, and technical assistance to, schools implementing an evidence-based multi- tiered behavioral framework for improving outcomes and learning conditions for all students. Act Locally – Document evidence of critical features of implementation. – Document characteristics of schools participating in implementation. – Document evidence of development of state, district, and local capacity. – Document answers to key stakeholder questions about implementation and outcomes related to it. Core Features of an Evaluation Report

13. Eighty-four of the 100 counties in the state have at least one school participating in the North Carolina Positive Behavior Support Initiative. ContextReporting Evaluation Findings

14. Input [What We Are Doing] Think Globally SCTG Project is grounded on critical features that guide implementation. Act Locally –Document who, what, where, when, how, and why for support (i.e., program) that was provided. o State, District, and School Leadership Teams o Local Capacity Building – Training – Coaching – Evaluation –Document use of data for decision making. –Document quality, usefulness, and relevance of support. Core Features of an Evaluation Report

15. Ratings of the Summer Institute training and trainers were generally high (i.e., above 3.0 on the 4-point Strongly Disagree-Strongly Agree scale. InputReporting Evaluation Findings

16. Fidelity [How We Are Doing] Think Globally Evidence continues to indicate that implementation fidelity is the critical predictor of program success. Act Locally –Document levels of implementation. o Self-Assessment o TFI o SET o BoQ –Document skill development across schools, teams, and participants. –Document skill development over time. Core Features of an Evaluation Report

17. Leadership teams are actively using information from self- assessments to develop and adjust action plans for implementation. Current levels of school-wide, non- classroom, classroom and individual support systems were better in schools that were implementing more than one year (Cohort A) than in schools implementing on one year (Cohort B). Reporting Evaluation FindingsFidelity

18. Impact [Difference we are making] Think Globally The SCTG Project provides a framework for establishing a school-wide culture with the supports needed for all students to achieve academic and social success. Act Locally –Document extent to which SCTG implementation influenced social and academic behavior. –Document extent to which outcomes are related to implementation fidelity. Core Features of an Evaluation Report

19. Averages rates of out-of-school suspension (OSS) per 100 students remained similar between 2013- 14 and 2014-15 for the state and for all schools participating in the project; and, they decreased for project exemplar schools. Reporting Evaluation FindingsImpact

20. Implications and Future Directions – Policy – Practice – Technical Assistance Recommendations Resources and Other Information Core Features of an Evaluation Report

21. Resources and Other Information Features of an Effective Evaluation Report Purpose Context What are domains of influence and core features? What are implementation goals? Input Who is implementing program? What, where, when, and why is program being implemented? Fidelity Was technical assistance intervention provided as intended? [Were people trained?] Was technical assistance intervention implemented as intended? [Did they implement the training?] Impact Is program resulting in intended changes and benefits? Are improvements needed? Are systems and outcomes sustainable? Implications and Future Directions Recommendations Resources and Other Information Core Features of an Evaluation Report

22. Resources and Other Information The OSEP Technical Assistance Center on Positive Behavioral Interventions and Supports has developed a document for individuals who are implementing School-wide Positive Behavior Intervention and Support (SWPBIS) in districts, regions, or states. The purpose of the “blueprint” is to provide a formal structure for evaluating if implementation efforts are (a) occurring as planned, (b) resulting in change in schools, and (c) producing improvement in student outcomes.OSEP Technical Assistance Center on Positive Behavioral Interventions and Supports North Carolina PBIS Web Site http://www.dpi.state.nc.us/positivebehavior/ North Carolina Annual Performance Reports Available http://www.dpi.state.nc.us/positivebehavior/data/evaluation/ Illinois Evaluation Reports http://pbisillinois.org/ Florida Evaluation System http://flpbs.fmhi.usf.edu/ Childs, K. E., Kincaid, D., & George, H. P. (2010). A model for statewide evaluation of a universal positive behavior support initiative. Journal of Positive Behavior Interventions, 12, 198–210. Missouri Schoolwide Positive Behavior Support http://pbismissouri.org/ PBIS Maryland http://www.pbismaryland.org/ Core Features of an Evaluation Report

23. Moving on…  Where are you in your implementation?  What did you learn?  Did you learn what you hoped to learn?  What will you do with what you learned? Taillights and Dust…