1. Bob Algozzine Rob Horner National PBIS Leadership Forum Chicago Hyatt Regency O’Hare October 8, 2009 http://www.dpi.state.nc.us/positivebehavior / http://www.dpi.state.nc.us/positivebehavior / IMPLEMENTING AN EVALUATION SYSTEM FOR SWPBS

2. Objectives  To provide brief overview of evaluation field  To describe NC PBS evaluation model  To illustrate North Carolina’s evaluation efforts o Context o Input o Fidelity o Impact

3. Evaluation  Collecting and Reporting Information  Who  Where  When  Why  What  How  “Make a Case”  What do we know?  What do we need to know?  How good is acceptable?  How bad is unacceptable?

4. GoalInformationAction ContextDefining [Who, Where, When, Why] InputObtaining [What] ProcessApplying [How] ProductAnalyzing [What difference did it make?] Evaluation Collecting and Reporting Information for Decision-Makers Describe Judge Stufflebeam, D. L., & Shinkfield, A. J. (2007). Evaluation theory, models, and applications. San Francisco, CA: Jossey Bass.

5. Evaluation Collection and Reporting of Information for Decision-Makers Defining Obtaining Applying Analyzing Project Context Input Process Product What How Who Where When Why What difference did it make? Are we doing it as planned? Outcome What should we do? How should we do it?

6. SWPBS/NC Evaluation Collection and Reporting of Information for Decision-Makers Plan Perform Measure Compare School-Wide Positive Behavior Support Context Input Fidelity Impact What How Who Where When Why What difference did it make?

7. NC PBS Evaluation Collection and Reporting of Information for Decision-Makers The North Carolina Positive Behavior Support Initiative is part of the North Carolina State Improvement Program funded through IDEA. VISION [IMPACT] All schools in North Carolina will implement Positive Behavior Support as an effective and proactive process for improving social competence and academic achievement for all students. MISSION [CONTEXT, INPUT, FIDELITY] Provide leadership, professional development, resources, and on-going assistance in order for schools to successfully implement Positive Behavior Support.

8. NC PBS Evaluation Collection and Reporting of Information for Decision-Makers Context Districts and schools participating in the North Carolina Positive Behavior Support Initiative regularly document the goals, objectives, and activities of their programs to establish the extent to which systems and practices that should be available are available.

9. NC PBS Evaluation Collection and Reporting of Information for Decision-Makers Context Who Where When Why STATEWIDE LEADERSHIP AND COORDINATION One full-time Consultant and eight part-time Regional Coordinators support implementation of PBS in NC. The Consultant is in the Behavior Support and Special Programs Section of the Exceptional Children Division at the NC Department of Public Instruction. The Regional Coordinators are hosted by LEAs or another agency in their region and spend 1/3 of their time working with the PBS implementation in that school system or Charter School. The primary role of these professionals is to coordinate training, support trainers/coaches/coordinators in LEAs, and facilitate the evaluation of the statewide initiative. http://www.ncpublicschools.org/positivebehavior/coordinator/

10. NC PBS Evaluation Collection and Reporting of Information for Decision-Makers Context Who Where When Why Eighty-four of the 100 counties in the state have at least one school participating in the North Carolina Positive Behavior Support Initiative.

11. NC PBS Evaluation Collection and Reporting of Information for Decision-Makers Context Who Where When Why Currently, 104 (90%) of the 115 counties and local education agencies in the state have at least one school participating in the North Carolina Positive Behavior Support Initiative.

12. NC PBS Evaluation Collection and Reporting of Information for Decision-Makers Context Who Where When Why Steady growth has been evident in the number of schools that have implemented PBS and current estimates suggest that about 85% are still implementing. TypeECH/PreschoolElemMid/Jr.HighK (8-12)Alt./Ctr.NC Total n3488175651234777 Percent of NC Total0.39%62.81%22.52%8.37%1.54%4.38%100.00%

13. NC PBS Evaluation Collection and Reporting of Information for Decision-Makers Context Who Where When Why

14. NC PBS Evaluation Collection and Reporting of Information for Decision-Makers Input The North Carolina Positive Behavior Support Initiative includes critical features of SWPBS. Districts and schools participating in the North Carolina Positive Behavior Support Initiative regularly assess the extent to which key features are being implemented and they use this information to develop and refine action plans for sustaining the effort.

15. NC PBS Evaluation Collection and Reporting of Information for Decision-Makers The North Carolina Positive Behavior Support Initiative includes critical features of SWPBS. Leadership TeamRepresentative stakeholders meets regularly to support effort. CoordinationState and local coordinators manage day-to-day operations. FundingImplementation supported with targeted state-approved funds. VisibilityOngoing local, state, and national dissemination efforts. Political SupportLeadership team reports annually on activities and outcomes. Training CapacityTeam of trainers is available to build and sustain practices. Coaching CapacityIndividuals are available to support emerging practices. DemonstrationsLarge number of schools available for model demonstrations. EvaluationProcess in place for collecting and reporting information on  Implementation of action plan.  Use of SWPBS.  Impact of SWPBS on student outcomes. Input What http://www.dpi.state.nc.us/docs/positivebehavior/implementation/brochure/brochure.pdf

16. NC PBS Evaluation Collection and Reporting of Information for Decision-Makers The North Carolina Positive Behavior Support Initiative focuses on four key elements and schools participating in the program intervene at three distinct, but related levels.  Universal Interventions  School-wide Rules & Procedures  Systematic Reinforcement  Social Skills Instruction  Culturally Responsive Practices  Data-Based Decision-making  Parent & Community Partnerships  Targeted Group Interventions  Social Skills Instruction  Reinforcement of Specific Skills  Group Behavior Strategies  Classroom Coaching  Intensive, Individual Interventions  Functional Behavior Assessment  Behavior Intervention Planning Input What http://www.dpi.state.nc.us/docs/positivebehavior/implementation/brochure/brochure.pdf

17. NC PBS Evaluation Collection and Reporting of Information for Decision-Makers Schools implementing PBS regularly assess the extent to which key features of PBS are being implemented and they use this information to develop action plans for refining and sustaining the effort. Input What

18. NC PBS Evaluation Collection and Reporting of Information for Decision-Makers Fidelity Schools participating in the North Carolina Positive Behavior Support Initiative regularly document the implementation levels of the support being provided and they use this information to refresh, redirect, and sustain their efforts. “Outcome evaluations should not be attempted until well after quality and participation have been maximized and documented in a process evaluation. Although outcome data can determine the effectiveness of a program, process data determine whether a program exists in the first place” (Gilliam, Ripple, Zigler, &Leiter, 2000, p. 56). Gilliam, W. S., Ripple, C. H., Zigler, E. F., & Leiter, V. (2000). Evaluating child and family demonstration initiatives: Lessons from the Comprehensive Child Development Program. Early Childhood Research Quarterly,15(1), 41-59.

19. NC PBS Evaluation Collection and Reporting of Information for Decision-Makers Average [08-09] scores on SET subscales varied across levels of schools (N=336) and were generally higher for elementary (n=226) than middle (n=68), high (n=24), or combined (n=18) enrollment schools. Fidelity How

20. NC PBS Evaluation Collection and Reporting of Information for Decision-Makers Fidelity How More schools reported SET scores and they were generally higher for 2008-2009 than for previous school years.

21. NC PBS Evaluation Collection and Reporting of Information for Decision-Makers Impact The North Carolina Positive Behavior Support Initiative is focused on improving social competence and academic achievement for all students. Information from impact evaluation indicators reflects the extent to which targeted outcomes are being and/or likely to be achieved.

22. NC PBS Evaluation Collection and Reporting of Information for Decision-Makers Office discipline referral data (majors) from schools implementing PBS in North Carolina [07-08] compare favorably with national averages. Impact What difference did it make?

23. NC PBS Evaluation Collection and Reporting of Information for Decision-Makers Impact What difference did it make? There has been a consistent decrease in reported suspensions across schools implementing PBS over the past five years.

24. NC PBS Evaluation Collection and Reporting of Information for Decision-Makers Impact What difference did it make? Levels of behavior risk in schools implementing PBS were comparable to widely-accepted expectations and better than those in comparison schools not systematically implementing PBS.

25. NC PBS Evaluation Collection and Reporting of Information for Decision-Makers Impact What difference did it make? Rates of office discipline referrals in schools that met SET expectations were lower than in schools that did not meet them.

26. NC PBS Evaluation Collection and Reporting of Information for Decision-Makers Impact What difference did it make? While achievement differences were not statistically significant across schools with different levels of implementation, effect sizes reflecting practical differences were large (.55 for reading and.98 for mathematics).

27. NC PBS Evaluation Collection and Reporting of Information for Decision-Makers Impact What difference did it make? [A]chievement causes [B]ehavior? [B]ehavior causes [A]chievement? [C]ontext causes [A]chievement and [B]ehavior?

28. NC PBS Evaluation Collection and Reporting of Information for Decision-Makers Sustainability  Continuing to Build Capacity o state o district o school  Expanding Role of Local Coaches  Expanding Efforts to Make Evaluation Easier o report and presentation templates o external support o www. pbsurveys.org o www. pbseval.org o www. pbsassessment.org

29. Web site http://www.dpi.state.nc.us/positivebehavior/ Annual Performance Reports Available http://www.dpi.state.nc.us/positivebehavior/data/evaluation/ NC PBS Evaluation Collection and Reporting of Information for Decision-Makers