1. Building Evaluation Capacity for States and Districts
Susan Barrett
Jerry Bloom
Sheppard Pratt Health System

2. Leadership Team Management Team
PBS Systems Implementation Logic
Marketing
Visibility
Funding
Braiding
Initiatives
Political Support
Leadership Team
Management Team
Active Coordination
Training
Coaching
Evaluation
Local School Teams/Demonstrations

3. Evaluation What are your questions? Do you have the tools to answer?
Can you get the answer quickly?
Easy, Efficient, Relevant
Economic Benefits
Behavior
Achievement
Regular Feedback to all Stakeholders- MD example

5. How many schools are implementing SW-PBS ?
Number by cohort
Number by local district
Number by implementation phase
Number of Active and Inactive by local district
Number of schools reporting data (rate of return)
Are schools implementing SW-PBS?
State and District needs to determine process tool (i.e. team implementation checklist, implementation inventory, benchmarks of quality, EBS self assessment)
Are schools in beginning or advances stages of implementation?
Implementation Phase Inventory used to track level
How do reported rates of problem behavior in local district or state compare to the national average? (SWIS)
by Elementary, Middle, High, Alternative
What is the percentage of students who receive 0 or 1 Office referrals/year? (SWIS)
By Elementary, Middle, High, Alternative
What are the problem behaviors? (SWIS)
Elementary, Middle, High, Alternative
Where are problem events occurring? (SWIS)
What time do problem events occur? (SWIS)
By Elementary, Middle , High, Alternative
What are the discipline patterns by gender? (SWIS)
What are the discipline patterns by ethnicity? (SWIS)
What are the discipline patterns for students on an IEP? (SWIS)

6. Evaluation Template
Fairfax County Example

7. Forms/Surveys and Checklists
Tier I
Surveying Staff- EBS Survey
Fidelity Checks- Team Implementation Checklist,
SET, BOQ, IPI
Safety Surveys- Oregon Safety Survey
Outcome Measures-
-ODR, Suspension, Attendance, Special Ed, GPA, Honor Roll, AYP

8. Forms /Checklists to TRAINING MATERIALS
EXAMPLE– MD Training Materials

9. Tools State Database- Data Entry, Report Generation Various Levels POC
Coach
State Team

10. Marketing and Visibility
Who are your stakeholders?
Do you have a spokesperson?
Using the data to create newsletters, presentations, fact sheets, elevator business cards- important you can get access to what you need to make your case on the fly!!
Maryland Newsletter
Be Careful

11. Multiple levels of Visibility
State and Local Level: Presentations, Trainings, Stakeholder meetings, Interagency efforts, (Transformation; Mental Health Integration; Wraparound)
Multiple Media: Visual, Face to Face, Written, Website
Multiple Audiences: School Administrations and Instructional Leaders; University staff; Legislators, Potential alternative funders; State and Local Political appointees; Juvenile Justice; Vendors in the System of Care; Parent and other advocacy organizations; Community Members

12. PBIS Maryland Partner Meeting
Management Team meets with Assistant Superintendent of Student Services and Special Education

13. Jerry

14. Jerry

15. 91% Rate of Return
Susan

16. Data Source: Implementation Phases Inventory
Evaluation Question: Are schools in beginning or advanced stages of implementation?
Data Source: Implementation Phases Inventory
Overall, the IPI data suggest a relatively advanced level of self-reported implementation among the schools in Maryland
Susan
Jerry, Can you include percentages? Thanks!

17. Evaluation Question: Are schools implementing School-wide PBIS
Evaluation Question: Are schools implementing School-wide PBIS? Data Source: Team Implementation Checklist (Target Criterion = 80%) Based on the total average of the 327 checklists submitted, school teams report that 78% of items are in place.
Susan
Jerry, can you add trend line?

18. Evaluation Question: Are schools implementing SW-Positive Behavior Support? Data Source: SET (Objective Criterion = 80%) -All regions met objective criterion -A pre-post comparison regional average shows a 69% increase.
Susan
Schools were able to establish 80% of the school-wide critical features within the first year of implementation.
222 schools have at least two SET scores
182 SETs were completed in 2007

19. Catherine-
Office Discipline Referrals are the “precursor” to suspension/expulsion. That is the data point that is analyzed most often in Universal PBIS—as the data can identify potential problem areas, students, teachers, times of day, etc. so that interventions can be put in place BEFORE it is a crisis.
Since much of this work is about developing positive relationships, let me share one anecdote about an administrative intervention to decrease ODR’s: One of our local facilitators was a PPW at a middle school. She shared with us that she watched one of her middle school principals establish a new practice. Before a teacher could refer a student to the office (removal from class) they had to be able to articulate 5 things about the student that are “non-academic” in nature. ODR’s dropped significantly and the message was clear—know our students!

20. Evaluation Question: Do 80% of the students in MD Elementary PBIS schools using SWIS receive 0 or 1 office referrals? Data Source: SWIS PBIS Elementary schools in Maryland using SWIS report that 92% of their students receive 0 or 1 office referral.
Susan
Growing the Green

21. Evaluation Question: Where is the location of behavior problems in Middle Schools? Data Source: SWIS The majority of the problem behaviors reported occur in the classroom followed by hallways.
Susan