Presentation on theme: "Building Evaluation Capacity for States and Districts"— Presentation transcript:
1 Building Evaluation Capacity for States and Districts Susan BarrettJerry BloomSheppard Pratt Health System
2 Leadership Team Management Team PBS Systems Implementation LogicMarketingVisibilityFundingBraidingInitiativesPolitical SupportLeadership TeamManagement TeamActive CoordinationTrainingCoachingEvaluationLocal School Teams/Demonstrations
3 Evaluation What are your questions? Do you have the tools to answer? Can you get the answer quickly?Easy, Efficient, RelevantEconomic BenefitsBehaviorAchievementRegular Feedback to all Stakeholders- MD example
5 How many schools are implementing SW-PBS ? Number by cohortNumber by local districtNumber by implementation phaseNumber of Active and Inactive by local districtNumber of schools reporting data (rate of return)Are schools implementing SW-PBS?State and District needs to determine process tool (i.e. team implementation checklist, implementation inventory, benchmarks of quality, EBS self assessment)Are schools in beginning or advances stages of implementation?Implementation Phase Inventory used to track levelHow do reported rates of problem behavior in local district or state compare to the national average? (SWIS)by Elementary, Middle, High, AlternativeWhat is the percentage of students who receive 0 or 1 Office referrals/year? (SWIS)By Elementary, Middle, High, AlternativeWhat are the problem behaviors? (SWIS)Elementary, Middle, High, AlternativeWhere are problem events occurring? (SWIS)What time do problem events occur? (SWIS)By Elementary, Middle , High, AlternativeWhat are the discipline patterns by gender? (SWIS)What are the discipline patterns by ethnicity? (SWIS)What are the discipline patterns for students on an IEP? (SWIS)
7 Forms/Surveys and Checklists Tier ISurveying Staff- EBS SurveyFidelity Checks- Team Implementation Checklist,SET, BOQ, IPISafety Surveys- Oregon Safety SurveyOutcome Measures--ODR, Suspension, Attendance, Special Ed, GPA, Honor Roll, AYP
8 Forms /Checklists to TRAINING MATERIALS EXAMPLE– MD Training Materials
9 Tools State Database- Data Entry, Report Generation Various Levels POC CoachState Team
10 Marketing and Visibility Who are your stakeholders?Do you have a spokesperson?Using the data to create newsletters, presentations, fact sheets, elevator business cards- important you can get access to what you need to make your case on the fly!!Maryland NewsletterBe Careful
11 Multiple levels of Visibility State and Local Level: Presentations, Trainings, Stakeholder meetings, Interagency efforts, (Transformation; Mental Health Integration; Wraparound)Multiple Media: Visual, Face to Face, Written, WebsiteMultiple Audiences: School Administrations and Instructional Leaders; University staff; Legislators, Potential alternative funders; State and Local Political appointees; Juvenile Justice; Vendors in the System of Care; Parent and other advocacy organizations; Community Members
12 PBIS Maryland Partner Meeting Management Team meets with Assistant Superintendent of Student Services and Special Education
16 Data Source: Implementation Phases Inventory Evaluation Question: Are schools in beginning or advanced stages of implementation?Data Source: Implementation Phases InventoryOverall, the IPI data suggest a relatively advanced level of self-reported implementation among the schools in MarylandSusanJerry, Can you include percentages? Thanks!
17 Evaluation Question: Are schools implementing School-wide PBIS Evaluation Question: Are schools implementing School-wide PBIS? Data Source: Team Implementation Checklist (Target Criterion = 80%) Based on the total average of the 327 checklists submitted, school teams report that 78% of items are in place.SusanJerry, can you add trend line?
18 Evaluation Question: Are schools implementing SW-Positive Behavior Support? Data Source: SET (Objective Criterion = 80%) -All regions met objective criterion -A pre-post comparison regional average shows a 69% increase.SusanSchools were able to establish 80% of the school-wide critical features within the first year of implementation.222 schools have at least two SET scores182 SETs were completed in 2007
19 Catherine-Office Discipline Referrals are the “precursor” to suspension/expulsion. That is the data point that is analyzed most often in Universal PBIS—as the data can identify potential problem areas, students, teachers, times of day, etc. so that interventions can be put in place BEFORE it is a crisis.Since much of this work is about developing positive relationships, let me share one anecdote about an administrative intervention to decrease ODR’s: One of our local facilitators was a PPW at a middle school. She shared with us that she watched one of her middle school principals establish a new practice. Before a teacher could refer a student to the office (removal from class) they had to be able to articulate 5 things about the student that are “non-academic” in nature. ODR’s dropped significantly and the message was clear—know our students!
20 Evaluation Question: Do 80% of the students in MD Elementary PBIS schools using SWIS receive 0 or 1 office referrals? Data Source: SWIS PBIS Elementary schools in Maryland using SWIS report that 92% of their students receive 0 or 1 office referral.SusanGrowing the Green
21 Evaluation Question: Where is the location of behavior problems in Middle Schools? Data Source: SWIS The majority of the problem behaviors reported occur in the classroom followed by hallways.Susan