1. Telling the Whole Story Alternative Accountability in Portland, Oregon Prepared for the Alternative Accountability Policy Forum November 2015

2. About us Christopher Mazzeo, Ph.D. Director of Evidence Use and Policy Education Northwest Director, REL Northwest Carla Gay, MSW Vice Principal, Reconnection Services Multiple Pathways to Graduation Portland Public Schools Matt Eide Senior Advisor Education Northwest

3. Goal Share Portland’s experience, and lessons learned on the development of multiple quantitative and qualitative accountability metrics for alternative schools.

4. Guiding Question How do you determine what quality looks like in an alternative program? Discuss this question with a neighbor

5. Just enough context…

6. The District Perspective Nagging question… How do we get accurate data to represent the work of our alternative schools and programs? But it was all below the line

7. The District Perspective Confluence of Events in 2012 brought the nagging question above the line... National Youth Taskforce Oregonian ArticleBudget Cuts

8. The District Perspective With the confluence of events, developing an accountability framework for alternative education options became an above the line priority…

9. The Alternative Schools’ Perspective Will the metrics tell the whole story? Can traditional metrics accurately determine quality programming and achievement in alternative settings? We don’t want your cookie cutter! Alternative programs in Portland are strong because of their diversity and specificity! But we believe in quality too.

10. The REL Perspective Help move the work forward An issue of equity An important issue in our region

11. What we did and what we learned…

12. The Portland Approach Quantitative Site VisitsQualitative

13. How did we do this? JFF pointed us in the direction of Denver and Chicago school districts Denver helped us map our process for a district wide segmentation analysis Chicago shared their accountability template We convened a group of stakeholders including policy, research and providers We convinced REL Northwest to facilitate...

15. Your turn… Use your dots to indicate which three metrics would be best to determine what quality looks like in an alternative program.

16. The Portland Framework Put it here METRIC DESCRIPTION Academic Progress SKILL GROWTH Percent of students who meet or exceed growth targets in Reading and Math on either MAP or CASAS CREDIT ATTAINMENT Percent of students who meet targets for the number of credits earned for length of enrollment or earn their maximum required credits while in school Successful Completion ONE YEAR HS GRADUATION RATE Percent of students eligible for graduation who graduate within one year COLLEGE READY GED ATTAINMENT RATE Percent of students who meet/exceed GED target composite score POSTSECONDARY READINESS Percent of students who meet/exceed target college readiness scores on COMPASS or ACT School Connection AVERAGE DAILY ATTENDANCEPercent of days attended by students enrolled at the school GROWTH IN ATTENDANCE Percent of students that show growth in their individual daily attendance rates compared to their individual daily attendance rate in the previous school year ANNUAL RETENTION RATE Percent of students enrolled at an alternative school and retained from the point of enrollment to the end of the year SCHOOL CLIMATETBD

17. The Report Card

18. What we learned Have a third party facilitate the process and analyze the data Make sure to attend to technical and leadership considerations Collaborative process Include state policymakers in the process Programs need access to real-time data

19. The Dashboard

20. What’s missing? Are these metrics sufficient to reliably determine alternative program quality? Discuss with your neighbor

21. Telling more of the story through qualitative data

22. Qualitative Process The Process Seven step iterative process: – needs sensing, instrument creation, instrument feedback collection, instrument revision, pilot testing, data analysis and reporting. Piloted focus groups and interviews with three alternative programs – Administrators, teachers, students, and parents

23. Qualitative Process The Instrument Focus group and interview questions aligned with PPS successful schools framework The Results Evaluation briefs highlighting program context

24. Qualitative Process Lessons Learned Challenging to schedule visits and interviews Aligning protocol with PPS Successful Schools Framework was useful but limited findings Student focus groups should include alumni and students who are struggling

25. Qualitative Process Lessons Learned The qualitative review process is labor intensive and time consuming Data triangulation and member check De-islandize alternative education Include as many stakeholders as possible

26. Questions?

27. Where to reach us Christopher Mazzeo Christopher.Mazzeo@educationnorthwest.org Carla Gay cgay@pps.net Matt Eide Matthew.Eide@educationnorthwest.org