1. Using Community and Technical College Student Achievement Data for State- Level Accountability Purposes in Washington State PNAIRP 2010 Conference Deb Stephens Washington State Board for Community and Technical Colleges

2. Government Management Accountability and Performance Governor implemented in 2005 – first in nation to apply model to state government performance Data-based management model to improve the results of statewide programs and services Requires state agencies to develop and use performance data to set targets and inform decisions What is GMAP?

3. Governor can hold state government accountable Public forums are held quarterly and are organized around priorities, including education GMAP supports: –Transparency to the Public (Annual Report to Citizens) –Evaluations of Program Effectiveness –Decisions about Operational Priorities and Resource Allocation What is GMAP? (cont’d)

4. Performance funding system for WA’s community and technical colleges –Improves public accountability –Rewards colleges for increasing student achievement Rewards are based on key academic benchmarks that improve students’ chances of college completion Shift from funding only inputs (enrollments) to also funding outcomes (student progress) Washington’s Student Achievement Initiative s

5. 4 Categories of Achievement measures –Building toward college level skills (basic skills, pre-college) –First year retention (first 15 and 30 college-level credits) –Completing college math –Completions (degrees, certificates, apprenticeship training) Measures are meaningful for all students –Across demographics (race, age, income, employment) –Across missions (academic, basic skills, workforce, remedial) –Regardless of enrollment patterns (full or part time) –Across types of institutions (rural, urban, community or technical) Student Achievement Initiative (cont’d)

6. 2006-07: first Baseline 2007-08: Learning year 2008-09: first Performance year with financial rewards to colleges (Rewards distributed Oct 2009) Evaluation is underway by the Community College Research Center (CCRC) at Columbia University and the Institute for Higher Education, Leadership and Policy at California State University Student Achievement Initiative (cont’d)

7. Measures are simple, understandable, and relevant to citizens, colleges, students, and policy makers Measures focus on shorter-term, intermediate outcomes that provide meaningful momentum towards college completion Measures can be influenced by colleges on a timely basis Student achievement successes at individual colleges provide evidence to support statewide application of effective strategies Why Student Achievement in GMAP?

8. Student Achievement in a Logic Model Context

9. All GMAP Student Achievement slides (as the Governor sees them) provided at the end of the presentation The slides are very data-rich….Pre-College slide shown as an example Information in slides allows Governor and Staff to understand trends, issues and what the college system is doing to improve performance GMAP Student Achievement Presentation Slides

10. 1.Graph shows total points for all students in each point category over 4 years shown on a graph (shows scale) 2.Chart shows change in relevant achievement points from baseline to first performance year for a significant subset of students (shows efficiency) 3.Summary Analysis describes trends and provides insight into strengths and challenges 4.Action Plan maps system activities and research surrounding key issues GMAP Student Achievement Slide Elements

11. Are underprepared college students succeeding in pre- college and college-level work? Pre-College Math Students’ Math Achievement Growth Over Baseline Measure: Number of pre-college math and English courses students pass with a qualifying grade to advance toward college-level work Source: SBCTC Data Warehouse Link to Agency Plan: http://www.sbctc.ctc.edu/docs/sbctc_system_direction_final.pdf Summary Analysis Nearly half (48%) of high school graduates who come to community and technical colleges directly from high school need math remediation. Colleges are working hard to get students through pre-college math quicker and ensure they successfully complete college- level math. There is much work to be done to increase math success. The graph to the left shows the number of pre-college gains completed by all students, and the table shows math achievement growth by those students who attempted pre- college math. Gains in pre-college math are keeping pace with the increase in the number of students (14% growth in each over the baseline); however, students are still behind in moving to college-level math. Through the Transition Math Project, Achieving the Dream and Gates funded work, colleges are evaluating and piloting ways to move more students into college math faster. Action Plan Provide financial rewards to colleges each year for increasing the number of pre-college courses completed with a qualifying grade. The new Re-Thinking Pre-College Math Initiative, funded by Gates Foundation grant, will support 7 colleges to improve curriculum, instructional practices and assessment approaches in pre-college math programs. Through Gates Foundation funding, expand I-BEST to 10 pilot colleges to integrate below college-level courses with college- level courses.

12. GMAP Slide Elements: The Main Graph Main graph shows trend in all college students who completed pre-college (both English and Math) courses over 4 years

13. Chart shows achievement gains of students who attempted pre-college math from baseline to first performance year Governor can see that pre-college math is keeping pace with increases in students but only half are making it through college-level math GMAP Slide Elements: The Chart Pre-College Math Students’ Math Achievement Growth Over Baseline

14. GMAP Slide Elements: Summary Analysis Nearly half (48%) of high school graduates who come to community and technical colleges directly from high school need math remediation. Colleges are working hard to get students through pre-college math quicker and ensure they successfully complete college-level math. There is much work to be done to increase math success. The graph to the left shows the number of pre-college gains completed by all students, and the table shows math achievement growth by those students who attempted pre-college math. Gains in pre-college math are keeping pace with the increase in the number of students (14% growth in each over the baseline); however, students are still behind in moving to college-level math. Through the Transition Math Project, Achieving the Dream and Gates funded work, colleges are evaluating and piloting ways to move more students into college math faster.

15. Provide financial rewards to colleges each year for increasing the number of pre- college courses completed with a qualifying grade. The new Re-Thinking Pre-College Math Initiative, funded by Gates Foundation grant, will support 7 colleges to improve curriculum, instructional practices and assessment approaches in pre-college math programs. Through Gates Foundation funding, expand I-BEST to 10 pilot colleges to integrate below college-level courses with college-level courses. GMAP Slide Elements: Action Plan

16. Total Student Achievement Points increased by 19% in 2008-09 over the baseline year. Basic skills points increased 36% over the baseline, improving achievement in basic skills and beyond. The number of basic skills students reaching the tipping point nearly quadrupled. Three quarters of students’ success in reaching the tipping point can be attributed to I-BEST programs. Highlights of Other Student Achievement Slides

17. Students starting the year with some (but less than 30) college credits grew primarily in the tipping point milestone. Workforce students generated most of this growth through short certificate growth in students with a year of college. While the percent of transfer students completing college math in 2008-09 over the baseline just kept pace with the increase in the number of students, pre-college math points increased at a faster rate. Highlights (cont’d)

18. Measures are real-time, process measures for public sector entity, something not historically available in education Highlights system challenges Governor can ask for and expect action items and quarterly updates Allows Governor and staff to see what the college system is doing NOW Generates timely information to support operational decisions and strategic planning Value of Information to Governor and Staff

19. Held up as a model performance funding mechanism Governor has encouraged other entities to consider a similar performance system, in education and elsewhere Governor immediately clued in on math Governor responded favorably to I-BEST Responses by Governor and Key Staff

20. Promotes credibility, particularly important in the current budget climate Many government agencies do not possess the same quality of data, so the community and technical college system stands out in a very positive way Other Considerations

21. Links and All GMAP Slides GMAP Website http://www.accountability.wa.gov Student Achievement Website http://www.sbctc.edu/college/e_studentachievement.aspx

22. Are two-year college students making academic progress? 2008-09 Change in Achievement Points Over Baseline Measure: Number of key academic benchmarks (achievement points) that students meet on the path to successfully completing degrees and certificates. Source: SBCTC Data Warehouse Link to Agency Plan: http://www.sbctc.ctc.edu/docs/sbctc_system_direction_final.pdf Summary Analysis The Student Achievement Initiative is a new performance funding system for community and technical colleges aimed at improving public accountability and providing financial rewards to colleges for increasing the levels of achievement attained by their students. Students come to colleges with varying levels of college readiness. The initiative rewards colleges when students pass through critical academic milestones: getting ready for college, succeeding in the first year of college, completing college math, and reaching the tipping point, a certificate with at least a year of college or a degree. SBCTC and national research demonstrates that college students experience a substantial boost in earnings gain when they reach the tipping point or beyond. The first performance awards in October totaled $1.8 million--$1 million from the Legislature and $0.8 million from the Gates and Ford Foundations. Compared to the 2006-07 baseline, Student Achievement Points increased by 19 percent overall in 2008-09, the first year colleges received a portion of their funding based on performance. ALL colleges increased performance and gained points as students are going further in college success. Action Plan Provide financial rewards to colleges each year for increasing the number of significant gains in critical academic milestones. Colleges will identify and share strategies that are working well to increase student achievement in an effort to foster continuous improvement.

23. Are adult literacy students increasing their achievement in basic skills and beyond? Measure: Number of times adult literacy students make nationally recognized standardized test gains in math or English as measured by pre- and post-testing or by earning a GED Source: SBCTC Data Warehouse Link to Agency Plan: http://www.sbctc.ctc.edu/docs/sbctc_system_direction_final.pdf Basic Skills Students’ Achievement Growth Over Baseline Summary Analysis The graph and table show gains made by those students who took basic skills courses, regardless of their purpose in college. Basic skills students come to college the furthest below college level readiness. Colleges have made substantial progress increasing basic skills students’ achievement. In 2008-09, 56,218 students earned a total of 100,000 points, 36% over the baseline. Basic skills students increased achievement in basic skills as well as beyond basic skills—the number of basic skills students reaching the tipping point nearly quadrupled. Three quarters of students’ success in reaching the tipping point can be attributed to I-BEST programs. Action Plan Provide financial rewards to colleges each year for increasing the number of significant gains made by ABE students on their way to college readiness. Expand programs that integrate basic skills education with workforce education (I-BEST) Provide training to colleges on using data to better understand factors that contribute to adult literacy students’ success. Provide training for new faculty on administration of the standardized test (CASAS) used to evaluate adult literacy student progress.

24. Are underprepared college students succeeding in pre- college and college-level work? Pre-College Math Students’ Math Achievement Growth Over Baseline Measure: Number of pre-college math and English courses students pass with a qualifying grade to advance toward college-level work Source: SBCTC Data Warehouse Link to Agency Plan: http://www.sbctc.ctc.edu/docs/sbctc_system_direction_final.pdf Summary Analysis Nearly half (48%) of high school graduates who come to community and technical colleges directly from high school need math remediation. Colleges are working hard to get students through pre-college math quicker and ensure they successfully complete college- level math. There is much work to be done to increase math success. The graph to the left shows the number of pre-college gains completed by all students, and the table shows math achievement growth by those students who attempted pre- college math. Gains in pre-college math are keeping pace with the increase in the number of students (14% growth in each over the baseline); however, students are still behind in moving to college-level math. Through the Transition Math Project, Achieving the Dream and Gates funded work, colleges are evaluating and piloting ways to move more students into college math faster. Action Plan Provide financial rewards to colleges each year for increasing the number of pre-college courses completed with a qualifying grade. The new Re-Thinking Pre-College Math Initiative, funded by Gates Foundation grant, will support 7 colleges to improve curriculum, instructional practices and assessment approaches in pre-college math programs. Through Gates Foundation funding, expand I-BEST to 10 pilot colleges to integrate below college-level courses with college- level courses.

25. Are college students increasing their achievement in the critical first year of college? Achievement Growth Over Baseline of Students Who Started the Year with at Least 15 But Less Than 30 College-Level Credits Measure: Number of students who complete the first 15 and 30 college-level credits Source: SBCTC Data Warehouse Link to Agency Plan: http://www.sbctc.ctc.edu/docs/sbctc_system_direction_final.pdf Summary Analysis The number of students reaching the first 15 and 30 college-level credits increased by 16% and 15% over baseline, respectively, as shown in the graph to the left. The achievement of a subset of those students, those who start the year with at least 15 but less than 30 credits, is shown in the table to the left. Those students are getting further in achievement in the performance year (2008-09) over the baseline, as total points increased 13% compared to 6% growth in the number of students. The tipping point was the fastest growing milestone for those students. Workforce students generated most of this growth as colleges added short certificates that students with 45 credits combined to earn a point in the tipping point category. These students will need to complete college- level math if they are to advance further. Action Plan Provide financial rewards to colleges each year for increasing the number of college students who reach the first 15 and first 30 college-level credits. Expand promising practices such as integrating achievement points into tracking and advising. Engage in global consortia to provide tutoring and virtual library services to all students around the clock to help them succeed in college.

26. Are college students increasing their achievement in college-level math? Math Achievement Growth Over Baseline of Transfer Students Starting the Year With No College Math Measure: Number of students who complete the first 5 credits in college-level math Source: SBCTC Data Warehouse Link to Agency Plan: http://www.sbctc.ctc.edu/docs/sbctc_system_direction_final.pdf Summary Analysis The graph to the left illustrates the 6% increase in the number of all CTC students who completed college-level math in 2008-09 over the baseline. College-level math is required for all students who are planning to transfer to a 4-year institution. The table to the left illustrates math achievement of transfer students. In 2008-09, nearly 100,000 students who were planning to transfer entered the year with no college math. While the percent of transfer students completing college math in 2008-09 over the baseline just kept pace with the increase in the number of students, pre-college math points increased at a faster rate. Colleges are focusing on making improvements in math remediation, advising students to take math sooner and are working with high schools to improve college prep. Action Plan Provide financial rewards to colleges each year for increasing the number of college students who reach the first 15 and first 30 college-level credits. Look for promising practices that prepare students for college level math and advise and guide those students into appropriate courses for their degrees, through the Gates- funded initiatives and college efforts. Colleges will work to provide students appropriate math options for specific fields of study.

27. Are more community and technical college students reaching the “tipping point”? Tipping Point Growth Over Baseline of All Students Measure: Number of students who complete a degree, a certificate backed by at least one year of college or an apprenticeship (the “tipping point”) Source: SBCTC Data Warehouse Link to Agency Plan: http://www.sbctc.ctc.edu/docs/sbctc_system_direction_final.pdf Summary Analysis The minimum threshold for the “tipping point” is a short-term certificate backed by at least a year of college. Most students go beyond that to complete a long-term certificate, a degree or an apprenticeship. SBCTC and national research demonstrates that college students experience a substantial boost in earnings gain when they reach the tipping point or beyond. In 2008-09, the number of students reaching the tipping point or higher increased by 11% over the baseline, as compared to 4% growth in the number of students. Action Plan Provide financial rewards to colleges each year for increasing the number of college students who reach the tipping point. Provide Opportunity Grants to help low-income adults reach the tipping point in an educational program leading to careers in high-demand fields.