1. WA Student Achievement Initiative: Preliminary Findings from Points Analysis and Field Research
Davis Jenkins
Community College Research Center
Teachers College, Columbia University
Presentation to WA CTC BAC and WARP Councils
July 14, Seattle

2. Washington State’s Student Achievement Initiative
Goal is to increase academic achievement at the state’s 34 community and technical colleges, so that all students go farther, faster
Funding awards are based on how well colleges perform relative to themselves
Initial awards (fall 2009) based on the change in total points at each college across six metrics between the baseline year (AY2007) and AY2009

3. Features Getting National Attention
Achievement measures recommended by taskforce and advisory group and informed by SBCTC and CCRC research
Measures encompass all mission areas, including adult basic skills and developmental education
Performance measures designed to be within colleges’ control
Colleges receive quarterly performance data to help decide where to invest resources to accelerate student achievement

4. CCRC-IHELP SAI Policy Study
3-year study funded by the Gates Foundation
Research questions:
Is the initiative shaping college practices related to student retention and success? What is the relative influence/value of the performance funding vs. the measures and data?
Have students moved farther, faster? Has the mix of students who reach the “tipping point” changed in a way the reflects the demographics of the state?
What distinguishes colleges that perform well according to the achievement point framework from those that do less well?
Is the SAI policy sustainable in WA and what aspects of the policy might be adopted by other states?
(becomes part of base budget)

5. CCRC-IHELP SAI Policy Study
Research methods
Builds on CCRC study during “learning year”
Quantitative analysis of student progression and college performance using SAI data
Interviews at sample of 17 colleges (spring 2010, and spring 2012)
Interviews with policy makers in WA and other states
Deliverables
Briefing for Gates and WA policy makers (Sept. 2010)
Reports on quantitative analysis of achievement points (1 report per year)
Guides for state policy makers and college leaders (Dec and Dec. 2012)
(becomes part of base budget)

6. Year 1 (2010) Research Activities
1) Quantitative analysis of achievement point changes during the initial funding period
2) Interviews on-site and by telephone with administrators, faculty, and staff at 17 colleges previously studied during learning year
3) Telephone interviews with policy makers in WA, OH, NC and TX

7. Initial Points Analysis
From baseline to AY2009:
Total points
Change in total points
Points per individual metric
Points per student (college-level)
Points per student (student-level)

8. Total Points and Change in Points
At baseline, average points total was 8,684; by 2009 it had risen to 10,365
Points growth of 1,681 points, or 19%.
Points growth (19%) exceeded enrollment growth (12%)
Two-thirds of points growth is attributable to more students; one-third to more points per student -- may indicate greater effectiveness

9. Change in Points

10. Influence of College Size
Enrollment and total points are strongly correlated: bigger colleges produce more points
Enrollment and change in total points are also positively correlated: bigger colleges gain more points
A college with 1,000 more students gained on average 52 more points per year (total gain is ~840 points per year)

11. Total Points per College (AY2009) and FTE Enrollment

12. Total Points per College (AY2009) and Total Expenditures

13. Change in Points per College and FTE Enrollment

14. Growing Influence of Basic Skills and College Readiness
Most points produced in basic skills and college readiness
By 2009, basic skills and college readiness were making an even larger contribution to total points
These two metrics accounted for 1,056 of the 1,681 extra points gained per college
Five colleges lost points in quantitative reasoning; this metric may be diminishing in importance
Student-level analysis indicates that the basic skills metric is the most powerful in encouraging enrollment from traditionally under-served groups

15. Source of Change in Points 2007-09

16. Heterogeneity of Effects
No simple model predicts likely points accumulation for any given student
Student characteristics – such as age, full-time status, race or prior education – do not strongly and clearly influence total points and the change in points
There are some significant influences on points across the six metrics, but these wash out in the aggregate
Overall, there is little evidence that colleges serving more at-risk are being penalized by the SAI awards method

17. Change in Points per College Effects
No correlation between change in points and:
% on financial aid, % prior education=dropout
% age under 20, % age over 65
% fulltime, % male, spending per FTE
% Asian, % Black, % Hispanic, % other race
Positive correlation between change in points and:
Enrollment 
Student socio-economic status (affects basic skills points)

18. Spring 2010 Field Research Preliminary Findings
Views on Achievement Point framework
Generally liked because it focuses attention on student progression, provides common vocabulary for retention efforts; yet awareness on most campuses is limited
Concerns re: Achievement Point measures
Disagreement re: emphasis on basic skills/college prep
Concern that larger colleges garner more points
Rewards enrollment growth, but does it reward improved student persistence and completion?
How to give colleges credit for students who “swirl”
Colleges having trouble replicating SAI results their with their data

19. 2010 Field Research Draft Findings
Views re: performance funding
Viewed more negatively than the points framework; many dislike the “skim” due to lack of new funds
Some correlation between attitude and ranking
Agreement among many that performance funding is here to stay in WA and will expand
Consensus that money talks, but current SAI funding level is not enough to change behavior
Concern however that more money could skew college priorities, promote unhealthy competition
Yet, few suggestions for a better model

20. 2010 Field Research Draft Findings
Use of SAI data for improvement
Use varies – influenced by IR staff attitudes re: the data
Use limited by early data concerns, uncertainty on how to use data to identify specific areas for improvement and “best practices”
SAI has caused some colleges to look more closely at their own data on student progression and success
Views on future of SAI
Uncertainty about source, stability of future funding
What happens when enrollment growth subsides?

21. Further Research Planned
Further quantitative analysis ( ):
Will early patterns be reinforced such that the points distribution becomes more uneven?
Do alternative measures of institutional performance yield different funding allocations?
“Highest momentum” -- Are students going farther and progressing faster?
Field research comparing high- and low-performing colleges (spring 2012)

22. For more information:
Please visit us on the web at
where you can download presentations, reports,
CCRC Briefs, and sign-up for news announcements.
Community College Research Center
Institute on Education and the Economy, Teachers College, Columbia University
525 West 120th Street, Box 174, New York, NY 10027
Telephone: