1. Policy Capturing Approaches to Setting Cut Scores for College Readiness Arthur Thacker CCSSO Student Assessment Conference June 2015

2. The Challenge  Kentucky colleges and universities use ACT scores to determine entrance, eligibility for scholarships, placement into advanced or non-credit bearing courses.  State assessment data were not always seen to support the story told by the ACT.  Typical standards setting methods rely on expert judgement regarding the content of test items to set standards (with some revisions based on impact data).  Using empirical data to set cut scores, without explicitly considering item content, can be a challenge for creating or directly linking performance/achievement level descriptors with classifications. 2

3. Linking Policy to Classification of Students  Kentucky schools are accustomed to interpreting ACT scores into policy actions.  A panel of experts including teachers, principals, superintendents, higher education, and guidance counselors were gathered to make cut score recommendations.  They discussed the issues and referenced scores back to the ACT.  The most easily determined cut score was for Proficient—the panelists agreed that proficient must mean college ready and that definition was based on ACT for Kentucky colleges. 3

4. What is “Proficient?”  ACT Sets benchmark scores based on how students perform in certain college courses  Kentucky’s Council for Post-secondary Education (CPE) set Kentucky-specific benchmarks 4 ACT AssessmentACT Benchmark CPE Benchmark English18 Math2219 Reading2120 Science2422

5. Beyond Proficient  Kentucky classifies students as Novice, Apprentice, Proficient, and Distinguished.  We needed 2 more cut scores to establish 4 performance categories.  Discussion around the Distinguished category centered on qualification for scholarships and college recruitment.  The Apprentice category was more challenging, and eventually tied to routine college entrance (community and technical colleges). 5

6. Grades  End-of-Course tests are expected to be included in student grades. State regulation requires them to be included at a minimum of 20% of the of the final grade).  Panelists quickly agreed that B had the essentially the same meaning as proficient.  Grades of A were judged to be lower hurdles than the Distinguished category.  Grades of C were judged to be roughly equivalent to the Apprentice category.  By policy, grades of D and F were distinguished by establishing a lower bound for Ds (around 1% Fs). 6

7. Results The ACT score ranges on which other Kentucky test scores were based. 7 EOC Exam Linked ACT Exam NoviceApprenticeProficientDistinguished English IIEnglish 0-15 (36.6%) 16-17 (11.0%) 18-26 (41.6%) 27-36 (10.8%) Algebra IIMath 0-15 (26.5%) 16-18 (33.1%) 19-26 (33.5%) 27-36 (6.9%) U.S. History Reading 0-15 (37.6%) 16-19 (22.9%) 20-26 (26.4%) 27-36 (13.1%) BiologyScience0-15 (24.9%) 16-21 (44.6%) 22-26 (23.8%) 27-36 (6.8%)

8. Extending these results to the rest of K-PREP  Once these proportions of students by category were established for high school, it was time to consider middle and elementary grades.  Obviously, the idea of “college ready” is tenuous at the elementary grades.  It is important that the system be coherent (wild differences in proportions of students by category by grade strain credibility).  Cut scores were determined for lower grades using the equipercentile method. 8

9. Wrap Up  Kentucky’s system is linked to ACT.  Monitoring is required to address changes to college- ready benchmarks (either by ACT or CPE).  A brief report is available describing the full methodology and results of this effort. http://education.ky.gov/AA/KTS/Documents/HumRRO%202 013-007%20Policy%20EOC%20and%20K- PREP%20Cut%20Scores.pdf  Questions? 9