Setting Accuplacer Cut Scores for WritePlacer Plus and College Math Becky J. Mussat-Whitlow, M.A., Ph.D. Director of Institutional Assessment Winston-Salem State University Robert Ussery, M.S. Assistant Vice Chancellor for Academic Affairs North Carolina A&T State University 18th Annual Accuplacer National Conference Fort Lauderdale, Florida June 26, 2008

Introduction Historical context Executive decision to use Accuplacer Local bubble sheet test for math placement ETS’s online Criterion for English placement Executive decision to use Accuplacer For math and English placement Single vendor

Project Timeline November, 2007 December, 2007 – Secure funding Assign project manager Complete project plan December, 2007 – Secure funding January, 2008 Provost appoint Placement Committee Recruit cut score study consultant February, 2008 Appoint cut score study panels and chairs Conduct Round 1 of student testing March, 2008 Conduct Cut Score study during Spring Break (March 3 – 7) Round 2 of student testing Cut score recommendations to Placement Committee

Creating Defensible Cut Scores The standard setting process should pay careful attention to: 1. Selection of panelists 2. Training 3. Aggregation of data into a final set of standards 4. Validation of performance standards 5. Careful documentation of the process. Reference: Hansche, L.N. (1998). Handbook for the development of performance standards. Bethesda, MD: US Department of Education, Council of Chief State School Officers. Mathematics Panel 12 2 Lecturers 4 Associate Professors 5 Assistant Professors 1 Professor Torry Burden, Lecturer, Center for Academic Excellence Shea Burns, Associate Professor, Mathematics Department Mingxiang Chen, Associate Professor, Mathematics Department Christopher Doss, Assistant Professor, Electrical Engineering Gregory Gibson, Assistant Professor, Mathematics Department Etta Gravely, Assistant Professor, Chemistry Department Yaw Kyei, Assistant Professor, Mathematics Department Janice Oldham, Associate Professor, Mathematics Department Devedas Pai, Professor, Mechanical Engineering Pat Shelton, Lecturer, Mathematics Department Katrina Staley, Assistant Professor, Mathematics Department Ron Steed, Lecturer, University Studies Vereda Williams, Associate Professor, Economics Department English Panel 3 Associate Professors 4 Assistant Professors 2 Professor 1 Unspecified Paul Ankomah, Associate Professor, Human Performance and Leisure Studies Department Betty Chapman, Assistant Professor, Business Administration Department Titilayo Evans, Lecturer, Center for Academic Excellence Jorge Gaytan, Business Education Department Lakecia Gunthrop, Lecturer, Center for Academic Excellence Beth Kaufka, Assistant Professor, University Studies Thelma King, Associate Professor, Business Education Department Elon Kulii, Professor, English Department Ewuuk Lomo-David, Associate Professor, Business Education Department Valerie Nieman, Assistant Professor, English Department Thomas Porter, Professor, History Department Sonya Ricks, Assistant Professor, History Department Panel Chairs English: Deborah Barnes, Associate Dean, University Studies Mathematics: Eric Cheek, Assistant Vice Chancellor, Summer Sessions and Outreach

At what point along the scale should the passing mark be set? Critical Question At what point along the scale should the passing mark be set?

Classification Error Cut scores split a continuous distribution of knowledge, skills, and abilities into separate regions. Need to determine the preference for classification error. Do you prefer to pass students who deserved to fail? OR Do you prefer to fail students who deserved to pass?

History of Standard Setting 2000 BC – Chinese Military Selection 1800s - Integration of Psychology and Statistics 1970s – Mandated pupil proficiency testing 1978 Journal of Educational Measurement 1982 ETS Passing Scores publication

“People have been setting cut scores for thousands of years, but it is only since the middle of the 20th century that measurement professionals began to pay much attention.” ~ From: Cizek, G.J. (2001). Setting Performance Standards. Hillsdale, NJ: Lawrence Erlbaum Associates. (page 20)

Overview of Standard Setting All standard setting methods involve judgment Performance standards (cut scores) may be set too high or low Need a reasonable process to arrive at decision Standard setting methods for performance assessments such as writing samples are not as well developed at this time.

Two Categories of Setting Performance Standards Test-based methods – methods in which panelists are focused on a review of test content Student based methods – methods that focus on students Types of Test-Based Methods: Nedelsky Ebel Angoff -- assign a probability to each dichotomously scored test item directly. Each probability is to be an estimate of the “borderline student.” Student based Methods: Borderline-Group: Requires that a description is prepared of each performance category. In practice, teachers are asked to submit a list of students whose performances would be so close to the standard or borderline that they could not be reliably classified. The test is administered to these “borderline” groups and the median test score for each group is taken as the standard. Contrasting groups: Working with the description of students in each performance category, teachers are asked to classify their students into these performance categories. The test is administered to the groups and the score distributions for the groups are compared. The point of intersection is often taken as the initial standard (Berk, 1976).

Guiding Principles to Ensure Fairness in Establishing Cut Scores Those who set the standards should be thoroughly knowledgeable about the content domain that is to be assessed, the population of examinees who are to take the assessment, and the uses to which the results are to be put. (p. 316) Proficiency classifications should have the same meaning for all sub-groups. (p. 317) ~ From: Bond, L. (1995). Ensuring fairness in the setting of performance standards. In Proceedings of Joint Conference on Standard Setting for Large-Scale Assessments (pp. 311-324). Washington, DC: National Assessment Governing Board and National Center for Education Statistics.

Guiding Principles to Ensure Fairness in Establishing Cut Scores If the assessment is to be used as a screen for future educational opportunities, the content of the assessment and the level of proficiency required should be demonstrably related to future success. (p. 317) Attention must be paid to the consequences of particular uses of an assessment. (p. 318) ~ From: Bond, L. (1995). Ensuring fairness in the setting of performance standards. In Proceedings of Joint Conference on Standard Setting for Large-Scale Assessments (pp. 311-324). Washington, DC: National Assessment Governing Board and National Center for Education Statistics.

Guiding Principles to Ensure Equity in the Process of Setting Cut Scores Adequate notice of proposed actions. Ample provision of opportunities for participation Adequate records of all discussions and decisions by the participants. Timely distribution of minutes and ballot results Careful attention to minority opinions. ~ From: Collins,B. L. (1995). The consensus process in standards development. In Proceedings of Joint Conference on Standard Setting for Large-Scale Assessments (pp. 203-220). Washington, DC: National Assessment Governing Board and National Center for Education Statistics.

Common Problems Use of ambiguous descriptions of performance standards Failure to involve key stakeholders in standard setting process Failure to devote sufficient time to establish cut scores Failure to document the process Failure to validate the process

Questions to Consider How many cut scores are needed? Two (1 to differentiate between remedial and regular, 1 to differentiate between regular and advanced) VERSUS One (to differentiate between remedial and regular course placement)

Desirable Characteristics of Cut Scores Should be understandable and useful for all stakeholders Clearly differentiate among levels Grounded in student work Built by consensus Focus on learning From p. 16 of Hansche 1998 Handbook

Contrasting Group Method A group of panelists qualified to assess the content domain and students being assessed are asked to classify students into two groups (master vs. non-masters) Panelists initial make judgments regarding grouping of students. Then, the performance of examinees empirically determined.

Contrasting Group Method Two distributions created to represent the students’ actual (obtained) scores on the assessment separately; one for those students judged to have acceptable skills by the standard setters and another for those students whose performances were judged to be unacceptable. The point at which the two distributions intersect may be chosen as the cut score location.

Hypothetical Illustration of Contrasting Groups Distributions Nonmasters Distribution Masters Score Scale Cχ ƒ

Modified Contrasting Group Method Students were classified into three broad performance categories of average, below average, or above average and administered the placement tests. Score distributions were plotted to represent the students’ actual (obtained) scores on the assessment separately The distribution plot was visually analyzed to identify an appropriate cut score. Phase III will consist of a student-based method of setting performance standards.  More specifically, in Phase III, faculty teaching remedial, regular, and advanced courses in English and math, will be asked to classify their students into three broad performance categories of average, below average, or above average.  For both the mathematics and English area, a minimum of three courses will be targeted such that students enrolled in remedial, regular, and advanced courses will be required to complete the Accuplacer test.  As such, students placed into a remedial math course who are classified as below average by their instructor, would be expected to score poorly on the college level math test.  Likewise, students enrolled in an advanced mathematics course who are classified as above average, would be expected to perform well on the college level math test.  The score distributions for each of the performance categories will be compared to determine the most appropriate point at which to place the cut scores.

Modified Contrasting Group Method Panelists completed the placement test role playing as a student with average and above average ability Score distributions were plotted to represent the panelists’ actual (obtained) scores The distribution plot was visually analyzed to help guide the establishment of an appropriate cut score. Phase III will consist of a student-based method of setting performance standards.  More specifically, in Phase III, faculty teaching remedial, regular, and advanced courses in English and math, will be asked to classify their students into three broad performance categories of average, below average, or above average.  For both the mathematics and English area, a minimum of three courses will be targeted such that students enrolled in remedial, regular, and advanced courses will be required to complete the Accuplacer test.  As such, students placed into a remedial math course who are classified as below average by their instructor, would be expected to score poorly on the college level math test.  Likewise, students enrolled in an advanced mathematics course who are classified as above average, would be expected to perform well on the college level math test.  The score distributions for each of the performance categories will be compared to determine the most appropriate point at which to place the cut scores.

Multi-Step Approach to Establish Cut Scores Panelists created performance level descriptions (PLDs) and mapped these PLDs to College Board proficiency level statements   Panelists reviewed student score distributions Panelists completed placement test and reviewed their score distributions. Additionally, for English, panelists retrospectively categorized papers into 2 groups (remediation required vs. no remediation required) and used this information to guide cut score establishment.      

Three Day Panel Sessions Panels met for 3 hour sessions on 3 consecutive days. Math Panel: 12 faculty English Panel: 12 faculty

Panel Session I Panel Session 1: Identified the number of cut scores needed and the different courses into which students would be placed Developed clear and concise performance level descriptions for each placement level three three-hour sessions during the week of March 3–7, 2008 Identified the number of cut scores needed and the different courses into which students would be placed i.e. remedial, regular, or advanced for math, and remedial or regular for English. They also developed clear and concise performance level descriptions for each placement level

Characteristics of Performance Level Descriptions Written in positive terms Parallel in organization, language, and style Written in clear and concise language without using unmeasurable qualifiers (e.g., often, seldom, etc.)

Example of Worksheet

Writing Performance Level Description FOCUS maintains consistent point of view/perspective and clearly communicates main point and unity of purpose ORGANIZATION demonstrates understanding of primary elements of the essay (introduction, body, conclusion), develops effective paragraphs, and uses transitions correctly and effectively sequences ideas DEVELOPMENT AND SUPPORT develops complete thoughts using appropriate vocabulary and contains relevant evidence and supporting details SENTENCE STRUCTURE demonstrates understanding of basic grammar and varied syntax MECHANICS demonstrates understanding of basic mechanical conventions such as spelling, punctuation, capitalization, and word choice

Math Performance Level Descriptions Perform addition, subtraction, multiplication and division of polynomials; identify factors, terms, constants, variables, exponents, and coefficients; and recognize graphs of polynomial functions. MATH 110 or 111 Perform addition, subtraction, multiplication and division on algebraic expressions involving integer exponents and radicals, solve linear and quadratic equations, evaluate, graph, and find the domain and range of functions including linear and quadratic functions. MATH 131 Perform addition, subtraction, multiplication, division, factoring, composition and the simplification of algebraic, trigonometric, exponential and logarithmic expressions and functions; use algebraic skills to solve linear, quadratic, rational, and absolute value equations and inequalities; and evaluate, graph and interpret functions as well as their transformations.

Panel Session II Panel Session 2: PLDs Approved Initial round of cut score setting based upon the results of student testing and faculty role play performance For English, student work categorized by panelists into 2 groups. Initial round of cut score setting based upon the results of student testing conducted on February 27, 2008 Students were classified into three broad performance categories of average, below average, or above average and administered the placement tests. Panelists were also required to complete the placement test a minimum of two times (role playing as a student with poor and above average ability).

Illustration of Score Distributions Students scoring 31 and less than 42 would qualify for placement in Math 101. Students scoring 42 and less than 65 would qualify for placement in Math 110 or Math 111. Students scoring 65 or higher would qualify for placement in Math 131.

Illustration of Score Distributions 2nd Cut Score = 42 Students scoring from 30 to 41 – Math 101 1st Cut Score = 30 Students scoring less than 30 – Remedial Placement 3rd Cut Score = 65 Students scoring from 42 to 64 – Math 110 or 111 Students scoring 65 or higher – Math 131 Students scoring less than 30 remedial placement. Students scoring 30 and less than 42 would qualify for placement in Math 101. Students scoring 42 and less than 65 would qualify for placement in Math 110 or Math 111. Students scoring 65 or higher would qualify for placement in Math 131.

English Panelist Rating Information Mean SD Remedial Indicated by Two or Fewer Panelists 7.85 1.37 Remedial Indicated by Three or More Panelists 6.90 1.07 4 raters per paper

English Panelist Rating Information Mean SD Remedial Indicated by None or Some Panelists 7.61 1.28 Remedial Indicated by All Panelists 5.0 1.01 4 raters per paper

Panel Session III Panel Session 3: Cut Scores were aligned across various methods used. Initial round of cut score setting based upon the results of student testing conducted on February 27, 2008 Students were classified into three broad performance categories of average, below average, or above average and administered the placement tests. Panelists were also required to complete the placement test a minimum of two times (role playing as a student with poor and above average ability).

PLDs Mapped To Proficiency Statements MATH 101 PLD: Perform addition, subtraction, multiplication and division of polynomials; identify factors, terms, constants, variables, exponents, and coefficients; and recognize graphs of polynomial functions. Lower than ACCUPLACER 40 Cut Score 30

Alignment of Recommendations Performance level descriptions written by NC A & T faulty were mapped to College Board proficiency level statements   Review score distributions for student performance groups to determine cut score Cut scores indicated on the basis of the student data were considered in conjunction with performance level descriptions.      In Phase IV, the data available from Phase II and III would be compared to obtain the best possible alignment for cut score recommendations.  As such, the performance level descriptions detailed by the panelists should coincide with the proficiency level statements provided by the College Board.  Moreover, the cut scores indicated on the basis of the student data should be consistent as well.     After discussion and consensus by the panelist, the final cut scores recommendations will be prepared.

Example of Alignment Process Based on mapping to Accuplacer Proficiency Statements. Cut score should fall between 40 & 63. Math 110 or 111

Follow-Up Testing and Cut-Score Refinement Additional Students Tested Cut Scores Revised Recommendations Made By Panelists A second round of student testing was conducted on March 12, 2008. Student performance results were used by panels in wrap-up sessions to refine and adjust their previous cut score recommendations. The English panel recommended a cut score of 6. Students scoring 6 or less would be placed in a remedial first English course. Students scoring 7 or above would be placed in a regular first English course. _______________________X_________________________ 2 6 12 The mathematics panel recommended three cut scores. Students scoring 30 or less would be placed in a remedial first math course. Students scoring 31 and less than 42 would qualify for placement in Math 101. Students scoring 42 and less than 65 would qualify for placement in Math 110 or Math 111. Students scoring 65 or higher would qualify for placement in Math 131. The panel recommended that students should have the option to register for a non-remedial first math course lower than their qualifying level. ___________X_____X___________X_____________________ 2 30 42 65 120

Project Timeline after Study March, 2008 Committee review and recommendations Executive review and policy April, 2008 Develop student score validation and upload procedure May, 2008 Train proctors and faculty advisors Final system test June Go live with Accuplacer placement testing 500 students per week July, 2008 Process evaluation and recommendations

Campus Collaboration Admissions – New students Student Affairs – New student orientation Registrar – Validate student data IT – Upload student data to campus system Academic Affairs - Proctors and advisors Institutional Research – Management and support

Accuplacer Support Helpline Online procedure guides

Early Results Five (5) orientation sessions in June Overall, the system works Some Accuplacer interface weaknesses

WHERE TO FROM HERE Develop alternative placement measures Implement off-site testing

Thank you for attending! Questions? NCA&T Support Site http://qed.ncat.edu/ir&p/availabl.htm

Proficiency Statements WritePlacer Plus Total Right Score of About 40 or Less These students should take the Elementary Algebra test before any placement decisions are finalized. Total Right Score of About 40 Students scoring at this level can: identify common factors factor binomials and trinomials manipulate factors to simplify complex fractions These students should be considered for placement into intermediate algebra. For further guidance in placement, have these students take the Elementary Algebra test. Total Right Score of About 63 Students scoring at this level can demonstrate the following additional skills: work with algebraic expressions involving real number exponents factor polynomial expressions simplify and perform arithmetic operations with rational expressions, including complex fractions solve and graph linear equations and inequalities solve absolute value equations solve quadratic equations by factoring graph simple parabolas understand function notation, such as determining the value of a function for a specific number in the domain have a limited understanding of the concept of function on a more sophisticated level, such as determining the value of the composition of two functions have a rudimentary understanding of coordinate geometry and trigonometry These students should be considered for placement into college algebra or a credit-bearing course immediately preceding calculus. Total Right Score of About 86 understand polynomial functions evaluate and simplify expressions involving functional notation, including composition of functions solve simple equations involving: • trigonometric functions • logarithmic functions • exponential functions These students can be considered for a precalculus course or a nonrigorous course in beginning calculus. Total Right Score of About 103 perform algebraic operations and solve equations with complex numbers understand the relationship between exponents and logarithms and the rules that govern the manipulation of logarithms and exponents understand trigonometric functions and their inverses solve trigonometric equations manipulate trigonometric identities solve right-triangle problems recognize graphic properties of functions such as absolute value, quadratic, and logarithmic These students should be considered for placement into calculus.

ACCUPLACER – Math Proficiency Statements Total Right Score of About 40 Students scoring at this level can identify common factors, factor binomials and trinomials, manipulate factors to simplify complex fractions Total Right Score of About 63 Students scoring at this level can demonstrate the following additional skills: work with algebraic expressions involving real number exponents, factor polynomial expressions, simplify and perform arithmetic operations with rational expressions, including complex fractions, solve and graph linear equations and inequalities, solve absolute value equations, solve quadratic equations by factoring, graph simple parabolas, understand function notation, such as determining the value of a function for a specific number in the domain , have a limited understanding of the concept of function on a more sophisticated level, such as determining the value of the composition of two functions, have a rudimentary understanding of coordinate geometry and trigonometry Total Right Score of About 86 Students scoring at this level can demonstrate the following additional skills: understand polynomial functions, evaluate and simplify expressions involving functional notation, including composition of functions, solve simple equations involving: trigonometric functions, logarithmic functions, exponential functions Total Right Score of About 103 Students scoring at this level can demonstrate the following additional skills: perform algebraic operations and solve equations with complex numbers, understand the relationship between exponents and logarithms and the rules that govern the manipulation of logarithms and exponents, understand trigonometric functions and their inverses, solve trigonometric equations, manipulate trigonometric identities, solve right-triangle problems, recognize graphic properties of functions such as absolute value, quadratic, and logarithmic Total Right Score of About 40 or Less These students should take the Elementary Algebra test before any placement decisions are finalized. Total Right Score of About 40 Students scoring at this level can: identify common factors factor binomials and trinomials manipulate factors to simplify complex fractions These students should be considered for placement into intermediate algebra. For further guidance in placement, have these students take the Elementary Algebra test. Total Right Score of About 63 Students scoring at this level can demonstrate the following additional skills: work with algebraic expressions involving real number exponents factor polynomial expressions simplify and perform arithmetic operations with rational expressions, including complex fractions solve and graph linear equations and inequalities solve absolute value equations solve quadratic equations by factoring graph simple parabolas understand function notation, such as determining the value of a function for a specific number in the domain have a limited understanding of the concept of function on a more sophisticated level, such as determining the value of the composition of two functions have a rudimentary understanding of coordinate geometry and trigonometry These students should be considered for placement into college algebra or a credit-bearing course immediately preceding calculus. Total Right Score of About 86 understand polynomial functions evaluate and simplify expressions involving functional notation, including composition of functions solve simple equations involving: • trigonometric functions • logarithmic functions • exponential functions These students can be considered for a precalculus course or a nonrigorous course in beginning calculus. Total Right Score of About 103 perform algebraic operations and solve equations with complex numbers understand the relationship between exponents and logarithms and the rules that govern the manipulation of logarithms and exponents understand trigonometric functions and their inverses solve trigonometric equations manipulate trigonometric identities solve right-triangle problems recognize graphic properties of functions such as absolute value, quadratic, and logarithmic These students should be considered for placement into calculus.