1. Setting Accuplacer Cut Scores for WritePlacer Plus and College Math
Becky J. Mussat-Whitlow, M.A., Ph.D.
Director of Institutional Assessment
Winston-Salem State University
Robert Ussery, M.S.
Assistant Vice Chancellor for Academic Affairs
North Carolina A&T State University
18th Annual Accuplacer National Conference
Fort Lauderdale, Florida
June 26, 2008

2. Introduction Historical context Executive decision to use Accuplacer
Local bubble sheet test for math placement
ETS’s online Criterion for English placement
Executive decision to use Accuplacer
For math and English placement
Single vendor

3. Project Timeline November, 2007 December, 2007 – Secure funding
Assign project manager
Complete project plan
December, 2007 – Secure funding
January, 2008
Provost appoint Placement Committee
Recruit cut score study consultant
February, 2008
Appoint cut score study panels and chairs
Conduct Round 1 of student testing
March, 2008
Conduct Cut Score study during Spring Break (March 3 – 7)
Round 2 of student testing
Cut score recommendations to Placement Committee

4. Creating Defensible Cut Scores
The standard setting process should pay careful attention to:
1. Selection of panelists
2. Training
3. Aggregation of data into a final set of standards
4. Validation of performance standards
5. Careful documentation of the process.
Reference: Hansche, L.N. (1998). Handbook for the development of performance standards. Bethesda, MD: US Department of Education, Council of Chief State School Officers.
Mathematics Panel 12
2 Lecturers
4 Associate Professors
5 Assistant Professors
1 Professor
Torry Burden, Lecturer, Center for Academic Excellence
Shea Burns, Associate Professor, Mathematics Department
Mingxiang Chen, Associate Professor, Mathematics Department
Christopher Doss, Assistant Professor, Electrical Engineering
Gregory Gibson, Assistant Professor, Mathematics Department
Etta Gravely, Assistant Professor, Chemistry Department
Yaw Kyei, Assistant Professor, Mathematics Department
Janice Oldham, Associate Professor, Mathematics Department
Devedas Pai, Professor, Mechanical Engineering
Pat Shelton, Lecturer, Mathematics Department
Katrina Staley, Assistant Professor, Mathematics Department
Ron Steed, Lecturer, University Studies
Vereda Williams, Associate Professor, Economics Department
English Panel
3 Associate Professors
4 Assistant Professors
2 Professor
1 Unspecified
Paul Ankomah, Associate Professor, Human Performance and Leisure Studies Department
Betty Chapman, Assistant Professor, Business Administration Department
Titilayo Evans, Lecturer, Center for Academic Excellence
Jorge Gaytan, Business Education Department
Lakecia Gunthrop, Lecturer, Center for Academic Excellence
Beth Kaufka, Assistant Professor, University Studies
Thelma King, Associate Professor, Business Education Department
Elon Kulii, Professor, English Department
Ewuuk Lomo-David, Associate Professor, Business Education Department
Valerie Nieman, Assistant Professor, English Department
Thomas Porter, Professor, History Department
Sonya Ricks, Assistant Professor, History Department
Panel Chairs
English: Deborah Barnes, Associate Dean, University Studies
Mathematics: Eric Cheek, Assistant Vice Chancellor, Summer Sessions and Outreach

5. At what point along the scale should the passing mark be set?
Critical Question
At what point along the scale should the passing mark be set?

6. Classification Error
Cut scores split a continuous distribution of knowledge, skills, and abilities into separate regions.
Need to determine the preference for classification error.
Do you prefer to pass students who deserved to fail? OR
Do you prefer to fail students who deserved to pass?

7. History of Standard Setting
2000 BC – Chinese Military Selection
1800s - Integration of Psychology and Statistics
1970s – Mandated pupil proficiency testing
1978 Journal of Educational Measurement
1982 ETS Passing Scores publication

8. “People have been setting cut scores for thousands of years, but it is only since the middle of the 20th century that measurement professionals began to pay much attention.”
~ From: Cizek, G.J. (2001). Setting Performance Standards. Hillsdale, NJ: Lawrence Erlbaum Associates. (page 20)

9. Overview of Standard Setting
All standard setting methods involve judgment
Performance standards (cut scores) may be set too high or low
Need a reasonable process to arrive at decision
Standard setting methods for performance assessments such as writing samples are not as well developed at this time.

10. Two Categories of Setting Performance Standards
Test-based methods – methods in which panelists are focused on a review of test content
Student based methods – methods that focus on students
Types of Test-Based Methods:
Nedelsky
Ebel
Angoff -- assign a probability to each dichotomously scored test item directly. Each probability is to be an estimate of the “borderline student.”
Student based Methods:
Borderline-Group: Requires that a description is prepared of each performance category. In practice, teachers are asked to submit a list of students whose performances would be so close to the standard or borderline that they could not be reliably classified. The test is administered to these “borderline” groups and the median test score for each group is taken as the standard.
Contrasting groups: Working with the description of students in each performance category, teachers are asked to classify their students into these performance categories. The test is administered to the groups and the score distributions for the groups are compared. The point of intersection is often taken as the initial standard (Berk, 1976).

11. Guiding Principles to Ensure Fairness in Establishing Cut Scores
Those who set the standards should be thoroughly knowledgeable about the content domain that is to be assessed, the population of examinees who are to take the assessment, and the uses to which the results are to be put. (p. 316)
Proficiency classifications should have the same meaning for all sub-groups. (p. 317)
~ From: Bond, L. (1995). Ensuring fairness in the setting of performance standards. In Proceedings of Joint Conference on Standard Setting for Large-Scale Assessments (pp ). Washington, DC: National Assessment Governing Board and National Center for Education Statistics.

12. Guiding Principles to Ensure Fairness in Establishing Cut Scores
If the assessment is to be used as a screen for future educational opportunities, the content of the assessment and the level of proficiency required should be demonstrably related to future success. (p. 317)
Attention must be paid to the consequences of particular uses of an assessment. (p. 318)
~ From: Bond, L. (1995). Ensuring fairness in the setting of performance standards. In Proceedings of Joint Conference on Standard Setting for Large-Scale Assessments (pp ). Washington, DC: National Assessment Governing Board and National Center for Education Statistics.

13. Guiding Principles to Ensure Equity in the Process of Setting Cut Scores
Adequate notice of proposed actions.
Ample provision of opportunities for participation
Adequate records of all discussions and decisions by the participants.
Timely distribution of minutes and ballot results
Careful attention to minority opinions.
~ From: Collins,B. L. (1995). The consensus process in standards development. In Proceedings of Joint Conference on Standard Setting for Large-Scale Assessments (pp ). Washington, DC: National Assessment Governing Board and National Center for Education Statistics.

14. Common Problems Use of ambiguous descriptions of performance standards
Failure to involve key stakeholders in standard setting process
Failure to devote sufficient time to establish cut scores
Failure to document the process
Failure to validate the process

15. Questions to Consider How many cut scores are needed?
Two (1 to differentiate between remedial and regular, 1 to differentiate between regular and advanced)
VERSUS
One (to differentiate between remedial and regular course placement)

16. Desirable Characteristics of Cut Scores
Should be understandable and useful for all stakeholders
Clearly differentiate among levels
Grounded in student work
Built by consensus
Focus on learning
From p. 16 of Hansche 1998 Handbook

17. Contrasting Group Method
A group of panelists qualified to assess the content domain and students being assessed are asked to classify students into two groups (master vs. non-masters)
Panelists initial make judgments regarding grouping of students. Then, the performance of examinees empirically determined.

18. Contrasting Group Method
Two distributions created to represent the students’ actual (obtained) scores on the assessment separately; one for those students judged to have acceptable skills by the standard setters and another for those students whose performances were judged to be unacceptable.
The point at which the two distributions intersect may be chosen as the cut score location.

19. Hypothetical Illustration of Contrasting Groups Distributions
Nonmasters
Distribution
Masters
Score Scale Cχ ƒ

20. Modified Contrasting Group Method
Students were classified into three broad performance categories of average, below average, or above average and administered the placement tests.
Score distributions were plotted to represent the students’ actual (obtained) scores on the assessment separately
The distribution plot was visually analyzed to identify an appropriate cut score.
Phase III will consist of a student-based method of setting performance standards.  More specifically, in Phase III, faculty teaching remedial, regular, and advanced courses in English and math, will be asked to classify their students into three broad performance categories of average, below average, or above average.  For both the mathematics and English area, a minimum of three courses will be targeted such that students enrolled in remedial, regular, and advanced courses will be required to complete the Accuplacer test. 
As such, students placed into a remedial math course who are classified as below average by their instructor, would be expected to score poorly on the college level math test.  Likewise, students enrolled in an advanced mathematics course who are classified as above average, would be expected to perform well on the college level math test.  The score distributions for each of the performance categories will be compared to determine the most appropriate point at which to place the cut scores.

21. Modified Contrasting Group Method
Panelists completed the placement test role playing as a student with average and above average ability
Score distributions were plotted to represent the panelists’ actual (obtained) scores
The distribution plot was visually analyzed to help guide the establishment of an appropriate cut score.
Phase III will consist of a student-based method of setting performance standards.  More specifically, in Phase III, faculty teaching remedial, regular, and advanced courses in English and math, will be asked to classify their students into three broad performance categories of average, below average, or above average.  For both the mathematics and English area, a minimum of three courses will be targeted such that students enrolled in remedial, regular, and advanced courses will be required to complete the Accuplacer test. 
As such, students placed into a remedial math course who are classified as below average by their instructor, would be expected to score poorly on the college level math test.  Likewise, students enrolled in an advanced mathematics course who are classified as above average, would be expected to perform well on the college level math test.  The score distributions for each of the performance categories will be compared to determine the most appropriate point at which to place the cut scores.

22. Multi-Step Approach to Establish Cut Scores
Panelists created performance level descriptions (PLDs) and mapped these PLDs to College Board proficiency level statements  
Panelists reviewed student score distributions
Panelists completed placement test and reviewed their score distributions.
Additionally, for English, panelists retrospectively categorized papers into 2 groups (remediation required vs. no remediation required) and used this information to guide cut score establishment.
     

23. Three Day Panel Sessions
Panels met for 3 hour sessions on 3 consecutive days.
Math Panel: 12 faculty
English Panel: 12 faculty

24. Panel Session I Panel Session 1:
Identified the number of cut scores needed and the different courses into which students would be placed
Developed clear and concise performance level descriptions for each placement level
three three-hour sessions during the week of March 3–7, 2008
Identified the number of cut scores needed and the different courses into which students would be placed
i.e. remedial, regular, or advanced for math, and remedial or regular for English. They also developed clear and concise performance level descriptions for each placement level

25. Characteristics of Performance Level Descriptions
Written in positive terms
Parallel in organization, language, and style
Written in clear and concise language without using unmeasurable qualifiers (e.g., often, seldom, etc.)

26. Example of Worksheet

27. Writing Performance Level Description
FOCUS
maintains consistent point of view/perspective and
clearly communicates main point and unity of purpose
ORGANIZATION
demonstrates understanding of primary elements of the essay (introduction, body, conclusion), develops effective paragraphs,
and uses transitions correctly and effectively sequences ideas
DEVELOPMENT AND SUPPORT
develops complete thoughts using appropriate vocabulary and
contains relevant evidence and supporting details
SENTENCE STRUCTURE
demonstrates understanding of basic grammar and varied syntax
MECHANICS
demonstrates understanding of basic mechanical conventions
such as spelling, punctuation, capitalization, and word choice

28. Math Performance Level Descriptions
Perform addition, subtraction, multiplication and division of polynomials; identify factors, terms, constants, variables, exponents, and coefficients; and recognize graphs of polynomial functions.
MATH 110 or 111
Perform addition, subtraction, multiplication and division on algebraic expressions involving integer exponents and radicals, solve linear and quadratic equations, evaluate, graph, and find the domain and range of functions including linear and quadratic functions.
MATH 131
Perform addition, subtraction, multiplication, division, factoring, composition and the simplification of algebraic, trigonometric, exponential and logarithmic expressions and functions; use algebraic skills to solve linear, quadratic, rational, and absolute value equations and inequalities; and evaluate, graph and interpret functions as well as their transformations.

29. Panel Session II Panel Session 2: PLDs Approved
Initial round of cut score setting based upon the results of student testing and faculty role play performance
For English, student work categorized by panelists into 2 groups.
Initial round of cut score setting based upon the results of student testing conducted on February 27, 2008
Students were classified into three broad performance categories of average, below average, or above average and administered the placement tests. Panelists were also required to complete the placement test a minimum of two times (role playing as a student with poor and above average ability).

30. Illustration of Score Distributions
Students scoring 31 and less than 42 would qualify for placement in Math 101. Students scoring 42 and less than 65 would qualify for placement in Math 110 or Math 111. Students scoring 65 or higher would qualify for placement in Math 131.

31. Illustration of Score Distributions
2nd Cut Score = 42 Students scoring from 30 to 41 – Math 101
1st Cut Score = 30 Students scoring less than 30 – Remedial Placement
3rd Cut Score = 65 Students scoring from 42 to 64 – Math 110 or 111
Students scoring 65 or higher – Math 131
Students scoring less than 30 remedial placement. Students scoring 30 and less than 42 would qualify for placement in Math 101. Students scoring 42 and less than 65 would qualify for placement in Math 110 or Math 111. Students scoring 65 or higher would qualify for placement in Math 131.

32. English Panelist Rating Information
Mean SD
Remedial Indicated by Two or Fewer Panelists
7.85
1.37
Remedial Indicated by Three or More Panelists
6.90
1.07
4 raters per paper

33. English Panelist Rating Information
Mean SD
Remedial Indicated by None or Some Panelists
7.61
1.28
Remedial Indicated by All Panelists
5.0
1.01
4 raters per paper

34. Panel Session III Panel Session 3:
Cut Scores were aligned across various methods used.
Initial round of cut score setting based upon the results of student testing conducted on February 27, 2008
Students were classified into three broad performance categories of average, below average, or above average and administered the placement tests. Panelists were also required to complete the placement test a minimum of two times (role playing as a student with poor and above average ability).

35. PLDs Mapped To Proficiency Statements
MATH 101
PLD: Perform addition, subtraction, multiplication and division of polynomials; identify factors, terms, constants, variables, exponents, and coefficients; and recognize graphs of polynomial functions.
Lower than ACCUPLACER 40
Cut Score 30

36. Alignment of Recommendations
Performance level descriptions written by
NC A & T faulty were mapped to College Board proficiency level statements  
Review score distributions for student performance groups to determine cut score
Cut scores indicated on the basis of the student data were considered in conjunction with performance level descriptions.     
In Phase IV, the data available from Phase II and III would be compared to obtain the best possible alignment for cut score recommendations.  As such, the performance level descriptions detailed by the panelists should coincide with the proficiency level statements provided by the College Board.  Moreover, the cut scores indicated on the basis of the student data should be consistent as well.     After discussion and consensus by the panelist, the final cut scores recommendations will be prepared.

37. Example of Alignment Process
Based on mapping to Accuplacer Proficiency Statements. Cut score should fall between 40 & 63.
Math 110 or 111

38. Follow-Up Testing and Cut-Score Refinement
Additional Students Tested
Cut Scores Revised
Recommendations Made By Panelists
A second round of student testing was conducted on March 12, Student performance results were used by panels in wrap-up sessions to refine and adjust their previous cut score recommendations.
The English panel recommended a cut score of 6. Students scoring 6 or less would be placed in a remedial first English course. Students scoring 7 or above would be placed in a regular first English course.
_______________________X_________________________
The mathematics panel recommended three cut scores. Students scoring 30 or less would be placed in a remedial first math course. Students scoring 31 and less than 42 would qualify for placement in Math 101. Students scoring 42 and less than 65 would qualify for placement in Math 110 or Math 111. Students scoring 65 or higher would qualify for placement in Math 131. The panel recommended that students should have the option to register for a non-remedial first math course lower than their qualifying level.
___________X_____X___________X_____________________

39. Project Timeline after Study
March, 2008
Committee review and recommendations
Executive review and policy
April, 2008
Develop student score validation and upload procedure
May, 2008
Train proctors and faculty advisors
Final system test
June
Go live with Accuplacer placement testing
500 students per week
July, 2008
Process evaluation and recommendations

40. Campus Collaboration Admissions – New students
Student Affairs – New student orientation
Registrar – Validate student data
IT – Upload student data to campus system
Academic Affairs - Proctors and advisors
Institutional Research – Management and support

41. Accuplacer Support
Helpline
Online procedure guides

42. Early Results Five (5) orientation sessions in June
Overall, the system works
Some Accuplacer interface weaknesses

43. WHERE TO FROM HERE Develop alternative placement measures
Implement off-site testing

44. Thank you for attending!
Questions?
NCA&T Support Site

45. Proficiency Statements WritePlacer Plus
Total Right Score of About 40 or Less
These students should take the Elementary Algebra test before any placement decisions are finalized.
Total Right Score of About 40
Students scoring at this level can:
identify common factors
factor binomials and trinomials
manipulate factors to simplify complex fractions
These students should be considered for placement into intermediate algebra. For further guidance in placement, have these students take the Elementary Algebra test.
Total Right Score of About 63
Students scoring at this level can demonstrate the following additional skills:
work with algebraic expressions involving real number exponents
factor polynomial expressions
simplify and perform arithmetic operations with rational expressions, including complex fractions
solve and graph linear equations and inequalities
solve absolute value equations
solve quadratic equations by factoring
graph simple parabolas
understand function notation, such as determining the value of a function for a specific number in the domain
have a limited understanding of the concept of function on a more sophisticated level, such as determining the value of the composition of two functions
have a rudimentary understanding of coordinate geometry and trigonometry
These students should be considered for placement into college algebra or a credit-bearing course immediately preceding calculus.
Total Right Score of About 86
understand polynomial functions
evaluate and simplify expressions involving functional notation, including composition of functions
solve simple equations involving:
• trigonometric functions
• logarithmic functions
• exponential functions
These students can be considered for a precalculus course or a nonrigorous course in beginning calculus.
Total Right Score of About 103
perform algebraic operations and solve equations with complex numbers
understand the relationship between exponents and logarithms and the rules that govern the manipulation of logarithms and exponents
understand trigonometric functions and their inverses
solve trigonometric equations
manipulate trigonometric identities
solve right-triangle problems
recognize graphic properties of functions such as absolute value, quadratic, and logarithmic
These students should be considered for placement into calculus.

46. ACCUPLACER – Math Proficiency Statements
Total Right Score of About 40
Students scoring at this level can identify common factors, factor binomials and trinomials, manipulate factors to simplify complex fractions
Total Right Score of About 63
Students scoring at this level can demonstrate the following additional skills: work with algebraic expressions involving real number exponents, factor polynomial expressions, simplify and perform arithmetic operations with rational expressions, including complex fractions, solve and graph linear equations and inequalities, solve absolute value equations, solve quadratic equations by factoring, graph simple parabolas, understand function notation, such as determining the value of a function for a specific number in the domain , have a limited understanding of the concept of function on a more sophisticated level, such as determining the value of the composition of two functions, have a rudimentary understanding of coordinate geometry and trigonometry
Total Right Score of About 86
Students scoring at this level can demonstrate the following additional skills: understand polynomial functions, evaluate and simplify expressions involving functional notation, including composition of functions, solve simple equations involving: trigonometric functions, logarithmic functions, exponential functions
Total Right Score of About 103
Students scoring at this level can demonstrate the following additional skills: perform algebraic operations and solve equations with complex numbers, understand the relationship between exponents and logarithms and the rules that govern the manipulation of logarithms and exponents, understand trigonometric functions and their inverses, solve trigonometric equations, manipulate trigonometric identities, solve right-triangle problems, recognize graphic properties of functions such as absolute value, quadratic, and logarithmic
Total Right Score of About 40 or Less
These students should take the Elementary Algebra test before any placement decisions are finalized.
Total Right Score of About 40
Students scoring at this level can:
identify common factors
factor binomials and trinomials
manipulate factors to simplify complex fractions
These students should be considered for placement into intermediate algebra. For further guidance in placement, have these students take the Elementary Algebra test.
Total Right Score of About 63
Students scoring at this level can demonstrate the following additional skills:
work with algebraic expressions involving real number exponents
factor polynomial expressions
simplify and perform arithmetic operations with rational expressions, including complex fractions
solve and graph linear equations and inequalities
solve absolute value equations
solve quadratic equations by factoring
graph simple parabolas
understand function notation, such as determining the value of a function for a specific number in the domain
have a limited understanding of the concept of function on a more sophisticated level, such as determining the value of the composition of two functions
have a rudimentary understanding of coordinate geometry and trigonometry
These students should be considered for placement into college algebra or a credit-bearing course immediately preceding calculus.
Total Right Score of About 86
understand polynomial functions
evaluate and simplify expressions involving functional notation, including composition of functions
solve simple equations involving:
• trigonometric functions
• logarithmic functions
• exponential functions
These students can be considered for a precalculus course or a nonrigorous course in beginning calculus.
Total Right Score of About 103
perform algebraic operations and solve equations with complex numbers
understand the relationship between exponents and logarithms and the rules that govern the manipulation of logarithms and exponents
understand trigonometric functions and their inverses
solve trigonometric equations
manipulate trigonometric identities
solve right-triangle problems
recognize graphic properties of functions such as absolute value, quadratic, and logarithmic
These students should be considered for placement into calculus.