Standard Setting Different names for the same thing Standard Passing Score Cut Score Cutoff Score Mastery Level Bench Mark

Standards exist for as long as there are tests Canada: 50 China: 60 America: 70 France: 2/3

Examinee-Centered Methods: Borderline Group Method Contrasting Groups Method Problem: How to identify the examinees. Test-Centered Methods: Nedelsky Method (1954) Angoff Method (1971) IDEA Method (2004) Problem: Judgmental errors.

Test-Centered Methods: Conceptualization of examinee competency Judgmental item analysis Aggregating item difficulty estimates

Training of Judges: Conceptualization of minimum competency The use of a standard-setting method Intrajudge inconsistency Iterative process Documentation

What do experts say? "We have come to realize that there is no objectively correct way to set standards. But we have also come to realize that there is nothing wrong with using judgments appropriately." (Zieky, 1995, p.5) "Determination of a minimum acceptable performance always involves some rather arbitrary and not wholly satisfactory decisions." (Ebel, 1972, p.492)

"Researchers agree with Glass (1978) that standards are all arbitrary. But they reject the notion that being arbitrary, in the sense of being judgmental, is a fatal flaw." (Zieky, 1994, p28) "If competence is a continuous variable, there is clearly no point on the continuum that would separate students into the competent and the incompetent. A right answer does not exist, except perhaps in the minds of those providing the judgments." (Jaeger, 1989, p.492)

1. All standard-setting is judgmental 2. Standard-setting leads to errors of classification 3. Standard-setting is and will remain controversial 4. There is no purely absolute standard. 5. There is no one right method 6. Choosing judges is more important than choosing methods 7. Standard-setting is a process

Failed Standard-Setting Exercises Due to Legal Matters Tenured teachers can not be decertified. Contracted teachers can not be decertified. Candidates for becoming teachers can be decertified Due to Psychometric Matters Practice analysis failed to support job-relatedness of the test. Teachers’ concern about the objective of the test was not addressed. Items did not reflect judgment of content committee. Teachers were excluded from the standard-setting processes. Changing cutscore without justifications.

Defensible Standard Setting Steps Subject matter experts are asked to review test items. Sensitivity review to check biases against certain groups. Documentation of the standard setting processes Description of subject matter experts Selection criteria and procedures Standard-setting methods and justifications Training procedures Independent evidence that the cutscore is “reasonable.” Any indices of reliability, item analysis information, distractor analysis Intrajudge and interjudge consistency evidence, e.g., split half

Steps and Procedures in Developing an Achievement Test 1. Define domain content Intelligence tests and theories,20% Personality tests,20% Item characteristics,10% Reliability,20% Validity,15% Test development,15% Table of specifications HighLowTotal Intelligence tests and theories5%15%20% Personality tests,5%15%20% Item characteristics,5% 10% Reliability,15%5%20% Validity,10%5%15% Test development,5%10%15%

3. Item Analysis Item variance Item difficulty Item discrimination 4. Setting Standards 5. Test reliability Test-retest Parallel form Split half-internal consistency 6. Test Validity various validity issues 2. Write Items Normally done by subject teachers.

"Anyone who expects to discover the "real" passing score by any of these approaches, or any other approach, is doomed to disappointment, for a "real" passing score does not exist to be discovered. All any examining authority that must set passing scores can hope for, and all any of their examinees can ask, is that the basis for defining the passing score be defined clearly, and that the definition be as rational as possible." (Ebel, 1972, p.496) "At a minimum, standard-setting procedures should include a balancing of absolute judgments and direct attention to passing rates." (Shepard, 1980, p.463)

Uses of Standards Exhortation To exhort, encourage or urge the students, schools, and public to exert more or different kinds of effort to achieve established standards of performance. Exemplification of Goals To provide clear specifications of the achievement levels that students are expected to attain. Accountability In the U.S., schools not students can obtain rewards or sanctions depending on the progress achieved in meeting performance standards. Certification of Achievement and Mastery

"The thing that hath been, it is that which shall be; and that which is done is that which shall be done: and there is no new thing under the sun." Ecclesiastes (1:9).