C R E S S T / CU University of Colorado at Boulder National Center for Research on Evaluation, Standards, and Student Testing Design Principles for Assessment and Accountability Systems Robert L. Linn AERA Symposium: Designing Ideal Assessment and Accountability Systems New Orleans, LA April 25, 2000

C R E S S T / CU Sources of Principles  Experience with Previous Assessment and Accountability Systems  Successes  Unintended Negative Effects  Test Standards  CRESST Model

C R E S S T / CU Validity, Fairness, Credibility, Utility Are Underlying Principles

C R E S S T / CU Identification of Intended Uses  Validity is specific to particular interpretations and uses of assessment results  Standard 1.1: “A rationale should be presented for each recommended interpretation and use of test scores, together with a comprehensive summary of the evidence and theory bearing on the intended use or interpretation” (p. 17).

C R E S S T / CU Education Broader than Assessment and Accountability  Potentially useful tools if supported by  Curriculum  Professional Development  Resources

C R E S S T / CU Symmetry  Educator Accountability  Teachers  Administrators  Student Accountability  Consequences  Opportunity  Policymaker and Public Accountability

C R E S S T / CU Develop standards, then assessments  How standards are formulated matters  How standards are assessed makes a difference.  WYTIWYG (What You Test Is What You Get) premise.

C R E S S T / CU Alignment  Content Standards  Curriculum  Instruction  Assessment

C R E S S T / CU Provide the resources and professional development to teachers required for students to meet the high expectations  Tests should be used for high stakes decisions about individuals only after they have been taught the knowledge and skills on which they will be tested. (Hauser & Heubert, High Stakes: Testing for Tracking, Promotion, and Graduation, National Academy of Sciences Press, 1998)

C R E S S T / CU Frames of Reference  Content and Performance Standards  Status  Progress  Normative Comparisons

C R E S S T / CU Set standards that are high, but obtainable  Educational standards at the national, state, and district levels are often inconsistent.  Standards are being set that seem out of reach - at least in the near term.  Holding all students to the same standards will either lead to a lowering of standards or untenable retention and failure rates.

C R E S S T / CU Attend to Both Status and Progress  Absolute targets for performance maintain the focus on goals for all students  Acknowledgement of progress allows recognition of improvement and reduces discouraging effects of goals that currently seem unobtainable

C R E S S T / CU Place more emphasis on comparisons of performance from year to year than from school to school  Comparisons among schools immediately raise questions of fairness and whether fairness requires taking context into account.  Use new high-quality assessments each year that yield comparable scores to those of previous years.

C R E S S T / CU Track progress for subgroups of students as well as the total group  Title I requirements for disaggregation.  Ensures attention to all students.

C R E S S T / CU Include all students in testing except those with the most severe disabilities  Who’s included and excluded in testing can produce different results - accountability incentives to distort results.  Traditionally excluded students are often capable of participating in assessments.  Use accommodated assessments for students who have not yet transitioned into English language programs or whose disabilities require it.

C R E S S T / CU Use multiple measures to make important decisions  Tests have validity only in relation to specific purposes.  Tests are not perfect - and neither are the alternatives.  No high-stakes educational decision about a test taker should be made solely or automatically on the basis of a single test score; other relevant information should also be taken into account. (Hauser & Heubert, High Stakes: Testing for Tracking, Promotion, and Graduation, National Academy of Sciences Press, 1998)

C R E S S T / CU Standard 13.7: “In educational settings, a decision or characterization that will have a major impact on a student should not be made on the basis of a single test score. Other relevant information should be taken into account if it will enhance the overall validity of the decision” (p. 146).

C R E S S T / CU Evaluate and Report on the Accuracy of Results  Student Classification  School Classification  Standard 13.14: “In educational settings, score reports should be accompanied by a clear statement of the degree of measurement error associated with each score or classification level and information on how to interpret the scores” (p. 148).

C R E S S T / CU Evaluate and Report on the Impact of the Assessment and Accountability System  Intended Effects  Instruction  Student learning  Unintended Effects  Instruction  Effects on Students

C R E S S T / CU Evaluate both the intended positive effects and the unintended negative effects of the assessment and accountability system  Gains in scores to not necessarily signal improved learning and achievement.