1. Beyond Peer Review: Developing and Validating 21st-Century Assessment Systems Is it time for an audit? Thanos Patelis Center for Assessment Presentation at the National Conference on Student Assessment New Orleans, June 25, 2014

2. Overview What is being reviewed? What is being reviewed? Assessment Quality Audits Assessment Quality Audits What are the criteria & how? What are the criteria & how? Requirements: Requirements: Theory of Action & Evidence Theory of Action & Evidence Sound methodology implemented by experts with transparency and external review Sound methodology implemented by experts with transparency and external review Recap Recap 2 Patelis – NCSA - 06/25/2014

3. 3 The need today for a mechanism to assure the quality of tests and their valid use flows, therefore, out of the tension between the critical consequences of present high-stakes tests for test takers and institutions and the public’s need for the protection testing affords. George Madaus, 1992, p.26 Patelis – NCSA - 06/25/2014

4. 4 The need for some sort of social intervention to monitor, audit, or review high-stakes testing programs is critical because the demands of test validation often clash with the commercial nature of applied testing. George Madaus, 1992, p. 27 Patelis – NCSA - 06/25/2014

5. What is being reviewed? 5 Patelis – NCSA - 06/25/2014 The reviews are of the assessment system. The reviews are of the assessment system. While assessment systems are components of other systems, these other systems will include other criteria and evidence. While assessment systems are components of other systems, these other systems will include other criteria and evidence. Accountability systems Accountability systems Educational evaluation systems Educational evaluation systems Because they are assessment systems, the review must go beyond the technical aspects of the instrument. Because they are assessment systems, the review must go beyond the technical aspects of the instrument. Context Context Purpose and use Purpose and use Score reporting Score reporting Data systems Data systems Operational features that interact with the assessment Operational features that interact with the assessment

6. Assessment Quality Audits 6 Patelis – NCSA - 06/25/2014 Peer Review: “To determine whether States have met NCLB standards and assessments requirements, the U.S. Department of Education (ED) will use a peer review process involving experts in the fields of standards and assessments. The review will evaluate States' assessment systems only against NCLB requirements.” USDOE, 2007, p. 7

7. Assessment Quality Audits The idea of having experts in the fields of standards and assessments is sound. The idea of having experts in the fields of standards and assessments is sound. USDOE, by only looking at compliance with the law, is missing the opportunity to implement a full review of the quality of assessments. USDOE, by only looking at compliance with the law, is missing the opportunity to implement a full review of the quality of assessments. While the USDOE should require quality, it is not in the best position to do it. While the USDOE should require quality, it is not in the best position to do it. Similarly, federal granting agencies require independent evaluators to evaluate the initiatives that they fund. But, they do not do the evaluations! Similarly, federal granting agencies require independent evaluators to evaluate the initiatives that they fund. But, they do not do the evaluations! 7 Patelis – NCSA - 06/25/2014

8. Criteria and Methodology 8 Patelis – NCSA - 06/25/2014 CCSSO published a set of assessment criteria “for high-quality state summative assessments aligned to college- and career-readiness standards” CCSSO published a set of assessment criteria “for high-quality state summative assessments aligned to college- and career-readiness standards” (CCSSO, 2013, p. 1)

9. Overview of Assessment Criteria 9 Patelis – NCSA - 06/25/2014 The criteria bring together and utilize The criteria bring together and utilize Standards of Educational and Psychological Testing (1999), Standards of Educational and Psychological Testing (1999), the Operational Best Practices for Statewide Large-Scale Assessment Programs (2010), the Operational Best Practices for Statewide Large-Scale Assessment Programs (2010), and a variety of recommendations of quality that implicitly allude to quality of data as seen from the Data Quality Campaign and principles of score reporting (Zenisky & Hambleton, 2012). and a variety of recommendations of quality that implicitly allude to quality of data as seen from the Data Quality Campaign and principles of score reporting (Zenisky & Hambleton, 2012). Examples of undertaking full reviews or audits of assessments programs can be found and are currently done. Examples of undertaking full reviews or audits of assessments programs can be found and are currently done. Buros Center, ETS. Buros Center, ETS. The operationalization of quality criteria is not trivial. The operationalization of quality criteria is not trivial. Examining a toolkit provided by Achieve partnering with CCSSSO and Student Achievement Partners recently for evaluating alignment of instructional and assessment material to the Common Core State Standards illustrates the hard work and expertise that is needed to implement the reviews effectively, fairly and truthfully. Examining a toolkit provided by Achieve partnering with CCSSSO and Student Achievement Partners recently for evaluating alignment of instructional and assessment material to the Common Core State Standards illustrates the hard work and expertise that is needed to implement the reviews effectively, fairly and truthfully.

10. Overview of Assessment Criteria 10 Patelis – NCSA - 06/25/2014 A. Meet overall assessment goals and ensure technical quality (7) B. Align to standards – English language arts/literacy (9) C. Align to standards – Mathematics (5) D. Yield valuable reports on student progress and performance (2) E. Adhere to best practices in test administration (1) F. State specific criteria (varies) Samples: Samples: Linkage to higher education Linkage to higher education Articulated assessments Articulated assessments Interoperability Interoperability

11. Requirements 11 Patelis – NCSA - 06/25/2014 Assessment Program: Assessment Program: Theory of action Theory of action Evidence Evidence Reviewers: Reviewers: Criteria Criteria Operationalization of the criteria Operationalization of the criteria Description Description Evidence needed Evidence needed Exemplars Exemplars Rubric Rubric Rating collection forms Rating collection forms Transparency of methodology including multiple external reviews Transparency of methodology including multiple external reviews Review panels Review panels Publication and dissemination Publication and dissemination

12. Recap 12 Patelis – NCSA - 06/25/2014 As new, important assessments are being launched, there is a heightened awareness for evaluating and ensuring quality of these assessments and all their components. As new, important assessments are being launched, there is a heightened awareness for evaluating and ensuring quality of these assessments and all their components. The USDOE is not in a position to do this, but should require that assessments show evidence of quality and be evaluated by an independent party. The USDOE is not in a position to do this, but should require that assessments show evidence of quality and be evaluated by an independent party. Substantial effort is needed to operationalize the criteria that have been offered by CCSSO. Substantial effort is needed to operationalize the criteria that have been offered by CCSSO. These criteria go beyond the technical quality of the instrument. These criteria go beyond the technical quality of the instrument. The methodology developed must realize the nature of the criterion. The methodology developed must realize the nature of the criterion. The same old methodology will not work. The same old methodology will not work.

13. References AERA, APA, NCME (1999). Standards for educational and psychological testing. Washington, DC: AERA. Council of Chief State School Officers & Association of Test Publishers (2010). Operational best practices for statewide large-sale assessment programs. Washington, DC: Council of Chief State School Officers. [ISBN: 978-1453716281] Council of Chief State School Officers (2013). Criteria for procuring and evaluating high-quality assessments. Washington, DC: Author. http://www.ccsso.org/Documents/2014/CCSSO%20Criteria%20for%20High%20Quality%20Assessments%2003242014.pdf http://www.ccsso.org/Documents/2014/CCSSO%20Criteria%20for%20High%20Quality%20Assessments%2003242014.pdf Madaus, G. F. (1992). An independent auditing mechanism for testing. Educational Issues: Measurement and Practice, 11(1), 26-31. Madaus, G. F. (1992). An independent auditing mechanism for testing. Educational Issues: Measurement and Practice, 11(1), 26-31. DOI: 10.1111/j.1745-3992.1992.tb00225.x Student Achievement Partners, CCSSO, Achieve (March, 2014). Toolkit for evaluating alignment of instructional and assessment materials to the Common Core State Standards. Washington, DC: Achieve. http://www.achieve.org/files/CompleteToolkit_March2014_highRes.pdf http://www.achieve.org/files/CompleteToolkit_March2014_highRes.pdf U.S. Department of Education (2009). Standards and assessment peer review guidance: Information and examples for meeting requirements of the No Child Left Behind Act of 2001. Washington, DC: Author. Zenisky, A. L. & Hambleton, R. K. (2012). Developing test score reports that work: The process and best practices for effective communication. Educational Measurement: Issues and Practice, 31(2), 21-26. 13 Patelis – NCSA - 06/25/2014

14. Questions and Comments? Center for Assessment www.nciea.org Thanos Patelis tpatelis@nciea.org 14 Patelis – NCSA - 06/25/2014