CCSSO Criteria for High-Quality Assessments Technical Issues and Practical Application of Assessment Quality Criteria

Background Test Validity Madaus call for Test Monitoring Board Unified Concept of Test Validity AERA/APA/NCME Standards for Educational and Psychological Testing Peer-Review Criteria for High-Quality Assessments

Technical Quality Indicate Progress toward College & Career Readiness Valid for Required and Intended Purposes Ensuring Reliability Valid and Consistent Test Score Interpretations within and across Years Accessibility to ALL students, including English Learners and Students with Disabilities Transparency of Test Design and Expectations Meeting Requirements for Data Privacy and Ownership

Progress Toward College & Career Readiness Description of process for developing performance level descriptors and setting performance standards, including: – Involvement of higher education & career experts – Use of external evidence to inform standards – Evidence that external benchmarks valid for intended purpose Description of studies to be conducted to evaluate validity of standards over time

Valid for Required and Intended Purposes Well-articulated validity evaluation that focuses on: – the validity of test score uses – scoring and reporting structures consistent with structure of state standards – total test and sub-scores are related to external variables as expected – assessments lead to intended outcomes – content validity of test forms and usefulness of score reports

Ensuring Reliability Reliability of test scores for total population and reported sub-populations Precision of scores at cut points and consistency of student classifications Generalizability for relevant sources – Variability within and across groups – Variability among schools – Consistency across test forms – Consistency across rater scores

Consistency in Score Interpretations Within and Across Years Assessment Forms – Comparability within years – Linking across years – Consistency in meaning across achievement level Score Scales – Method used to transform raw to scale score coherent with test design and intended claims – Scaling procedures support intended interpretations – Evidence supports validity of vertical scales

Accessibility for ALL Students Principles of Universal Design – Description of item development process used to reduce construct irrelevance – Sample items and interfaces that reflect principles of Universal Design Appropriate Accommodations – Accessibility features – Access to translations and definitions – Construct validity of accessibility features Evidence that the test items and accessibility features permit ELL and SWD to demonstrate their knowledge, skills, and abilities

Transparency of Test Design Test blue prints demonstrate range of standards covered Release plan yields representative sample of items on regular basis and across grades Sample items with annotations and scoring rubrics available Item development specifications available

Data Privacy Assurance of student privacy protection that complies with all federal and state requirements Assurance of state ownership of all data State is provided all underlying data in timely and usable manner to support secondary state analyses Description of secure data management procedures

Challenges to Implementation Validity and Reliability are not dichotomous and acceptable levels vary based on test use – Guidance on what is acceptable levels for different uses Timing of availability of evidence – New Programs may only have descriptions of plans – Existing Programs have student data to perform analyses Objectivity, Transparency, and Reliability of Quality Review

Proposed Phased Approach Phase I: Content and Test Design – Focus on alignment with standards, item quality, and accessibility features – Occurs after initial item development and test form construction Phase II: Test Characteristics – Focus on validity, reliability, and generalizability – Occurs after field testing and/or first operational use Phase III: Program Implementation – Focus on Test Administration, Reporting, and Test/Item Pool Maintenance – Occurs after second or third year of operational administration