1. Innovative Approaches for Examining Alignment
Validity in the Context of Computer Adaptive Testing
Patricia Reiss, Ph.D.
National Conference on Student Assessment (NCSA)
Tuesday, June 21, 2016

2. Alignment Context: Traditional Approach
Agenda
Alignment Context: Traditional Approach
Alignment Approaches to State Assessments in the Context of Computer Adaptive Testing
Alignment of Smarter Balanced Assessments by HumRRO
Simulation Evaluated Alignment Connection

3. Evidence Based on Test Content Evidence Based on Response Processes
Validity
The Standards of Educational and Psychological Testing (AERA, APA, and NCME, 2014) refer to four types of evidence:
Evidence Based on Test Content
Evidence Based on Response Processes
Evidence Based on Internal Structure
Evidence Based on Relations to Other Variables

4. Evidence Based on Test Content Alignment
A comparison of traditional alignment models can be found on the CCSSO website at:
ment_Analysis/Models/
Webb’s (1997, 2002) alignment model

5. Alignment The assessments
cover the full range of content specified in the State’s academic content standards.
measure both the content (what students know) and the process (what students can do) aspects of the academic content standards.
reflect the same degree and pattern of emphasis apparent in the academic content standards.
reflect the full range of cognitive complexity and level of difficulty of the content standards.
Yield results that represent all achievement levels
US Department of Education, Standards and Assessment Peer Review Guidance, Revised June 25, 2007.

6. From: Webb (July 25, 2005) TILSA Alignment Tool Dissemination Workshop

7. Webb’s Alignment Criteria
Categorical Concurrence
Depth of Knowledge Consistency
Range of Knowledge Correspondence
Balance of Representation and Source of Challenge

8. Alignment Approaches to State Assessments in the Context of Computer Adaptive Testing
Various approaches
Examination of alignment of blueprint and item pool
Examination of alignment of a sample of actually administered tests

9. Alignment
U.S. Department of Education Non-Regulatory Guidance, September 25, 2015.

10. HumRRO’s Alignment Study of the Smarter Balanced Assessments
Alignment Study Report posted at
tStudyReport.pdf
Comprehensive evaluation based on evidence-centered design of the summative assessment system

11. Connections Examined in the HumRRO Alignment Study of the Smarter Balanced Assessments
HumRRO (2016) Smarter Balanced Assessment Consortium Alignment Study Report

12. Alignment Connections Examined
Content Specifications and the CCSS (A)
CCSS and the Content Specifications (A)
Evidence Statements and Content Specifications (B)
Content Specifications and Test Blueprints (C)
Evidence Statements and Items (D)
Items/Performance Tasks and Content Specifications (G)

13. Alignment Criteria
Content Representation
DOK Distribution
DOK Consistency
Agreement between Reviewers’ and Content Specifications/Developers’ Ratings
Agreement among Reviewers’ Ratings

14. Simulated tests included both CAT and PT Evaluated 3 item pools
Alignment Connection (F) Simulation-Based Evaluation of the Smarter Balanced Summative Assessments by CRESST
content/uploads/2016/05/Simulation-based- Evaluation.pdf
Simulations conducted for both ELA/L and Mathematics in grades 3-8 and HS
Simulated tests included both CAT and PT
Evaluated 3 item pools
General
Spanish
Braille

15. Simulation Description
1000 simulated administrations
Evaluated blueprint match
Evaluated item exposure rate
Examined how well true proficiency scores were recovered, and
Examined precision of the score estimates

16. Alignment Connection F CRESST Findings
CAT engine
Good estimation of student proficiency
Low item exposure rates
CRESST results similar to AIR results
For PT – difficulty in fulfilling blueprint requirements related to the number of items in the PT section for each claim .

17. Questions?