1. Considerations of Content Alignment in CAT
Steven L. Wise, Northwest Evaluation Association
G. Gage Kingsbury, Independent Consultant
CCSSO 2016 National Conference on Student Assessment, Philadelphia

2. The problem for the day
How do we know that our adaptive test covers the content that it should cover for each student?

3. Do current alignment practices fit CAT?
CAT is individualized, creating an assessment for each student
Current content alignment practices
Are designed for fixed-form tests
Which commonly aim at the average student
And use grade specific content standards
So a new approach is needed, that takes into account the complexity of an adaptive test, and the overall purpose of the test

4. Do current alignment practices fit education?
Not really –
Within any grade level, students have a wide variety of achievement
The mission for any educational agency is to help every student learn
So a test that fits nominal content standards for a grade level will fit many students poorly.

5. So the new model
Needs to take into account the purpose of the adaptive test
Needs to understand that each student is important and may need different content
A new model is needed
That includes content appropriate to each student
That includes the purpose of the assessment
That is pragmatic for an adaptive test

6. Alignment in CAT
Alignment: to what degree does a test event cover the desired content?
The interpretation of desired content, however, depends largely on the purpose of the test. What inferences are to be made on the basis of the test results?
Understanding the test’s purpose is crucial to understanding how alignment should be evaluated.

7. Two General Types of Score Inferences
Standards-focused. Inferences are to be made about the degree to which students achieve relative to a predetermined set of learning expectations. These are commonly used in accountability systems. Often, the focus is on the aggregated scores from a classroom, grade level, or school.
Student-focused. Inferences are to be made about what individual students know and can do in a particular subject area. This assumes that there is a learning continuum, and that the purpose of testing is to identify each student’s position on that continuum.

8. Implications about Alignment
These two types of score inferences require different interpretations about alignment.
The first inference prioritizes item content related to a grade-level specific set of content standards
The second inference prioritizes content and item difficulty appropriate to the particular student.
Thus, any evaluation of alignment should be guided by the type of score inferences that are to be made.

9. Evaluating Alignment
First step is for the test owner to develop an alignment argument for the test
What inferences are to be made?
What content domain is to be addressed by the test?
Other specific requirements the test owner feels are important to be represented in a test event.
Examples: specifications of (or restrictions on) the learning standards to be addressed, content goals/sub-goals, item complexity, depth of knowledge, item types, etc.

10. Three Evaluation Components
Reasonableness of the alignment argument should be assessed.
Item pool should be examined to evaluate how capable it is to provide test events congruent with the alignment argument.
A set of test events should be examined to assess the degree to which the specifications outlined in the alignment argument have been satisfied.

11. So what does this mean? Need alignment approach that is adoptable
So that peer review in ESSA works with an alignment model that works for CAT
We need to make this argument that the approach is appropriate
Why a CAT?
Naturally compatible with the needs of education, and the needs of each student
How do we make this practical?

12. Our Paper
A detailed description of a model for the evaluation of content alignment for an adaptive test is seen in
Wise, S. L., Kingsbury, G. G., & Webb, N. L. (2015). Evaluating content alignment in computerized adaptive testing. Educational Measurement: Issues and Practice, 34(4),