1. Item pool optimization for adaptive testing
Marty McCall
Director of Psychometrics, Smarter Balanced
National Conference on Student Assessment
June, Philadelphia, PA

2. Construct modeling and test design
Content specifications derived from CCSS
Assessment standards modeled on instructional standards
Grade-to-grade progressions modeled
Test structure follows content specs
Item specifications address specific content spec elements
Test blueprints derived from
content specifications
Depth and balance
Time/cost constraints

3. Test Blueprint What do you want every student to get?
Content – categories and proportions
Cognitive characteristics
Item types
How many items in each test event?
What are you going to report? For individuals? For groups?
Overall scores
Sub-scores
Achievement category

4. Adaptive test: Item pool + test blueprint + algorithm
Constrained CAT
Test blueprint is a design for the student test event. It specifies the content for every student taking the test
Item pool distributed to supply tests at every score level
The algorithm uses the item pool to find the most informative test for each student within blueprint constraints

5. Pool design Reckase (2007) – P-optimal pool evaluation.
“It’s unrealistic to expect that every value of theta will have a maximally informative item.”
p-optimal means assuring an item within a specified proportion of full optimality for each item requested by the algorithm.
Theta-scale is represented as adjacent “bins”
Width of bins determined by level of optimality for desired SEM

6. Smarter Adaptive Test Blueprint
Hierarchical relationships
Claims
ELA-1. Reading, 2. Writing, 3. List/Spk, 4. Research
Math-1. Concepts/Procedures, 2. Problem Solving, Communicating Reasoning, 4. Modeling/dData Analysis
Target Clusters
Targets
Global rules
DOK
item type
passage rules(ELA)
Claims consistent across grades, targets consistent across grades in ELA, vary in math

7. Mathematics, Grade 5 Blueprint Target Specifications

8. Mathematics, Grade 5 Blueprint Target Specifications

9. Examples of non-hierarchical rules
In grades 6-8, up to one CAT item per student may require hand-scoring (from either Claim 3 or Claim 4
Claim 2 (Problem Solving) and Claim 4 (Modeling and Data Analysis) have been combined .... There are still four claims, but only three claim scores will be reported with the overall math score.
DOK: The CAT algorithm will be configured to ensure the following:
For Claim 1, each student will receive at least 7 CAT items at DOK 2 or higher.
For combined Claims 2 and 4, each student will receive at least 2 CAT items at DOK 3 or higher.
For Claim 3, each student will receive at least 2 CAT items at DOK 3 or higher.

10. Item pool specification sheet; aka The Beast

11. Mathematics, grade 5 Claim 1
Beast Detail
Mathematics, grade 5 Claim 1
Total N broken out by the number of bins

12. Bins
Bin boundaries are artificial (Van der Linden method continuous, but similar)
Item writers can’t predict at the fine levels required by bins
They can respond to student levels
Made estimates of easy, medium and hard
In general, underestimated item difficulty, which has caused us to examine scale score characteristics through item mapping

13. Expanded pools Pools are evaluated by grade level
For students at high or low extremes, the pool expands to adjacent grade levels to provide better measurement
After about 2/3 of test has been administered
For students who are far above or below standard
Pool includes items from adjacent grades that content experts have judged to be measuring the grade level target
Items with the most information are chosen from expanded pool

14. Pool evaluation and maintenance
Uses the bin method to estimate needs
Compares needs to operational pool
Gives recommendations for item writing, item retirement, etc
Data structure connected to the Item database for constant updating
Clear and practical method

15. Thank you for your attention
Questions?