1. Liru Zhang, Delaware DOE Shudong Wang, NWEA Presented at the 2015 NCSA Annual Conference, San Diego, CA 1

2.  At the heart of the Common Core State Standards (CCSS) in English language arts/Literacy (ELA/LIT) is the shift of instruction to center on text. The standards focus on the growing complexity of texts (or passages) and using evidence collected from texts to present analyses and well- defended claims.  To align the CCSS in K-12 assessments, students are expected to read and comprehend grade-appropriate passages across content categories in a variety of topics and respond a range of passage-dependent questions that require inferences based on the passage instead of questions that can be answered from prior knowledge and experiences (Coleman and Pimentel, 2012). 2

3.  With innovative technology, advantages of online testing, and encouragement of educational policy, the computerized-adaptive testing (CAT) has been greatly implemented in K-12 assessment.  In CAT, the adaptive process is typically based on item-level selection. The ultimate goals are to satisfy the test specifications, match provisional student ability, and control item exposure rate.  In reading comprehension, students are expected to read and comprehend grade-appropriate passages across a range of genres with a variety of topics. Some texts, such as in philosophy, literature or scientific research, may be more difficult than the others to comprehend because of knowledge, multiple resources, structure, and/or features.  3

4.  The current study is an investigation whether the item-level selection could achieve the content balance at both the item-level and the passage-level to align the CCSS for the passage-based reading comprehension.  According to the Standards for the Educational and Psychological Testing (1999), the study collected supporting evidence (e.g., in item selection procedure, item exposure rate) and validity evidence in parallel construct across individual tests to assure that content standards are adequately represented.  The CAT reading comprehension is Rasch-based with fix-length of 50 on- grade and off-grade items. Test results are reported on the vertical scale and in four performance levels. Students’ responses are collected from the grades 5 and 9 in the fall and spring operations. 4

5.  Item Pool Review and Evaluation: Two overlapping item pools, the initial pool used in the fall and the enhanced pool used in the spring, per grade. Item pool evaluation focused on the sufficiency to support the constraints in test specifications.  Empirical Analyses focused on the adequacy of validity in content balance and parallel construct across individual tests by student achievement level in, such as, number of passages and associated items per passage, content category, topic, gender, Lexile, and difficulty level of passages, uses of on- /off-grade items/passages, and conditional exposure and overlapping rates.  Expert Review focused on the content balance at the passage and the item- levels based on a randomly selected sample of 100 individual tests per grade per operation by student achievement level. 5

6. Grade Item Content CategoryPassage TypeItem Cognitive Level StandardMin.Max.%TypeMin.Max.%DOKMin.Max.% 5 2304060-80Inf.455010200-30 4102020-40Lit.455020400-60 Total8103060-10 On-Gr4050MC4950 Off-Gr010TE01 9 2304060-80Inf.677010200-30 4102020-40Lit.233020400-60 Total8103060-10 On-Gr4050 MC4950 Off-Gr010TE01 6

7. 7 Constraints Test Spec.Grade 5 N%Pool (%) Operation (N) MeanMin.-Max. Passage-Level8-107.1-8.96-13 Informational4-550523.7-5.02-5 Literary4-550483.5-3.92-8 Item-Level50 Std. 230-4060-8078-8036.4-37.231-40 Std. 410-2020-4020-2212.8-13.610-19 On-Gr 40-5080-10044-4843.3-43.839-50 Off-Gr 0-100-2052-566.2-6.70-11 MC49-5098-10049.7-49.948-50 TE0-10-20.1-0.30-2

8. 8 Constraints Test Spec.Grade 9 N%Pool (%) Operation MeanMin.-Max. Passage-Level8-107.2-8.06-10 Informational6-77070-764.1-4.83-7 Literary2-33024-303.1-3.21-5 Item-Level50 Std. 230-4060-807837.3-37.831-40 Std. 410-2020-402212.2-12.710-19 On-Gr 40-5080-10043-5244.839-50 Off-Gr 0-100-2048-575.20-11 MC49-5098-10049.6-49.944-50 TE0-10-20.1-0.40-6

9. 9 Content CategoryN. ItemsTypeGenderOn-GradeLexileLength Topic Animal7INY1020L815 Career6INY1250L825 Genre Biography1IFY880L632 Biography6INY1110L791 Legend-Folktale1LMY870L582 Realistic Fiction1LMN500L341 Realistic Fiction5LMY1130L668 Realistic Fiction1LFY740L981 Realistic Fiction1LFY800L898 Realistic Fiction6LFN570L984 Structure Pair8LMY890L112 Pair7IMY1290L681 Total12505/74/5/310/2CCSS: 830L-1010L

10. 10 ContentTopicN. ItemsTypeGenderOn-GradeLexileLength Topic Career6INY1250L825 Sports1IMY910L717 FormatHow-to-Do8INY970L679 Genre Poem6LNY1100L100 Realistic Fiction8LNN960L575 Realistic Fiction7LFY970L1011 Structure Pair6IMY1000L550 Pair8LMY890L1012 Total8504/41/3/47/1 CCSS: 830L-1010L

11. 11 ContentTopicN. ItemsTypeGenderOn-GradeLexileLength Topic Food8INN1290L609 History7IMY970L1135 Science7INY1320L1171 FormatHow-to-Do7INY1160L1588 Genre Realistic Fiction 8LFY1060L821 Realistic Fiction 2LMY1280L900 Structure Pair2LNN1610L991 Pair9LFY820L1031 Total8504/42/2/46/2 CCSS: 1050L- 1260L

12. 12 ContentTopicN. ItemsTypeGenderOn/OffLexileLength Topic Entertainment5INY1070L1188 Entertainment4INY1090L1130 Environment7INY1300L1048 Health8INY1290L761 Genre Biography6IMN1210L376 Realistic Fiction7LMY970L800 Realistic Fiction7LMY980L1136 Realistic Fiction6LFY930L Total8505/31/3/47/1CCSS: 1050L-1260L

13. 13  Compared with the test specifications, the content balance is satisfied at the item level in Standards 2 & 4 on the average as well as within the min./max. limitations in both grades. The proportion of on-grade and off- grade items is generally met in the fix-length test.  At the passage level, the total number of passages varies greatly from student to student, as shown in the sample test from 8 for an achievement level 4 to 12 for an achievement level 1. The proportion of the two types of passages, informational and literary, are failed to meet the target in operation.  In reading, passage and associated items are related with each other, but each has its unique coding category and evaluation system. To address all constraints and balance them in both levels is much more complicated to accomplish in reality than presumed. Otherwise, the compensation for the constraints at one level could be compromised at another level.

14. 14  When students repeatedly received reading passages from certain genres with similar topics or in the same format or structure, it not only limits the breadth of their exposure in reading, but also introduces bias in testing.  According to the content expert review, one pairing per test is desirable. This is because paired passages increase the reading demands with an additional passage. More importantly, the cognitive load is increased as students are asked to make inferences and draw conclusions across passages, not just within each passage.  When students read passage(s) with only 1-2 associated items to satisfy the test specifications in a fixed-length test, the reading demands are unexpectedly swelled, especially for young readers.

15. 15  To achieve content balance in passage-based adaptive reading tests, an indispensable condition is that all constraints at the item level and the passage level must be considered simultaneously..  In CAT, sufficient item pools and well established content constraints in the test specifications are the necessity for ensuring the adequacy in content balance and comparable construct across individual tests.

16. Thank you! 16