1. Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved. 10/7/2015 A Model for Scaling, Linking, and Reporting Through-Course Summative Assessments Rebecca Zwick Robert J. Mislevy Educational Testing Service NCSA – Orlando, June 20, 2011

2. Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved. Overview of Presentation 10/7/2015 Desired properties of a model for analyzing through-course summative assessments (TCAs) Description of the proposed model A possible simplification Recommendations 2

3. Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved. Desired Properties The model must be able to: Yield proficiency estimates for individuals & groups, accommodating different patterns of instruction Provide end-of year summaries & growth measures Provide results that are comparable across classrooms, schools, districts, & states Incorporate items that vary in instructional sensitivity (implies multidimensionality) 10/7/2015 3

4. Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved. Model Characteristics Accommodates inferences desired from TCAs; can be a framework for studying plausible submodels Has 2 components: – Multidimensional item response theory (MIRT) component specifies dependency of item responses on proficiency – Population component models association between proficiency and background variables. Builds on NAEP, TIMSS, & PISA experience 10/7/2015 4

5. Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved. Notation: 10/7/2015 = vector of student proficiencies (note that is multidimensional) = vector of item responses for student i (includes dichotomous & polytomous) =vector of background variables for student i (more later on this) 5

6. Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved. Bayesian model (see Mislevy, 1985): 10/7/2015 posterior distribution of for student i “ is proportional to” MIRT model conditional distribution of, given background variables 6

7. Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved. Model Assumptions Our model relies on the traditional IRT assumption of conditional independence: i.e., item responses are independent, conditional on, which implies that the same item response model holds across groups and over time. MIRT makes this possible; our goal is to find an interpretable model that satisfies the assumption. Model testing is needed. 10/7/2015 7

8. Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved. Why include any background variables in the model? Improves precision of estimation and avoids biases, especially when item data are sparse. includes both demographic variables and instructional variables, such as nature and order of curriculum. ( will vary over time.) For fairness reasons, demographic variables are not included when estimating individual scores. 10/7/2015 8

9. Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved. Estimating individual and group proficiency For individual estimates, use mean or mode of posterior distribution Group characteristics are NOT estimated using aggregates of individual estimates Distribution of optimal individual estimates is not the optimal estimate of the population distribution. 10/7/2015 9

10. Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved. Market-basket (MB) reporting (Mislevy, 2003) Report results in terms of a scale based on a “market- basket” of selected tasks (must be calibrated). Using observed data, we can generate predicted responses to these MB tasks: 10/7/2015 10

11. Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved. MB Reporting - Example Suppose 100 items are to be administered during a year; these could be the market-basket Assume 4 TCAs, each with 25 of these 100 items For each TCA, each student has actual responses for 25 items and predicted responses for 75 items Score is expectation of sum of responses over all 100 items 10/7/2015 11

12. Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved. Market-basket reporting (cont.) “Behind-the-scenes” machinery is complex, but resulting scores “look like” ordinary test scores: multivariate is mapped onto unidimensional scale. Can use for year-end summary of TCAs and for growth measurement (e.g., last TCA minus first TCA). Can predict end-of-year performance, achieved by adjusting values of. Given his current, how will Johnny perform with a whole year of instruction? 10/7/2015 12

13. Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved. Can a simpler model work? Simplifications become more feasible if demands on model are scaled down. Example: Use a more traditional assessment for comparisons across schools, districts and states – Machine-scoreable items; no complex tasks – Administer to a random sample only – Might eliminate need for a population model Could then use less constrained test forms, including complex tasks, to inform instruction 10/7/2015 13

14. Confidential and Proprietary. Copyright © 2010 Educational Testing Service. All rights reserved. Recommendations 1.Use pilot and field test periods to test the model and explore simplifications 2. Recognize that a tradeoff exists between inferential demands and procedural simplicity. Reducing demands makes simpler approaches more feasible 10/7/2015 14