1. National Accessible Reading Assessment Projects Research on Making Large-Scale Reading Assessments More Accessible for Students with Disabilities June 7, 2007 Deborah Dillon, Jamal Abedi, Cara Cahalan Laitusis, and Linda Cook Discussants: Geneva Haertel and Gerald Tindal

2. National Accessible Reading Assessment Projects Designing Accessible Reading Assessments (DARA) Partnership for Accessible Reading Assessment (PARA) Technology Assisted Reading Assessment (TARA)

3. National Accessible Reading Assessment Projects NARAP Projects’ Goals 1.Develop a definition of reading proficiency 2.Research the assessment of reading proficiency 3.Develop research-based principles and guidelines making large-scale reading assessments more accessible for students who have disabilities that affect reading 4.Develop and field trial a prototype reading assessment

4. National Accessible Reading Assessment Projects Partnership for Accessible Reading Assessments (PARA) Collaboration of National Center on Educational Outcomes and U of MN Department of Curriculum and Instruction, CRESST, U of CA Davis, and Westat Focus on all disabilities that impact reading, particularly: –Learning disabilities –Speech or language impairments –Mental retardation –Deafness or hard of hearing

5. National Accessible Reading Assessment Projects Designing Accessible Reading Assessments (DARA) Educational Testing Service (ETS) Focuses on students with learning disabilities Focuses on component approach to assessing reading skills. Primary focus are: –Word Recognition –Reading Fluency –Vocabulary Knowledge –Comprehension

6. National Accessible Reading Assessment Projects Technology Assisted Reading Assessment (TARA) ETS, NCEO and Center for Applied Special Technology (CAST) Focus on students with visual impairments Focus on: –Examining the performance of operational ELA tests for students with visual impairments –Development of prototype Technology Assisted Reading Assessment –Inclusion of VI students in NARAP field test

7. Partnership for Accessible Reading Assessment Partnership for Accessible Reading Assessment (PARA) Deborah Dillon and Jamal Abedi

8. Partnership for Accessible Reading Assessment PARA Research Program Assumptions Assumption #1: We do not know everything about what goes into accessible reading assessment yet. Assumption #2: Preliminary research must inform design of accessible assessment. Assumption #3: Both preliminary and experimental research will inform development of Principles and Guidelines for future assessments.

9. Partnership for Accessible Reading Assessment Three Major Research Question Areas What characteristics of current assessment practices hinder accessibility? What characteristics of students require more accessible assessment of reading? What characteristics would accessible assessment have?

10. Partnership for Accessible Reading Assessment Purpose: To examine whether improving the motivational characteristics of a large-scale reading assessment increases its accessibility for students with disabilities, and in so doing provides a more valid assessment of these students’ reading proficiency due to their increased engagement. Motivation Study

11. Partnership for Accessible Reading Assessment Research Questions 1. Does the option of exercising choice in the selection of reading comprehension passages, which is hypothesized to improve student motivation and engagement on a large-scale assessment, produce significantly higher measured reading comprehension for all students? 2. Is there a significant difference in reading scores of students with disabilities versus general education students on large-scale reading assessments? 3. Is there a significant difference in student performance on text type (literary-fiction versus informational- exposition passages) on large-scale reading assessments?

12. Partnership for Accessible Reading Assessment Research Questions 4. Is there an interaction effect between choice, type of text, and type of student? 5. Is there a correlation between students’ general motivation to read (e.g., as measured by the MRQ-Motivation to Read Questionnaire) and their performance on a large-scale reading assessment? Are participants who are more motivated to read (as measured by the MRQ), more likely to benefit from the choice option on a large scale reading assessment?

13. Partnership for Accessible Reading Assessment Design: Calibration Study Prior to the motivation study, 32 reading passages (16 each for grades 4 and 8) with 10 items each, will be calibrated to empirically determine the comparability of the passages Calibration requires the placement of all passages & questions on a common IRT-based equal-interval measurement scale, the development of passage scoring tables, & the use of a mechanism for equating scores across passages –Item fit analysis will be employed to determine which items will be retained or eliminated –Students will rate passages to indicate interest and difficulty Participants include 400 students in grades 3-5 and 400 students in grades 7-9 from intact classrooms, including students with and without disabilities

14. Partnership for Accessible Reading Assessment Design: Motivation Study Participants include 280 students who are fluent in English—140 students from 4th grade and 140 from 8th grade, including targeted samples of students representing a range of disability groups The motivation assessment includes 2 literary- fiction and 2 informational-expository passages for both grade 4 & grade 8; each passage will be followed by 5-6 multiple choice items; the assessment is untimed & completed on a computer-based platform Interviews will be conducted after testing with subsets of students from the control and experimental groups at both grade levels

15. Partnership for Accessible Reading Assessment Attending to Issues of Motivation General motivation will be measured prior to the test to obtain information on students’ feelings about “self as reader” ( e.g., Motivation for Reading Questionnaire-MRQ ). Situated motivation will be measured using questions woven into the test booklets for the choice and no choice conditions (placed after the comprehension items); specific questions will tap –students’ perceptions of the texts they read (e.g., difficulty; interest), and –students’ sense of self-efficacy in reading and completing the items following the passage (the task).

16. Partnership for Accessible Reading Assessment Design &Analysis: Motivation Study A counterbalanced stratified random assignment design will be used with experimental choice (C) groups that select reading passages for the assessment (“design your own assessment”) and control no choice (NC) groups that do not select passages The dependent measure is comprehension performance; the factors include choice condition (choice/ no choice), disability status (youth with disabilities/ youth without disabilities) and text type (literary-fiction/informational- exposition)

17. Partnership for Accessible Reading Assessment Design &Analysis: Motivation Study The design is a split-plot design with two between-subjects factors (A = passage choice and B = disability status), one within-subjects factor (C = text type), one blocking variable (S = subject), and one covariate (X = motivation as assessed on the MRQ) at the between-subject level; A, B, C, and X are fixed effects, and S is a random effect Analysis of variance will be used to evaluate various effects; correlations of students’ performance on the comprehension test & responses on the MRQ and situated motivation questions will be calculated