1. National Accessible Reading Assessment Projects Defining Reading Proficiency for Accessible Large Scale Assessments Principles and Issues Paper American Educational Research Association April 10, 2006 Deborah R. Dillon, University of Minnesota & John Sabatini, ETS

2. National Accessible Reading Assessment Projects Overview of the Project A collaboration between two projects funded to conduct research and development on accessible reading assessments for students with disabilities that affect reading. Goals include producing research findings and assessment techniques that demonstrate how large-scale assessments can become more accessible and valid.

3. National Accessible Reading Assessment Projects A Vision for Accessible Assessments for Students with Disabilities Students deserve the opportunity to show their proficiency as readers – to show what they know and are able to do; this should be reflected in the way reading proficiency is defined. Expectations for students reading performance should not be lowered; rather, accommodations should be built into the test itselfvia universal designallowing participation of the widest range of students & more valid inferences about performance. Flexible expressions of reading will be necessary to allow students who may not be proficient on all components of reading to demonstrate the skills they do know.

4. National Accessible Reading Assessment Projects First Task in the Project To design accessible reading assessments we need precise definitions of the constructs being measured. Step one involves developing a definition of reading proficiency. To accomplish this task we synthesized information and described: (a) a set of principles that will guide the research and development phases of our projects, and (b) unresolved issues that need to be addressed related to each principle.

5. National Accessible Reading Assessment Projects Sources for the Development of the Three Principles and Related Issues A review of existing definitions of reading proficiency (e.g., reports such as NRP, RAND, PISA, PIRLS) A panel of experts to provide input Focus groups

6. National Accessible Reading Assessment Projects Overview of Principles Principle 1: Definitions of reading proficiency must be consistent with core NCLB provisions. Principle 2: Reading proficiency must be defined in such a way that flexible expressions of reading are allowed while preserving the essential nature of reading. This is crucial as we seek to make assessments accessible to students with a variety of disabilities. Principle 3: Definitions of reading proficiency must reflect both comprehension and foundational skills.

7. National Accessible Reading Assessment Projects Principle 1: Definitions of reading proficiency must be consistent with core NCLB provisions. Accessible large-scale assessments developed by NARAP must adhere to two requirements of NCLB: to provide (a) a valid measure of proficiency against academic standards, and (b) individual interpretive, descriptive, and diagnostic reports for the full range of students with disabilities that affect reading. NCLB requires states to address areas of reading proficiency and component skills through grade-level definitions of content standards; NARAP cannot impose a particular standardwe must allow for the variability among states.

8. National Accessible Reading Assessment Projects Principle 1cont. Access to, participation in, and progress in the general curriculum is a foundational requirement. The 2009 NAEP assessment, which is required to be used by all states, focuses on grade-level cognitive targets, and defines these as the mental processes or kinds of thinking that underlie reading comprehension. Assessments used primarily for accountability should also provide useful information to educators as they plan instructional improvements.

9. National Accessible Reading Assessment Projects Issues Under Principle 1 How do the important reading skills vary as a function of grade level? How do we determine which measure may be appropriate for use at a specific grade level? How much can achievement levels vary and still meet the requirements of grade-level content? How are differences in reading achievement standards (e.g., modified or alternate achievement standards) developed and defined? How are these varying achievement standards reflected in definitions of reading proficiency?

10. National Accessible Reading Assessment Projects Principle 2: Reading proficiency must be defined in such a way that flexible expressions of reading are allowed while preserving the essential nature of reading. This is crucial as we seek to make assessments accessible to students with a variety of disabilities. Proficient readers, including students with disabilities, may rely on any set of component proficiencies to read and may compensate for some skills they lack by drawing on othersdepending on their disability ( e.g., a student with congenital deafness may comprehend what he/she reads, but need to deploy alternate strategies to understand sound-symbol relationships ) Public views of what constitutes reading are reflected in various ways; these are indicated in how states allow/do not allow different approaches to assessments in reading and if assessments are allowed without restrictions

11. National Accessible Reading Assessment Projects Principle 2:--cont. Federal statutes allow a range of options in types of assessments used and the achievement standards applied to students with disabilities. Some students can take assessments based on modified achievement standards or alternate assessments based on grade level standards or on alternate achievement standards. Alternate achievement standards must be aligned to grade-level content standards but may differ from grade level achievement standards in breadth, depth, or complexity.

12. National Accessible Reading Assessment Projects Issues under Principle 2 How broadly can we define what constitutes reading and still have the definition based on grade-level achievement standards? Can what constitutes as reading for standards- based assessments differ by disability category or by needed accommodation? When do the concepts of modified and alternate achievement standards apply to grade-level reading standards? How so students with disabilities compensate for weaknesses in specific reading proficiency components due to their disability or multiple disabilities?

13. National Accessible Reading Assessment Projects Principle 3: Definitions of reading proficiency must reflect both comprehension and foundational skills. Panels of researchers and practitioners indicated that the NARAP core construct of a definition of reading should include both foundational reading skills and comprehension. NARAP is basing its work on the NCLB definition, which emphasizes foundational reading skills, and the 2009 NAEP framework, which stresses comprehension and indicates that students must apply foundational skills to comprehend a variety of texts for various purposes and situations.

14. National Accessible Reading Assessment Projects Principle 3cont. It could be argued that if a child demonstrates proficiency on an assessment focused on comprehension (e.g., 2009 NAEP), that no other assessment is necessary; conversely, foundational reading skills may not be developed or are in process for the population NARAP is studying. The overall goal is to develop flexible assessments to assess reading comprehension and/or foundational skills based on student performance.

15. National Accessible Reading Assessment Projects Issues under Principle 3 To what degree can component skills be measured independently? Comprehension is the primary goal for readers. If students are proficient in this area with accommodations, do we need to measure the foundational skills? If comprehension is our primary goal, should the comprehension score be weighted more heavily when foundational skills are also assessed? What feasible techniques are available for measuring foundational skills such as fluency or phonics knowledge in the context of large-scale assessment?

16. National Accessible Reading Assessment Projects Issues under Principle 3cont. For students who do not achieve grade-level proficiency, what processes can be developed or applied to aggregate their performance on component skills into an overall measure of reading proficiency? Some components appear to be problematic for certain disabilities (e.g. phonemes and deaf students). Do we develop alternate definitions of proficient reading for these populations based on a better understanding of reading processes and performances? Are some skills less critical to measure than others?

17. National Accessible Reading Assessment Projects Issues under Principle 3cont. If foundational skills were only assessed after a student had performed below proficient on comprehension, what proportion of ALL students (with and without disabilities) would be assessed on each of the foundational skills? Is this proportion small enough to assess students in small groups, individually, or via computer? Can some foundational skills be assessed together (e.g., decoding and phonemes)? If foundational skills are going to be measured only for students who are not proficient on an assessment of comprehension, can accommodations be allowed that invalidate the foundational skills (e.g., read aloud for decoding or extra time for fluency). If a student is not proficient on a measure of reading comprehension, should listening comprehension be assessed prior to measuring foundational skills?

18. National Accessible Reading Assessment Projects Conclusions The three principles will guide NARAP in formulating the definitions of reading proficiency used within the studies. The definition of reading proficiency will be an organizing framework that supports states. Research will be conducted consistent with Principle 1 and addressing issues listed under Principles 2 & 3; several issues cannot be resolved through empirical research; these require information from NARAP to impact policies on assessing students with disabilities that impact reading.

19. National Accessible Reading Assessment Projects NARAP Web Site The Principles & Issues Paper is posted at: http://www.narap.info/