Presentation is loading. Please wait.

Presentation is loading. Please wait.

National Accessible Reading Assessment Projects Introduction to Accessible Reading Assessment June 14, 2008 CCSSO National Conference on Student Assessment.

Similar presentations


Presentation on theme: "National Accessible Reading Assessment Projects Introduction to Accessible Reading Assessment June 14, 2008 CCSSO National Conference on Student Assessment."— Presentation transcript:

1 National Accessible Reading Assessment Projects Introduction to Accessible Reading Assessment June 14, 2008 CCSSO National Conference on Student Assessment Martha L. Thurlow

2 National Accessible Reading Assessment Projects Todays Purpose 1.Highlight the challenges in reading assessment for students with disabilities 2.Provide an overview of relevant research design and analysis 3.Present research findings from projects funded to research and develop accessible reading assessments 4.Identify implications of the research for you 5.Share Principles and Guidelines of based on our research and other resources 2

3 National Accessible Reading Assessment Projects Designing Accessible Reading Assessments (DARA) Partnership for Accessible Reading Assessment (PARA) Technology Assisted Reading Assessment (TARA) 3

4 National Accessible Reading Assessment Projects NARAP Goals 1.Develop a definition of reading proficiency 2.Research the assessment of reading proficiency 3.Develop research-based principles and guidelines making large-scale reading assessments more accessible for students who have disabilities that affect reading 4.Develop and field trial a prototype reading assessment 4

5 National Accessible Reading Assessment Projects Designing Accessible Reading Assessments (DARA) Educational Testing Service (ETS) Focuses on students with learning disabilities Focuses on component approach to assessing reading skills. Primary focus are: – Word Recognition – Reading Fluency – Vocabulary Knowledge – Comprehension 5

6 National Accessible Reading Assessment Projects Partnership for Accessible Reading Assessments (PARA) Collaboration of National Center on Educational Outcomes and U of MN Department of Curriculum and Instruction, CRESST, U of CA Davis, and Westat Focus on all disabilities that impact reading, particularly: – Learning disabilities – Speech or language impairments – Mental retardation – Deafness or hard of hearing 6

7 National Accessible Reading Assessment Projects Technology Assisted Reading Assessment (TARA) ETS, NCEO and Center for Applied Special Technology (CAST) Focus on students with visual impairments Focus on: – Examining the performance of operational ELA tests for students with visual impairments – Development of prototype Technology Assisted Reading Assessment – Inclusion of VI students in NARAP field test 7

8 National Accessible Reading Assessment Projects Background All projects focus on standards-based general assessments based on grade-level achievement standards – the regular assessment! Not focused on alternate assessments based on alternate achievement standards Not focused on alternate assessments based on modified achievement standards Still, work may sometimes be applicable to these too 8

9 National Accessible Reading Assessment Projects Who We Are Martha Thurlow (PARA, TARA) Cara Cahalan Laitusis (DARA, TARA) Linda Cook (DARA, TARA) David OBrien (PARA) Jamal Abedi (PARA) Discussant – Peggy Carr 9

10 National Accessible Reading Assessment Projects Who Are You? 10

11 National Accessible Reading Assessment Projects Plan for Today 1:00 – 2:30 Introduction Issues for Students with Disabilities Research Design and Analysis 2:30 – 2:45BREAK 2:45 – 3:35Identifying Less Accurately Measured Students Impact of Motivation and Engagement 3:35 – 3:50BREAK 3:50 – 4:30Segmented Reading Passages Principles and Guidelines 4:30 – 5:00Peggy Carr, Discussant 11

12 National Accessible Reading Assessment Projects Workshop Notebook Notebook Tabs – Guide to Workshop Process Agenda Each Topic Powerpoint presentation Resource materials Biographies for Presenters Notepaper 12

13 National Accessible Reading Assessment Projects Ground Rules Ask questions for clarification Interact with us! Take care of own needs 13

14 National Accessible Reading Assessment Projects Contact Information Martha L. Thurlow University of Minnesota 207 Pattee Hall 150 Pillsbury Drive SE Minneapolis, MN

15 National Accessible Reading Assessment Projects Issues in Assessing Reading of Students with Disabilities June 14, 2008 CCSSO National Conference on Student Assessment Martha L. Thurlow

16 National Accessible Reading Assessment Projects Poor performance of students with disabilities is a big indicator that there are issues in assessing their reading performance Data from state reading assessments shows that this is so.... 2

17 National Accessible Reading Assessment Projects Gaps in Performance on Reading Assessments Elementary School 3

18 National Accessible Reading Assessment Projects Gaps in Performance on Reading Assessments Middle School 4

19 National Accessible Reading Assessment Projects Gaps in Performance on Reading Assessments High School 5

20 National Accessible Reading Assessment Projects Gaps exist in reading performance at all school levels: They increase as the grade level increases They vary by state, but the variability seems to be more a function of the difficulty of the test than its accessibility (states with the lowest and highest average scores for students with disabilities have smaller gaps – probably due to ceiling and floor effects) 6

21 National Accessible Reading Assessment Projects Disabilities affect reading in many ways – we explored the ways in which disabilities may affect reading for 7 categories and developed a report about each: Visual Impairments Deaf or Hard of Hearing Autism Learning Disabilities Mental Retardation Speech or Language Impairments Emotional or Behavioral Disabilities 7

22 National Accessible Reading Assessment Projects Purpose To provide general information about specific disabilities and how they interact with reading, so that reading professionals and others who might contribute to the development of accessible reading assessments understand some of the challenges that need to be addressed. 8

23 National Accessible Reading Assessment Projects Disclaimer Papers clearly state that the purpose is to begin a discussion of the issues surrounding reading and students with each disability. The papers were not intended to be comprehensive research reviews. We have clarified that the papers are for people who do not know the disabilities or for those who have not considered the interaction of disabilities with reading. 9

24 National Accessible Reading Assessment Projects Students Receiving Special Education Services Overview 10

25 National Accessible Reading Assessment Projects Reading and Students with Visual impairments Most students with visual impairments are not blind. Tactile (braille) and auditory methods of accessing text are most common. Common classroom supports and accommodations may not be available for state assessment. 11

26 National Accessible Reading Assessment Projects Reading and Students who are Deaf or Hard of Hearing Age of onset of hearing loss and other factors shape educational and communication experiences. Many communication forms (e.g., American Sign Language, Manually Coded English, lip reading); cochlear implants have raised new issues. State assessment policies vary in whether they allow commonly used accommodations. 12

27 National Accessible Reading Assessment Projects Reading and Students with Autism Many students with Asperger Syndrome can decode words well, but may lack comprehension skills (Barnhill, 2004). Students with autism may find it difficult to screen out distractions. Accommodations are not as often designated toward this group. 13

28 National Accessible Reading Assessment Projects Reading and Students with Specific Learning Disabilities 90% of students with learning disabilities identify reading as their primary difficulty (Presidents Commission on Excellence, 2003). The read aloud accommodation is one of the most common and controversial accommodations provided for these students. 14

29 National Accessible Reading Assessment Projects Reading and Students with Mental Retardation Historically, educators often skipped academics (including reading) in favor of functional, social, or motor skills. Despite wide variety of characteristics that can influence reading (poor short-term memory, low-level meta-cognition), reading skills can be mastered by many students with mental retardation. Access to the general curriculum, broader accommodations, and alternate assessments are aspects of reading achievement for students with mental retardation. 15

30 National Accessible Reading Assessment Projects Reading and Students with Speech or Language Impairments Since reading is a language-based skill, students without strong language skills may be at-risk. Accommodations for these students reflect reading strategies used with them – read aloud, assistive augmentative communication devices, and frequent breaks during assessment. 16

31 National Accessible Reading Assessment Projects Reading and Students with Emotional or Behavioral Disorders Students need to compensate for lack of attention, distractibility, etc. Some accommodations that are needed are generally acceptable (breaks, quite room), while other are questionable (motivational prompts, calming music). 17

32 National Accessible Reading Assessment Projects Accommodations Accommodations are more of an issue for reading assessments than for other content areas. This occurs because many students use those accommodations that may produce invalid scores (known as modifications in most places). 18

33 National Accessible Reading Assessment Projects Percentages of Students Using Certain Accommodations Not all states publicly report on the use of accommodations, much less the specific accommodations or percentages of students. States that do include: Colorado North Carolina Based on data 19

34 National Accessible Reading Assessment Projects The Challenge for Accountability Students who use accommodations that produce invalid scores – modifications – will now count as nonparticipants in the assessment. 20

35 National Accessible Reading Assessment Projects Accessibility An accessible assessment is one that reveals the knowledge and skills of students whose characteristics create barriers to accurate measurement of these on traditional reading assessments It measures the same knowledge and skills, at the same level It may reduce the need for accommodations 21

36 National Accessible Reading Assessment Projects Implications It is important to understand the characteristics of all students taking assessments, including those that may affect performance but are not what is being measured (e.g., short term memory) Disabilities do not are not the cause of poor performance – most students with disabilities can perform at levels comparable to their peers – if we make sure they get access to the curriculum, instruction, accommodations, and accessible assessments! 22

37 National Accessible Reading Assessment Projects Implications Accommodations are an important part of accessibility – more important now than ever before to address them (which can be incorporated into the assessment, which really produce invalid scores) We need to explore innovative approaches to improving accessibility – things that in the end may benefit all students 23

38 National Accessible Reading Assessment Projects Contact Information Martha L. Thurlow University of Minnesota 207 Pattee Hall 150 Pillsbury Drive SE Minneapolis, MN

39 National Accessible Reading Assessment Projects Research Design and Analysis June 14, 2008 CCSSO National Conference on Student Assessment Cara Cahalan Laitusis and Linda Cook

40 National Accessible Reading Assessment Projects Overview Types of Questions Research Designs and Analyses Case Example from DARA project – Can read aloud and standard scores be reported on the same scale? Questions and answers 2

41 National Accessible Reading Assessment Projects Research Questions Are test scores from accommodated and non- accommodated tests: – Psychometrically comparable? – Measuring the same construct? – Equally valid predictors of the construct? What changes to test items (or administration) can: – Increase/decrease accessibility? – Increase/decrease validity of scores? – Engage students with disabilities? – Provide useful feedback to teachers? 3

42 National Accessible Reading Assessment Projects Research Studies Opinion Research Item Tryouts Experimental Studies Analysis of Operational Test Data 4

43 National Accessible Reading Assessment Projects Opinion Research Types of Opinion Research – Surveys – Interviews – Focus Groups Potential uses: – Explore the types of changes to test items (or administration) that may be worthy of additional research – Identify problems in assessment design or administration 5

44 National Accessible Reading Assessment Projects Item Tryouts Cognitive Labs (Think Alouds) Pilot Testing Field Testing 6

45 National Accessible Reading Assessment Projects Item Tryouts Cognitive Labs (Think Alouds) – 9-20 students per subgroup – Requires one-on-one administration – Qualitative analysis of responses 7

46 National Accessible Reading Assessment Projects Item Tryouts Pilot Testing – students per subgroup – Group administration – Qualitative and Quantitative analysis of responses 8

47 National Accessible Reading Assessment Projects Item Tryouts Field Testing – 100 students per subgroup – Group administration – Quantitative analysis of responses 9

48 National Accessible Reading Assessment Projects Experimental Studies Are test scores from accommodated and non- accommodated tests: – Psychometrically comparable? – Measuring the same construct? – Equally valid predictors of the construct? What changes to test items (or administration) can: – Increase/decrease accessibility? – Increase/decrease validity of scores? – Engage students with disabilities? – Provide useful feedback to teachers? 10

49 National Accessible Reading Assessment Projects Requirements of Experimental Studies Large sample sizes Random assignment of students to experimental groups to eliminate – Form effects – Order effects (test form or accommodation) Examinations of change generally require: – Two samples (students with and without disabilities) – Two testing conditions (standard and test change) – Two equated test forms 11

50 National Accessible Reading Assessment Projects Pros and Cons Experimental Design Pros – Disentangle accommodation or test change from disability – Impact of test change on total test score can be directly measured – Impact of other effects (order, test form, disability- accommodation interactions) can be mitigated Cons – Expensive – Time consuming – May not be able to simulate testing environment for high stakes testing 12

51 National Accessible Reading Assessment Projects Differential Boost Data Collection Design Group Session 1Session 2 Form Accommodation/ ModificationForm Accommodation/ Modification 11Standard2Accommodation 21 2Standard 32 1Accommodation 42 1Standard 13

52 National Accessible Reading Assessment Projects Analyses Does test change result in differential performance gains for students with disabilities? – Repeated Measures Analysis of Variance – Sample sizes vary based on degree of change expected Power.8, significance level.05, sample size needs to be 175 per group to detect and effect size of.20 Select degree of change that is practically significant rather than just statistically significant: 14

53 National Accessible Reading Assessment Projects Analyses Which score is a better predictor of construct? – Collect alternative data on construct Teachers Ratings Grades Alternate test of same construct Future performance (if test predicts readiness) – Analyze data using Regression Analyses Minimum sample size of 100 per group 15

54 National Accessible Reading Assessment Projects Studies Using Operational Data Using operational test data to study the validity and fairness of assessments for students with disabilities 16

55 National Accessible Reading Assessment Projects Studies Based on Operational Test Data (Overview) Using operational test data to study fairness and validity Studies that use differential item functioning Studies that use factor analysis 17

56 National Accessible Reading Assessment Projects Using Operational Test Data to Study Fairness and Validity Willinghams definition of fairness and validity It seems clear that the overriding issue is the comparability of tests administered to people with disabilities to those administered to others. Marks of Comparability – Reliability – Factor structure – Item functioning 18

57 National Accessible Reading Assessment Projects Using Operational Test Data to Study Fairness and Validity Effective program of research might use Willinghams framework to compare psychometric properties and internal structure of a state test for: – Students without disabilities who take the test under standard conditions; – Students with disabilities who take the test under standard conditions; and – Students with disabilities who take the test with accommodations and/or modifications 19

58 National Accessible Reading Assessment Projects Pros and Cons of Using Operational Test Data Pros – Readily available – Large sample sizes – Less expensive – Realistic Cons – Disability may be poorly or inaccurately described – Accommodations are bundled; not always described accurately 20

59 National Accessible Reading Assessment Projects Comparing the Internal Structure of an Assessment Differential item functioning (DIF) Factor analysis 21

60 National Accessible Reading Assessment Projects What is Differential Item Functioning Reference Group – Students whose performance is used as the standard for the DIF comparison Focal Group – Students whose item performance is the focus of the study Test takers matched on proficiency level 22

61 National Accessible Reading Assessment Projects Pros and Cons of Using Differential Item Functioning Pros Well established procedures Can be used with relatively small samples Analyses are simple and inexpensive 23

62 National Accessible Reading Assessment Projects Pros and Cons of Using Differential Item Functioning Cons/issues Results sometimes difficult to interpret Non-uniform DIF Differences in ability level of focal and reference groups 24

63 National Accessible Reading Assessment Projects Pros and Cons of Using Differential Item Functioning (cont.) Cons/issues Matching criterion Confounding of disability and accommodation 25

64 National Accessible Reading Assessment Projects What is Factor Analysis A statistical method used to explain relationships among variables Variables can be test scores or item scores Exploratory analyses Confirmatory analyses Single-group and multiple-group analysis 26

65 National Accessible Reading Assessment Projects Pros and Cons of Using Factor Analysis Pros – Reduction in number of variables – Identification of groups of interrelated variables 27

66 National Accessible Reading Assessment Projects Pros and Cons of Using Factor Analysis Cons – Requires relatively large samples – More than one interpretation can be made of the same data factor analyzed in the same way – Factor analysis cannot identify causality 28

67 National Accessible Reading Assessment Projects Case Example Designing Accessible Reading Assessment Project 29

68 National Accessible Reading Assessment Projects DARA Project Primary Question: – Can we assess some components of reading (decoding and comprehension) in isolation using multi-stage test design? Primary Focus: – Students with learning disabilities, particularly those who receive read aloud accommodations 30

69 National Accessible Reading Assessment Projects Reading Comprehension Routing Test Reading Fluency Decoding and Extended Comprehension Test with Audio Extended Comprehension Test with Audio Extended Reading Comprehension Test 31

70 National Accessible Reading Assessment Projects Possible Solutions Increase the reliability of test scores for students scoring at the chance level (or below) on current state assessments Allow students with reading-based disabilities to receive scores on separate components of reading (potentially comprehension, vocabulary, decoding, fluency) as well as total test score for accountability purposes. Allows state to count scores of students that receive read aloud modification for AYP because it includes a separate measure of fluency and decoding. Holds teachers accountable for both decoding and comprehension instruction. 32

71 National Accessible Reading Assessment Projects Research Questions 1.Are test scores from standard and read aloud administrations psychometrically comparable? 2.Does the read aloud administration offer an unfair advantage to test takers with disabilities? 3.Is the audio + fluency route comparable to standard administration route in terms of predicting teachers ratings of reading comprehension? 33

72 National Accessible Reading Assessment Projects Research Studies Experimentally designed differential boost study – RM ANOVA – Regression – DIF – Factor Analysis Analysis of operational test data – DIF – Factor Analysis – Simulation of multistage design Cognitive labs 34

73 National Accessible Reading Assessment Projects Questions 1 Does the read aloud administration offer an unfair advantage to test takers with disabilities? 35

74 National Accessible Reading Assessment Projects Differential Boost Data Collection Design Group Session 1Session 2 Form Accommodation/ ModificationForm Accommodation/ Modification 1SStandardTAudio 2S TStandard 3T SAudio 4T SStandard 36

75 National Accessible Reading Assessment Projects Data Collected Primary Measure 2 Reading Comprehension Tests (Form S and T) – Extra time – Extra time with Read Aloud via CD Additional Measures 2 Fluency Measures 2 Decoding Measures (4 th grade only) Student Survey Teacher Survey 37

76 National Accessible Reading Assessment Projects Sample th grade students – 527 students with reading based learning disabilities (RLD) – 654 students without a disability (NLD) th Graders – 394 RLD – 461 NLD 38

77 National Accessible Reading Assessment Projects Results of the Differential Boost Study RM-ANOVA indicated that students with reading disabilities had a significantly larger boost from an audio (read-aloud) accommodation than students without disabilities – findings consistent with Fletcher et al and Crawford & Tindal, 2004) Other Findings – Controlling for other factors (e.g., reading fluency, decoding)) using RM-ANCOVA does not change these findings – Controlling for ceiling effects does not change these findings 39

78 National Accessible Reading Assessment Projects Questions 2 Are test scores from standard and read aloud administrations psychometrically comparable? 40

79 National Accessible Reading Assessment Projects Analyses of ELA Assessment Using Operational Data Analyses – Factor analyses – Differential item functioning analyses and distractor analyses Groups of Interest – Students with learning disabilities who took the test with and without a change in testing conditions Test – Grade 4 and grade 8 English-language Arts (ELA) assessment Focus – Determine if the test measures the same constructs for Examinees without disabilities Examinees with learning disabilities who took the test with and without a change in testing conditions 41

80 National Accessible Reading Assessment Projects42

81 National Accessible Reading Assessment Projects Differential Item Functioning (DIF) Analyses The purpose of the DIF study was to examine whether or not the ELA assessment measured the same construct (s) for the groups in our study Used Mantel-Haenszel procedure with total score as criterion Mantel-Haenszel categorization – Anegligible DIF – Bslight to moderate DIF – Cmoderate to large DIF Direction of DIF Flags – Negative favors reference group – Positive favors focal group 43

82 National Accessible Reading Assessment Projects Results of the DIF Study Fourth grade results – 1 C DIF item, 10 B DIF items – 5 B DIF items were reading items that favored students with disabilities who took test with read-aloud change in testing conditions – 5 B DIF items (3 reading and 2 writing) favored students without disabilities 44

83 National Accessible Reading Assessment Projects Results of the DIF Study Eighth grade results – 1 C DIF item, 7 B DIF items – Five B DIF items (4 reading and 1 writing) favored students who took test with read- aloud change in testing conditions 45

84 National Accessible Reading Assessment Projects Factor Analyses of ELA Assessment Purpose: to examine whether or not the ELA assessment measured the same construct (s) for the groups in our study Exploratory analyses (separately in each group) – how many factors Confirmatory (multi-group) – Establish base-line model – Confirm number of factors needed to describe data across all groups 46

85 National Accessible Reading Assessment Projects Results of Factor Analysis of ELA Assessment Compared the internal structure of the grade 4 and grade 8 ELA assessment – Students without disabilities – Students with disabilities (no test conditions changes) – Students with disabilities (504/IEP accommodations) – Students with disabilities (read-aloud change in testing conditions) Results suggest test measures same single dimension for all groups 47

86 National Accessible Reading Assessment Projects Questions 3 Is the audio + fluency route comparable to standard administration route in terms of predicting teachers ratings of reading comprehension? 48

87 National Accessible Reading Assessment Projects Predictive Validity of Scores Regression analyses were conducted to examine which test scores captured the most variance in teachers ratings of reading comprehension by grade and disability group. Tested 4 models: – Standard – Standard + Fluency – Audio – Audio + Fluency 49

88 National Accessible Reading Assessment Projects Predictive Validity of Scores Tests taken with read-aloud do not predict teachers ratings of reading comprehension as well as tests taken under standard conditions. However combining read-aloud scores with reading fluency scores results in equal (or better) predictions of teacher ratings than tests taken under standard conditions. 50

89 National Accessible Reading Assessment Projects Lessons Learned and Implications Reading fluency is an important element of reading comprehension based on teachers ratings – States should consider administering reading fluency measure when read aloud is used on reading comprehension assessments Scores from standard administration and read aloud condition are fairly comparable psychometrically – States should consider replicating these DIF and Factor Analyses studies to provide some justification for reporting read aloud and standard scores on the same scale 51

90 National Accessible Reading Assessment Projects Lessons Learned and Implications 4 th and 8 th graders have no problems using individualized CD players – States should consider this type of standardized administration instead of human readers Students were better than their teachers in predicting if scores would improve with read aloud – Students should be included in the decision making process on the use of read aloud accommodations 52

91 National Accessible Reading Assessment Projects Questions? Contact Information – Cara Cahalan Laitusis – Linda Cook 53

92 National Accessible Reading Assessment Projects Identifying Less Accurately Measured Students June 14, 2008 CCSSO National Conference on Student Assessment Ross Moen, Martha Thurlow, Kristi Liu

93 National Accessible Reading Assessment Projects As was previously described, typical reading assessments have limitations for assessing the reading skills of students with disabilities. Is accessible reading assessment a way to increase test scores for all students with disabilities? Higher Scores for All? 2

94 National Accessible Reading Assessment Projects Interaction Hypothesis Might Suggest All SWD Scores Rise Typical Assessment Accessible Assessment Students With Disabilities Students Lacking Disabilities 3

95 National Accessible Reading Assessment Projects Some students with disabilities already score well despite their disabilities. Some students with disabilities truly cannot do what a States standards require. – Regardless of where the fault lies - whether with the instruction, the student or elsewhere – assessments should show if a student cannot do what is required. But Reality Is More Complicated 4

96 National Accessible Reading Assessment Projects Assessment Scores Should Rise For Some Less Accurately Measured Students Clear High Scores Clear Low Scores Questionable Low Scores Less Accurately Measured Students (LAMS) 5

97 National Accessible Reading Assessment Projects Sources of Reduced Accuracy Cheating, Narrow Teaching to the Test Lucky Learning, Good Guesses Bias, Inappropriate Obstacles Bad Day, Bad Guesses, Test Taking Errors Too High Scores Too Low Scores Systematic Error Random Error 6

98 National Accessible Reading Assessment Projects ? Compare test results with (what?) other information Match How Can We Identify Potential LAMS? Mismatch Compare Match 7

99 National Accessible Reading Assessment Projects Compare Tests with Teacher Judgment? ? = 8

100 National Accessible Reading Assessment Projects How well can teachers identify LAMS? – Do they say they can? – Can they distinguish reasons for LAMS? – Can they provide supporting evidence? – Do brief supplemental examinations match teacher judgments? What can we learn from teachers LAMS? – What do they say they need or want? – What do we observe in assessment situations? Teacher Nominations: Study Goals 9

101 National Accessible Reading Assessment Projects Teachers complete LAMS nomination questionnaire – 4th and 8th grade classroom, reading, English/language arts, special education teachers Researchers meet with teachers – Structured interview & examine supporting evidence Researchers meet with students – 4th through 8th grade, native speakers of English – Structured interview and differentiated assessment Teacher Nominations: Study Procedures 10

102 National Accessible Reading Assessment Projects Two phases separated by adjustments in meeting procedures 21 teachers at 10 sites completed LAMS nomination questionnaires on 77 students Average misrepresentation (1-5): 3.89 First phase, met with 2 teachers and 6 students Second phase, met with 7 teachers and 17 students – all elementary Teacher Nominations: Study Results 11

103 National Accessible Reading Assessment Projects Reasons for Identifying Students as LAMS Count*Percentage* Fluency Limitations Obscure Comprehension Skills % Some Comprehension Limitations Obscure other Skills % Test Fails to Reveal Non-Tested Strengths % Responds Poorly to Testing Circumstances or Materials % Other 5 6.5% * Note duplicate counts on 77 students sum to a total count of 108 and total percentage of 140% 12

104 National Accessible Reading Assessment Projects Teacher Ratings of Hindrances to Student Performance Hardly At AllA LittleSomeQuite a BitA LotBlankMean Fluency limitations %0.0%23.5%35.3%23.5%0.0% Comprehension limitations %5.9%29.4%41.2%23.5%0.0% Low motivation for the test %5.9%23.5%5.9%23.5%0.0% Keeping attention focused on the test %29.4% 11.8% 0.0% Getting worn out by the test %23.5%11.8%17.6% 0.0% Anxiety %17.6%35.3%0.0%11.8%5.9% Other: %5.9%0.0%11.8%41.2% 13

105 National Accessible Reading Assessment Projects Student Attitudes Toward Reading and Tests Hardly At AllA LittleSomeQuite a BitA LotBlankMean How much do your read not for school? %23.5%41.2%5.9%17.6%5.9% How much do you Like reading? % 52.9%23.5%17.6%5.9% How hard is reading for you? %11.8%41.2%23.5%0.0%5.9% How well do tests show your reading? %5.9%35.3%29.4%11.8%17.6% 14

106 National Accessible Reading Assessment Projects Student Ratings of What Might Help Hardly At AllA LittleSomeQuite a BitA LotBlankMean Shorter reading passages %11.8%23.5%41.2%5.9%17.6% More interesting passages %17.6%5.9%23.5%35.3%17.6% Computer instead of paper and pencil %5.9%11.8%23.5% Entire test read aloud by CD etc % 41.2%11.8%17.6% Computer pronounces or explains words you pick % 5.9%35.3%41.2%17.6% Other ideas you have %5.9%0.0%5.9%29.4%58.8% 15

107 National Accessible Reading Assessment Projects Qualitative Analysis: Teachers LAMS confirmed? Clear Bulls Eye Consensus between researchers & teacher n = ? Off Target Indications that student is not a LAMS n = ? Seems Close Differ on why LAMS n = ? Seems Close Weak confirmation n = ? 16

108 National Accessible Reading Assessment Projects For More Information, Contact: Ross Moen 17

109 National Accessible Reading Assessment Projects The Impact of Motivation and Engagement on Assessing Reading Comprehension June 14, 2008 CCSSO National Conference on Student Assessment Deborah R. Dillon and David G. OBrien

110 National Accessible Reading Assessment Projects Construct of Comprehension Current assessments of reading comprehension are inadequate (Sweet, 2005) The current knowledge base on reading comprehension is sizeable but too sketchy to provide a foundation for a systematic instructional agenda (RAND Reading Study Group, 2002) 2

111 National Accessible Reading Assessment Projects Relation of Motivation and Engagement to Comprehension The RAND RSG initiated its work by generating this definition of reading comprehension: Reading comprehension is the process of simultaneously extracting and constructing meaning through interaction and involvement with written language (Snow, 2002, p.11). Guthrie and Wigfield (2005) built upon this definition, contending that involvement with written language connotes motivation 3

112 National Accessible Reading Assessment Projects Motivation and Comprehension Involvement assumes an active, intentional stance toward the text, enabling one to both persevere in getting information from text and using both the textual information and cognitive processes to make meaning Without motivation, specifically the intention and persistence to the goal of understanding texts for various purposes, there is little comprehension Thus, Guthrie and Wigfield (2005) argued that definitions of reading comprehension should include motivation 4

113 National Accessible Reading Assessment Projects Motivation and Comprehension Guthrie and Wigfield (1999) outlined a motivational- cognitive model of reading with import for the development of accessible reading assessments. In the model they posit that both cognitive and motivational processes influence reading comprehension Wang and Guthrie (2004) found that intrinsic motivation for reading was highly predictive of reading comprehension test performance. Beliefs about reading and perceptions of oneself as a reader impacts whether students expect reading to be useful and whether they want to be effective readers (Guthrie & Wigfield, 2005; OBrien & Dillon, 2008). 5

114 National Accessible Reading Assessment Projects Motivation Constructs with Potential for Enhancing Accessibility Guthrie and Wigfield (2005) argued that the validity of comprehension assessment can be improved by enabling students to: – read with clearly defined purposes – take positive stances that support self-efficacy – exert autonomy through choice to better employ their cognitive competencies in the testing situation. 6

115 National Accessible Reading Assessment Projects Interest and Choice Enhance Accessibility Interest and choice to are especially compelling – research indicates that text interest may be more important for lower achievers than for more proficient readers – choice may have a greater impact on these readers comprehension (deSousa & Oakhill, 1996). However Schiefles (1999) findings indicated that the interest effect is independent of prior knowledge or verbal ability. 7

116 National Accessible Reading Assessment Projects Interest and Choice Enhance Accessibility This positive impact on comprehension is particularly true if readers perceive the texts to be attractive – appealing visual elementsfonts, illustrations, layout, – full of interesting details--desirable length and difficulty level of (Schraw, Bruning, & Svobada, 1995). We acknowledge that prior knowledge, which is correlated with both situational interest and reading achievement, may be a confounding variable 8

117 National Accessible Reading Assessment Projects Interest and Choice Enhance Accessibility The NAEP Reader Study: The Effects of Choice in Reading AssessmentResults from The NAEP Reader Special Study of the 1994 National Assessment of Education Progress (Campbell & Donahue, 1997) – U.S. Department of Education, Office of Educational Research and Improvement ((NCES ) special study – Examined the feasibility and measurement impact of offering grade 8 and grade 12 test takers a choice of reading material on an assessment of reading comprehension. 9

118 National Accessible Reading Assessment Projects Interest and Choice Enhance Accessibility – In the design, a group of readers who could exert choice in selecting from among seven stories to read as part of the 1994 NAEP, was compared with a group who were assigned stories. – In the choice condition, the researchers found no significant effect for choice for twelfth graders and slightly lower performance in the choice condition for eighth graders 10

119 National Accessible Reading Assessment Projects Interest and Choice Enhance Accessibility Our Expansion on the NAEP Reader Study – Assessment matching the goals of the 2009 NAEP Framework in terms of the text types and the cognitive targets assessed (Reader study used assessed more generic reading comprehension constructs following all passages) – Using both literary-fiction and informational- exposition passages (Reader study used narrative only) – Items following the reading of passages match the content and key ideas of individual passages 11

120 National Accessible Reading Assessment Projects Interest and Choice Enhance Accessibility Respondents exert choice choose your own assessment before assessment is administered (Reader study included reading passage summary and choice as part of testing time) 12

121 National Accessible Reading Assessment Projects Calibration Study The purpose of the study is to scale or calibrate the measurement tools that will be used in a large-scale accessible reading assessment for students with disabilities. This process allows investigators to empirically determine the comparability of passages and items used in the reading assessment study by placing all passages and questions on a common IRT (item response theory)-based equal-interval measurement scale. 13

122 National Accessible Reading Assessment Projects Research Questions 1.What is the difficulty of each reading passage (based on a passage total score, which, in turn, is based on performance on all passage comprehension items/questions) and each comprehension item/question? 2.How well can the reading passages be placed on a common interval measurement scale to allow scores from different passages (of equal or unequal difficulty) to be compared and equated? 3.Based on IRT item fit statistics, what multiple choice items should be retained and which should be eliminated? 4.Which reading passages do students prefer to read? 14

123 National Accessible Reading Assessment Projects Design Representative total sample of 1200 students, representing a range of reading ability and including students with disabilities for 4 th and 8 th grade Selection of 40 Passages, including 10 literary/fiction and 10 informational/exposition administered five forms 10 items for each passage using 2009 NAEP cognitive targets 15

124 National Accessible Reading Assessment Projects Analysis This preliminary item/passage psychometric calibration study allows for: 1.the placement of all passages/questions on a common equal-interval measurement scale, 2.the development of passage scoring tables by which to assign subjects reading ability scores, and 3.provision of a mechanism for equating scores across different passages. 4.This item fit analysis will determine which items will be retained and those that will be eliminated. 16

125 National Accessible Reading Assessment Projects Motivation Choice Study Purpose: To examine whether improving the motivational characteristics of a large- scale reading assessment increases its accessibility for students with disabilities, and in so doing provides a more valid assessment of these students reading proficiency due to their increased engagement. 17

126 National Accessible Reading Assessment Projects Research Questions 1.Is there an interaction effect between choice, type of text, and type of student? 2.Is there a correlation between general motivation to read and performance on a large-scale comprehension assessment? 3.Are participants who are more motivated to read more likely to benefit from the choice option on a large-scale assessment? 4.Does the option of exercising choice improve comprehension for general education students and for students with disabilities? 18

127 National Accessible Reading Assessment Projects Participants and Design Students fluent in English (140 4 th graders; th graders) Targeted samples representing range of disability groups Random assignment to treatment (motivation- choice) and control (no choice) reading 2 literary fiction and 2 informational-expository passages followed by 5-6 multiple choice items Untimed administration Assessment of general and situational motivation 19

128 National Accessible Reading Assessment Projects Design & Procedures cont. Post-assessment interviews conducted with subsets of students from the control and experimental groups at both grade levels. Students from the various disabilities groups as well as regular education students selected for interviews (16 students from 4th grade and 16 from 8th grade) 20

129 National Accessible Reading Assessment Projects Analysis A split-plot design will be used – two between-subjects factors (A = passage choice & B = disability status), – one within-subjects factor (C = text type), one blocking variable (S = subject), & one covariate (X = motivation as assessed on the MRQ) at the between-subject level (A, B, C, and X are fixed effects, and S is a random effect) 21

130 National Accessible Reading Assessment Projects Analysis--contd. Analysis of variance will be used to evaluate various effects; correlations of students performance on the comprehension test & responses on the MRQ and situated motivation questions will be calculated Various analytic deduction approaches will also be used to analyze the post assessment interview data and a mixed-design approach will be used to integrate the overall quantitative and qualitative findings 22

131 National Accessible Reading Assessment Projects Implications Motivation can lead to increased engagement, which can lead to higher, more valid comprehension performance on high stakes assessments In constructing bias-free assessments, test designers may eliminate engaging passages or design features either unwittingly or in the interest of psychometrics Students with disabilities may be more likely to engage with, persevere with, and exert more effort with more motivating and engaging passages, defining these assessments as more accessible than typical reading assessments 23

132 National Accessible Reading Assessment Projects Contact Information Deborah R. Dillon University of Minnesota, Twin Cities 330B Peik Hall, 159 Pillsbury Dr. S. E. Curriculum & Instruction Department University of Minnesota, Twin Cities Minneapolis, MN David G. OBrien 125 Peik Hall, 159 Pillsbury Drive Curriculum and Instruction Department University of Minnesota, Twin Cities Minneapolis, MN

133 National Accessible Reading Assessment Projects Segmented Text Study June 14, 2008 CCSSO National Conference on Student Assessment Jamal Abedi, Seth Leon & Jenny Kao

134 National Accessible Reading Assessment Projects Purpose To determine if reducing the length of reading passages by segmenting them would impact performance of students with disabilities. To examine the impact of segmenting on students non-cognitive domains such as anxiety, fatigue, frustration and motivation. 2

135 National Accessible Reading Assessment Projects Background of Study From chunking to segmented text – Chunking in past literature deals with working memory capacity, with the hypothesis that reading material chunked into meaningful units facilitates reading comprehension and efficiency. – However, chunking in the literature refers to chunking sentences. – We use segmented text to refer to how passage segments are grouped with their corresponding items on the test page. 3

136 National Accessible Reading Assessment Projects Segmented Text Segmented text also serves as built-in test breaks, possibly reducing the need for accommodation 4

137 National Accessible Reading Assessment Projects Participants 738 Grade 8 students from ten public schools in California 620 non-SD, 117 SD Of the 117 SD: – 107 specific learning disabilities – 2 deaf/hard of hearing – 3 autistic – 2 speech/language impairment – 4 other health impairments 5

138 National Accessible Reading Assessment Projects Reading Test Three reading comprehension passages were obtained from publicly-released tests from two states. Two versions of the test were created: Original (version A) and Segmented (version B) Test designed to be completed in one classroom period (approx. 50 min.) 6

139 National Accessible Reading Assessment Projects Passages All passages were informational (i.e., not fiction or literature). First passage was 700 words, other two passages were about 550 words each. Each passage had 8 multiple-choice items with 4 possible answer choices (24 total MC items). 7

140 National Accessible Reading Assessment Projects Process of Segmenting Segments were grouped with corresponding test items Each passage was broken down into 3 to 4 segments; each segment contained 1-3 questions Inferential questions appeared at the end Test items appeared in the same order in both versions 8

141 National Accessible Reading Assessment Projects Other Instruments Teacher Ratings Emotion/Mood Inventory Motivation Scale 9

142 National Accessible Reading Assessment Projects Teacher Ratings Asked teachers to rate each of their students. Corresponds with Calif. (CST) proficiency levels. In your opinion, how would you rate this students reading comprehension ability? Advanced Proficient Basic Below basic Far below basic 10

143 National Accessible Reading Assessment Projects Emotion/Mood Inventory Asked students after each passage: How does taking the test make you feel? Please circle all the words that describe how you feel. There is no right or wrong answer. If none of these words describe how you feel, please circle NONE. goodtired energetic upset boredconfident frustratedokay happy stressed blanked outinterested relaxedbad NONE 11

144 National Accessible Reading Assessment Projects Motivation Scale Post-test (printed at the end of the test booklets) 10-item, 4-point Likert-type, combining importance and effort questions 12

145 National Accessible Reading Assessment Projects Research Questions Accessibility – Segmented Text and Reliability – Segmented Text and Performance – Segmented Text and the correlation between teacher ratings, English language arts (ELA) achievement test level and reading performance Affective Factors – Segmented Text and Motivation – Segmented Text and Emotion/Mood Inventory 13

146 National Accessible Reading Assessment Projects Segmented Text and Reliability Findings The original version of the assessment was more reliable for non-SDs than for SDs. This reliability gap decreased on the segmented version (no longer a significant difference). This suggests the segmented version may be more accessible for SD students Reliability limits validity, because rxy < rxx (Allen & Yen, p. 113) Groups ReliabilityValidity SD/Original (n=53) SD/Segment (n=62) Non- SD/Original (n=312) Non- SD/Segment (n=305)

147 National Accessible Reading Assessment Projects Segmented Text and Performance No significant differences in reading performance of either group due to segmenting Groups MeanSDn SD/Original SD/ Segment Non-SD/ Original Non-SD/ Segment

148 National Accessible Reading Assessment Projects Motivation Results Summary of descriptive analyses for the motivation section No significant differences GroupMeanSDn Students with disabilities, original Students with disabilities, seg Students with disabilities, total Non-disabled, original Non-disabled, seg Non-disabled, total Original version, total Segmented version, total Total

149 National Accessible Reading Assessment Projects Conclusions Results suggest segmented text may be more accessible to students with disabilities Segmenting did not affect performance of non- SD students; therefore, it did not alter the reading construct Segmenting did not affect performance of SD students either Segmenting improves psychometric characteristics of reading comprehension assessments 17

150 National Accessible Reading Assessment Projects Implications for the large scale reading assessments The study could help states in identifying factors that affect accessibility of reading assessment It provides methodological paradigm for the study of accessibility of reading It examined important factors that affects presentation of test items Since segmenting passages improves the reliability of reading assessment without altering the construct, states can apply this feature into their assessments. This study encourages states and test publishers to look into test characteristics in a more comprehensive way to identify factors affecting accessibility of assessments for all students particularly for students with disabilities. 18

151 National Accessible Reading Assessment Projects Contact Information Jamal Abedi 19

152 National Accessible Reading Assessment Projects Accessibility Principles for Reading Assessments June 14, 2008 CCSSO National Conference on Student Assessment Martha L. Thurlow

153 National Accessible Reading Assessment Projects Purpose of the Accessibility Principles To identify supported* principles and guidelines for making large scale assessments of reading proficiency more accessible for students who have disabilities that affect reading, while maintaining a high level of validity for all students taking the assessments. Support = research, standards, and theory 2

154 National Accessible Reading Assessment Projects Accessibility An accessible assessment is one that reveals the knowledge and skills of students whose characteristics create barriers to accurate measurement of these on traditional reading assessments 3

155 National Accessible Reading Assessment Projects Intended Use Large-scale reading assessments Assessments focused on grade-level content standards based on grade-level achievement standards (regular or alternate) Reading portion of normative assessments included in state assessment programs Reading component of English language proficiency assessments 4

156 National Accessible Reading Assessment Projects Review Process NARAP Technical Advisory Committees NARAP Principles Committee Workshop at annual conference of the Association of Test Publishers Interactive session at National Conference on Student Assessment* *Attend interactive session on Monday, June 16 at 8:15 am (Florida Ballroom I) 5

157 National Accessible Reading Assessment Projects Organization of Accessibility Principles Principles are rules that define the overarching goals to achieve accessibility A rationale is provided for each principle to justify why it is included Guidelines under each principle address the implementation Support is provided for each guideline via an annotated bibliography 6

158 National Accessible Reading Assessment Projects Overview of Principles 1.Reading assessments should be accessible to all students in the testing population, including students with disabilities 2.Reading assessments should be grounded in the field of reading 3.Reading assessments should be developed with accessibility as a goal throughout rigorous and well-documented test design, development, and implementation procedures 7

159 National Accessible Reading Assessment Projects Overview of Principles 4.Reading assessments should reduce the need for accommodations, yet be amenable to accommodations that are needed to make valid inferences about a students performance 5.Reporting of reading assessment results should be designed to be transparent to relevant audiences and to encourage valid interpretation and use 8

160 National Accessible Reading Assessment Projects Contact Information Martha L. Thurlow University of Minnesota 207 Pattee Hall 150 Pillsbury Drive SE Minneapolis, MN


Download ppt "National Accessible Reading Assessment Projects Introduction to Accessible Reading Assessment June 14, 2008 CCSSO National Conference on Student Assessment."

Similar presentations


Ads by Google