1. Copyright © 2004 Educational Testing Service Listening. Learning. Leading. Using Differential Item Functioning to Analyze a State English-language Arts Assessment Linda Cook Fred Cline Educational Testing Service

2. Copyright © 2004 Educational Testing Service Page 2 Relationship to DARA Investigating how tests work for students with disabilities now is one way to determine how to build more appropriate tests in the future. Analyzing tests as a whole and item by item can be done via Factor Analysis and Differential Item Functioning

3. Copyright © 2004 Educational Testing Service Page 3 Differential Item Functioning (DIF) DIF refers to a difference in item performance between two comparable groups of test takers DIF exists if test takers who have the same underlying ability level are not equally likely to get an item correct

4. Copyright © 2004 Educational Testing Service Page 4 Why DIF Procedures are Useful DIF studies first carried out on a frequent basis in 1960s –Evaluate ethnic differences –Identify and remove biased items Examples –Deaf and HH examinees –Test item draws on knowledge of popular music

5. Copyright © 2004 Educational Testing Service Page 5 Some Issues Related to Using DIF Procedures For Students With Disabilities Definition of groups Sample Size Matching criterion Ability differences between groups Very little research focusing on DIF for students with disabilities

6. Copyright © 2004 Educational Testing Service Page 6 Applications of DIF Procedures to Evaluate Assessments for Students With Disabilities Lewis, Green & Miller, 1999 –Read aloud and extended time –35 different groups with disabilities –More ELA items had DIF than Math items

7. Copyright © 2004 Educational Testing Service Page 7 Applications of DIF Procedures to Evaluate Assessments for Students With Disabilities(cont.) Bielinski, Thurlow, Ysseldyke, Freidebach & Freidebach, 2001 –Read aloud accommodations –41 multiple choice grade 3 reading items –32 multiple choice grade 4 math items –DIF for 30 reading and 7 math items –Results indicated read aloud accommodation may affect comparability of items

8. Copyright © 2004 Educational Testing Service Page 8 Applications of DIF Procedures to Evaluate Assessments for Students With Disabilities(cont.) Bolt,2004 –Read aloud accommodation is less appropriate for reading tests than for non-reading tests –Accommodations are more appropriate for students with sensory and physical disabilities than for students with cognitive disabilities

9. Copyright © 2004 Educational Testing Service Page 9 Applications of DIF Procedures to Evaluate Assessments for Students With Disabilities(cont.) Barton& Finch, 2004 –Second edition of the TerraNova –Language and math subtests for grades 3,5,8 –DIF for items with high amount of text provided via read aloud conditions to accommodated students –Concluded some level of boost for students with disabilities with read aloud accommodation

10. Copyright © 2004 Educational Testing Service Page 10 Current Study (ETS/DARA) Grade 4 and Grade 8, State English Language Arts test Evaluated three criteria –Total Test –Reading –Writing Analyzed items in total test

11. Copyright © 2004 Educational Testing Service Page 11 Description of the Tests Grade 4 ELA test contains reading and writing strands for a total of 75 items Reading subtest has three strands for a total of 42 items Writing subtest has two strands for a total of 33 items

12. Copyright © 2004 Educational Testing Service Page 12 Description of the Sample Four groups of students –Students without disabilities –Students with LD who took the test without an accommodation –Students with LD who took the test with an accommodation defined by 504 plan or IEP –Students with LD who took the test with a read-aloud accommodation

13. Copyright © 2004 Educational Testing Service Page 13 Comparison Groups Used for DIF Analyses Reference GroupFocal Group Without disabilitiesLD, no accommodations Without disabilitiesLD, IEP/504 accommodations Without disabilitiesLD, read- aloud accommodation LD, no accommodationsLD, IEP/504 accommodations LD, no accommodationsLD, read-aloud accommodation

14. Copyright © 2004 Educational Testing Service Page 14 Characteristics of Groups Used for DIF Studies GroupSample SizeMeanStandard Deviation No Disability30,2254814 LD, no Accommodation 9,0452912 LD, 504/IEP Accommodation 4,7242710 LD, Read-aloud accommodation 1,3672911

15. Copyright © 2004 Educational Testing Service Page 15 No. of DIF Items Identified Using Total Test as Criterion Reference GroupFocal GroupB DIFC DIF No DisabilityLD, No Accommodation 1R No DisabilityLD, 504/IEP Accommodation 1R, 1W No DisabilityLD, Read-aloud Accommodation 6R1W LD, No Accommodation LD, 504/IEP Accommodation LD, No Accommodation LD, Read-aloud Accommodation 1R, 1W

16. Copyright © 2004 Educational Testing Service Page 16 No. of DIF Items Identified Using Reading Subtest as Criterion Reference GroupFocal GroupB DIFC DIF No DisabilityLD, No Accommodation 1R, 3W1W No DisabilityLD, 504/IEP Accommodation 2R, 4W1W No DisabilityLD, Read-aloud Accommodation 6R, 10W2W LD, No Accommodation LD, 504/IEP Accommodation LD, No Accommodation LD, Read-aloud accommodation 1R, 1W

17. Copyright © 2004 Educational Testing Service Page 17 No. of DIF Items Identified Using Writing Subtest as Criterion Reference GroupFocal GroupB DIFC DIF No DisabilityLD, No Accommodation 5R1R No DisabilityLD, 504/IEP Accommodation 5R, 1W2R No DisabilityLD, Read-aloud Accommodation 8R, 2W1W LD, No AccommodationLD, 504/IEP Accommodation LD, No AccommodationLD, Read-aloud accommodation 2R, 2W1R

18. Copyright © 2004 Educational Testing Service Page 18 Summary of Results Criterion –Total test score most reliable –Writing subtest score least reliable –Total test may be multi-dimensional –Total test identifies least amount of DIF items –Reading and writing subtests identify similar amounts of DIF –Using Reading as the criterion identifies mostly writing items as having DIF and using writing items as the criterion identifies mostly reading items as having DIF

19. Copyright © 2004 Educational Testing Service Page 19 Summary of Results (cont.) When reference group is students without disabilities, students with disabilities who took test with accommodations showed more DIF than students with disabilities who took test without accommodations Read-aloud accommodations result in increased DIF DIF is decreased for accommodated groups if reference group is students with disabilities

20. Copyright © 2004 Educational Testing Service Page 20 Conclusions Choice of matching criterion impacts results Disability, alone, results in DIF Accommodations result in DIF Read-aloud accommodations result in the most DIF Accommodations specified in 504/IEP do not result in DIF when reference group is students with disabilities who took test without accommodations