1. Modifying Achievement Test Items: A Theory-Guided & Data-Based Approach Stephen N. Elliott Learning Sciences Institute and Department of Special Education Vanderbilt University CCSSO 2009 National Conference on Student Assessment

2. CCSSO NSA Conference / Elliott 20092 Projects & Partners CAAVES: Consortium for Alternate Assessment Validity and Experimental Studies –USDE funded; 2006-2009 –Partners: AZ, HI, ID, & IN + Vanderbilt Measurement Group + Discovery Education Assessment CMAADI: Consortium for Modified Alternate Assessment Development and Implementation –USDE funded; 2007-2010 –Arizona Dept. of Education –Indiana Dept. of Education Visit Websites for Resources Discussed Today –http://peabody.vanderbilt.edu/LSI_Projects/CAAVES_Project_Home.xml –http://peabody.vanderbilt.edu/LSI_Projects/C-MAADI_Project_Home.xml –http://peabody.vanderbilt.edu/LSI_Projects/CAAVES_Project_Home/TAMI_Project.xml

3. CCSSO NSA Conference / Elliott 20093 Key Terms AccessAccess: the opportunity for test-takers to demonstrate proficiency on the target construct of a test or a test item. In essence, complete access is manifest when a test-taker is able show the degree to which he/she knows the tested content. Access, therefore, must be understood as an interaction between individual test-taker characteristics and features of the test itself. AccommodationAccommodation: wide ly recognized in state testing guidelines as individualized changes to the setting, scheduling, presentation format, or response format of an assessment. ModificationModification: alterations or adjustments of test items to facilitate access for virtually all test takers. Appropriate modifications …. –Remove extraneous material, –Maintain the same depth of knowledge (DOK), –Do NOT change the grade-level construct being measured, and –Increase the validity of the inference from the test score.

4. CCSSO NSA Conference / Elliott 20094 Evolving Modification Paradigm Step 1. Evaluate original item accessibility. Step 2. Reduce sources of construct-irrelevant variance in items. Step 3. Document changes to items. Step 4. Pilot test with student cognitive labs & post-assessment focus groups. Step 5. Field test with large sample of students. Step 6. Conduct psychometric & related analyses.

5. CCSSO NSA Conference / Elliott 20095 Goal #2 of the CAAVES Project is to “investigate feasibility of item modification strategies for future alternate assessments.” This goal was accomplished by: (a) developing a common set of test items from existing reading and mathematics tests using modification principles that facilitate reading access and valid responses and (b) using a computer-based delivery system to experimentally examine student preferences, score comparability, and item statistics of the modified items for students with and without disabilities. CAAVES Project Goal #2

6. CCSSO NSA Conference / Elliott 20096 Motivating Questions We wanted to answer the following questions about item modifications: 1.Will modifications in testing conditions change the skill (construct) being measured? 2.Will taking the test under modified conditions change the resulting scores? 3.Will non-disabled examinees benefit if allowed the same modifications?

7. CCSSO NSA Conference / Elliott 20097 Procedures for Addressing Questions We completed the following….. Modified a common set of existing reading and math items to create items designed to be more accessible and still measure the same grade-level content as the original items. Conducted a cognitive lab study with a small sample of students with and without disabilities to gain their insights into which item modifications are preferred and most likely to improve test access for students whose disability involves reading difficulties. Conducted a cross-state experimental study to compare the effects of tests with and without modified items on students’ test performances and test score comparability. Conducted post-assessment survey of all students concerning their perceptions of item types and cognitive ease.

8. CCSSO NSA Conference / Elliott 20098 Participation Criteria for Students with Disabilities

9. CCSSO NSA Conference / Elliott 20099 Anatomy of an Item

10. CCSSO NSA Conference / Elliott 200910 Item Summary Reports: An Example

11. CCSSO NSA Conference / Elliott 200911 Overview of Results Elliott, et al. (in press) Exceptional Children

12. CCSSO NSA Conference / Elliott 200912 Tools to Analyze Items & Guide Modifications The Test Accessibility and Modification Inventory (TAMI; 2008) and TAMI Accessibility Rating Matrix (ARM; 2009) were developed as decision-making tools to facilitate the analysis of new and existing tests and test items with the purpose of enhancing their accessibility.

13. CCSSO NSA Conference / Elliott 200913 TAMI Item Accessibility Ratings TAMI Item Accessibility Ratings

14. CCSSO NSA Conference / Elliott 200914 TAMI Item Accessibility Ratings TAMI Item Accessibility Ratings

15. CCSSO NSA Conference / Elliott 200915 TAMI Item Accessibility Audits TAMI Item Accessibility Audits IN CMAADI Item Review, 2009

16. CCSSO NSA Conference / Elliott 200916 Method for Documenting OTL AZ Cog Lab Study, 2008

17. CCSSO NSA Conference / Elliott 200917 Cognitive Labs Excerpted from Kettler, Elliott, & Beddow, in press Peabody Journal of Education

18. CCSSO NSA Conference / Elliott 200918 Post-Assessment Focus Groups Post-Assessment Focus Groups AZ CMAADI Pilot Study, 2009

19. CCSSO NSA Conference / Elliott 200919 Theory-Guided & Data-Based Modifications

20. CCSSO NSA Conference / Elliott 200920 Review: Evolving Modification Paradigm Step 1. Evaluate original item accessibility. Step 2. Reduce sources of construct-irrelevant variance in items. Step 3. Document changes to items. Step 4. Pilot test with student cognitive labs & post-assessment focus groups. Step 5. Field test with large sample of students. Step 6. Conduct psychometric & related analyses.

21. CCSSO NSA Conference / Elliott 200921 Thanks! Thank you very much for your time and joining us for this session. Please provide follow-up questions and suggestions in writing to:Steve.elliott@vanderbilt.edu http://peabody.vanderbilt.edu/LSI_Projects/CAAVES_Project_Home.xml