1. Alternate Assessments of Modified Achievement Standards: Research on More Accessible & Less Difficult Items Stephen N. Elliott Vanderbilt University Designing & Evaluating Modified Items for Students with Disabilities: Research Results

2. NCME 20092 Project, Partners & Presenters CAAVES = Consortium for Alternate Assessment Validity and Experimental Studies (USDE funded; October 2006 - March 2009; 6 states involved) Partners = AZ, HI, ID, & IN + Vanderbilt Measurement Group + Discovery Ed. Assessment Presenters  Andrew Roach, Georgia State University  Ryan Kettler, Vanderbilt University  Michael Rodriguez, University of Minnesota

3. NCME 20093 Terminology Modification: alternations or adjustments of test items to facilitate access. Appropriate modifications  Remove extraneous material,  Maintain the same depth of knowledge (DOK),  Do NOT change the grade-level construct being measured, and  Increase the validity of the inference from the test score.

4. NCME 20094 Session Goals: Federal Policy, Item Modifications, & Results Briefly review AA-MAS policy from NCLB Review item modification development approaches & methods Discuss results and share observations about the findings and the implications for the design and use of alternate assessments of modified achievement standards

5. NCME 20095 Policy Context for Our Study Our research has the potential to inform current testing practices and policies concerning students with disabilities who have experienced persistent academic difficulties and poor performance on statewide assessments. USDE Regulations 34 CFR Part 200 (2007)

6. NCME 20096 2% Alternate Assessment Recent (i.e., 2007) NCLB regulations have permitted states to develop an alternate assessment based on modified academic achievement standards (AA-MAS) and report proficiency for 2% of the population based on the results of this new test.* The assessment must test grade-level content standards. To be eligible for participation in the 2% test, students must:  Have an IEP;  Be receiving high-quality instruction at grade-level;  Have a history of poor test performance;  Have persistent academic difficulties.

7. NCME 20097 CAAVES Project Goal #2 Goal #2 of the CAAVES Project is to “investigate feasibility of item modification strategies for future alternate assessments.” This goal was accomplished by (a) developing a common set of test items from existing reading and mathematics tests using modification principles that facilitate reading access and valid responses and (b) using a computer-based delivery system to experimentally examine student preferences, score comparability, and item statistics of the modified items for students with and without disabilities.

8. NCME 20098 To Accomplish Goal #2 We proposed and completed the following….. Modified a common set of existing reading and math items to create items designed to be more accessible and still measure the same grade-level content as the original items. Conducted a cognitive lab study with a small sample of students with and without disabilities to gain their insights into which item modifications are preferred and most likely to improve test access for students whose disability involves reading difficulties. Conducted a cross-state experimental study to compare the effects of tests with and without modified items on students’ test performances and test score comparability.

9. NCME 20099 Original Motivating Questions We wanted to answer the following questions about item modifications: 1. Will modifications in testing conditions change the skill being measured? 2. Will taking the test under modified conditions change the resulting scores? 3. Will non-disabled examinees benefit if allowed the same modifications?

10. NCME 200910 Participation Criteria for Students with Disabilities

11. NCME 200911 Multi-State Sample

12. NCME 200912CAAVES Item Mod Data 200812 Order of Forms and Conditions Students were randomly assigned to one of 36 possible reading and math tests comprised of 39 items that represented three types of multiple choice items: unmodified, modified, and modified with reading support.

13. NCME 200913 TAMI: A Tool to Guide Item Modifications The Test Accessibility and Modification Inventory (TAMI; Beddow, Kettler, & Elliott, 2008) was developed as a decision-making tool to facilitate the analysis of new and existing tests and test items with the purpose of enhancing their accessibility. The TAMI was influenced by 4 primary areas of study: 1. Universal design principles; 2. Cognitive load theory; 3. Research on test and item development; and 4. Guidance on web and computer accessibility.

14. NCME 200914 Anatomy of an Item Stimulus Visual Stem Answer Choices key (B) and distractors (A & C) Page Layout X X

15. NCME 200915 Passage / Item Stimulus 1. Passage / Item Stimulus 1.1 Passage and/or item stimulus vocabulary and sentence complexity are grade- appropriate. 1.2 Paragraphs are appropriate in length. 1.3 Each line in poems is numbered. 1.4 Idioms or jargon are avoided (unless they are defined and related to the item content). 1.5 Sentence structure is as simple as possible given the construct being measured. 1.6 Text includes only words essential for responding to the item(s), with minimal extraneous verbiage. 1.7 Essential words or vocabulary terms use bold font to facilitate identification. 1.8 Readability analyses indicate appropriate grade level.1.9 Passage or stimulus does not require construct irrelevant knowledge and skills for item response. 1.10 Passage and/or stimulus is viewable on the same page as visuals, item stem, and answer choices. 1.11 Other: ___________________________________________

16. NCME 200916 Item Stem 2. Item Stem 2.1 Item stem reflects intended content standard(s) and/or objective(s). 2.2 Text includes all requisite information for responding. 2.3 Text includes only words essential for responding to the item, with minimal extraneous verbiage. 2.4 Readability analyses indicate appropriate grade level.2.5 Item vocabulary and sentence complexity are grade-appropriate. 2.6 Item stem is as direct as possible and uses the active voice.2.7 Idioms or jargon are avoided (unless defined and related to the item content). 2.8 Essential words or vocabulary terms use bold font to facilitate identification. 2.9 Item stem is positively worded (i.e., avoiding not questions). 2.10 When the item references a specific part of a passage or poem, the relevant sentence or excerpt is included with the item stem. 2.11 Other: ___________________________________________

17. NCME 200917 Visuals 3. Visuals 1 3.1 Visuals are included when necessary. 3.2 Visuals are relevant to essential item content. 3.3 Visuals are connected to the text (e.g., pictures, graphics, direction icons). 3.4 Visuals clearly represent intended images. 3.5 Visuals use colors with sufficient contrast. 3.6 Visuals convey information using more than distinctions between colors. 3.7 Essential words or vocabulary terms use bold font to facilitate identification. 3.8 Visuals include labels and supportive text if necessary (e.g., titles, row and column headers for graphs). 3.9 Circles, arrows, or highlighting are used for important elements of visuals. 3.10 Supportive text is located close to related visuals. 3.11 When visuals are self-explanatory, words are not added. 3.12 Other: ___________________________________________

18. NCME 200918 Answer Choices 4. Answer Choices 4.1 Text includes only words essential for responding to the item, with minimal extraneous verbiage. 4.2 Answer choices are about equal in length. 4.3 All distractors are plausible (“attractors”). 4.4 All answer choices (of a multiple choice item) are necessary. 4.5 No distractors could be eliminated without reducing the validity of the inference(s) intended by the item. 4.6 The order and construction of the answer choices are logical and deliberate. 4.7 Other: ____________________________________________

19. NCME 200919 Page / Item Layout 5. Page / Item Layout 5.1 All text is printed in standard typeface, using a minimum of 12-point text. 5.2 There is sufficient space between lines (leading). 5.3 There is high contrast between text and background colors. 5.4 All text can be read in a traditional, left-to-right, top-to-bottom way, with consistent left- margins. 5.5 Text is formatted appropriately with regard to breaks and unbroken blocks. 5.6 The entire item (stimulus, visuals, stem, and answer choices) is visible on the same page/screen. 5.7 Page includes ample white space to prevent the item from appearing cluttered. 5.8 Right margin is unjustified (staggered). 5.9 Visuals are integrated into the item stimulus and item stem, rather than placed off to the side. 5.10 Other: ___________________________________________

20. NCME 200920CAAVES Item Mod Data 200820 Item Summary Reports: An Example

21. NCME 200921 Group by Condition Overview In the remainder of this session we focus on two sets of initial analyses: (1) group performance comparisons on reading and math tests and (2) item difficulty and distractor analyses.

22. NCME 200922 Thanks! Thank you very much for your time and joining us for this session. Please provide follow-up questions and suggestions in writing to: Steve.Elliott@vanderbilt.edu http://peabody.vanderbilt.edu/LSI_Projects/CAAVES_Project_Home.xml