Students in the Gap: Understanding Who They Are & How to Validly Assess Them

Findings from Two Projects  Montana Modified Achievement Standards Pilot 1 Year Technical Assistance on State Data Collection Grant  New England Compact Study of Students in the Gap NH, ME, RI, VT 2 Year Enhancing State Assessment Grant

Montana Modified Achievement Standards Overview  Developed and administered a pilot alternate assessment  Based assessment on modified achievement standards  Evaluated feasibility of incorporating it into the statewide assessment system

New England Compact (NEC): Students in the Gap Looking at Gaps in Large-Scale Assessment Systems: Two Percent… or Two Gaps?

NEC Enhanced Assessment Grant  Funded by US Department of Education  Four states: NH, ME, RI, VT   Challenge: describe students in gap, and design an assessment that will meet needs of students in gap  8 th grade mathematics –8 th grade to look at complexity –Mathematics to avoid reading comprehension issues

Project history  Original goals −Identify students in the gaps −Develop varied assessment modules −Pilot/validate assessment modules  Issues we faced –Not easy to identify the gap, or the students –Impossible to develop an assessment without knowing target students’ needs

Revised goals Identify students in the gaps through multiple methods, triangulating evidence Define common criteria for identifying students in the gap Plan and develop task module assessment strategies (assessment prototypes) Recommend core components of an assessment structure that would lessen the gaps Disseminate products to others considering assessments for students in the gap

Accountability context Project began February 2005 Modified achievement standards announced April 2005 Proposed “2%” regulations released December 2005 Studies were designed before 2%, not in response to 2% Findings speak to needs of all students not effectively assessed in current system, not necessarily dovetailing with 2% definitions

Big questions Who are the students in the gaps? Of all the students who are not proficient, how can states identify those who are in the assessment gaps? What are the attributes of students in the gaps, and how do these students perform? What issues in the assessments themselves contribute to the gaps? Are there specific aspects of multiple- choice items used in state assessments that contribute to the assessment gaps?

Gap identification process Conduct exploratory interviews with teachers to identify the assessment gaps Review student assessment data Review teacher judgment data Operationalize gap criteria Conduct focused teacher interviews to confirm gap criteria Parker and Saxon: Teacher views of students and assessments Bechard and Godin: Finding the real assessment gaps

The process for investigating gap profiles Bechard and Godin: Who are students in gaps? Conduct focused teacher interviews to confirm gap criteria Investigate characteristics of students in gap 1 Investigate characteristics of students in gap 2 Investigate achievement patterns of students in gap 1 Investigate achievement patterns of students in gap 2 Develop profiles of students in gap 1 Develop profiles of students in gap 2

Alternative test items Hypothesize alternate test items Decompose items into requisite skills/ knowledge Provide alternative formats: Item format Item content Visuals Multimedia Review with mathematics experts Pilot and evaluate items Russell and Famularo: Utility of a prototype assessment Dolan et. al.: Providing students with choice

Session Goals Consider:  How we define “students in the gap”  If and how we alter achievement standards for “students in the gap”  How we assess “students in the gap”

Participants Judy Snow, MT Office of Public Instruction –Lessons from MT Modified Achievement Standards Project Carrie Parker, EDC –Teachers’ Views of Gaps in Large-Scale Assessment Sue Bechard, Measured Progress –Using State Assessment Data to Identify Students in the Gap Bob Dolan, CAST –What Literature Tells Us about Math Instruction and Assessment Mary Ann Snider, RI Department of Education –State Director of Assessment Gaye Fedorchak, NH Department of Education –Moderator, State Supervisor of Alternate Assessments

?

New England Compact Students in the Gap: Final Questions How can we discriminate between the instruction gap and the assessment gap? Should students be taught the skills that help to close the context gap between classroom and assessment? Are different assessment approaches needed for either gap group?

Final Questions (cont.) Can a test developed for the 2% option stretch low enough to reach students in gap 2? How should expectations be adjusted? The items? The accommodations or choices provided? The achievement standards? If you were in charge of a state assessment program, what else would you need to know before deciding how to proceed? What would the students say?

2% Flexibility Students in Gap 2 not tested on what they are taught Students in Gap 2 are performing far below grade level Regulations won’t allow off-grade level tests for Gap 2 A modified assessment to Gap 1 or non-Gap 1 would mean lower standards for students who could be at grade level Conclusion: The 2% flexibility option will not increase the validity or relevance of the test for students in either gap