Presentation on theme: "Adding Value to Accommodations Decision-Making (AVAD) 2006 Enhanced Assessment Instruments Grant Project Elizabeth Jones and Christopher Webster, South."— Presentation transcript:
Adding Value to Accommodations Decision-Making (AVAD) 2006 Enhanced Assessment Instruments Grant Project Elizabeth Jones and Christopher Webster, South Carolina Department of Education
2006 Enhanced Assessment Instruments Grant Award The South Carolina Department of Education received a 2006 Enhanced Assessment Instruments Grant award to undertake the AVAD project.
Overall goal of AVAD: To ensure the valid, effective use of accommodations designed to increase access to large-scale assessments for English language learners and students with disabilities
AVAD Project Partners South Carolina Department of Education, lead state and fiscal agent LEP SCASS ASES SCASS Michigan State University: Rebecca Kopriva University of Oregon: Gerald Tindal University of Nebraska/Psychometric Inquiries: Barbara Plake Phoebe Winter, project evaluator
AVAD Project Period Current project period from October 1, 2006 to March 31, 2008 One-year no-cost extension will be requested SCASS grant activities will begin in spring/summer 2007 SCASS grant activities will be scheduled during or immediately before/after regularly scheduled SCASS meetings
AVAD’s Objectives: 1)to validate and enhance the Selection Taxonomy for English Language Learner Accommodations (STELLA) and publish the program on the SDE’s Web site 2)to create and validate a decision-making taxonomy for the Accommodation Station (AS) based on the ASES Accommodations Manual and publish the program on the SDE’s Web site
AVAD’s Objectives, cont.: 3) to develop 20 prototypes of computer-based access enhanced science items for grade four, and 4) to disseminate results through reports, the SDE’s Web site, User and Technical Manuals for STELLA and AS, the ASES Professional Development Guide video, and Web-Ex teleconferences
What role does the LEP SCASS play? LEP SCASS will serve as expert panel to provide input and feedback on STELLA and computer-based access- enhanced prototype items Opportunity to participate in a subgroup of states within the SCASS focusing on accommodations issues Accommodations subgroup will commit to participation in AVAD meetings/conference calls Updates on AVAD will be provided to the larger SCASS group (ASES SCASS will play a similar role in Accommodation Station design and development)
What is STELLA? STELLA=Selection Taxonomy for Testing English Language Learners an online system to assist educators in assigning testing accommodations to English language learners (ELL) so that individual students receive appropriate accommodations developed by a 2003 Enhanced Assessment Instruments grant
Data Collection Teacher Form Parent Form Records Form Preloaded in System Student Information Variables Test Accommodations Conversion and Consolidation Rules Decision-Making Rules Output Student Profile Accommodation Decisions for Each Student Associated Materials Data Collection Manual Output Interpretation Manual Technical Manual STELLA Overview
Current Accommodations loaded in STELLA system Pre-Test Best Practice Accommodations Family Assessment Night Tailored Classroom Support Forms Standard or some Universal Design Forms Access-based Form L1 or Side-by-Side forms as available Tools Bilingual word list, general or test specific Picture-word dictionary Problem solving tools
Current Accommodations loaded in STELLA system Administration Small group Individual Administration Oral English Oral L1 Language Liaison Extra time More frequent breaks Response Written L1 or code switching Oral English Oral L1 or code switching Demonstrated or modeled response
STELLA Screenshots What does STELLA look like?
Records Form Language of instruction English language proficiency test L1 proficiency test
Records Form Type of ELL program
L1 Proficiency in four domains Parent/Guardian Form
Attendance in U.S. schools Schooling in native country
Classroom equipment, books, and supplies in native country Types of assessments in native country
Grading and assignments Test scores and experiences with testing
English and L1 proficiency (L1 includes ‘don’t know’ option) Standardized score accuracy and judgments about reasons for inaccuracy Student’s experience with standard test formats Student’s perceived purpose of standardized testing Classroom test condition options Condition options that help student on classroom tests, evaluations Teacher Form
Experience with standardized testing Testing conditions used in the classroom
Beneficial testing conditions in the classroom
System Output Individual Student Profile English language proficiency, 4 domains L1 proficiency, 4 domains Cultural proximity, 3 variables: 1. Time in U.S. 2. Experience with testing procedures 3. Schooling proximity, native country to U.S. experience
System Output: STELLA, Teacher Recommended and SEA/LEA Allowed Since different states allow different accommodations, STELLA program output is set up to identify the subset of allowable accommodations for each student. Non allowable accommodations selected for each student based on research and other best practice literature are also displayed in gray. This provides guidance to teachers and others about additional accommodations that could be useful for ELL students with specified needs.
Test Conversion Rules Currently, output from 4 English language proficiency tests (including ELDA) is preloaded. There is also the opportunity for personnel to add results from another test. In all cases score conversion rules place students on a common scale with four levels: Beginner ELL Low intermediate ELL High intermediate ELL Grade-level-competitive ELL
Student Information Consolidation Rules In several cases, more than one piece of student information is used to make judgments about his or her level on relevant variables. Two types of consolidation rules are part of STELLA. These are: Consolidation of data from related items Consolidation of information from more than one source
Time in School = LOW If the student has been in the US less than 1 academic year. OR If the student has been in US between 1 and 2 years AND has missed more than two months of school per year for 1 or more years. OR If the student has been in US between 2 and 3 years AND has missed more than two months of school per year for more than one year. Combining Data from More Than One Question: An Example
Combining Data from More Than One Source: An Example Consolidation rules are also used to determine L1 proficiency and English language proficiency. For L1 proficiency, consolidation rules are applied to information from teacher, parent and records forms.
Parents describe student as performing above grade level in L1 Teacher describes student as performing below grade level in L1 Parents trump teacher’s knowledge of L1, but test scores of L1 proficiency trump parents. What Trumps What? Consolidation Rules Example
Test Accommodation Decision-Making Rules A beginning set of decision-making rules were developed and tested. These rules take relevant student information and pair it with relevant accommodation factors for individual students. The accommodation factors identify which student needs the specific accommodation was designed to remediate. In this way, individual students are matched with accommodations appropriate to their particular needs.
LEP SCASS Validation and Enhancement of STELLA Activities The LEP SCASS accommodations sub-group will validate existing accommodations decision-making rules by participating in a two-meeting and online expert panel process led by Barbara Plake. Profiles of ELL students will be matched with specific sets of accommodations; accommodations for individual student profiles will be rated for validity and effectiveness; results will be compared to existing STELLA accommodations decision-making rules. STELLA decision-making structure will be tweaked based on results of expert panel process. STELLA interface and forms will be reviewed and revised.
Computer-based access-enhanced (CAE) science items review CAE items aim to reduce barriers to access that English language learners may encounter in typical large-scale assessment items without altering the construct being measured or the rigor of the items South Carolina will develop 20 prototypes of computer-based access-enhanced science items for grade four and place them online LEP SCASS will review the CAE items and provide feedback
Benefits of Participation LEP SCASS member states will have input into the design and development of STELLA LEP SCASS member states (and any other interested parties) will have free access to the STELLA once it is completed and placed on the SDE Web site LEP SCASS member states will be able to adapt the STELLA to best fit the policy and needs of their states
ASES SCASS: Parallel Activities The ASES SCASS will serve as an expert panel for the development and review of the Accommodation Station (AS), an online decision-making model designed to assist IEP teams in assigning valid and appropriate accommodations to individual students with disabilities. Current AS collects and consolidates information but does not include a decision-making algorithm. The ASES manual and materials will serve as the basis for an algorithm that will be built and incorporated into the current AS. A professional development video component will be developed to accompany existing ASES materials. Sharon Hall of ASES will share additional details.
Questions? Please contact Therese Gleason Carr at or (803) Thank you very much for your collaboration!