Presentation on theme: "1 Marylands Alternate Assessments based on Grade-Level Content Standards & Modified Academic Achievement Standards Trinell Bowman, MSDE Carolyn Wentzel,"— Presentation transcript:
1 Marylands Alternate Assessments based on Grade-Level Content Standards & Modified Academic Achievement Standards Trinell Bowman, MSDE Carolyn Wentzel, Lora Monfils, and Lisa Baumann, ETS MAG Conference, 2008
2 ETS and MSDE staff will give an overview and answer all your questions about the new Modified High School Assessment – the Mod-HSA. What is the Mod-HSA? How was it developed? How can it benefit schools and students? When and how will it be administered? How will it be reported? Overview
3 Requirements for Testing Students with Disabilities Students with disabilities will participate in general state and district wide assessments, with appropriate accommodations, where necessary. Sec. 612(a)(17)(A) For students with disabilities who are determined to be unable to participate in general assessments, alternate assessments will be used. Sec. 612(a)(17)(A)(ii) IDEA REQUIREMENTS
4 Requirements for Testing Students with Disabilities Beginning in , all students in grades 3-8, including students with disabilities (with reasonable adaptations and accommodations, as necessary) will participate in annual assessments in, at a minimum, mathematics and reading/language arts, and, beginning no later than , in science. Sec. 1111(b)(3)(C) Accommodations, guidelines, and alternate assessments must be provided in the same manner as under Sec. 612(a)(17) of the IDEA. Sec. 1111(b)(2)(I) NCLB REQUIREMENTS
5 Goals for Inclusion of Students with Disabilities in the Maryland Assessment Program Provide ALL students access to the general curriculum Provide ALL students with assessments aligned with grade-level content standards Ensure both special and general educators set high expectations for students with disabilities
6 A students disability must not determine which assessment the student will take. There is no limit on the number of students TAKING any particular assessment. A students IEP team will make the decision as to which assessment a student will take. Goals for Inclusion of Students with Disabilities in the Maryland Assessment Program
7 Maryland State Assessments Maryland Model for School Readiness (MMSR) Alternate MSA (Alt-MSA) Maryland School Assessment (MSA) Modified MSA (Mod-MSA) High School Assessment (HSA) Modified HSA (Mod-HSA) 7
8 A students disability must not determine which assessment the student will take. There is no limit on the number of students TAKING any particular assessment. A students IEP team will make the decision as to which assessment a student will take. Mod-HSA Considerations
9 Provide a measure of students with disabilities performance compared with grade-level academic content standards Designed to meet the needs of students specific instructional characteristics, such as the need for: Shorter reading passages More visual cues Scaffolding questioning Few items presented on a page What are Modified Achievement Standards?
10 Provide measure of students with disabilities compared with grade-level academic content standards Based on the same academic content standards as the states regular assessment (MSA) Provide teachers and parents with information that will help guide access to the general education curriculum and instructional strategies What are Modified Achievement Standards?
11 What have we learned so far? WHO ARE THE STUDENTS?
12 Who Are The Students? What have we learned so far? Students with all categorical disabilities Students who have difficulty processing complex information Students who are easily distracted Students who have difficulty completing written tasks Students who require multiple accommodations such as extended time, verbatim reading, calculator, graphic organizers, multiple or frequent breaks
13 A student with an IEP working on grade level content and working towards completion of the high school diploma A student making progress on grade-level academic IEP goals, but not a year for a year Has had multiple interventions over consecutive years Does not qualify for Alt-MSA eligibility (not a student with significant cognitive disabilities) Who is a Mod-Eligible Student?
14 Modified Assessments and General Requirements Students may be eligible to take the Mod-HSA in one content area and take the HSA in another content area.
15 What was the Process for Test Development of the Mod-HSAs? Teams of General Educators and Special Educators received Initial training: overview and a public release SR, SPR and BCR were modified and then another similar item allowed the teams to practice, discuss and share their modifications. This process was done in several stages that allowed the teams to look at specific indicators and their assessment limits and skill statements to ensure continued alignment with the Core Learning Goals, Expectations, and Indicators.
16 What was the Process for Test Development of the Mod-HSAs? Alterations of test items were based on: Adhering to the CLG assessment limits and skill statements Simplified language while not compromising the content vocabulary Lowered the cognitive demand while not compromising the content such as using friendlier numbers in presentation of algebra or biology data charts
17 What was the Process for Test Development of the Mod-HSAs? Converted all items to SRs with 3 answer choices Used data to delete an answer choice from SRs Stem: Extra line space between stimulus and answer choices Chunked information when appropriate. Extra line space added before test question.
18 What was the Process for Test Development of the Mod-HSAs? Passages must often be presented in full for a student to be able to respond to certain questions (e.g., main idea); but when an item is based on only part of the passage, present just the relevant section of the passage again immediately before the item Avoid using A, B, or C to label information in an item stem…in the stem when the answers are numbers or something else. BUT if the question asks the student to select a method or company… then it is appropriate to use A, B, C as labels.
19 What was the Process for Test Development of the Mod-HSAs? Selected Responses only; no Brief Constructed Responses (BCRs), no Extended Constructed Responses (ECRs) Deleted extraneous information when possible Use pictures to support passage Online Assessment using Pearsons Test Nav system (currently used for the MSA Science and Mod-HSAs)
20 Mod-HSA Algebra/Analysis Test Format Sample Test Item Look at the equation below. 125 x N = 375 What value of N makes this equation correct? Sample Modified Test Item Look at the equation below. 125 x N = 375 What value of N makes this equation correct? What is the value of N? (distracter)
21 What are the Benefits for Students? Eligible to take the Mod-HSA in one subject and take the HSA in another subject Test is more accessible and students with IEPs will do better All accommodations available Students do not need to pass to benefit from participation in assessment Assist in increasing points towards composite score of 1602
22 Michelle Daley Division of Special Education/Early Intervention Services Trinell Bowman Division of Accountability & Assessment Contact Information
23 Psychometric Analyses to Support the Modified-HSAs (Mod-HSAs) Carolyn Wentzel and Lora Monfils, ETS MAG Conference, 2008
24 Introduction Overview Psychometric Goals Test Structure & Student Groups Psychometric Analyses Comparability of Mod-HSAs & HSAs Confirmation of Cut Scores Test Score Summary Results
25 Overview Mod-HSAs are modified versions of the HSAs comprised of SR items only Designed to be taken in place of one or more of the HSAs Two administration options: online & paper and pencil First administered in May 2008
26 Psychometric Goals for May 2008 Administration Build Mod-HSA operational forms to the HSA blueprints using 50 modified HSA SR items Link Mod-HSA scores to the HSA reporting scale ( ) Confirm use of HSA cut scores for Mod-HSAs
27 Test Structure 2 forms per content area with modified HSA items + 24 unmodified HSA items, common to both forms 50 items per form would be selected and retained as operational items
28 HSA Linking Items 24 HSA items, in each content area, used to link the Mod-HSA to the HSA scale Items selected to be representative of the blueprint so as to comprise a mini-test Items selected across a range of difficulties Linking items did not count toward final scores
29 Student Groups Two groups of students took May 2008 Mod-HSAs: 1) Target population: Special education students eligible to take the Mod-HSA based on IEP and progress in learning course content (2,000 – 2,700 students) 1) Linking samples: Subset of students who took the May HSA in the same content area (1,900 – 3,000 students)
30 Psychometric Analyses: IA & DIF Classical item analysis (IA) Obtain item raw score statistics Percent of students obtaining correct response Correlation of correct and incorrect responses with test scores Omit rates IA statistics obtained for both groups Differential item functioning analysis (DIF) Compare item performance between two groups matched on the total score of a test DIF statistics obtained for both groups
31 Psychometric Analyses: IRT Scaling Data from linking samples used for Item Response Theory (IRT) scaling Items calibrated concurrently using three parameter logistical (3PL) item response model HSA linking items had two sets of parameters : Parameters estimated during the current calibration Banked parameters expressed on the HSA reporting scale Stocking and Lord (1983) procedure used to align the TCCs based on the two sets of linking item parameters Resulting constants used to transform the Mod-HSA parameters and express them on the HSA score scale
32 Criteria to Select Operational Items Blueprint coverage maintained IA statistics within recommended range 0.10 < P-values < 0.90 Item-total correlations 0.10 for correct responses Item-total correlation weak or negative for incorrect responses Percent of students omitting an item 5% DIF: No evidence of differential item performance IRT: Well-estimated during scaling
33 Comparability of Mod-HSAs & HSAs for Linking Samples Correlations ranged from 0.70 (English) to 0.76 (Biology) Average performance similar Variability greater for Mod-HSAs
34 Scatterplot of Mod-HSA & HSA Scale Scores for Algebra
35 Confirmation of Cut Scores Standard settings conducted. Panels consisting of general educators and special educators. Verified that cut scores established for HSAs could be used. Common scale and cut scores permits Mod- HSA and HSA to be used to create composite scores.
36 Results: Score Summary Statistics by Group
37 Additional Information Mod-HSA technical report available early 2009 Posted to:
38 Test Administration Mod-HSA follows the same testing calendar as the HSA Mod-HSA test takers are to be included in the same pretest file as the HSA test takers Mod-HSA is available for online testing or paper and pencil testing Mod-HSA has an online practice test
39 Test Administration Mod-HSA has the Large Print, Braille and Kurzweil versions of the test forms and an audio test form is also available for online Mod-HSA has a set of Examiners Manuals
40 Questions & Reflections
41 ETS Contact Information Lisa Baumann Carolyn Wentzel Lora Monfils