1. Part 1: Overview Mimi Fuhrman, American Institutes for Research Part 1: Overview Mimi Fuhrman, American Institutes for Research

2. How to develop effective assessments ? Assessments should be RELIABLE and VALID Reliability: A reliable assessment is one that provides consistent “scores”. Validity: An assessment is valid when it accurately measures what it purports to measure. Assessments should be RELIABLE and VALID Reliability: A reliable assessment is one that provides consistent “scores”. Validity: An assessment is valid when it accurately measures what it purports to measure.

3.  If poorly designed, often limited to measuring simple recall or trivia  Not appropriate for measuring ability to synthesize ideas, to write effectively, or to perform certain types of problem- solving operations  Difficulty in making the assessment “authentic”  If poorly designed, often limited to measuring simple recall or trivia  Not appropriate for measuring ability to synthesize ideas, to write effectively, or to perform certain types of problem- solving operations  Difficulty in making the assessment “authentic” Multiple-choice test disadvantages:

4. Example of an “authentic” MC item A stratigrapher observes a series of cyclothems, each between 5 and 20 meters thick consisting of, from top to bottom: TopRed, bioturbated shales with small, irregular, calcite nodules Thin-bedded, ripple-laminated, fine-grained sandstone Thick-bedded, medium- to coarse-grained sandstones with unimodal trough cross-bedding BottomErosion surface overlain by coarse, pebbly sandstone What depositional environment is represented? (A)Eolian (B)Littoral (C)Alluvial (D)Deltaic (E)Neritic A stratigrapher observes a series of cyclothems, each between 5 and 20 meters thick consisting of, from top to bottom: TopRed, bioturbated shales with small, irregular, calcite nodules Thin-bedded, ripple-laminated, fine-grained sandstone Thick-bedded, medium- to coarse-grained sandstones with unimodal trough cross-bedding BottomErosion surface overlain by coarse, pebbly sandstone What depositional environment is represented? (A)Eolian (B)Littoral (C)Alluvial (D)Deltaic (E)Neritic

5.  High reliability; good sampling of content  Ability to obtain a wide range of scores  Easier to frame questions so that all examinees will address themselves to the same problem  Low cost (time)  High reliability; good sampling of content  Ability to obtain a wide range of scores  Easier to frame questions so that all examinees will address themselves to the same problem  Low cost (time) Multiple-choice test advantages

6. Short answer vs. Multiple-choice 15 short answer questions administered followed immediately by identical questions, multiple-choice. Scores were 43% higher on multiple- choice version. (from Shea, 1992) “What portion of a parent isotope remains after four half lives have elapsed?” 15 short answer questions administered followed immediately by identical questions, multiple-choice. Scores were 43% higher on multiple- choice version. (from Shea, 1992) “What portion of a parent isotope remains after four half lives have elapsed?”

7. What portion of a parent isotope remains after four half lives have elapsed?  Short answer responses:  15  Daughter isotope  Nucleus  Uranium  10 million years  Short answer responses:  15  Daughter isotope  Nucleus  Uranium  10 million years  Multiple-choice options A. one-fourth B. one-eighth C. one-sixteenth D. virtually none

8. Scoring

9.  Free- or constructed response  Use of rubrics for holistic or analytical scoring  Most time investment AFTER administration  Multiple-choice  Options determine what is correct and what is incorrect  Scoring decisions and major time investment when the items are developed, BEFORE administration  Free- or constructed response  Use of rubrics for holistic or analytical scoring  Most time investment AFTER administration  Multiple-choice  Options determine what is correct and what is incorrect  Scoring decisions and major time investment when the items are developed, BEFORE administration Scoring IS an issue for multiple-choice :

10.  Alignment: to content, learning outcomes/goals, cognitive demand, difficulty  Importance: testing trivia is a waste of resources; concentrate on fundamental concepts  Clarity: the intent of the task and the meaning of the options must speak for themselves and be interpreted in the same way by all examinees  Item construction: well-constructed items can be answered successfully by examinees who have the knowledge or skill you are testing, and NOT by examinees who are lacking the skill or knowledge  Alignment: to content, learning outcomes/goals, cognitive demand, difficulty  Importance: testing trivia is a waste of resources; concentrate on fundamental concepts  Clarity: the intent of the task and the meaning of the options must speak for themselves and be interpreted in the same way by all examinees  Item construction: well-constructed items can be answered successfully by examinees who have the knowledge or skill you are testing, and NOT by examinees who are lacking the skill or knowledge What makes a “Good” Multiple-choice item?

11. Performance Data : Example item from 8 th grade test How does most sand get to the ocean shore? A) Boulders near the ocean break down into sand and are deposited on the shore. B) Rocks break down into smaller pieces as they are transported by rivers to the ocean shore.* C) Rocks dissolve in rivers and then crystallize as sand when the river reaches the ocean shore. D) Wave action breaks down ocean floor rock and pushes the resulting sand onto the ocean shore. How does most sand get to the ocean shore? A) Boulders near the ocean break down into sand and are deposited on the shore. B) Rocks break down into smaller pieces as they are transported by rivers to the ocean shore.* C) Rocks dissolve in rivers and then crystallize as sand when the river reaches the ocean shore. D) Wave action breaks down ocean floor rock and pushes the resulting sand onto the ocean shore. Performance Data: percent answeredbiserial 1 A) 9.0953-0.0875 B) 16.6909*-0.0560* C) 17.9971-0.0291 D) 48.0890+ 0.2290 1: biserial = correlation between performance on item and performance on test

12.  Requirements for community use --- use across classrooms, instructors, institutions Agreement about learning goals Context/scenarios should be equally familiar for all examinees Vocabulary/terminology should be standard or clearly explained Maximize usefulness by sharing performance data  Requirements for community use --- use across classrooms, instructors, institutions Agreement about learning goals Context/scenarios should be equally familiar for all examinees Vocabulary/terminology should be standard or clearly explained Maximize usefulness by sharing performance data Consideration for Community Tools

13.  If the community can agree on a set of common learning goals --- “core” content/concepts/skills MC instruments can be used to measure how widespread misconceptions are Results can guide instruction, on a local level AND in the community (compare with Force Concept Inventory used in physics)  The development of the instruments/items themselves can serve as a focus the community: agreement = importance  If the community can agree on a set of common learning goals --- “core” content/concepts/skills MC instruments can be used to measure how widespread misconceptions are Results can guide instruction, on a local level AND in the community (compare with Force Concept Inventory used in physics)  The development of the instruments/items themselves can serve as a focus the community: agreement = importance How multiple-choice items may be useful for Community Tools