Presentation on theme: "Writing Constructed Response Assessment Items:"— Presentation transcript:
1Writing Constructed Response Assessment Items: One Element of a Comprehensive Assessment System
2What are Constructed Response Items? Open-ended QuestionsRequire several sentences or brief paragraphRequire higher level thinking (than simple recall) and the application of students’ knowledgeMaking ComparisonsIdentifying PatternsEvaluating Points of ViewMaking GeneralizationsSynthesizing InformationAllow for the examination of Student ThinkingScored using a Rubric that provides varying degrees of CreditWisconsin Dept. of Public Instruction, May 2007
3Why Use Constructed Response Items? “The notion that learning comes about by accretion of little bits is outmoded learning theory. Current models of learning … contend that learners gain understanding when they construct their own knowledge and develop … interconnections among facts and concepts …”(Shepard, quoted in Michigan Curriculum Framework)
4Constructed Response as part of a Coordinated Assessment System: “An assessment system is an ongoing coordinated process of collecting information for the purpose of continuously improving student learning.”Michigan Curriculum Framework
5Elements of a Coordinated Assessment System: Fixed Response:Simple items that assess factual information / discrete skills. Typically there is a single best answer.Constructed Response:Short answers, explanations, essays or diagrams that involve analysis or evaluation. Require judgment-based scoring (i.e. rubrics).Oral Questioning in Class: Could be fixed or constructed response.Teacher Observation: Typically involve a simple checklist.Performance: Speeches, experiments, debates, etc. Require judgment-based scoring.Project: Designing and/or building useful things. May be done individually or collectively.Portfolio: A collection of student work over time – written work, artistic creations, project reports, etc.Standardized Monitoring: e.g. MEAP/MME, ACT, SAT, NWEA, etc.Adapted from Michigan Department of Education, Science Education Guidebook; and Wiggins, G. & McTighe, J (1998). Understanding by Design. Association for Supervision and Curriculum Development (ASCD).
6WHEN to use Constructed Response: Consider the VERB of the Content Expectation:B2.5i Relate cell parts/organelles to their function.P2.1A Calculate the average speed of an object using the change of position and elapsed time.C5.5A Predict if the bonding between two atoms of different elements will be primarily ionic or covalent.E5.3e Determine the approximate age of a sample, when given the half-life of a radioactive substance along with the ratio of daughter to parent substances present in the sample.Can be assessed with FIXED RESPONSE Items
7WHEN to use Constructed Response: Consider the VERB of the Content Expectation:B1.1A Generate new questions that can be investigated in the laboratory or field.P1.1g Use empirical evidence to explain and critique the reasoning used to draw a scientific conclusion or explanation.C5.8B Draw isomers for simple hydrocarbons.E5.4C Analyze the empirical relationship between the emissions of carbon dioxide, atmospheric carbon dioxide levels and the average global temperature over the past 150 years.Best assessed with CONSTRUCTED RESPONSE Items
8WHEN to use Constructed Response: Consider the VERB of the Content Expectation:B1.1C Conduct scientific investigations using appropriate tools and techniques.P1.2C Develop an understanding of a scientific concept by accessing information from multiple sources. Evaluate the scientific accuracy and significance of the information.Best assessed with a PERFORMANCE exerciseE1.2D Evaluate scientific explanations in a peer review process or discussion format.Best assessed through TEACHER OBSERVATION
9HOW to use Constructed Response Items: Standards for Assessment Tasks: 1. Organization of Information —The task asks students to organize, synthesize, interpret, explain, or evaluate complex information in addressing a concept, problem or issue.2. Consideration of Alternatives —The task asks students to consider alternative solutions, strategies, perspectives, or points of view in addressing a concept, problem or issue.3. Disciplinary Content —The task asks students to show understanding and/or use ideas, theories, or perspectives considered central to an academic or professional discipline.4. Disciplinary Process —The task asks students to use methods of inquiry, research, or communication characteristic of an academic or professional discipline.5. Elaborated Written Communication —The task asks students to elaborate on their understanding, explanations, or conclusions through extended writing.6. Problem Connected to the World Beyond the Classroom —The task asks students to address a concept, problem or issue that is similar to the one that they have encountered or are likely to encounter in life beyond the classroom.7. Audience Beyond the School —The task asks students to communicate their knowledge, present a product or performance, or take some action for an audience beyond the teacher, classroom, and school building.Michigan Curriculum Framework
10HOW to Develop Constructed Response Items: The Specifics Set the ContextSpecify the knowledge to be brought to bearSpecify the ReasoningUse specific verbs e.g. analyze, cite, describe…Point the WayInform students of the criteria that will be applied to evaluate their responsesDevelop the Scoring RubricClear articulation of the appropriate evaluation criteria by which to judge the quality of student responses.Adapted from Classroom Assessment for Student Learning, Richard Stiggins, et. al. 2006
11Examples: 5th Grade Science E.ST Explain moon phases as they relate to the position of the moon in its orbit around the Earth, resulting in the amount of observable reflected light.Sample Item:“Sometimes the Moon looks like a full circle, sometimes it looks like a half circle, and sometimes it looks like a crescent (Set the Context). Explain why (Specify the Reasoning) the Moon appears to be different shapes at different times. You may use labeled drawings in your explanation (Point the Way).”Released NAEP Item (http://nces.ed.gov)
12Scoring Rubrics: Two Types Generic Rubrics“…define what content understanding looks like, in general, for any body of knowledge.”“3” The response is clear, focused, and accurate.“2” The response is clear and somewhat focused, but not compelling.“1” The response misses the point, contains inaccurate information, or otherwise demonstrates lack of mastery of the material.Classroom Assessment for Student LearningRichard Stiggins, et. al. 2006
13Generic Rubrics: Another Example: Sophisticated: an unusually thorough, elegant, and inventive account; fully supported, verified, and justified; deep and broad: goes well beyond the information given.In-Depth: an atypical and revealing account, going beyond what is obvious or what was explicitly taught; makes subtle connections; well supported by argument and evidence; novel thinking displayed.Developed: reflects some in-depth and personalized ideas; going beyond the given – there is supported theory here, but insufficient or inadequate evidence and argument.Intuitive: an incomplete account but with apt and insightful ideas; extends and deepens some of what was learned; some “reading between the lines”; account has limited support/argument/data or sweeping generalizations. There is a theory, but one with limited testing and evidence.Naïve: a superficial account; more descriptive than analytical or creative; a fragmentary or sketchy account of facts/ideas or glib generalizations; a black-and-white account; less a theory than an unexamined hunch or borrowed idea.Understanding by DesignGrant Wiggins & Jay McTight, 1998
14Scoring Rubrics: Exercise-Specific “Points are awarded when specific information appears in students’ responses.”Classroom Assessment for Student LearningRichard Stiggins, et. al. 2006
15Examples: 5th Grade Science Sometimes the Moon looks like a full circle, sometimes it looks like a half circle, and sometimes it looks like a crescent. Explain why the Moon appears to be different shapes at different times. You may use labeled drawings in your explanation.__________________________________________ALL examples taken from NAEP Released Items
16Exercise-specific Scoring Rubric Complete (3) Student explanation includes all the points given below. Student can provide a drawing correctly illustrating the phases of the moon.The Moon is visible because it reflects (or is illuminated by) sunlight.The Moon revolves around the Earth.The portion of the illuminated half of the Moon that is visible from Earth changes, thus making the Moon appear to change shape.Partial (1-2) Student explains 1 or 2 aspects of the causes of the phases of the Moon without major misconceptions.Unsatisfactory/Incorrect (0) Student does not correctly explain any aspect of the phases of the Moon, or explains aspects but includes major misconceptions.
20Examples: 7th Grade Science S.IP Construct charts and graphs from data and observations.A student took a sample of water from a pond and examined it under a microscope. She identified several species of protozoans, including two species of Paramecium that are known to eat the same food. The student decided to examine the water sample every day for a week. She added food for the Paramecia each day and counted the number of each species. Her findings are summarized in the table below.
21Examples: 7th Grade Science NUMBER OF PARAMECIA IN POND WATER SAMPLE Day Species S Species T Using the axes below, construct a graph showing the number of each species of Paramecium the student found each day. Be sure to label the axes.
22Exercise-specific Scoring Rubric Complete (3): Student scales, plots, and labels the graph correctly. Essential (2): Student scales and plots S and T correctly; one or both axis labels or plot labels are missing (or incorrect). Partial (1): Student scales and plots either S or T correctly or scales and plots S and T combined (for example, adds data for each into one data point). No labels on graph. Unsatisfactory/Incorrect (0): Student fails to plot data for either S or T correctly or produces an illogical graph.
25Sample Student Response: Essential (2/3) “Provides correct scales and plots for both sets of data, but lacks a label for the y-axis.”
26Sample Student Response: Essential (2/3) “Provides correct scales and plots for both sets of data, but lacks labels for both axes.”
27Sample Student Response: Partial (1/3) “Using different types of graphs, both responses scale and plot only the data for species S. They also both lack labels for the axes.”
28Sample Student Response: Partial (1/3) “Using different types of graphs, both responses scale and plot only the data for species S. They also both lack labels for the axes.”
29Sample Student Response: Unsatisfactory (0/3) “Illogical graph of the data.”
30Sample Student Response: Unsatisfactory (0/3) “Illogical graph of the data.”
31Examples: High School Science B1.2g Identify scientific tradeoffs in design decisions and choose among alternative solutions.ANDB4.2h Recognize that genetic engineering techniques provide great potential and responsibilities.“Some people believe that recombinant DNA technology has serious disadvantages. Describe one disadvantage that might result from the use of recombinant DNA technology. Then describe a plan or a policy for dealing with the disadvantage that could be followed by research scientists, doctors, public officials, or other people who are involved with recombinant DNA technology and its uses.”
32Exercise-specific Scoring Rubric Complete (3): Student response describes a reasonable disadvantage of recombinant DNA technology and provides a clear description of a plan for dealing with the disadvantage. Credited disadvantages (1 pt) include:Regulation of new strainsProduction of dangerous organismsGenetic Similarity - loss of diversityRegulation of applications/patentsCredited acceptable plans (2 pts) include:Informed consentRegulationThorough testingOversight committeeEssential (2): Student response describes a reasonable disadvantage and attempts a brief description of a plan for dealing with this disadvantage (e.g., test or observe, research further). OR Student response provides only a description of a plan. Partial (1): Student response describes a reasonable disadvantage of genetic technology but does not develop a plan for dealing with the disadvantage. Unsatisfactory/Incorrect (0): Student response states that there are no disadvantages, or states a disadvantage that is inaccurate or unreasonable.
33Sample Student Response: Complete (3/3) “Student response states that making new kinds of viruses and mutations are a disadvantage, and outlines a plan that involves experimentation with human cells outside the body.”
34Sample Student Response: Essential (2/3) “Student response discusses the production of dangerous viruses, and attempts a brief description of a plan.”
35Sample Student Response: Essential (2/3) “Student response describes a disadvantage in general terms only, but does outline a plan that involves the setting of limitations.”
36Sample Student Response: Partial (1/3) “Student response explains that the intermixing of genes could result in the production of nontreatable diseases. No plan is given.”
37Sample Student Response: Partial (1/3) “Student response explains that the virus may have side effects, such as changing or altering a regular body function. No plan is given.”
38Sample Student Response: Unsatisfactory (0/3) “Student response does not state a clear disadvantage or outline a coherent plan.”
39Sample Student Response: Unsatisfactory (0/3) “Student response does not give a clear disadvantage, and states merely that these people should stop messing around with recombinant DNA technology.”
40Activity #1: Write a Scoring Rubric Question 1 – 4th Grade Earth ScienceQuestion 2 – 5th Grade Life ScienceQuestion 3 – 7th Grade Physical ScienceQuestion 4 – 7th Grade Physical ScienceQuestion 5 – High School BiologyQuestion 6 – High School Earth Science
41Activity #1: Write a Scoring Rubric E.ST Explain that the spin of the Earth creates day and night.“Everyone knows about day and night. Write what you think makes day and night. Draw a picture to show what you think.”NAEP Released Items
42Activity #1: Write a Scoring Rubric “Everyone knows about day and night. Write what you think makes day and night. Draw a picture to show what you think.”2 pts. The response indicates that the Earth turns so that the same face is not always facing the Sun. Example: “The Earth turns every 24 hours, and for 12 hours we are facing the Sun.”1 pt. The response indicates that the Moon and Sun are on different sides of the Earth and the Earth rotates facing one and then the other. There is no implication that the Sun moves. Example: “In the day we face the Sun and in the night we turn to face the Moon.”0 pt. The response indicates that the Sun moves (possibly across the sky) to cause night and day. Example: “The Sun moves and makes way for the Moon.” – PLUS –1 pt. Diagram correctly illustrates Sun and Earth, with the side facing the Sun illuminated (day) and the side away from the Sun darkened (night).Adapted from Classroom Assessment for Student Learning, Stiggins, et. al. 2006
43Activity #1: Write a Scoring Rubric L.OL Explain how animal systems (digestive, circulatory, respiratory, skeletal, muscular, nervous, excretory, and reproductive) work together to perform selected activities. Sample Item: “When you exercise strenuously, your body produces excess heat. Describe at least two things your body does to help prevent your temperature from rising excessively, and explain why the body's response is effective.”
44Activity #1: Write a Scoring Rubric Complete (3) Student indicates that the body keeps its temperature from rising through sweating or by blood vessels dilating AND states how these are effective – cooling by evaporation or transferring heat in blood to the air surrounding the skin. Partial (1-2) Student indicates one or two methods the body keeps its temperature from rising but does not explain fully how the mechanisms work. Unsatisfactory/Incorrect (0) Student provides little or no evidence of knowledge of any mechanism for losing heat during exercise.
45Activity #1: Write a Scoring Rubric S.IP Design and conduct scientific investigations.S.IP Use tools and equipment (spring scales, stop watches, meter sticks and tapes, models, hand lens, thermometer, models, sieves, microscopes, hot plates, pH meters) appropriate to scientific investigations.Sample Item: “Explain how (Specify the Reasoning) you can find out the volume of a solid object, such as a small rock (Set the Context), using only water and either a measuring cup or a graduated cylinder (Point the Way).”
46Activity #1: Write a Scoring Rubric Complete (3) Student correctly describes how to determine the volume of a solid object using water and a measuring cup or graduated cylinder. Comparison or change in volume of water should be explicit.Method A:Pour some water into the graduated cylinderRecord the water level (1 pt.)Then put the rock in the graduated cylinderRecord the water level again (1 pt.)The difference between the first and second volume measurements is the volume of the rock (1 pt.)Method B:Spillage Responses = Fill cup with water (to the top) (1 pt.), add rock, catch the water that overflows in separate container, and measure the overflow (1 pt.). Overflow = volume of the rock (1 pt.).Partial (1-2) Student gives the initial steps involved in measuring the volume of a solid object, but does not compare or discuss the change in water level, volume of overflow, or volume of the rock.Unsatisfactory/Incorrect (0) Student demonstrates no understanding of how to use water and a graduated cylinder to measure volume.
47Activity #1: Write a Scoring Rubric S.IA Analyze information from data tables and graphs to answer scientific questions.“One characteristic that can be used to identify pure metals is density. If you determine the density of a pure metal, you can determine what the metal is, as shown in the table below.Suppose that you determine that a metal ring has a density of 15.3 grams/cm3. Assume that the ring is a mixture of some combination of the metals listed in the table. What can you determine about its composition from its calculated density? Explain your answer.”
48Activity #1: Write a Scoring Rubric Complete (3) Student response states that the ring is not pure gold but must contain some gold (1 pt.). Response may reason that the density of the ring (15.3 g/cm3) is less than the density of gold (19.3 g/cm3), but more than the density of any of the other metals (1 pt.). Response must also state that the identity of the other metals in the ring cannot be determined from the given information (1 pt.). Partial (1-2) Student response identifies gold as being one of the metals in the mixture and may choose others based on such factors as averaging densities. Response does not state that the identity of the other metals in the ring cannot be determined from the given information. Unsatisfactory/Incorrect (0) Student response does not identify gold as one of the metals in the mixture, or states that the ring is made of pure gold.
49Activity #1: Write a Scoring Rubric B4.3g Explain that cellular differentiation results from gene expression and/or environmental influence (e.g., metamorphosis, nutrition). Sample Item: “Biologists know that nearly all cells in a person's body contain the same genes. For example, kidney cells contain the same genes as the cells that normally make hemoglobin. Given these facts, explain why kidney cells do not make hemoglobin even though they contain the hemoglobin gene.”
50Activity #1: Write a Scoring Rubric Complete (3) Student response states that even though all cells in a person's body contain the same genetic information (1 pt.), different cells "use" different parts of this information at different times (1 pt.), i.e., the gene for hemoglobin may be "turned on" in hemoglobin-manufacturing (e.g. bone marrow) cells, but "turned off" (1 pt.) in kidney cells. Unsatisfactory/Incorrect Student response demonstrates little or no understanding of what causes different cells to perform different functions.
51Activity #1: Write a Scoring Rubric E1.2g Identify scientific tradeoffs in design decisions and choose among alternative solutions.E2.4A Describe renewable and nonrenewable sources of energy for human consumption (electricity, fuels), compare their effects on the environment, and include overall costs and benefits.“For each of the sources of electrical energy listed below, describe an advantage and a disadvantage of relying on that energy source for a large part of our country’s electrical energy.”a. Solar b. Nuclearc. Hydroelectric d. Fossil FuelsNAEP Released Items
52Activity #1: Write a Scoring Rubric “For each of the sources of electrical energy listed below, describe an advantage and a disadvantage of relying on that energy source for a large part of our country’s electrical energy.”SourceAdvantageDisadvantageSolarLess air pollutionExpensive, Clouds interfereNuclearLess air pollution, large amount of energy per mass of fuelRadioactive wastes, Possibility of accident or meltdown, Public acceptanceHydroelectricLittle pollutionLimited access to rivers, Reservoirs flood landFossil fuelsAbundant relatively cheap fuelNonrenewable fuel source, Air pollution, Greenhouse gasesComplete (3) Includes 7-8 of the essential partsPartial (2) Includes 5-6 of the essential partsUnsatisfactory (1) Includes 3-4 of the essential partsIncorrect (0) Includes less than 3 of the essential parts
53Developing Constructed Response Assessments: Plan the AssessmentDevelop the Assessment(including the Scoring Rubric!)Critique the AssessmentAdminister the Assessment(Student Performance Data)Revise the AssessmentClassroom Assessment for Student LearningRichard Stiggins, et. al. 2006
54Step 1: Plan the Assessment. How Many Items? Aligned to which Content Expectations?GradeScience Unit 1Science Unit 2Science Unit 3Science Unit 45thForces & MotionAnimal Body SystemsAdaptations & Traits of OrganismsMoon & Planets6thMatter & EnergyEcosystemsSoils, Rocks & FossilsPlate Tectonics & Earth’s Magnetic Field7thEnergy & WavesElements & CompoundsCells, Plants & PhotosynthesisAtmosphere, Weather & Climate, Water Cyclee.g. 2 CR Items per Unit = 8 Total
55Step 1: Plan the Assessment. High School PHYSICS Units1Motion2Two-Dimensional Motion and Forces3Dynamics4Momentum5Periodic Motion6Mechanical Energy7Mechanical Waves8Electromagnetic Waves9Electric Forces10Electric Current11Energy Transformations12Energy and SocietyHow Many Items?Aligned to which Content Expectations?e.g. 2 CR Items per Unit = 24 Total Constructed Response Items per Physics Course
56Steps 2 & 3: Developing & Critiquing the Assessment. Eight Guidelines for Writing Constructed-Response Items:Assess understanding beyond rote recall.There should be more than one way to answer a question.There should be opportunities for students to earn partial credit.Harcourt Assessment, Inc. andMichigan Department of Education, 2005
57Steps 2 & 3: Developing & Critiquing the Assessment. Keep the item within a reasonable scope.Avoid questions that are so broad that a knowledgeable person could write multiple pages on the subject.EXAMPLE:Poor: Explain kinetic energy and gravitational potential energy.Better: A pencil rolls across a tabletop and then falls to the floor. Describe the changes in the kinetic energy and gravitational potential energy of the pencil as it rolls, falls, and lands on the floor.
58Steps 2 & 3: Developing & Critiquing the Assessment. Define the task specifically.Don’t expect students to “read between the lines.”EXAMPLE:Poor: Describe the differences between various types of rocks.Better: Describe three differences between igneous and sedimentary rocks.
59Steps 2 & 3: Developing & Critiquing the Assessment. Break a complex task into parts.Makes an item more accessible to students.Put tasks in a logical sequence (first part is often at lower cognitive level than later parts).Avoid redundancy.EXAMPLE:Poor: Juan and Valerie are designing an experiment to test whether a pesticide affects tomato plant growth. Identify four possible variables in this experiment. Choose one of these and explain how it can be controlled and how the results might change if it were not controlled.Better: Juan and Valerie are designing an experiment to test whether a pesticide affects tomato plant growth.Identify four possible variables in this experiment.Choose one of these and explain how it can be controlled.Explain how the results of their experiment might change if this factor were not controlled.
60Steps 2 & 3: Developing & Critiquing the Assessment. Use verbs that discourage one-word responses.Avoid questions that can be answered simply “yes” or “no.”EXAMPLES:“Explain” or “Illustrate” vs. “Name” or “List”
61Steps 2 & 3: Developing & Critiquing the Assessment. Use caution when asking subjective questions.Items should not intrude on student privacy.Do not ask students how they feel.Do not ask students to relate things to personal experience.Any explanation or justification for a student’s response should be based on the stimulus material.
62Steps 2 & 3: Developing & Critiquing the Assessment. Write the scoring rubric at the same time as the item.Include examples of “correct” or “partial” responses.Critique and confirm that the item elicits the intended response.Aligned to a specific Content Expectation.A common error is to ask one question, but base the scoring rubric on an answer that really corresponds to another related question that goes into more depth than what is asked.Harcourt Assessment, Inc. andMichigan Department of Education, 2005
63Step 4: Administer the Assessment (Examine Student Data). Sometimes the Moon looks like a full circle, sometimes it looks like a half circle, and sometimes it looks like a crescent. Explain why the Moon appears to be different shapes at different times. You may use labeled drawings in your explanation.8th Grade NAEP Test, 2005 National ResultsComplete %Partial 18%Unsatisfactory / Incorrect 76%Omitted / Off Task 3%
64Step 4: Administer the Assessment (Examine Student Data). NUMBER OF PARAMECIA IN POND WATER SAMPLE Day Species S Species T Using the axes below, construct a graph showing the number of each species of Paramecium the student found each day. Be sure to label the axes.12th Grade NAEP Test, 2005 National ResultsComplete 56%Essential 23%Partial %Unsatisfactory / Incorrect 13%Omitted / Off Task %
65Step 4: Administer the Assessment (Examine Student Data). “When you exercise strenuously, your body produces excess heat. Describe what your body does to help prevent your temperature from rising excessively, and explain why the body's response is effective.”2005 National Results8th Grade th GradeNAEP Test NAEP TestComplete 2% 9%Partial 66% 72%Unsatisfactory /Incorrect 23% 12%Omitted / Off Task 9% 7%
66Step 5: Revise the Assessment. Flaws in the assessment will become very clear.Can also see if and where instruction has “fallen short” and allow you to make improvements for your current and future sets of students.
67Contact InformationTom Wessels, DirectorGrand Traverse Regional Math and Science CenterTraverse Bay Area Intermediate School District1101 Red Drive PO Box 6020Traverse City, MI