Presentation on theme: "Sue Brookhart December 1, 2014"— Presentation transcript:
1 Sue Brookhart December 1, 2014 Test ConstructionSue BrookhartDecember 1, 2014
2 Introductions Sue Brookhart, Ph.D. Juliette Lyons-Thomas, Ph.D. (Fellow, Regents Research Fund)
3 Webinar Norms All phones will be placed on mute If you have a question, you can type into the chat box, and your question will be addressed during a breakThe chat box icon is located at the top right hand corner of your screen (remember to direct your chat to “Everyone”)At the end of the webinar, you will be asked to fill out a survey based on your experience today
4 Learning OutcomesThe TITC grant has emphasized that in some cases, districts may need to create or alter existing assessments, based on the results of the assessment reviewThe purpose of this webinar is to help attendees to better understand the components of test construction, including test blueprints, linking item performance to learning objectives, and item writing.
5 A Sad TaleEvery Friday Story Test 15 points – vocabulary 5 points – comprehension 20 points total
6 Example – A Test Blueprint for Friday Story Test Learning ObjectiveRe-memberUnder-standAnalyze/CreateTotalKnow new vocabulary words55 (17%)Use new vocabulary words in sentencesUnderstand the main points in the story1010 (33%)Connect elements from the story (character, plot, or setting) with own life or other texts.15 (50%)30 (100%)
7 Example – A Simpler Blueprint for Friday Story Test ContentRe-memberUnder-standAnalyze/CreateTotalVocabulary words510 (33%)Elements from the story (character, plot, or setting)1020 (67%)5 (17%)15 (50%)30 (100%)
8 What is a Test Blueprint? A table with rows and columns that is a plan for the way in which the questions in a test will be distributedRows show number of questions and marks for each topic/standardColumns show number of questions and marks for depth of thinking levelsOther names for this -- test specification, specification matrix, test plan
9 Use a Blueprint To plan individual tests Blueprint (or table of specifications) includes:ContentThinking skillsSpecific learning targetsEmphasis (weight)This information helps you write a test that is interpretable as you intend
10 Example of a Test Blueprint for a Middle School Science Unit Content OutlineRememberUnderstandApplyTotal Points%Basic Parts of CellName and tell function of nucleus, cytoplasm, cell membrane; Label parts of a cell on a line drawing(12 points)Given photos of actual plant and animal cells, label the parts(4 points)1640Plant vs. Animal CellsExplain differences between plant & animal cells;Describe cell walls & cell membrane410Cell MembraneDefine diffusion; List substances diffused and not diffused by cell membrane(6 points)Distinguish between diffusion and osmosis(2 points)820Division of CellsDefine division, chromosomes, and DNAExplain differences between plant and animal cell divisionGiven the numbers of chromosomes in a cell before division, state the number in each cell after division1230221005525Name at least three things you notice.
11 Example of a Different Format for a Test Blueprint for a Middle School Science Unit Learning ObjectiveRe-memberUnder-standApplyTotalIdentify basic parts of cell12416 (40%)Distinguish between plant & animal cells4 (10%)Describe diffusion and the function of cell membrane628 (20%)Understand the process of cell division12 (30%)22 (55%)10 (25%)40(100%)How is this example different from the previous example?
12 Example of a Different Format for a Test Blueprint for a Middle School Science Unit Learning ObjectiveRe-memberUnder-standApplyTotalIdentify basic parts of cell12416 (40%)Distinguish between plant & animal cells4 (10%)Describe diffusion and the function of cell membrane628 (20%)Understand the process of cell division12 (30%)22 (55%)10 (25%)40(100%)Unit learning outcomes go here.
13 What is a Multiple-Choice Item? A multiple-choice item consists of one or more introductory sentences followed by a list of two or more suggested responses. The student must choose the correct answer.
14 Align requirements for multiple choice item performance to learning objectives ContentPerformanceThinking skillsObjective items, especially multiple choice, can tap higher-order thinking if carefully writtenClear presentation to students of what is required for each task or item
15 Multiple Choice ItemsWhich president of the United States was elected to four terms? a. Abraham Lincoln b. Theodore Roosevelt *c. Franklin D. Rooseveltstemdistractorsalternativeskey
16 Guidelines for writing Multiple Choice Items Assess an important aspect of the unit’s instructional targets. Match your assessment plan in terms of performance, emphasis, and number of points. Ask a direct question or set a specific problem. Put the alternatives at the end. Put repeated words in the stem.
17 Guidelines for writing Multiple Choice Items Place the word in the stem and definitions in the alternatives, if testing definitions. Avoid “cluing” and “linking” (where the correct answer of one item depends on another item). Avoid textbook wording. Use simple vocabulary and sentence structure.
18 Guidelines for writing Multiple Choice Items Use consistent, correct punctuation and grammar relative to the stem. Avoid phrasing the item so the student’s personal opinion is an option. Arrange alternatives in a logical order. Have distractors that would be plausible to non-knowledgeable students.
19 Guidelines for writing Multiple Choice Items Have homogenous alternatives. Have distractors based on common errors or misconceptions if possible. Have 3 to 5 functional alternatives. Have one correct or best answer. Avoid “all of the above” and use “none of the above” sparingly.
20 Evaluate the Stems Why did housing prices drop so rapidly in 2008? Which one of the following statements is true about housing prices?Which one is better? Why?
21 Evaluate the Stems An orangutan is a Orangutans are classified as Which one is better? Why?
22 Evaluate the StemsBrass, which is used in decoration, musical instruments, and plumbing supplies, to name only a few, is made fromBrass is made fromWhich one is better? Why?Which one is better? Why?
23 Evaluate the Stems The man who first explored Florida was The Spaniard who first explored Florida wasWhich one is better? Why?Which one is better? Why?
24 Evaluate the StemsWhich of the following illustrates what is meant by condensation?Which of the following does not illustrate what is meant by condensation?Which one is better? Why?Which one is better? Why?
25 A question for youWhich of the following provides the best stem for a multiple-choice item? a. Penicillin is b. Penicillin was discovered by c. Penicillin, which has many uses in medicine, was discovered by d. Who discovered penicillin?
26 A question for youWhich of the following provides the best stem for a multiple-choice item? a. Which of the following did not contribute to the great depression? b. One major factor that contributed to the great depression is c. The great depression was d. The great depression was caused by
27 A question for youWhat is wrong with the stem of the following multiple-choice question? “Which of the following states is the largest state in the United States?”Largest can be measured either geographicallyor by population.b. It measures opinion rather than fact.c. It measures only a lower order skill.d. It should be posed as a statement instead of aquestion.
28 A question for youWhich of the following sets of alternatives would be best for a multiple-choice item about a battle in the Civil War? a. Davis, Grant, Lincoln, none of the above b. Lincoln, Mason-Dixon Line, Sherman, Vicksburg c. Grant, Jackson, Lee, Sherman d. Jefferson, Lincoln, Roosevelt, Washington
29 A question for youWhich of the following sets of alternatives is best for the following multiple-choice item: "The perimeter of a rectangle 4 inches long and 2 inches wide is ______"? a. 6 inches, 8 inches, 12 inches b. 2 inches, 12 inches, 24 inches c. 11 inches, 12 inches, 13 inches
31 Context-dependent item sets Use introductory materialReadingsTables, graphs, or chartsPicturesFormulas, lists of terms or symbolsWrite a set of items requiring students to interpret the material
32 Why use Introductory Materials? Introductory materials (readings, graphs, tables, and maps) used in context-dependent item sets can help assess higher order thinking.Because they give students something to think about.
33 Context-dependent item sets A good way to assess higher-order thinkingThe introductory material allows you to present novel material to studentsThe questions can then be about interpreting, not recalling, material
34 Jennifer drew what the Moon looked like just after sunset every third or fourth night. Her drawings for the nights she observed the Moon are shown below. On Night 11 the clouds were so thick that Jennifer could not see the Moon. Based on the drawings for the other nights, what would Jennifer have seen on Night 11 if the sky were clear? A. C. C. B. D. D.
35 Cars Iron Aluminum Coal According to the map, which of the following does the United States both export to Canada and import from Canada?CarsIronAluminumCoal
36 Frontier Women Which statement best describes the frontier women? Like the early colonial women settlers of the backwoods, frontier women made everything their families needed. Most began work at daybreak and did not rest until late evening. They cooked, spun cloth, made clothing, raised children, and tried to keep their dirt homes clean. They cleared and plowed fields, tended and harvested crops, milked the cows, raised hogs, rode and trained horses, and did just about every chore on the farm. The women not only worked, they also made most of their own tools. To make pitchforks, they attached handles to deer antlers. Many of the women learned to use a knife well enough to carve spoons, forks, and bowls out of animal bones. They fashioned cups and containers out of vegetable gourds and animal horns.Which statement best describes the frontier women?They lived dangerous lives and tamed the West.They hunted to provide food for their families.They frequently worried about the safety of their homes.They worked hard and possessed many skills.
37 What is a constructed-response item? Constructed response test items ask students to compose their responses, and are scored with a judgment of the quality of those responses.
38 Types of Constructed Response Items Restricted response essay items limit both the content of students’ answers and the form of their written responses.Extended response essay items require students to express their own ideas and to organize their own answers.Show-and-explain-the-work problems on math and science tests are also constructed response items.
39 Restricted Response Essays Limit the responsesStill should require higher-order thinking, not recallCan include interpretive materialSeveral restricted response essays usually yield better information about student understanding than one extended essay
40 Essay ItemsWrite items that require students to explain a process, defend a position, etc. – something worth writing about, NOT just “coming up with” facts and conceptsWrite scoring scales or rubrics that match the learning objective(s)Usually best to score all answers to one question before scoring the next
41 ExampleA bird-watcher wants to see many birds in a one-hour period. She decides to investigate which type of food will attract more birds in her backyard.She has a choice of two types of bird food.Sunflower seedsThistle seedsDescribe a fair test the bird-watcher could conduct to help her decide which food will attract more birds. What information should the bird-watcher collect from her test to help decide which type of food attracts more birds?
42 ExampleThe two statements below represent contrasting views regarding the rapid development of the Brazilian rain forest. For each view, explain one probable reason for the speaker's attitude, and give one possible argument the speaker might make to defend his or her point of view.Brazilian developer: "Our nation’s prosperity depends on developing the rich resources of the rain forest.“European diplomat at an international conference on Earth’s environment: "There is certainly a need for an international agreement on the responsible development of the rain forest."
43 Show-and-explain-the-work Problems An amusement park has games, rides, and shows. The total number of games, rides, and shows is 70. There are 34 rides. There are two times as many games as shows. How many games are there? ______________________ How many shows are there? ______________________ Use numbers, words, or drawings to show how you got your answer.
44 Guidelines for Writing Essay Items Assess an important aspect of the unit’s instructional objective(s).Match your assessment plan in terms of performance, emphasis, and number of points.Require students to apply their knowledge to a new or novel situation.
45 Guidelines for Writing Essay Items Define a task with specific directions (rather than leave the task so broad that virtually any response can satisfy it).Use a level of complexity appropriate for students’ level of maturity.Require the student to demonstrate more than recall of facts, definitions, generalizations or other ideas.
46 Guidelines for Writing Essay Items Word questions in a way that leads all students to interpret the item in the way you intended.Make clear to the students all of the following: (a) length of the required writing, (b) purpose for which they are writing, (c) amount of time to be devoted to answering this item, and (d) the basis on which their answers will be evaluated.
47 Guidelines for Writing Essay Items For essays requiring students to state and support their opinions on controversial matters, make clear to the students that their assessment will be based on the logic and evidence supporting their arguments, rather than the actual position taken or opinion stated.
48 A question for youIdentify the flaw(s) in the following essay question: “List the major exports of Chile.”Requires only the recall of facts.Does not specify length or criteria for evaluation.Does not require students to support an opinion.Only (a) and (b): “requires only recall” and “does not specify criteria”
49 A question for youIn a current events unit in a Social Studies class (where the learning goals are for students to understand current events in the news), students are asked to write an essay on the question, “Who do you think should be the next President of the United States?” Which of the following is the best critique of this essay question?It’s a bad question, because it calls for an opinion.It’s a bad question, because it asks about events that haven’t happened yet.It’s a good question, and should be given as an in-class essay after some directions about length and evaluation criteria have been added.It’s a good question for an out-of-class essay, and students should be asked to support their opinion with material from the news media and other sources.
50 A question for youWhat is the MOST important flaw in this essay question: “We have studied the organization of the federal government. Explain each step in the process for passing a bill into law.”Does not specify the content the essay is to be about.Does not require application or higher-order thinking.Does not provide criteria for evaluation.Does not give time limits.
51 A question for youA teacher gave her students this essay question: “Evaluate the effect of air pollution on the quality of life in the western part of this state.” One student wrote, “It’s horrible!” Which of the flaws in this question allowed for such a response?The question calls for an opinion.The question is too broad.No directions were given.
53 Example General scoring rubric for an essay question 2 1 Main Idea and Supporting DetailsAn important main idea is clearly stated. Supporting details are relevant and convincing.A main idea is stated. Supporting details are mostly relevant.A main idea is not stated, or is not correct. Supporting details are not relevant or are missing.ExplanationHow the evidence supports the main idea is clear, reasonable, and well explained.How the evidence supports the main idea is mostly clear and reasonable. Some explanation is given.How the evidence supports the main idea is not clear, not reasonable, and/or not explained.
54 Thank youThe slides and a video of this webinar will be posted at https://www.engageny.org/resource/teaching-core-assessment-literacy-series-materialsNext webinar:Action Plan and Professional Development3:30pm-4:30pm on December 15th, 2014Feedback:https://www.surveymonkey.com/s/testconstruction