1. Selected Response Tests Maureen Spelman, Ed. D. Saint Xavier University EDU 343 – Online Meeting # 1

2. ‘5 Commandments’ – for improving your assessments!

3. 1.Thou shall not provide opaque directions to students. * Unclear test-taking directions can result in confused test takers. * Most teachers design test items for topics they know very well - and often provide only sketchy directions for students.

4. 2. Thou shall not employ ambiguous statements in your assessment items. * You know what you mean, however, your students need to be sure about what you mean. * Sentences in which pronouns are used can be unclear as to whom the pronoun refers.

5. “Leaders of developing nations have tended to distrust leaders of developed nations due to their imperialistic tendencies.” [ ????who is the referent for “their”???? Leaders of developing nations OR leaders of developed nations?????]

6. 3. Thou shall not provide students with unintended clues regarding appropriate responses. * Inexperienced test writers often tend to make the correct answer to multiple-choice items twice as long as the incorrect answers. Students who do not know the answer may infer that if the teacher went into great detail for one answer – there must be something ‘special’ about that choice option.

7. * Absolute qualifiers such as never and always often appear in true-false tests. * The focus of good assessment should be on finding out what students know and do not know... not on trick questions.

8. Unintended clues occur when test writers provide an incomplete statement such as “The bird in the story was an...” - and then provide options like A. Falcon, B. Hawk, C. Robin, D. Owl. Those who know there grammar will immediately choose “D” – they do not have to know the story at all to guess correctly!

9. 4. Thou shall not employ complex syntax in assessment items. * An assessment item is not the place to do your “James Joyce” imitation! * Avoid complicated sentence construction – use very simple sentences.

10. 5. Thou shall not use vocabulary that is more advanced than your audience. * It is crucial that you eschew obfuscative verbiage! * Use vocabulary that is suitable for the students who will be taking your test.

11. Four Basic Categories of Selected-Response Assessments Binary-Choice Items Multiple Binary-Choice Items Multiple-Choice Items Matching Items

12. Binary Choice Items True – false Yes – no Right – wrong Correct – incorrect Fact – opinion Etc.

13. Advantages Binary-choice items are typically terse – so students can respond to many items in a relatively short period of time. Binary-choice items make it possible to cover a large amount of content in a brief assessment session.

14. Disadvantages With binary-choice items there are only two options Students have a 50-50 chance of guessing correctly (this does diminish with an increasing number of items).

15. Item Writing Guidelines for Binary-Choice Assessments Phrase items to elicit thoughtfulness – * although brief, binary-choice items need not be simplistic * get students to think about your test items * phrase items so that a superficial analysis would result in an incorrect answer

16. Minimize the use of negatives – * avoid the temptation to simply insert a not into an otherwise true statement * items including double (or triple) negatives should be avoided

17. Avoid items that include more than one concept * focus on a single concept in each test item * two concepts (even if they are both right/wrong) tend to confuse students and thereby your test results

18. Balance response categories – * aim to keep an approximately even number items representing the two response categories * it is not necessary to have exactly the same number of true/false items, but strive to keep a balance

19. Maintain a similar length for both categories of items – * you don’t want to give away clues with the item length (there is a tendency to include qualifying clauses for true statements so that they are truly true!)

20. PAUSE! Please go to our discussion site – You will find 4 discussion prompts related to Binary Choice test items. In your discussion group find the flaws in each item. Be sure to use evidence from the ‘5 Commandments and/or the Item Writing Guidelines to support your comments.

21. Multiple Binary-Choice Items are... Items in which a cluster of items is presented to students, requiring a binary response to each of the items in a cluster. Multiple Binary-Choice items are formatted to look like traditional multiple choice test items. In a regular multiple choice test students must choose one answer from several options...

22. ... However, in the multiple binary-choice test a student must make a response for each statement in the cluster. Here is an example of a well constructed multiple binary-choice test item.

23. * * * Twelve students scored as follows on a 10 item test... 5,6,7,7,7,7,8,8,8,8,9,10 Answer True or False. 1.The median for your students’ scores is 7.5. 2.The mode for the set of scores is 8.0. 3.The range of students’ scores is 5.0 4.The median is different than the mean.

24. Item-Writing Guidelines for Multiple Binary-Choice Items Be sure to separate new “clusters” of items to make it clear to students that a new set of items is beginning. (Note the three asterisks in the previous slide.)

25. In regular multiple-choice items the first part of the item is called the stem. In multiple binary-choice items the first part of the item cluster is known as the cluster stem or stimulus material.

26. Make certain that each item meshes well with the cluster’s stimulus material.

27. Regular multiple-choice items students generally rely on memorized information. That is rarely the case with multiple binary-choice items. In fact, since these items depend on applying knowledge to new situations... students will need to function at higher levels on Bloom’s Cognitive Taxonomy!

28. Multiple binary-choice items are much more intellectually demanding than typical true/false items.

29. * * * Twelve students scored as follows on a 10 item test... 5,6,7,7,7,7,8,8,8,8,9,10 Answers. 1.The median for your students’ scores is 7.5. (TRUE) 2.The mode for the set of scores is 8.0. (FALSE) 3.The range of students’ scores is 5.0. (TRUE) 4.The median is different than the mean. (FALSE)

30. PAUSE, again! With your discussion group – design and post a multiple binary-choice assessment item. Then review the postings of your classmates and offer a peer evaluation using both the ‘5 Commandments’ and Item Writing Guidelines.

31. Multiple-Choice Items Tend to dominate achievement testing in the U.S. Can be used to measure a student’s possession of knowledge OR a student’s ability to engage in higher order thinking skills.

32. Multiple-choice items can contain answers differing in their relative correctness, requiring students to make subtle distinctions among answer options. A weakness, however, is that students merely need to recognize – not generate a correct answer.

33. The first part of a multiple-choice item is referred to as the item’s stem, and potential answer options are described as item alternatives. Incorrect alternatives are usually referred to as the item’s distracters.

34. Creating multiple-choice items One alternative is to create an item stem that is a direct question. This option works well with younger and/or less able students. Here is an example of a well constructed multiple choice item.

35. Which of the following modes of composition would be most effective in explaining to someone how a bill becomes law in the U.S.? 1.Narrative 2.Expository 3.Persuasive 4.Descriptive

36. With older students and/or more capable students... one can use either the direct-question form OR the incomplete-statement form. Here is an example of a well constructed multiple choice item using the incomplete statement format.

37. Mickey Mouse’s nephews are named 1.Huey, Dewey, & Louie 2.Mutt and Jeff 3.Larry, Moe, and Curly 4.Morty and Ferdie *

38. Item Writing Guidelines for Multiple-Choice Assessments The stem should consist of a self-contained question or problem. * A well constructed multiple-choice stem presents a clearly described task so that the student can get to work. * Lengthy stems and terse alternatives are preferred over skimpy stems and long alternatives.

39. As a rule, avoid negatively stated stems. * a single not tossed into a test item can be very confusing for students * If there is a compelling reason for using a negative in the stem – be sure to highlight that negative with italics, boldface type, or underlining – so that students will have a fair chance to answer the item correctly.

40. Item writers frequently fail to realize that the length of a multiple-choice item can give an unintended clue to the correct answer. * Students can figure that if the teacher has given so much attention to constructing an answer options there must be something special about it.

41. * Try to keep all alternatives at about the same length. * If that proves too difficult, then be sure that at least two alternatives are of equal length. * Generally 3 alternatives are used for younger students 4 – 5 for older students.

42. Make sure you scatter correct answers among your alternatives. * Don’t allow students to “guess” their way to a high score simply because they have figured out that your favorite correct answer spot is choice C. * Aim to assign roughly 25% of your correct answers to each of the 4 positions (A, B, C, D).

43. Consider using “... of-the-above” alternatives carefully!! * When you include “all-of-the above” – a student who knows only two of the alternatives are correct can guess that the other two are correct as well. * “None-of-the above” should be used only when that alternative will help you make a test-based inference about student knowledge. (i.e., math)

44. PAUSE, ONCE AGAIN! Please go to our discussion site – You will find 4 discussion prompts related to Multiple Choice test items. In your discussion group find the flaws in each item. Be sure to use evidence from the ‘5 Commandments and/or the Item Writing Guidelines to support your comments.

45. Matching Items Consist of two parallel lists of words or phrases requiring the student to match entries on one list with appropriate entries on the second list. Entries in the list for which a match is sought are referred to as premises. Entries in the list from which selections are made are referred to as responses.

46. An advantage of matching items is that their compact form takes up little space on the printed page. Matching items can be easily scored. A disadvantage of matching items is that they sometimes encourage students’ memorization of low-level factual information.

47. Item Writing Guidelines for Matching Items Homogeneity is an important attribute of well constructed matching items - for instance... * You might want to make all of the left hand entries (the premises) U.S. military conflicts and all the entries in the right hand column (the responses) names of U. S. presidents.

48. Directions: On the line to the left of each military conflict listed in Column A, write the letter of the U.S. President from Column B who was in office when that conflict concluded. Names from Column B may be used only once. Column A ___1. World War I ___2. World War II ___3. Korea ___4. Vietnam ___5. 1 st Persian Gulf Column B A.Bush (the father) B.Clinton C.Eisenhower D.Johnson E.Nixon F.Roosevelt G.Truman H.Wilson

49. Go for relative brevity *From the perspective of the student, it is easier to respond to matching items if the entries on both lists are relatively few in number – about 10 items should be the maximum.

50. * If you have a number of items – say 24 – then best practice would indicate that you break that into three separate assessments of 8 items. *Generally it is better to place the shorter words or phrases to the right – in other words make the responses brief.

51. Loading up on responses – generally more than premises. * Make sure there are at least a few extra responses – otherwise a student who knows 80% of the matches to the premises may be able to figure out the remaining matches - guessing and/or process of elimination. Either way – this does not give you accurate data about the student.

52. Ordering responses logically * To avoid giving any unintended clues – consider ordering the responses in some sort of logical fashion – for example – alphabetical or chronological order. (Notice that in the previous example the names of U.S. presidents were listed alphabetically.)

53. Clearly describe the task for your students. * The directions for an assessment should always make explicit the basis on which matches are to be made – AND, the number of times a response can be used.

54. Formatting your assessment items ALL ON ONE PAGE. * Make sure all premises and responses are listed on a single page. * This practice not only eliminate disruptive page turning – but it will also decrease the likelihood that students will overlook correct answers simply because they are on “the other page”.

55. PAUSE AND RETURN TO OUR DISCUSSION... Just one this time!

56. A Few Things to Remember about Selected- Response Tests... Selected-Response tests can be used to assess student mastery of large amounts of content. Although some believe that selected-response tests only measure lower level skills, inventive teachers can create selected-response tests that elicit very high levels of cognitive skills from students.

57. Remember that the more assessment options you have at your disposal, the more appropriately you will be able to assess those student variables in which you need to gather data.

58. References Marzano, R.J. (2006). Classroom assessment & grading that work. Alexandria, VA: Association for Supervision and Curriculum Development. Popham, W.J. (2008). Classroom assessment: What teachers need to know. New York: Allyn & Bacon, Pearson. Popham, W. J. (2003). Test better, teach better: The role of instructional assessment. Alexandria, VA: Association for Supervision and Curriculum Development.