1. 2. Format of stem 1. Balance of content level What is a “good” (or “bad”) multiple-choice item? (Trade secrets from a professional) Psychology and applied psychometrics Rationale for multiple-choice testing Item quality that can be assessed before the test: Item quality that can be assessed after the test: 1. How difficult was the item? 2. How well did the item discriminate?! The Item Characteristic Curve (ICC) The point-biserial coefficient (r)

2. a.It can have extreme emotional content. b.It has only a single function. c.It cannot be varied or stopped once triggered. d.It is relatively independent of learning or experience. e.All of the above do not follow from that fact. What statement below does not follow from the fact that an FAP is ballistic? Wall’s Bad Example 1 (too many negative stems)

3. In accordance with a decay theory of forgetting, if a participant learning a list of words was then subjected to one of the following procedures, he would forget the greatest number of words if_______ a. he slept for four versus two hours b. he learned other lists for four versus two hours c. he performed arithmetic problems for four versus two hours d. he performed crossword puzzles for four versus two hours e. All of the above would be comparable. Wall’s Bad Example 2 (complex fill-in-the-blank formats (“unquestionized”))

4. Of the five properties listed below, which are the most important in relation to the function of the plasma membrane in living cells? i.Selective permeability ii. Strength iii. Elasticity iv. Hydrophilicity v. Fluidity a.i, ii, and iii b.i and v c.ii and iii d.iii and iv e.iv and v Wall’s Bad Example 3 (Compound questions)

5. Scatterplot for TT3 (W’04) r = +0.65 Score on essay Score on multiple-choice

6. Cognitive Levels Student ActivityWords in Item Stems Judging based on established set of criteria Appraise, evaluate, justify Producing something new from component parts Design, develop, create, formulate, construct Breaking down material to see relationships, hierarchy Differentiate, compare- contrast, relate Using a concept to solve a problem Solve, show, demonstrate, compute Explaining-interpreting Explain, predict, infer, account for, summarize Remembering facts, terms, concepts, definitions Define, list, state, identify, label Evaluation Synthesis Analysis Application Comprehension Knowledge

7. “QUESTIONIZED” Why does damage in the language center of a left- handed person often have less drastic consequences than in a right-handed person? FILL-IN-THE-BLANK Damage in a left-handed person in the language center often has less drastic consequences than in a right-handed person because _______ a. left-handers are slower to develop language function. b. in left-handers, recovery in the left hemisphere is more rapid. c. highly developed visual-spatial abilities compensate for language loss. d. in left-handers, the hemispheres are not as functionally lateralized. e. left-handers have several language centers.

8. Multiple-Choice Item Analysis......... N = 335 Index of “difficulty” or “easiness”

9. A hypothetical item analysis for a single multiple- choice item: The item characteristic curve (ICC) FDCBA 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 Student-performance groups Proportion getting item correct Chance performance Perfect performance

10. FDCBA 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 Student-performance groups Proportion getting item correct Chance performance Perfect discriminator (between below-C and above-C students) A hypothetical item analysis for a single multiple- choice item: The item characteristic curve (ICC)

11. FDCBA 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 Student-performance groups Proportion getting item correct Ogive ?!! A hypothetical item analysis for a single multiple- choice item: (Possible item characteristic curves (ICC’s) )

12. FDCBA 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 Student-performance groups Proportion getting item correct Actual Item Analyses: good discriminators Item 11 Item 7 Item 8 Item 43

13. FDCBA 1.0 0.9 0.8 0.7 0.6 0.5 0.4 0.3 0.2 0.1 0 Student-performance groups Proportion getting item correct Actual Item Analyses: poor discriminators Item 6 Item 12 Item 10 Item 42

14. The issue of discriminability: Biserial statistics......... N = 335 ?

15. Illustration of obtaining the point-biserial correlation for a single item: A 0 54 B 1 76 C 1 87 D 0 58 etc. Student Score on item (0 or 1) Overall score on test XY