2. Lesson Nine Test Evaluation: The JCEE Samples

3. Contents ► How to Decide P and D How to Decide P and D How to Decide P and D ► Seeing-Eye Dogs Seeing-Eye Dogs Seeing-Eye Dogs ► Painting Painting ► Female Caring Nature Female Caring Nature Female Caring Nature

4. Criterion Level for Difficulty Index ► How to decide the criterion levels for difficulty and discrimination index? ► P depends on the testing purpose  screening test: value of difficulty index should come closest to the desired selection ratio  Selection ration: around 40% at time of doing research  Penalized if a distractor is chosen  lower P by.03 ► So, P>=.37

5. Criterion Level for Discrimination Index ►.25 as the minimally accepted discrimination index in my study ► So, D>=.25 ► The upper and lower group are defined respectively as the top and bottom 33% of the test takers

6. Seeing-Eye Dogs (1) ► P=.40; D=.64; undesirable distractor: (D) UG=.14, LG=.14 ► Which of the following statements is NOT true about Buddy? (A) Buddy was a male German shepherd. (B) Buddy was not frightened by the heavy traffic in New York. (C) Buddy was trained by Mrs. Eustis. (D) Buddy came from Switzerland.

7. Seeing-Eye Dogs (2) ► A tricky task: negatively stated task is unnatural to the human mind. ► Likely to miss the cue “ her ” at the very end of passage, but still choose (A) for the wrong reason ► A problematic task: all the distractors could be either right or wrong since info. provided in passage is not enough. (Logically, anything uncertain or not definitely true could possibly be false, and thus can be considered “ not true. ” )

8. Seeing-Eye Dogs (3) ► (B): possible answer because it ’ s hard to tell if Buddy was frightened by the heavy traffic of NY (i.e., confidently ≠ not frightened) not frightened) ► (C) either right or wrong: because the owner is not necessarily the trainer. ► (D) also ambiguous: because a German shepherd could come from Germany originally before she got to Switzerland.

9. Seeing-Eye Dogs (4) Conclusion: ► Testing strategy: sticking to the passage or making inferences (assumptions allowed?)? ► Test makers with assumptions made are actually inviting inferences  violate their belief in “ autonomy of reading text ” ► Inappropriate focus of the reading task: on the low level detail of a pronoun “ her ” ► A tricky task: negatively stated task is unnatural to the human mind.

10. Painting (1) ► P=.30, D=.27, (A).42 (UG).21 (LG) ► Which part of the text tells you that Pamela was painting Carrie?? (A) “ Move your head a bit to the left, would you? ” Carrie did as she was told. (B) She was quite an experienced artist ’ s model by now and knew exactly what Pamela wanted. (C) “ Why didn ’ t you dance with Richard at the party? ” said Carrie. “ You know that poor boy likes you. ” (D) “ Oh, no! said Pamela under her breath, glaring at the empty tube.

11. Painting (2) ► Ambiguous question open to more than one interpretation:  Use of past progressive ( “ was painting ” ) sets up a time frame that this event (painting) was happening (an ongoing action)  (A)  Sentences should not be read in isolation. Readers would not be sure that what Pamela was doing is “ painting ” until reaching “ the empty tube ” at the end.  (D) ► “ artist ’ s model ” in (B) automatically & indisputably leads to the idea that “ Pamela was painting Carrie ” ?  test makers violate the autonomous model of literacy

12. Painting (3) ► Conclusion: ► Ways of interpretation lead to different responses:  “ Which part of the text tells you that Pamela was painting Carrie? ” ► A modified question:  At which part of the text, does it become unambiguously clear that Carrie was being painted by Pamela? ► Excerpted narratives: abrupt beginning and no end  autonomous model of literacy assumed in all reading tests. But, this item goes against this assumption of autonomy of text

13. Love Letters (1) ► P=.35, D=.29, (A).23 (UG),.16 (LG) ► A disparity of perspectives between the passage and the stem.  “ a matter of life and death ” : subjective thinking  The stem: objective prediction  Modification: “ According to Miss Harris ” added at the beginning of the stem

14. Love Letters (2) ► A disparity of perspectives between the stem and the distractors & between the target response & the distractors  “ from her point of view ” not stated in the stem  All distractors focus on what she felt  (C): sth. Bad would happen to her  independent of “ her point of view ”  Modification: change (c) into “ disastrous for her

15. Love Letters (3) ► A shift of perspective between the stem and distractor A and the target response.  “ a matter of life and death ” used in an exaggerated way in the passage, but taken literally in the task  “ vital ” in (A) and “ disastrous ” in (c) not taken literally ► “ consequence ” in the stem  With negative connotation  “ a disastrous event is extremely bad in its results and consequences ”  Modification: “ consequence ”  “ it ”

16. Female Caring Nature (1) ► #1: P =.22, D=.30, (A).14 (UG),.12 (LG), (B).25 (UG),.24 (LG); ► #2: P=.19, D=.06, (A).10 (UG),.09 (LG), (C).40 (UG),.16 (LG); ► #3: P=.33, D=.26, (A).07 (UG),.06 (LG), (B).16 (UG),.11 (LG) ► Give two examples of “ female caring nature ” based on the passage. ► Give two examples of “ male aggressiveness ” based on the passage. ► Recall the passage.

17. Female Caring Nature (2) ► Problems in general:  Many unfamiliar words/phrases in the passage and tasks.  Difficult to make a link between the prototypical and extended meanings of the “ female caring nature ” ► Task 1:  Must distinguish between different viewpoints presented in the passage, and hold back their views on this controversial issue

18. Female Caring Nature (3) ► Task 2:  A shift of perspectives between the task and the expected answer  “ a sign of ”  a task that tests skills necessary for interpreting “ sign ” ► Task 3:  An uncomfortable target response for many readers due to the imbalanced lengths of the two paragraphs involved