1. David DiBattista, Ph.D. Brock University Psychology Department Encouraging Higher-order Thinking Using Multiple-choice Questions Toronto, Ontario February, 2012 ©D. DiBattista 2012

2. OVERVIEW  Using MCQs for summative purposes  The Revised Bloom’s Taxonomy  Getting beyond remembering with MCQs  Using MCQs for formative purposes $ The rationale $ Some strategies ©D. DiBattista 2012

3. What is Summative Assessment? A summative assessment is a test that is given primarily: a) to assess a student’s achievement in a course, and b) to assign a grade that reflects the student’s level of achievement. MCQs are the most commonly used form of selected-response technique on classroom tests.

4. Selected quotes from the media All who take part in MC tests are wasting their time. MC tests are a stupid way to test what students know. MC testing is not a good indicator of conceptual understanding. [MC tests] are a lazy, one-size-fits-all substitute for actually assessing what students have learned. Results [of MC tests] tend to depend on luck, not on intelligence and hard work. Multiple choice tests are games with a premium on strategic guessing. ©D. DiBattista 2012 MC questions: Controversial !

5. The real value of MC items…is their applicability in measuring higher-level objectives, such as those based in comprehension, application, and analysis. (Burton, et al., 1991) [Constructed-response items] take more examinee time and resources to measure essentially the same thing as [MCQs, but they do so] more poorly than the latter. (Wainer & Thissen, 1993) ©D. DiBattista 2012 But on the other hand… Let's look at one of the most common criticisms of MCQs.

6. “It seems quite evident from many studies that the basic behavior evaluated (by MCQs) is pure recall of information.” (Srivastava, et al., 2004) Overemphasis on memory-based MCQs  Poor content-related validity of the test and Promotion of surface learning at the expense of deep learning ©D. DiBattista 2012 We need a framework that lets us discuss this issue in depth. 

7. OVERVIEW  Using MCQs for summative purposes  The Revised Bloom’s Taxonomy  Getting beyond remembering with MCQs  Using MCQs for formative purposes $ The rationale $ Some strategies ©D. DiBattista 2012

8. Factual Knowledge Dimension Conceptual ProceduralMetacognitive Cognitive Process Dimension Remember Understand Apply Analyze Evaluate Create The Revised Bloom’s Taxonomy Anderson and Krathwohl, 2001 ©D. DiBattista 2012 This chart is in today’s handout!

9. Factual Knowledge Dimension Conceptual ProceduralMetacognitive Cognitive Process Dimension Remember Understand Apply Analyze Evaluate Create The Revised Bloom’s Taxonomy Anderson and Krathwohl, 2001  These are all ACTION verbs—i.e., things students can DO with their knowledge. ©D. DiBattista 2012

10. Good tests allow us to determine what our students are capable of. …but they can be used effectively to assess all of the other cognitive processes. MC questions are not useful for assessing creativity… Factual Knowledge Dimension Conceptual ProceduralMetacognitive Cognitive Process Dimension Remember Understand Apply Analyze Evaluate Create The Revised Bloom’s Taxonomy Anderson and Krathwohl, 2001 Can you remember this? Can you understand this? Can you apply this? Can you evaluate this? Can you create this? Can you analyze this?      ©D. DiBattista 2012  These are all ACTION verbs—i.e., things students can DO with their knowledge.

11. Factual Knowledge Dimension Conceptual ProceduralMetacognitive Cognitive Process Dimension Remember Understand Apply Analyze Evaluate Create The Revised Bloom’s Taxonomy Anderson and Krathwohl, 2001 ©D. DiBattista 2012 Some thoughts about REMEMBER …

12. ALL assessment tasks involve using memory to at least some degree. BUT “If assessment tasks are to tap higher-order cognitive processes, they must require that students cannot answer them correctly by relying on memory ALONE.” —Anderson and Krathwohl, 2001, page 71 A simple, unfortunate fact: Creating MC items that rely on memory alone is far easier than creating higher-level items. ©D. DiBattista 2012

13. OVERVIEW  Using MCQs for summative purposes  The Revised Bloom’s Taxonomy  Getting beyond remembering with MCQs  Using MCQs for formative purposes $ The rationale $ Some strategies ©D. DiBattista 2012

14. In classical conditioning, what name is given to a stimulus that elicits a particular response even in the absence of any prior training? A. conditioned stimulus B. unconditioned stimulus C. activational stimulus D. discriminative stimulus “Do you remember…?” ©D. DiBattista 2012

15. Right after a rat smells menthol, it is always given Drug X, which causes osmometric drinking. Eventually, the rat drinks whenever it smells menthol, even when it is not injected with Drug X. In this situation, what is the role of Drug X? A. conditioned stimulus B. unconditioned stimulus C. activational stimulus D. discriminative stimulus Note the important role of NOVELTY. ©D. DiBattista 2012

16. To learn about the attitudes of Canadian voters toward a government bill limiting auto emissions, Gary gave a questionnaire to 200 members of a group called Friends of a Cleaner Environment. He found that every single one of them strongly supported the bill. Based on this research, which of the following is the most appropriate conclusion? ©D. DiBattista 2012

17. A. The questionnaire is extremely reliable. B. The questionnaire items are not written very clearly. C. The majority of Canadian voters strongly support the bill. D. The results tell us little about the attitudes of Canadian voters. ©D. DiBattista 2012 This item deals with sampling bias, but sampling bias is never mentioned. Note again the importance of novelty.

18. Your friend Anne reads that census data indicate that people are having fewer children nowadays than they did 40 years ago. Anne does not believe this because the young couple who live next door to her are both under 30 and already have four children. If Keith Stanovich were told about this, what might you reasonably expect him to say? ©D. DiBattista 2012

19. A. The census data must be wrong. B. Anne’s comment illustrates valid probabilistic reasoning. C. Anne’s comment illustrates the use of person-who statistics. D. The young couple provide an exception that actually serves to prove the rule. ©D. DiBattista 2012

20. Suppose that you have performed a Gram stain on a mixed culture of Gram-positive and Gram-negative cells. Afterwards you realize that you omitted the iodine step. What do you expect to see when you look at the slide under the microscope? ©D. DiBattista 2012 An MCQ from the test bank for Russell, et al., Biology: Exploring the Diversity of Life.

21. A.colourless Gram-positive cells and pink Gram-negative cells B.colourless Gram-positive cells and colourless Gram-negative cells C.pink Gram-positive cells and pink Gram-negative cells D. purple Gram-positive cells and pink Gram-negative cells ©D. DiBattista 2012 And see the handout for more examples… Now let’s see how MCQs can be used in a formative fashion.

22. OVERVIEW  Using MCQs for summative purposes  The Revised Bloom’s Taxonomy  Getting beyond remembering with MCQs  Using MCQs for formative purposes $ The rationale $ Some strategies ©D. DiBattista 2012

23. What is Formative Assessment? A formative assessment is one that is primarily intended to promote students’ learning rather than to assign a grade.

24. ©D. DiBattista 2012 “The key idea underlying our research is that frequent classroom testing (and student self-testing) can greatly improve education from kindergarten through university. This is a bold claim that runs counter to current wisdom in educational circles…. [We] emphasize testing as an aid to learning, a practice that should be part and parcel of a good educational system.” Roediger, et al., 2006

25. Formative Uses of MCQs ©D. DiBattista 2012  MCQs during class  MCQs outside of class $ Aplia $ Online quizzes

26. The so-called “10-minute rule” ©D. DiBattista 2012 Attributed to Hartley & Davis, 1978 Some classic data on attention…

27. ©D. DiBattista 2012 “More students will maintain interest for a longer time when the topic is interesting and delivered with clarity, enthusiasm, and drama, with a cognitive break now and then.” Wilson & Korn, 2007

28. ©D. DiBattista 2012 “Change grabs attention.” Dan Willingham University of Virginia “More students will maintain interest for a longer time when the topic is interesting and delivered with clarity, enthusiasm, and drama, with a cognitive break now and then.” Wilson & Korn, 2007

29. ©D. DiBattista 2012 PRESENT NEW MATERIAL MCQ PRESENT NEW MATERIAL DEMO PRESENT NEW MATERIAL MCQ A role for clickers! BUT Clickers work well only when the questions asked are used effectively. Lecture time

30. ©D. DiBattista 2012 Because we use clickers, we have better discussions of class material. Dol & DiBattista, 2011

31. ©D. DiBattista 2012 The questions asked when we use clickers encourage me to think more deeply about the material. Dol & DiBattista, 2011

32. ©D. DiBattista 2012 The clicker questions allow me to learn from my mistakes. Dol & DiBattista, 2011

33. ©D. DiBattista 2012 Retrieval-practice questions Remember or higher-level Fuzzy best-answer questions More than one defensible answer Student perspective questions Opinions and experiences Misconception questions Directly address errors of thought Four Types of Clicker Questions

34. ©D. DiBattista 2012 Retrieval-practice questions Remember or higher-level Fuzzy best-answer questions More than one defensible answer Student perspective questions Opinions and experiences Misconception questions Directly address errors of thought Which of the following defines microevolution? Which of the following provides an example of microevolution? Four Types of Clicker Questions

35. Retrieval-practice questions Remember or higher-level Fuzzy best-answer questions More than one defensible answer Student perspective questions Opinions and experiences Misconception questions Directly address errors of thought ©D. DiBattista 2012 What research strategy would be most effective for answering the question that Burton has posed? Four Types of Clicker Questions

36. Retrieval-practice questions Remember or higher-level Fuzzy best-answer questions More than one defensible answer Student perspective questions Opinions and experiences Misconception questions Directly address errors of thought ©D. DiBattista 2012 Which of the following best describes what you believe about the origins of human beings? Four Types of Clicker Questions

37. ©D. DiBattista 2012 Would you expect the vapour pressure of these two flasks of water to differ? If so, how? AB Retrieval-practice questions Remember or higher-level Fuzzy best-answer questions More than one defensible answer Student perspective questions Opinions and experiences Misconception questions Directly address errors of thought Four Types of Clicker Questions

38. Formative Uses of MCQs ©D. DiBattista 2012  MCQs during class  MCQs outside of class $ Aplia $ Online quizzes

39. Animations and videos clarify complex biological processes. Active learning opportunities include automatically graded questions and detailed explanations. Questions focus on data analysis, experimental and observational approaches to research, real- world applications, global problems, and more. Outside of Class: Aplia APLIA ©D. DiBattista 2012

40. Outside of Class: Online Quizzes ©D. DiBattista 2012 The course $PSYC 2F23: Statistics and Research Design $Full year, 4 hr/wk of contact $Required course; unpopular topic $Tests and exams: 78% of course grade

41. The online quizzes $ Purpose: Progress check and exam prep $Both REMEMBER and HIGHER-LEVEL MCQs $Six quizzes via Sakai; 12 items/quiz $Low stakes (2 marks per quiz) $10-day window for each quiz $Unlimited attempts $Only highest mark counts $Feedback after each quiz attempt ©D. DiBattista 2012

42. The research $Students completed an extensive survey at end of course. $ Participation rate: 69% (287 out of 413 students) $Obtained self-reports re: Course liking, Quiz liking, Quiz Motivation, and Learning Value of Quizzes ©D. DiBattista 2012

43. Major findings $ 94%: Quizzes cover important material. $ 87%: Quizzes help me know what to study more. $ 17%: Did not like having the quizzes. $ 79%: Would like to have them in other courses. $ 53%: Would not even look at quizzes if they did not count for any marks. ©D. DiBattista 2012

44. $ Total quiz score and Exam grades: r = +0.32 (Partialed out: GPA, Course Liking, Quiz Liking, Quiz Motivation) $ 79%: Quizzes have Learning Value. The online quizzes help me to learn the material relating to…  Strongly related to Quiz Motivation When I take an online quiz, I work hard to answer every single item correctly. If the online quizzes did not count for any marks, I doubt that I would even look at them. (R) ©D. DiBattista 2012

45. Learning Value YES NO Quiz Motivation Positive Negative $97% of those with positive Quiz Motivation agreed that the quizzes promoted their learning. $In contrast, only 65% of those with negative Quiz Motivation felt this way. $They also liked the course less, and they did less well on exams. ©D. DiBattista 2012 152 8446 5

46. ©D. DiBattista 2012 One final thought… We must remember to be respectful of students’ time.

47. Two Major Conclusions  In a summative context, MCQs can be used to assess students’ learning both at the remember level and at higher cognitive levels.  Well-chosen MCQs can be used in a formative fashion to promote students’ learning both inside and outside of the classroom. ©D. DiBattista 2012

48. David DiBattista, Ph.D. Brock University Psychology Department Encouraging Higher-order Thinking Using Multiple-choice Questions Toronto, Ontario February, 2012 ©D. DiBattista 2012