1. Writing MCQs Susan Chamberlain Dept of OBS/GYN Queen’s University Queen’s University

2. Writing MCQs At the end of this session, the participant will be able to Identify different types of MCQ items Identify different types of MCQ items Describe the structure of an MCQ Describe the structure of an MCQ List the technical flaws common to MCQ’s List the technical flaws common to MCQ’s write one well-structured MCQ testing appropriate content write one well-structured MCQ testing appropriate content

3. Writing MCQs At the end of this session, the participant will be able to Identify different types of MCQ items Identify different types of MCQ items Describe the structure of an MCQ Describe the structure of an MCQ List the technical flaws common to MCQ’s List the technical flaws common to MCQ’s write one well-structured MCQ testing appropriate content write one well-structured MCQ testing appropriate content All of the above All of the above

4. MCQ Check-list  Why we write them  What type is best  What flaws are to be avoided  What content to test  How to construct it

5. Why do we test? Communicate to the students what material is important (formative vs summative) Communicate to the students what material is important (formative vs summative) Motivate students to study Motivate students to study Identify areas of deficiency in need of remediation or further learning Identify areas of deficiency in need of remediation or further learning Determine final grades or make promotion decisions Determine final grades or make promotion decisions Identify areas where the course/curriculum is weak Identify areas where the course/curriculum is weak

6. What Should Be Tested? Exam content should match course objectives Exam content should match course objectives Important topics should be weighted more heavily than less important topics Important topics should be weighted more heavily than less important topics The testing time devoted to each topic should reflect the relative importance of the topics The testing time devoted to each topic should reflect the relative importance of the topics The sample of items should be representative of the instructional goals The sample of items should be representative of the instructional goals

7. What Should Be Tested? Sampling Sampling “Performance on the sample provides a basis for estimating achievement in the broader domain that is actually of interest.” Questions sample: Questions sample: –TOPICS –SKILLS ie determining diagnosis, next step in management, etc.

8. MCQ ‘Items’ Should test important information – either common or life-threatening Should test important information – either common or life-threatening Should be well-structured Should be well-structured –Flaws benefit test-wise examinees –Irrelevant difficulty does not test examinees’ specific knowledge

9. Anatomy of an MCQ Item Stem Stem Lead-in Lead-in Options Options –One correct answer –Distractors

10. Types of MCQ’s True/False family Assess recall of isolated facts Assess recall of isolated facts –Stems must be clear and unambiguous –Options must be absolutely true or false or examinee must decide how true options are Generally not recommended – usually have subtle flaws Generally not recommended – usually have subtle flaws

11. Types of MCQ’s: True/False Which of the following statements is true regarding contraception? A) oral contraceptives should not be initiated until six weeks after therapeutic abortion to avoid the risk of hypercoagulability B) an intrauterine device can provide good contraceptive protection 5 years after insertion C) oral contraceptives should not be prescribed to individuals under the age of 15 without parental consent D) lesbian women do not require counseling about contraception or sexually transmitted diseases E) oral contraceptives should not be prescribed for women over the age of 35

12. Types of MCQ’s One-Best-Answer family (A type) Options may be diagrammed: Options may be diagrammed: D C A E B Correct answer is the most likely correct Correct answer is the most likely correct Better application of knowledge, integration, synthesis and judgment Better application of knowledge, integration, synthesis and judgment Least correctMost correct

13. Types of MCQ’s: One-Best-Answer family A 24 year old university student came to see you because she is missing several days of school each month because of severe dysmenorrhea unresponsive to the OCP and NSAIDS. She also complains of deep dyspareunia and has a tender uterosacral nodule on the pelvic examination. Which one of the following conditions is most likely responsible for her symptoms? A 24 year old university student came to see you because she is missing several days of school each month because of severe dysmenorrhea unresponsive to the OCP and NSAIDS. She also complains of deep dyspareunia and has a tender uterosacral nodule on the pelvic examination. Which one of the following conditions is most likely responsible for her symptoms? A)Chronic pelvic inflammatory disease B)Endometriosis C)Adenomyosis D)Fibroids

14. MCQ Check-list Why we write them Why we write them What type is best What type is best  What flaws are to be avoided  What content to test  How to construct it

15. MCQ Technical Flaws Issues Related to Test-Wiseness Issues Related to Test-Wiseness Issues Related to Irrelevant Difficulty Issues Related to Irrelevant Difficulty

16. MCQ Technical Flaws Issues Related to Test-Wiseness Issues Related to Test-Wiseness –provide clues to the right answer –examinee can eliminate options based on flaws and not on knowledge Issues Related to Irrelevant Difficulty Issues Related to Irrelevant Difficulty

17. Item Flaws: Test-wiseness Grammatical cues Grammatical cues Logical cues Logical cues Absolute terms Absolute terms Long correct answer Long correct answer Word repeats Word repeats Convergence strategy Convergence strategy

18. Item Flaws: Test-wiseness Grammatical cues One or more distractors don’t follow grammatically from the stem One or more distractors don’t follow grammatically from the stem

19. Item Flaws: Test-wiseness Grammatical cues A 24 year old patient presents with a one year history of dysmenorrhea, mid-cycle pain, dyspareunia, and painful bowel movements with her periods. The most likely diagnosis is: A)Fibroid uterus B)Adenomyosis C)Endometriosis D)Psychosomatic condition

20. Item Flaws: Test-wiseness Logical cues A subset of the options are collectively exhaustive A subset of the options are collectively exhaustive

21. Item Flaws: Test-wiseness Logical cues Arterial blood gases in a normal pregnant woman, compared to her non-pregnant state may show which of the following? Arterial blood gases in a normal pregnant woman, compared to her non-pregnant state may show which of the following? A)Decreased pO 2 B)Increased pCO 2 C)Decreased pCO 2 D)Decreased pH E)No difference in pCO 2

22. Item Flaws: Test-wiseness Logical cues One or more distractors don’t follow logically from the stem One or more distractors don’t follow logically from the stem

23. Item Flaws: Test-wiseness Logical cues An important tool for assessing fetal well-being is the Non Stress test. A well baby demonstrates the following fetal heart rate pattern: A)A baseline heart rate above 120 bpm B)Evidence of fetal movement C)Baseline variability D)Two fetal heart rate accelerations in 20 minutes E)Fetal breathing movements

24. Item Flaws: Test-wiseness Absolute terms Always, never Always, never Not a problem if stem is long and lead-in asks a precise question Not a problem if stem is long and lead-in asks a precise question

25. Item Flaws: Test-wiseness Absolute terms In patients with a previous Cesarean section: A)Vaginal birth is NEVER an option B)Repeat Cesarean section is less dangerous to the mother C)The uterine scar may rupture during labour D)Care should ALWAYS be supervised by an obstetrician E)No further pregnancies should EVER be considered

26. Item Flaws: Test-wiseness Long correct answer Correct answer is longer, more specific, or more complete than other options Correct answer is longer, more specific, or more complete than other options

27. Item Flaws: Test-wiseness Long correct answer Which of the following problems in vaginal breech delivery is most likely to cause serious long-term complications? Which of the following problems in vaginal breech delivery is most likely to cause serious long-term complications? A)Failure to descend B)Cord prolapse C)Nuchal arms D)Entrapment of the after coming head with an incompletely dilated cervix E)1 minute Apgar score of 4

28. Item Flaws: Test-wiseness Word repeats A word or phase is included in the stem and in the correct answer A word or phase is included in the stem and in the correct answer

29. Item Flaws: Test-wiseness Word repeats A 24 year old with a previous history of chronic PID presents with primary infertility. Select the procedure that would be most appropriate to evaluate the status of her tubes and pelvic anatomy A 24 year old with a previous history of chronic PID presents with primary infertility. Select the procedure that would be most appropriate to evaluate the status of her tubes and pelvic anatomy A)Post-coital test B)Endometrial biopsy C)Laparoscopy with dye insufflation of the fallopian tubes D)Hysterosalpingogram

30. Item Flaws: Test-wiseness Convergence strategy The correct answer includes the most elements in common with the other options The correct answer includes the most elements in common with the other options

31. Item Flaws: Test-wiseness Convergence strategy Normal bladder neck mobility is measured by: A)A Q-tip angle measurement under 40 degrees B)A Q-tip angle measurement under 30 degrees C)A bladder neck ultrasound angle less than 30 degrees D)A Q-tip angle measurement under 20 degrees

32. MCQ Technical Flaws Issues Related to Test-Wiseness Issues Related to Test-Wiseness Issues Related to Irrelevant Difficulty Issues Related to Irrelevant Difficulty – do not assess knowledge of intended topic

33. Irrelevant Difficulty Options are long, complicated or double Options are long, complicated or double Extraneous details in stem Extraneous details in stem Numeric data not stated consistently Numeric data not stated consistently Vague frequency terms Vague frequency terms Non-parallel language in options Non-parallel language in options Stems are tricky or unnecessarily complicated Stems are tricky or unnecessarily complicated ‘None of the above’ ‘None of the above’

34. Irrelevant Difficulty Options are long, complicated or double Extraneous details in stem Shifts difficulty from content knowledge to speed reading Shifts difficulty from content knowledge to speed reading

35. Irrelevant Difficulty Options are long, complicated or double Mrs. Brown is a 70 year old woman who had a stroke 6 months ago and is still impaired. Her daughter tells you that her mother’s recent urinary incontinence makes her care unmanageable. She is seeking permanent placement. Mrs. Brown complains of urinary frequency, nocturia, and urge incontinence. Which of the following statements is true? Mrs. Brown is a 70 year old woman who had a stroke 6 months ago and is still impaired. Her daughter tells you that her mother’s recent urinary incontinence makes her care unmanageable. She is seeking permanent placement. Mrs. Brown complains of urinary frequency, nocturia, and urge incontinence. Which of the following statements is true? A) Neurological conditions are involved in the majority of women with symptoms like those presented by Mrs. Brown B) Diabetes has a greater impact on quality of life than Mrs. Brown’s condition. C) The pharmacological blockage of muscarinic receptors in the detrusor muscle controls symptoms compatible with Mrs. Brown’s condition D) Overactive bladder is defined as urinary incontinence accompanied by or immediately preceded by urgency E) Alpha stimulants are effective in treating this condition

36. Irrelevant Difficulty Extraneous details in stem You see in your office a 24 year old woman for her routine postpartum visit, at 6 weeks. Her baby is irritable and cries “all the time’ (including during her visit.) As she is changing the diaper during her visit, you notice that the child has a very red bum but upon questioning, she tells you she did not notice anything abnormal. Her partner is away in Afghanistan and she does not have a lot of friends, as they just recently were posted in Kingston. She states that she has a hard time getting dressed in the morning, cries easily and feels like she does not love her child. She denies any medical or psychiatric history. Her exam is essentially normal. She weighs 2 kg less than at her first antenatal visit at 10 weeks. You most likely clinical diagnosis is: You see in your office a 24 year old woman for her routine postpartum visit, at 6 weeks. Her baby is irritable and cries “all the time’ (including during her visit.) As she is changing the diaper during her visit, you notice that the child has a very red bum but upon questioning, she tells you she did not notice anything abnormal. Her partner is away in Afghanistan and she does not have a lot of friends, as they just recently were posted in Kingston. She states that she has a hard time getting dressed in the morning, cries easily and feels like she does not love her child. She denies any medical or psychiatric history. Her exam is essentially normal. She weighs 2 kg less than at her first antenatal visit at 10 weeks. You most likely clinical diagnosis is: A)Postpartum hypothyroidism B)Hyperthyroidism C)Postpartum blues D)Postpartum depression E)Psychotic depression

37. Irrelevant Difficulty Numeric Data Not stated consistently Not stated consistently Mixed format Mixed format Illogical order Illogical order

38. Irrelevant Difficulty Numeric Data not stated consistently Following a second episode of salpingitis, what is the likelihood that a woman is infertile? Following a second episode of salpingitis, what is the likelihood that a woman is infertile? A)Less than 20% B)20 – 30 % C)Greater than 50 % D)90% E)75%

39. Irrelevant Difficulty Vague frequency terms Often, usually, rarely, frequently Often, usually, rarely, frequently Not consistently defined Not consistently defined *may be used in stem if otherwise carefully constructed unambiguous question

40. Irrelevant Difficulty Vague frequency terms Gonorrhea is A) sometimes asymptomatic B) treated with cefixime 450 mg orally C) caused by a gram positive intracellular diplococcus D) diagnosed by a DNA test E) usually associated with vulvar ulcers

41. Irrelevant Difficulty Non-parallel language in options Non-parallel language in options Non-logical order Non-logical order

42. Irrelevant Difficulty Non-parallel language in options In a vaccine trial, 200 2-year-old boys were given a vaccine against a certain disease and then monitored for five years for occurrence of the disease. Of this group, 85% never contracted the disease. Which of the following statements concerning these results is correct? A)No conclusion can be drawn, since no follow-up was made of non-vaccinated children B)The number of cases (ie 30 cases over five year) is too small for statistically meaningful conclusions C)No conclusions can be drawn because the trial involved only boys D)Vaccine efficacy (%) is calculated as 85-15/100

43. Irrelevant Difficulty “None of the above” Problematic where judgment is involved or if options not absolutely true or false Problematic where judgment is involved or if options not absolutely true or false Turns question into true/false question Turns question into true/false question Confuses very knowledgeable students Confuses very knowledgeable students

44. Irrelevant Difficulty “None of the above” The main reason that fertility is reduced in a woman in her early 40’s is: The main reason that fertility is reduced in a woman in her early 40’s is: A)Irregular cycles make it hard to time intercourse B)Chromosomal damage adversely affects oocytes’ genetic complement C)Male erectile difficulties increase D)Hormone imbalance leads to poor endometrial development E)None of the above

45. Irrelevant Difficulty Stems are tricky or unnecessarily complicated Arrange the parents of the following children with Down’s syndrome in order of the highest to lowest risk of recurrence. Assume that the maternal age in all cases is 22 years and that a subsequent pregnancy occurs within 5 years. The karyotypes of the daughters are: Arrange the parents of the following children with Down’s syndrome in order of the highest to lowest risk of recurrence. Assume that the maternal age in all cases is 22 years and that a subsequent pregnancy occurs within 5 years. The karyotypes of the daughters are: I: 46, XX, –14,+T (14q21q) pat II: 46, XX, –14,+T (14q21q) de novo III: 46, XX, –14,+T (14q21q) mat IV: 46, XX, –21,+T (14q21q) pat V: 46, XX, –21,+T (21q21q) (parents not typed) A) III, IV, I, V, II B) IV, III, V, 1, II C) III, I, IV, V, II D) I, V, III, I, V, II E) III, IV, I, II, V

46. MCQ Check-list Why we write them Why we write them What type is best What type is best What flaws are to be avoided What flaws are to be avoided  What content to test  How to construct it

47. What to test? Items classified by cognitive processes required to answer the question: Recall (Memory)– tests knowledge of isolated facts Recall (Memory)– tests knowledge of isolated facts Interpretation (Comprehension)– review information to reach conclusion, eg diagnosis Interpretation (Comprehension)– review information to reach conclusion, eg diagnosis Problem solving (Reasoning) – situation requires action, eg next step in management Problem solving (Reasoning) – situation requires action, eg next step in management –Process depends on background of trainee

48. What to test? Simpler classification based on task: Application of knowledge item ** Application of knowledge item ** –Reach conclusions –Make predictions –Select course of action Recall item Recall item –Test knowledge of isolated facts

49. What to test? Test application of knowledge using clinical vignettes Test application of knowledge using clinical vignettes –Adds ‘face-validity’ –Usually selects more important, less trivial –Identifies student who’ve memorized but do not understand BUT experience in patient care should not be necessary

50. What to test - summary Focus item on key concepts and principles that are essential information (without access to references) for all examinees to understand Focus item on key concepts and principles that are essential information (without access to references) for all examinees to understand Test material that is relevant to learning in clinical clerkships and beyond Test material that is relevant to learning in clinical clerkships and beyond Avoid items that only require recall of isolated facts Avoid items that only require recall of isolated facts Avoid esoteric or interesting topics that are not essential Avoid esoteric or interesting topics that are not essential

51. MCQ Check-list Why we write them Why we write them What type is best What type is best What flaws are to be avoided What flaws are to be avoided What content to test What content to test  How to construct it

52. Writing the Questions Construct stem –include all necessary information to arrive at the right answer Construct stem –include all necessary information to arrive at the right answer Choose lead-in – pose a clear question Choose lead-in – pose a clear question Choose distractors Choose distractors

53. Stem* Usually a clinical scenario Usually a clinical scenario Clear, unambiguous Clear, unambiguous Should be long relative to options Should be long relative to options Includes all pertinent information Includes all pertinent information –Patient’s age/gender –Clinical setting –Complaint –Other important info – history, physical findings, test results

54. Lead-in* Follows stem Follows stem In the form of a question In the form of a question Should relate to the stem Should relate to the stem Should be answerable without looking at options Should be answerable without looking at options –“Cover the Options test”

55. Distractors Each should be selected by some, therefore all plausible, none obviously incorrect Each should be selected by some, therefore all plausible, none obviously incorrect 3 or 4 choices 3 or 4 choices –Common misconceptions –Faulty reasoning

56. Distractors Homogeneous in content to the correct answer Homogeneous in content to the correct answer Should be in same category as correct answer– diagnosis, treatment, diagnostic test Should be in same category as correct answer– diagnosis, treatment, diagnostic test Clearly incorrect or inferior to the correct answer Clearly incorrect or inferior to the correct answer Plausible and attractive to the uninformed Plausible and attractive to the uninformed Similar to the correct answer in construction and length Similar to the correct answer in construction and length Grammatically consistent and logically compatible with the stem Grammatically consistent and logically compatible with the stem

57. General Guidelines for Item Construction Include as much of the item as possible in the stems: Long stems, short options Include as much of the item as possible in the stems: Long stems, short options Avoid negatively phrased items (except or not in the lead-in). If you must, use only with short options Avoid negatively phrased items (except or not in the lead-in). If you must, use only with short options Avoid writing question of the form: Avoid writing question of the form: –Which of the following statements is correct? –Each of the following statements is correct EXCEPT »Unfocussed »Heterogeneous options

58. “Top-down” versus “Bottom-up” Top-Down Knowledge or recall question Knowledge or recall question Cite a disease and then ask what patient findings are expected Cite a disease and then ask what patient findings are expected Structured like textbooks Structured like textbooks Clinically backward

59. “Top-down” versus “Bottom-up” Bottom-up Application of knowledge Application of knowledge Gives findings and asks examinee to indicate the disease Gives findings and asks examinee to indicate the disease Examinees need to be able to synthesize information from several pages of a textbook (or notes) to answer these questions Examinees need to be able to synthesize information from several pages of a textbook (or notes) to answer these questions Clinically realistic

60. Guidelines for Clinical Sciences ie Clerkship and beyond… Test application of knowledge using clinical vignettes to pose medical decisions in patient care situations Test application of knowledge using clinical vignettes to pose medical decisions in patient care situations Focus items on common or potentially catastrophic problems Focus items on common or potentially catastrophic problems Pose clinical decisions that would be expected of a successful examinee Pose clinical decisions that would be expected of a successful examinee Avoid clinical situations that would be handled by a (sub)specialist Avoid clinical situations that would be handled by a (sub)specialist

61. MCQ Check-list Why we write them Why we write them What type is best What type is best What flaws are to be avoided What flaws are to be avoided What content to test What content to test How to construct it How to construct it

62. Basic Rules for MCQ’s (A type)* Focus item on important concept Focus item on important concept Item should assess application of knowledge, not recall of an isolated fact Item should assess application of knowledge, not recall of an isolated fact Stem must pose clear question – ‘Cover the options’ test Stem must pose clear question – ‘Cover the options’ test All distractors should be homogeneous All distractors should be homogeneous Avoid technical item flaws Avoid technical item flaws

63. Reference Constructing Written Test Questions For the Basic and Clinical Sciences –Case & Swanson, NBME 1996