1. Test Item Writing: Higher Order Thinking

2. Agenda Assessment vs. Evaluation Forms of assessment Item mechanics Best practices Leveled questioning Practice

3. Assessment Writing: I also want to know …

4. Assessment vs. Evaluation Assessment Focuses on learning, teaching, and outcomes Provides information for improving learning and teaching. An interactive process between students and faculty that informs faculty how well their students are learning what they are teaching. Evaluation – Focuses on grades or scores. – May reflect classroom components other than course content and mastery level.

5. Forms of Assessment Case studies Direct observation Examinations Knowledge surveys Self assessments Simulations Portfolios Presentations Performance projects

6. Spotlight: Examinations Multiple Choice Questions (MCQ) Short Answer (SA) Essay (vignettes, scenarios, case studies) Performance (demonstration, presentation, case studies)

7. Terminology Question Mechanics A 30-yr old man presented with a 4-month history of dyspnea, low-grade fever, cough, and fatigue. Given the following chest radiograph, what is the most likely diagnosis? a)adult respiratory distress syndrome b)pulmonary alveolar proteinosis c)pulmonary edema d)streptococcal pneumonia Item Relate to instructional objectives Test at the same level of learning as objectives Construct items to measure different levels of learning (i.e. Bloom’s Taxonomy) Item StemOptions Keyed response Distractors

8. Test Writing Best Practices Stem Use complete statements Determine level of complexity (Bloom’s taxonomy) Include only relevant information and graphics Keep succinct Question for the correct answer, not the ‘wrong’ answer Avoid absolutes: “always”, “never”, “all”, or “none” Avoid imprecise terms: “seldom”, “rarely”, “occasionally”, “sometimes”, “few”, or “many” Avoid cues: “may”, “could”, or “can” Define all eponyms, acronyms, or abbreviations

9. Test Writing Best Practices: MCQ Options Keep grammatically consistent with stem Incorrect options should be clearly incorrect, but plausible Ensure options are categorized together (all diagnoses, tests, etc.) All options similar to the correct answer regarding lengthy, grammar, and complexity Avoid “none of the above” or “all of the above” Place options in logical order (numerical, chronological, etc.) Ensure options are independent not overlapping Provide 3 to 5 options Vary keyed responses (B, C, D, A instead of B, B, A, B,)

10. A (describe the patient with age, gender, and ethnicity, if pertinent) presents with (give the signs, symptoms or other pertinent findings) to the (list site of presentation) in (describe the condition of the patient). Which of the following is the most likely caused this presentation? List 4 or 5 Appropriate Options of roughly equal length (homogeneous, i.e., list of five drugs or conditions or disorders responsible for the presentation cited). Item Patterning: Medical Education A basic pattern can help formulate a good stem You may want to start here Follow guidelines for answer options carefully Compliments of NBOME, 2010.

11. A (patient description) presents complaining of (somatic/visceral symptoms). A structural examination reveals (structural and palpatory findings). The most likely diagnosis is or the most appropriate treatment for this presentation is? Option list (list five treatments as options). Item Patterning Example Compliments of NBOME, 2010.

12. Leveled Question Formatting for Higher Order Thinking Remembering factual knowledge: Write the question as a factual statement. Offer similar, but discernable options Ask: Provide the answer using a series of words or phrases (select) Understanding conceptual knowledge: Write the question as a factual statement. Ask: Provide the answer using fill-in-the-blank. (identify) Applying procedural knowledge: Provide a diagram or data set and ask for application Ask: Given the following chart, which of the following is most likely to occur next? (predict) Analyzing conceptual knowledge Provide a diagram or data set and ask for analysis. Ask: Given the following data, which illustrates a significant savings? (calculate) Evaluating procedural knowledge Provide a case study, scenario, or current event situation Ask: Which of the following would have been a better plan of action? (compare/contrast) Creating a solution Provide a case study, scenario, or current event situation Ask: Develop a plan to resolve this problem. (create)

13. Leveled Questions: Application Remember/Understand

14. Leveled Questions: Analysis Apply

15. Leveled Questions: Evaluation Analyze

16. Leveled Questions: Creation Evaluate

17. MCQ Critique 1.Identify the problem areas of each question 2.Modify the question so that it is correctly written

18. Test Writing Best Practices: Short Answer and Essay Exams Stem Consistent with MCQ best practices Provide only pertinent information Present in a clear and consistent manner Option Response: (rubrics, checklists) Defined criteria Scaled using levels of mastery to determine quality of the area being rated Set target goals (consistency)

19. SA and Essay Question Format Example 1.Identify 1 concept you want your students to be able to do/know in your program. Apply appropriate policy management techniques 2.List what these key words are or would entail: policy management techniques: planning organizing decision making staffing motivating budgeting 3.Identify when this concept would be used Developing and implementing new federal policies that implement campaign promises acquired from the newly elected President. 4.Identify someone who may do this in your profession and a situation where this may occur Elvis, is the Assistant Director of the ASPA.

20. Item Example Elvis, is the Assistant Director of the ASPA. He has been asked to help develop and implement new federal policies that implement campaign promises acquired from the newly elected President. Discuss how Elvis would accomplish this task by applying his knowledge of appropriate policy management techniques.

21. Rubric Response Rubric Component (A-B) Descriptors (C) Key Concepts Beginning 1 Developing 2 Exemplary 3 Concept #1 (Apply policy and management skills) planning organizing decision making staffing motivating budgeting The student was only able to discuss the following (1- 2) key concepts in applying policy management skills relating to the given scenario. planning organizing decision making staffing motivating budgeting The student was only able to discuss the following (3- 5) key concepts in applying policy management skills relating to the given scenario. planning organizing decision making staffing motivating budgeting The student was able to appropriately discuss the following (6) key concepts in applying policy management skills related to the given scenario. planning organizing decision making staffing motivating budgeting

22. Performance Critique - handout 1.Identify the problem areas of each question 2.Modify the question or answer so that it is correctly written

23. Revise in Small Doses

24. References Collins J. Education techniques for lifelong learning: writing multiple-choice questions for continuing medical education activities and self-assessment modules. Radiographics. 2006;26:543–51. JozefowiczRF, Koeppen BM, Case S, Galbraith R, Swanson D, Glew H. The quality of in-house medical school examinations. Academy of Medicine 2002;77: 156–161. Kemp JE, Morrison GR, Ross SM. Developing evaluation instruments. In: Designing Effective Instruction. New York, NY: MacMillan College Publishing Company, 1994:180-213. Palmer E, Devitt PG. Assessment of higher order cognitive skills in undergraduate education: modified essay or multiple choice questions? Research paper. BMC Med Educ. 2007 Nov 28;7:49. Schuwirth LWT, van der Vleuten CPM. Different written assessment methods: what can be said about their strengths and weaknesses? Medical Education 2004; 38:974-979. The National Board of Medical Examiners. Constructing written test questions for the basic and clinical sciences. Available at: http://www.nbme.org/publications/item-writing-manual.html Accessed October 20, 2010.http://www.nbme.org/publications/item-writing-manual.html

25. More Information… Test Item Writing http://digitalcommons.hsc.unt.edu/test_items/ Christine Savi, PhD CLD & QEP: Assessment 817.735.2942