1. March 11, 2013 Chicago, IL American Board of Preventive Medicine American Board of Preventive Medicine Clinical Informatics Examination Committee Measurement Research Associates, a division of Measurement Incorporated

2. Goals and Important Measurement Considerations

3. Considerations Validity Reliability Objectivity

4. Validity   Does the test measure what it is supposed to measure with regard to: Construct Content

5. Validity Exam content guidelines Distribution of items ensures the test is representative of the field. A sufficient number of items cover all relevant areas of practice appropriately

6. Reliability Confidence in test scores Measurement error – “uncontrolled noise”

7. Objectivity Criterion-Referenced Testing Ascertains an individual’s ability level relative to performance on the content domains as represented by the test items The measurement of a candidate is independent of other test-takers

8. Criterion Standard   Collective expectation of the level of knowledge and skills needed to practice safely and effectively in the field Determined by experts   A criterion-referenced standard allows any candidate that can achieve the standard to pass. All candidates can pass All candidates can fail

9. Developing and Reviewing Multiple-Choice Items

10. Clinical Informatics Item Bank   Goal: Compile and maintain a pool of exam items which are appropriate to measure the knowledge and skills necessary for safe and effective performance in the field of practice.

11. Taxonomy   Taxonomy refers to the level of cognitive skill required to answer the item correctly. 1. 1. Recall 2. 2. Interpretive 3. 3. Problem Solving

12. Parts to an Item 1. 1. The stem 2. 2. The responses   Correct response   Distractors

13. The Stem   Format 1. 1. Ask a question Which of the following microscopic subtypes of ameloblastoma is most common? 2. 2. Give an incomplete statement The most common microscopic subtype of ameloblastoma is: 3. 3. Scenario with a question or an incomplete statement A 25 year-old man is brought to the emergency room. He was found lying unconscious on the sidewalk. After ascertaining that the airway is open, the next step in management should be:

14. Item Options O ptions are all the possible answers for a stem. – One correct (best) answer – Three distractors T he best answer is agreed upon by experts. T he distractors are logical misconceptions of the best answer.

15. Developing Items I tems should have one best answer. Avoid items based on opinion or for which there is not an accepted answer. I tems must focus on a single issue, fact, or problem.

16. Developing Items   Items should test important and pertinent material while avoiding trivial facts.   Attempt to write interpretation and problem solving items.   Items should be developed utilizing good grammar, punctuation, and spelling.

17. Stem Construction A void overly specific knowledge, excess information, and teaching in the stem. I nclude the central idea and most text in the stem T he stem should be stated positively; avoid negative phrasing.

18. Stem Construction U se terminology common to practice and avoid verbatim textbook phrasing. A void personal pronouns (i.e., you).

19. Response Construction Avoid “all of the above” and “none of the above.” Avoid absolutes such as “always” and “never.” Responses should be – organized in a logical order – independent and not overlapping – fairly consistent in length – homogeneous or parallel in content – plausible

20. Questions or Comments