March 11, 2013 Chicago, IL American Board of Preventive Medicine American Board of Preventive Medicine Clinical Informatics Examination Committee Measurement Research Associates, a division of Measurement Incorporated

Goals and Important Measurement Considerations

Considerations Validity Reliability Objectivity

Validity   Does the test measure what it is supposed to measure with regard to: Construct Content

Validity Exam content guidelines Distribution of items ensures the test is representative of the field. A sufficient number of items cover all relevant areas of practice appropriately

Reliability Confidence in test scores Measurement error – “uncontrolled noise”

Objectivity Criterion-Referenced Testing Ascertains an individual’s ability level relative to performance on the content domains as represented by the test items The measurement of a candidate is independent of other test-takers

Criterion Standard   Collective expectation of the level of knowledge and skills needed to practice safely and effectively in the field Determined by experts   A criterion-referenced standard allows any candidate that can achieve the standard to pass. All candidates can pass All candidates can fail

Developing and Reviewing Multiple-Choice Items

Clinical Informatics Item Bank   Goal: Compile and maintain a pool of exam items which are appropriate to measure the knowledge and skills necessary for safe and effective performance in the field of practice.

Taxonomy   Taxonomy refers to the level of cognitive skill required to answer the item correctly Recall Interpretive Problem Solving

Parts to an Item The stem The responses   Correct response   Distractors

The Stem   Format Ask a question Which of the following microscopic subtypes of ameloblastoma is most common? Give an incomplete statement The most common microscopic subtype of ameloblastoma is: Scenario with a question or an incomplete statement A 25 year-old man is brought to the emergency room. He was found lying unconscious on the sidewalk. After ascertaining that the airway is open, the next step in management should be:

Item Options O ptions are all the possible answers for a stem. – One correct (best) answer – Three distractors T he best answer is agreed upon by experts. T he distractors are logical misconceptions of the best answer.

Developing Items I tems should have one best answer. Avoid items based on opinion or for which there is not an accepted answer. I tems must focus on a single issue, fact, or problem.

Developing Items   Items should test important and pertinent material while avoiding trivial facts.   Attempt to write interpretation and problem solving items.   Items should be developed utilizing good grammar, punctuation, and spelling.

Stem Construction A void overly specific knowledge, excess information, and teaching in the stem. I nclude the central idea and most text in the stem T he stem should be stated positively; avoid negative phrasing.

Stem Construction U se terminology common to practice and avoid verbatim textbook phrasing. A void personal pronouns (i.e., you).

Response Construction Avoid “all of the above” and “none of the above.” Avoid absolutes such as “always” and “never.” Responses should be – organized in a logical order – independent and not overlapping – fairly consistent in length – homogeneous or parallel in content – plausible

Questions or Comments