Developing High Quality Clinical Skills Assessments University of North Carolina – Chapel Hill School of Medicine November 10, 2011 Ann Jobe, MD, MSN Clinical Skills Evaluation Collaboration Philadelphia, Pennsylvania Philadelphia, Pennsylvania

The most important consideration in developing a clinical skills assessment is to keep in focus the purpose of the assessment The most important consideration in developing a clinical skills assessment is to keep in focus the purpose of the assessment

Formative or Summative? Formative: to provide feedback for improvement of performance, usually midway through a course of study. Formative: to provide feedback for improvement of performance, usually midway through a course of study. Summative: to distinguish between those who are competent, and those who aren’t, usually at the end of a course of study. Summative: to distinguish between those who are competent, and those who aren’t, usually at the end of a course of study.

Test Design Begin at the beginning… Begin at the beginning… What are you trying to assess?What are you trying to assess? What is the level of the examinee?What is the level of the examinee? How much time will the examinee have? Can they complete the task in the time allotted?How much time will the examinee have? Can they complete the task in the time allotted?

Test Design What are you trying to teach and assess?What are you trying to teach and assess? Objectives for assessment – history taking skills, PE skills, communication skills, data interpretation skills? Objectives for assessment – history taking skills, PE skills, communication skills, data interpretation skills? What is the level of the examinee?What is the level of the examinee? Focus and length of checklists will be different for 1 st year medical students and individuals seeking board certification Focus and length of checklists will be different for 1 st year medical students and individuals seeking board certification

USMLE Step 2 Clinical Skills Purpose is to assess ability to: Gather information from patients Gather information from patients Correctly perform physical examination maneuvers Correctly perform physical examination maneuvers Synthesize and communicate findings to patients and colleagues Synthesize and communicate findings to patients and colleagues Examinees are moving to supervised patient care (PGY-1)

Development Of An Examination Blueprint A test blueprint defines the requirements for each examination, regardless of where or when it is administered. A test blueprint defines the requirements for each examination, regardless of where or when it is administered. Commonly seen cases. Commonly seen cases.

Blueprint (continued) Identify the criteria used to define an exam blueprint Identify the criteria used to define an exam blueprint Examination lengthExamination length Case contentCase content Examinee tasks (e.g. history, physical, communication)Examinee tasks (e.g. history, physical, communication) Setting: inpatient or outpatientSetting: inpatient or outpatient Patient gender, agePatient gender, age

Case Selection Documentation of commonly seen cases Documentation of commonly seen cases if specialty focused assessment, cases will be those common to that specialtyif specialty focused assessment, cases will be those common to that specialty Most common presenting complaints in clinical setting Most common presenting complaints in clinical setting Health Department population statistics – especially for region, state or local area Health Department population statistics – especially for region, state or local area

Case Pool Size Dependent on blueprint criteria Dependent on blueprint criteria Security concerns (more is better!) Security concerns (more is better!)

Case Content Cardiovascular Respiratory Gastrointestinal Musculoskeletal Constitutional Neurological Psychiatric Genitourinary Women’s health Other Form Patient age Age less than 18 Age 18 – 44 Age 45 – 64 Age 65 + Patient Gender Male Female Case Acuity Acute Subacute/Chronic

Blueprint Criteria Define Case Needs This information is provided to faculty to guide their case development This information is provided to faculty to guide their case development Case content: GastrointestinalCase content: Gastrointestinal Case acuity: AcuteCase acuity: Acute Patient age: 18-44Patient age: Patient gender: FemalePatient gender: Female Complete a “Medically Relevant Case Details” sheet Complete a “Medically Relevant Case Details” sheet

Case Development Committee Physicians (medical school faculty): Physicians (medical school faculty): content experts content experts SP trainers: training experts SP trainers: training experts SPs: portrayal experts SPs: portrayal experts Process evolved from experience Process evolved from experience

Usually break into several groups Usually break into several groups Each group focuses on developing several cases to fit blueprint needs Each group focuses on developing several cases to fit blueprint needs

Case Development Process Chief Complaint Chief Complaint “Medically Relevant” case details “Medically Relevant” case details Patient Personality Profile Patient Personality Profile Case checklist Case checklist Examinee instructions Examinee instructions Patient note development Patient note development

Chief Complaint/ Differential Diagnoses Brainstorming process Brainstorming process List all of the possibilities List all of the possibilities Then narrow the choices as development proceeds Then narrow the choices as development proceeds

Medically Relevant Case Details Patient /Case Name Patient /Case Name Race Race Any Any African AmericanAfrican American CaucasianCaucasian Other _________________Other _________________ BMI BMI Any Any Weight proportionate to height Weight proportionate to height Overweight: > 24 Overweight: > 24 Other Other

Medically Relevant Case Details Gender Gender Male Female Either Male Female Either Age Range Age Range 10 year age range – example 30 years old = year age range – example 30 years old = Acuity Acuity Acute Acute Subacute/Chronic Subacute/Chronic Category (Blueprint) Category (Blueprint) Exclusionary Medical conditions/scars Exclusionary Medical conditions/scars Differential Diagnosis Differential Diagnosis

Medically Relevant Case Details Communication Tasks Communication Tasks Fostering the Relationship Fostering the Relationship Gathering Information Gathering Information Providing Information Providing Information Making Decisions Making Decisions Supporting Emotions Supporting Emotions Overview of Case Stimuli for Communication Tasks Overview of Case Stimuli for Communication Tasks Thread of Encounter Thread of Encounter Primary MD Primary MD

Personality Profile Giving life and individuality to the patients who are being seen – more like “real patients” Giving life and individuality to the patients who are being seen – more like “real patients” Not verbatim statements from a script – more general ideas of each patient’s perspective Not verbatim statements from a script – more general ideas of each patient’s perspective

Personality Profile Who am I and how would I describe myself? Who am I and how would I describe myself? Why am I here? Why am I here? What made me come in today? What made me come in today? What do I think is going on and why? What do I think is going on and why? What are my expectations for the visit and how will I react if my expectations are not met? What are my expectations for the visit and how will I react if my expectations are not met?

Personality Profile What questions do I have? What questions do I have? What concerns/anxiety/fears do I have? What concerns/anxiety/fears do I have? How has this illness impacted my life? How has this illness impacted my life? What do I bring with me? What do I bring with me? What do I look like? What do I look like? How do I behave/what’s my communication style (mood/attitude related to my illness)? How do I behave/what’s my communication style (mood/attitude related to my illness)?

Personality Profile What is my level of trust of the medical profession? What is my level of trust of the medical profession? What type of communication style do I appreciate in a health care provider? How will I respond to different styles of communication? How much information do I want the doctor to provide? What type of communication style do I appreciate in a health care provider? How will I respond to different styles of communication? How much information do I want the doctor to provide?

Personality Profile What words won’t I understand (jargon)? What words won’t I understand (jargon)? What are my own words I use to describe what is going on? What are my own words I use to describe what is going on? How motivated am I to accept medical advice? How motivated am I to accept medical advice? How do I feel about the use of complementary/alternative medicine? How do I feel about the use of complementary/alternative medicine?

Case Checklist Recording instrument Recording instrument Checklist length Checklist length One concept per item One concept per item Evidence-based items Evidence-based items Use lay language Use lay language Example: Onset Example: Onset (“I’ve been coughing for about a month.”) (“I’ve been coughing for about a month.”)

Refining the Checklist Brainstorm a large number of items Brainstorm a large number of items Role play Role play Checklist revised Checklist revised A – essentialA – essential B – importantB – important C – relevantC – relevant F - deleteF - delete

Examinee Instructions Name Name Age Age Setting Setting Presentation problem Presentation problem Vital signs Vital signs Examinee Tasks Examinee Tasks Refers to the skills that the examination assesses Refers to the skills that the examination assesses

Role Play with SP A physician from another group “sees” the patient A physician from another group “sees” the patient Following the encounter, the physician provides his/her differential diagnoses (see if it matches what the group selected): face validity Following the encounter, the physician provides his/her differential diagnoses (see if it matches what the group selected): face validity Revise the checklist based on observing the physician Revise the checklist based on observing the physician

Role Play with SP This allows an SP to provide feedback on any difficulties that arise in portrayal of the case – especially if an SP needs to repeat the case several times in an assessment session This allows an SP to provide feedback on any difficulties that arise in portrayal of the case – especially if an SP needs to repeat the case several times in an assessment session

Development of Patient Note Key essentials and case-based scoring guidelines Key essentials and case-based scoring guidelines

Questions?