Evaluating procedural skills competency – moving beyond see one, do one, teach one. Stephen Ritz, DO Gregory Smith, MD UPMC St. Margaret Family Medicine Residency Pittsburgh, PA

Learning objectives: Analyze the challenges to defining competence in the performance of procedural skills.Analyze the challenges to defining competence in the performance of procedural skills. Identify the major elements in the development of a procedural skills credentialing process.Identify the major elements in the development of a procedural skills credentialing process. Compare the components of several procedural assessment tools.Compare the components of several procedural assessment tools. Analyze the relationship between instruction, evaluation, assessment.Analyze the relationship between instruction, evaluation, assessment.

Workshop agenda: Part I. Why is this necessary? Why is this necessary? RRC requirements. RRC requirements. Challenges. Challenges. What is “competence”? What is “competence”? How do you credential procedural competence? How do you credential procedural competence? Evaluation methods. Evaluation methods.

Part II. Small group breakout:Small group breakout: - development of an assessment tool - development of an assessment tool - develop a brief curriculum, including teaching methods / resources for your procedure. - develop a brief curriculum, including teaching methods / resources for your procedure.

Part III. Discussion /presentation of assessment tools and curricula Curriculum designCurriculum design ChallengesChallenges Resources / methods usedResources / methods used Arriving at consensusArriving at consensus User friendly / “real world” applicabilityUser friendly / “real world” applicability

I. RRC requirements: 1. Family Medicine programs should provide opportunity to learn in multiple settings those procedures that are within the scope of Family Medicine.1. Family Medicine programs should provide opportunity to learn in multiple settings those procedures that are within the scope of Family Medicine.

RRC requirements: 2. Residents must receive training to perform those clinical procedures required for their future practices in the ambulatory and hospital environments.2. Residents must receive training to perform those clinical procedures required for their future practices in the ambulatory and hospital environments.

RRC requirements: The program director/faculty should develop a list of procedural competencies required for completion by all residents in the program prior to their graduation.The program director/faculty should develop a list of procedural competencies required for completion by all residents in the program prior to their graduation. The list must be based upon the anticipated practice needs of all FM residents. The faculty should consider the current practices of the program graduates, national data, and the needs of the community served. The list must be based upon the anticipated practice needs of all FM residents. The faculty should consider the current practices of the program graduates, national data, and the needs of the community served.

RRC requirements: Procedural skill documentation should indicate when the resident is capable of independent performance of the procedure.Procedural skill documentation should indicate when the resident is capable of independent performance of the procedure.

RRC requirements: Residency programs should devise a method by which all procedures are supervised and evaluated.Residency programs should devise a method by which all procedures are supervised and evaluated. They must devise a credentialing process to establish whether or not a resident is competent to perform a procedure. They must devise a credentialing process to establish whether or not a resident is competent to perform a procedure.

Overall: No specific list of “required” procedures yet.( Except OB )No specific list of “required” procedures yet.( Except OB ) Programs must devise a process to credential residents in the performance of procedures.Programs must devise a process to credential residents in the performance of procedures. Documentation / credentialing process should indicate when a resident is capable of independent performance of a procedure.Documentation / credentialing process should indicate when a resident is capable of independent performance of a procedure.

AAFP – Process framework for procedural skills training: 1.Background 2.Indications 3.Contraindications 4.Alternatives 5.Complications 6.Informed consent patient counseling 7.Patient preparation 8.Anesthesia /analgesia 9.Equipment selection 10. Equipment sterility 11. Patient positioning 12. technique: - steps - observation - supervised performance - practice of skills 13. Pathology recognition 14. Complication mgmt. 15.Practice mgmt aspects 16. Pt monitoring/resusc. 17. Outcome evaluation

Procedural skills teaching principles: Plan ahead Plan ahead Demonstrate procedures Demonstrate procedures Observe learner in action Observe learner in action Provide feedback Provide feedback Engage self –assessment Engage self –assessment Allow for practice under less than ideal conditions Allow for practice under less than ideal conditions Prepare to modify approach: unprepared learner, different sites, opportunistic learning Prepare to modify approach: unprepared learner, different sites, opportunistic learning McLeod, et. al. Seven principles for teaching procedural and technical skills. Academic Medicine Vol. 76, No. 10 ; Oct 2001

AAFP Residency criteria for procedural skills training The training must include some form of evaluation including :The training must include some form of evaluation including : 1. Cognitive knowledge 1. Cognitive knowledge 2. Psychomotor skills / technique 2. Psychomotor skills / technique 3. Visual recognition of pathology 3. Visual recognition of pathology ( if applicable) ( if applicable)

ACOFP – ( Osteopathic) List of 25 procedures ( handout )List of 25 procedures ( handout ) Program director signs off when “competent”Program director signs off when “competent” Credentialing process up to individual programCredentialing process up to individual program

Components of competence for procedural skills. “Clinical competence exists when a practitioner has sufficient knowledge and skills such that a procedure can be performed to obtain intended outcomes without harm to the patient.”“Clinical competence exists when a practitioner has sufficient knowledge and skills such that a procedure can be performed to obtain intended outcomes without harm to the patient.” Miller,MD. Office procedures. Education,training and proficiency of procedural skills. Primary Care; clinics in office practice.24 (2): , 1997 June

Components of competence for procedural skills 1.Knowledge 2.Clinical decision making 3.Judgment 4.Technical Skills 5.Attitudes 6.Professional habits 7.Interpersonal skills 8.Consistency * Adapted from Miller,MD. Office procedures. Education,training and proficiency of procedural skills. Primary Care; clinics in office practice.24 (2): , 1997 June * Adapted from Miller,MD. Office procedures. Education,training and proficiency of procedural skills. Primary Care; clinics in office practice.24 (2): , 1997 June

Implementation – Applying the competency concepts 1.Knowledge :Learner needs assessment 1.Knowledge :Learner needs assessment A. self assessment : formal and informal inquiries: group ings, surveys of current and former residents, focus groups A. self assessment : formal and informal inquiries: group ings, surveys of current and former residents, focus groups B. Faculty assessment - review of procedure logs (by individual and class), query of affiliated specialists ( e.g. surgery, derm ) B. Faculty assessment - review of procedure logs (by individual and class), query of affiliated specialists ( e.g. surgery, derm ) survey of practicing physicians, research data survey of practicing physicians, research data

Implementation 2. Clinical decision making / judgement: - Direct observation - Direct observation - group learning activities - group learning activities - Simulated clinical scenarios ( OSCE ) - Simulated clinical scenarios ( OSCE ) - input from faculty / preceptors - input from faculty / preceptors

Implementation:Technical Skills A. Knowledge – knowledge set necessary for successful procedural performance A. Knowledge – knowledge set necessary for successful procedural performance - Resources: - Resources: Books ( e.g Pfenninger ) Books ( e.g Pfenninger ) CD ROM / DVD CD ROM / DVD Web based ( AAFP, ACOFP ) Web based ( AAFP, ACOFP ) Articles Articles

Implementation:Technical skills B. Hands – On: (psychomotor)B. Hands – On: (psychomotor) - workshops - workshops - 1:1 precepting with selected faculty - 1:1 precepting with selected faculty - Office experiences w/ community specialists ( derm ) - Virtual reality ( future ) - Office experiences w/ community specialists ( derm ) - Virtual reality ( future )

Implementation: Attitudes/Habits/Interpersonal skills Direct observationDirect observation Review of assessment toolsReview of assessment tools Review of evaluations – ACGME/AOA competency basedReview of evaluations – ACGME/AOA competency based Self assessment – what are they comfortable with currentlySelf assessment – what are they comfortable with currently 360 evaluations / Nursing staff assessments360 evaluations / Nursing staff assessments

5. Consistency Serial evaluationSerial evaluation Multiple rater evaluationMultiple rater evaluation Standardized, objective assessment toolsStandardized, objective assessment tools - assists in identifying the gifted or problem learner - assists in identifying the gifted or problem learner

Competency assessment: Process model 1.Learner pre - assessment - Self - assessment inventory (handout) - Self - assessment inventory (handout) - knowledge/skills assessment - knowledge/skills assessment 2. Knowledge assessment 3. Psychomotor skills assessment 4. Performance debriefing /feedback 5. Documentation of encounter (assessment tool) 6. Review and summary of encounters 7. Sign -off by faculty /program director

Assessment tools: Purposes / benefits Standardized Standardized Promotes objectivity Promotes objectivity Teaching tool ( prompts) Teaching tool ( prompts) Facilitates feedback process Facilitates feedback process Facilitates objective, specific feedback Facilitates objective, specific feedback Documentation of experiences by learner / class Documentation of experiences by learner / class Objective information for procedural curriculum needs Objective information for procedural curriculum needs

Challenges / Barriers – Assessment tools No national standardsNo national standards TimeTime Yet another paperwork documentation burdenYet another paperwork documentation burden Faculty / resident buy –inFaculty / resident buy –in Paper or electronic?Paper or electronic? Procedure specific vs. genericProcedure specific vs. generic Redundancy – e.g procedure logsRedundancy – e.g procedure logs Clerical resourcesClerical resources

Assessment tools - components Identify resident, patient, preceptor, dateIdentify resident, patient, preceptor, date Knowledge components:Knowledge components: 1. Pre –procedural: 1. Pre –procedural: - indications - indications - contraindications - contraindications 2. procedural: 2. procedural: - relevant anatomy - relevant anatomy - medications / dosages - medications / dosages - instruments - instruments - site prep - site prep Psychomotor components:Psychomotor components: - instrument handling - instrument handling - motor skills - motor skills Post procedural - overall assessmentPost procedural - overall assessment

VI.Credentialing process For “independent” performance of the procedure.For “independent” performance of the procedure. Rationale:Rationale: * Fulfills RRC requirements * Fulfills RRC requirements * More objective endorsement of procedural skills competence * More objective endorsement of procedural skills competence * Help PD verify resident competence for outside entities * Help PD verify resident competence for outside entities ( e.g. hospital credentialing, legal system ) ( e.g. hospital credentialing, legal system ) * Provides data for curriculum design and review * Provides data for curriculum design and review

Our system Resident and preceptor completes assessment tool together following each procedure performedResident and preceptor completes assessment tool together following each procedure performed Tool is submitted to clerical staffTool is submitted to clerical staff Evaluation result is recorded on compilation sheet for each resident; paper copy of tool is filed.Evaluation result is recorded on compilation sheet for each resident; paper copy of tool is filed. Upon receiving 3 “ competent “ ratings for a given procedure, resident is credentialed for that procedure if advisor and PD agree.Upon receiving 3 “ competent “ ratings for a given procedure, resident is credentialed for that procedure if advisor and PD agree. Compilation sheet data are reviewed by faculty, advisor and PD by resident and by class.Compilation sheet data are reviewed by faculty, advisor and PD by resident and by class. Results are discussed at annual resident promotional review and at advisee meetings.Results are discussed at annual resident promotional review and at advisee meetings.

Part II : Small Group Session Appoint a leader / scribe for your groupAppoint a leader / scribe for your group Devise the following for your procedure:Devise the following for your procedure: 1. Curriculum : elements 1. Curriculum : elements 2. Learner knowledge / skills assessment – 2. Learner knowledge / skills assessment – * Make an assessment tool * Make an assessment tool 3. Competency credentialing 3. Competency credentialing *Write out on poster paper * Prepare to present to larger group

Discussion Points – What was hardest part of the exercise?What was hardest part of the exercise? Was consensus hard to achieve for curriculum? For the assessment tool ?Was consensus hard to achieve for curriculum? For the assessment tool ? Was it difficult to devise a method to “certify” competence?Was it difficult to devise a method to “certify” competence? What procedural teaching methods have worked best for you?What procedural teaching methods have worked best for you? What methods are you currently using to credential your residents for independent performance of procedures?What methods are you currently using to credential your residents for independent performance of procedures? Identify barriers / advantages of implementing your curriculum and assessment tool in the “real world”Identify barriers / advantages of implementing your curriculum and assessment tool in the “real world”