1. Jose Y. Cueto Jr., MD, MHPEd Chairman, 2012 Professional Regulatory Board of Medicine Professional Regulation Commission

2. Objectives for the session: 1. Discuss the educational principles behind the acquisition and development of psychomotor skills 2. Examine required number of operations for OB-GYN residents 3. Discuss research studies relevant to training and how to utilize research data 4. Formulate a system of determining and validating requirements in number of operations in residency training (both for graduation and for certification) 5. Formulate a system of documentation and reporting

3. Objectives for the session: 6. Determine how the number of operations performed during residency will affect credentialing and privileging in hospitals 7. Discuss performance-based evaluation, specifically the Objective Structured Clinical Exam (OSCE)

4. Main Objective of Training NOVICE COMPETENT EXPERT SURGEON

5. I. Educational Principles Fitt and Posner (1967): 3 phases of skills acquisition Miller’s Pyramid

6. Fitt and Posner: 3 Phases 1. Cognitive phase 2. Associative phase 3. Autonomous or Fixation phase

7. Fitt and Posner: 3 Phases PHASES COMPONENTS / FEATURES 1.COGNITIVE PHASE (the mental part) Understanding the principle behind the procedure Knowing the indications and contraindications Recognizing the risks and complications Identifying and analyzing the steps and their proper sequence Identifying the critical parts of the procedure Internalizing what are observed during assists and what are discussed during rounds and conferences

8. Fitt and Posner: 3 Phases PHASES COMPONENTS / FEATURES 2. ASSOCIATIVE PHASE (the action or activity part) Converting the mental picture into actual action Performing the procedure according to determined sequence Ensuring guidance and supervision by Consultant Obtaining feedback on what was done right and what needs to be corrected Allowing adequate practice to polish rough movements Developing ability to concentrate on the procedure

9. Fitt and Posner: 3 Phases PHASES COMPONENTS / FEATURES 3. AUTONOMOUS or FIXATION PHASE (the refined, smooth part) Mastering the steps and correct sequence Developing smooth movements with minimal wasted moves Making the skill become “automatic” Modifying the procedure when conditions require it Precaution: If uncorrected during the 2 nd phase, there is danger of incorporating erroneous habits which will be difficult to unlearn

10. Miller’s Pyramid Does Shows how Knows how Knows

11. Progression of Skills Acquisition FITT and POSNER: 3 PHASES MILLER’S PYRAMID Cognitive PhaseKnows Knows how Associative PhaseShows how Autonomous or Fixation PhaseDoes

12. Progression of Skills Acquisition and Development 2 nd Assist (mainly exposure) 1 st Assist (limited participation in procedure) Surgeon (under direct supervision) Surgeon (independently performing)

13. Tracking Progression and Scheme of Reporting: 3 columns Operation / Procedure Number of First Assists Number of supervised operations / procedures Number of independently performed operations / procedures 1. Total abdominal hysterectomy 2. Cesarian hysterectomy 3. Adnexal surgery 4. Cesarian section 5. Vaginal Hysterectomy

14. II. Handbook of PBOGS, 2006 PROCEDURE 1st 2nd 3rd 4th TOTAL NSD 50 20 70 Episiorrhaphy 25 10 35 Outlet forceps / vacuum extraction 15or 15 15 Dilatation and Curettage 20 40 Manual extraction of placenta 3 3 Partial breech extraction 7 or 8 8 Cesarian section 10 20 30 Abdominal hysterectomy 5 5 8 Adnexal surgery 5 5 10 Vaginal hysterectomy 1 or 1 1 Cesarian or postpartum hysterectomy 1 or 1 1 Evacuation of H. mole 1 or 1 1

15. My Observation Description of The Residency Training Programs should include a section on Psychomotor Skills Include an explanation on the required number of operations using statistics Clarify the progression of operative cases handled by residents and provide the basis Documentation and reporting found in “Basic Requirements for Examination”, page 42; but not discussed in the curriculum (same requirements for graduation and certification)

16. General Principles Progression: Trainees first learn simple tasks, before they progress to procedures of medium difficulty, and finally to complicated procedures Transfer of learning / transfer of skills: what trainees learn from simple skills are utilized in the performance of medium and complicated skills Trainees need to learn to pay attention to details, follow proper sequence, observe meticulous technique

17. Application of Principles 1. The simpler the procedure, the shorter it takes to learn, and the less number of cases needed to master it 2. The more complicated the procedure, the longer it takes to learn, and the more number of cases needed to master it 3. The more complicated the procedure, the higher the need for supervision and guidance

18. Implementation Residency Training Committee 1 st yr 2 nd yr3 rd yr4 th yr

19. Roles according to year level Operation 1 st yr 2 nd yr 3 rd yr 4 th yr Requirement for Abdominal hysterectomy 5 5 Role 1 2 nd assist 1 st assist surgeon Role 2 2 nd assist 1 st assist (12) Surgeon, supervised (3) Surgeon, independent (7)

20. Evaluating Phases of Learning Fitt’s 3 Phases Miller’s Pyramid Method of Evaluation/Setting 1. Cognitive PhaseKnows Knows how Discussion Pre-op conference Q and A OR Record review Ward / office 2. Associative PhaseShows howDirect observation Skills lab (Supervised by Simulations Consultant) OR 3. Autonomous PhaseDoesDirect observation OR Record review Ward Outcome evaluation Rounds Conference

21. III. Researches 1.Does Residency Training Improve Cognitive Competence in Obstetric and Gynaecologic Surgery?, Balayla, Abenhaim and Martin, McGill University, Montreal, Can, J Obstet Gynaecol Can 2012;34(2):190–196 2. Competency-based Residency Training: The Next Advance in Graduate Medical Education, Donlin Long, MD, PhD, Johns Hopkins University School of Medicine, Academic Medicine, Dec 2000

22. Researches 3. Factors Associated with a Successful Outcome in the PBS Certifying Examinations, Crisostomo and Marfori, Philippine Journal of Surgical Specialties, Oct-Dec, 2010

23. Does Residency Training Improve Cognitive Competence in Obstetric and Gynaecologic Surgery? (JOGC,2012) Objectives: 1.To develop an operative knowledge assessment tool to evaluate the cognitive competence of trainees in obstetric and gynaecologic surgery 2.To determine the rate of change in competence during a five-year residency program.

24. J Obstet Gynaecol Can Methods: Twenty-eight participants in five training groups (PGY-1 to PGY-5) in McGill University’s residency program in obstetrics and gynaecology Evaluation based on surgical cognitive competence (SCC) assessment tools Three different obstetric and gynaecologic operations: open total abdominal hysterectomy (TAH), Caesarean section, and laparoscopic bilateral tubal sterilization (BTL)

25. J Obstet Gynaecol Can Performance of an operation/procedure Three fundamental components: 1. cognitive factor 2. technical element 3. judgment component

26. J Obstet Gynaecol Can COMPONENT FEATURE 1. cognitive factor the knowledge of the theoretical steps of the procedure 2. technical element takes the theoretical steps into account and translates them into the performance of the operation 3. judgement component comes from surgical experience and allows a surgeon to rely on his or her own intuition to determine the appropriate operative course of action on a case by case basis

27. J Obstet Gynaecol Can Summary of Findings: OPERATION PERCENTAGE INCREASE IN COGNITIVE COMPETENCE PER YEAR 1. Total Abdominal Hysterectomy 15.73% 2. Cesarian Section 8.06% 3. Laparoscopic BTL 16.31%

28. J Obstet Gynaecol Can Findings: At level of PGY-5, residents had 100% surgical cognitive competence This type of information may be helpful in ascertaining how long a residency program should be

29. Competency-based Residency Training: The Next Advance in Graduate Medical Education (AM, 2000) Donlin Long, MD, PhD Professor of Neurosurgery, Johns Hopkins University Studied NSS residents Introduced competency-based program

30. Academic Medicine Traditional Program Competency-based Program 1. Fixed number of years 2. Residents have to learn all specified knowledge and skills in the allotted time 3. Problem of evaluating competence of every resident 4. Graduate may not be competent to perform required procedure or manage particular patients. 1. Specifies maximum duration 2. Time taken to acquire knowledge and skills is based on the abilities of individual trainees 3. Evaluation of every resident in every procedure 4. Resident is evaluated and certified to have acquired competence and confidence to practice independently

31. Factors Associated with a Successful Outcome in the PBS Certifying Examinations (PJSS, 2010) Objective: To determine the factors associated with a successful outcome in the PBS certifying exams (written and oral) Method: Retrospective, cross-sectional study utilizing 370 candidates from 2006-2009, with 137 (37.0%) successful outcomes

32. PJSS Significant Factors: 1. Younger age of examinees 2. Previous performance in the RITE 3. Taking the exam within a year of completion of residency 4. Training in a university-based program 5. Undertaking subspecialty fellowship during the examination year

33. PJSS Other Factors (Not Significant) 1. Sex (M-F) 2. Marital Status 3. Case volume 4. Continuous/interrupted program 5. Location of training program (MM vs. outside)

34. PJSS CASE VOLUME performed during residency (major operative procedures): 1. High volume: 299 0r more cases (upper 3 rd ) 2. Medium volume: 171-298 cases (middle 3 rd ) 3. Low volume: less than 171 cases (lower 3 rd ) NOTE: did NOT influence performance in written and oral exams

35. Open for further research Relationship of case volume to: 1. surgical cognitive competence 2. technical element 3. judgment component

36. Utilizing Research Data (PJSS) FAVORABLE FACTORS (passing the PBS Certifying Exam) Accreditation Committee Residency Training Committee DECISIONS / ACTIONS 1. Graduation from a university-based training program Closely monitor the residents and graduates from gov’t and private institutions 2. Satisfactory performance in the Residency In-training Exam Institute remedial measures for residents with low scores in the RITE (identify topic areas) 3. Taking the exam within one year after end of residency Encourage / require graduates to take the certifying exam within one year after end of training

37. Utilizing Research Data (JOGC) FACTORS IDENTIFIED DECISIONS / ACTIONS 1. Percentage increase in surgical cognitive competence per year; reaches 100% at level PGY5 Policy: convert all 4-year programs to 5- year programs 2. Ability of the resident to identify the most critical steps (given 24-32 total steps in certain procedures) Provide in-depth discussions and adequate exposure and practice prior to allowing resident to perform actual operation 3. The cognitive factor and judgment component are more important than the technical factor in the performance of procedures Evaluate the development of cognitive factor and judgment component of operative skills

38. Utilizing Research Data (AM) FACTOR IDENTIFIED DECISION / ACTION Abilities of residents and pace of acquiring knowledge and learning operative skills differ. Provide flexibility in the duration of a training program but place a limit or maximum duration. Evaluate competencies in each procedure before certifying for promotion to higher level or for graduation. When in doubt, provide extension for additional operative cases

39. The essential messages 1. The process of acquiring and developing operative skills is more important than the output (number of operations performed). 2. Quality is more important than quantity. 3. Supervision and feedback are critical. There may be institutions where residents get to perform so many operations by themselves, but they never get to know which steps are done correctly or incorrectly.

40. The Essential Messages 4. The cognitive factor and the judgment component are very critical in the performance of procedures 5. The fixed duration of residency training may not be appropriate for a number of residents

41. Determining Requirements Factors to consider: 1. Degree of difficulty: simplicity / complexity of procedure /low-risk / high-risk *The simpler the procedure, the lower the number required to acquire competence *The more complex / complicated the procedure, the higher the number required to acquire competence

42. Determining Requirements Factors to consider: 2. Trainee factor: fast / slow learner dexterity with procedures *The fast learner and the trainee with dexterity / adeptness at performing procedures will require lower number of cases *The slow learner and the trainee with “clumsiness” in performing procedures will need a higher number of cases

43. Determining Requirements Factors to consider: 3. Institutional factor: high volume vs. low volume private vs. charity/service patients *The resident belonging to a high-volume hospital will require a lower number of cases (reinforcement) *The resident belonging to a low-volume hospital will require a higher number of cases (too few and far in-between)

44. Determining required number of operations 1. Use of the Delphi technique 12-30 Experts List of operations 3 rounds Questions to answer

45. Questions Based on your experience and expert opinion: How many times should a resident assist in the following procedure before he can be given his first case? How many times should a resident perform this procedure under direct supervision before he is allowed to perform it independently? How many times should a resident perform this procedure before he can acquire the competence and the confidence to perform it safely on his own?

46. Three Rounds Round 1: 12-30 experts give their proposed number of operations, based on the questions; without them communicating with each other Round 2: the experts are given feedback on how their colleagues answered the questions (tabulation of results); afterwards they are asked for their modified list of proposed number of operations Round 3: the experts are gathered and they are asked to arrive at a consensus regarding required number of operations

47. Validation Stage 1. Identify institutions: Government University-based Private 2. Conduct a parallel research study: longitudinal tracking of residents until they get to the certifying exams 3. Based on results, modify/maintain the requirements

48. Detailed Documentation of Operative Experience (35 cases) List of Operations Number of Assists Number of supervised operations Number of independently performed operations Total abdominal hysterectomy 15 Cesarian/postpartum hysterectomy 1 Vaginal hysterectomy 1 Adnexal procedures 10 Vaginal extraction of H. mole 1 Indicated manual extraction of placenta 1 Breech deliveries 1 Outlet forceps or vacuum 5

49. My Observation Notation: Starting 2006, only 70% of cases with complete transfer of technical responsibility is allowed What is the basis for the policy? What problem does it solve? Lack of residents’ cases? Consultants’ cases given to residents: fall under 2 nd column (supervised cases) Private cases: no complete transfer of responsibility demo cases to show residents how procedures should be done

50. Residents’ Responsibilities 1. provide preoperative evaluation, assessment of risks 2.identify indications/contraindications to planned procedure, possible complications 3. perform the procedure, modifying it in presence of unforeseen conditions 4. providing immediate and long-term care. Note: For private patients, decisions will always be made by the Attending Consultant (The judgment component is lacking)

51. Deficiencies and Actions What happens if residents do not meet requirements? Extension of rotation in a particular service where deficiencies occurred. Reduce the number of residents or admit residents every other year. Terminate program after adequate opportunities for correcting deficiencies have been given

52. Implications for Future Credentialing and Privileging Operations / Number Performed / Required Allowed Not Allowed 1. Dilatation and curettage 50 (40) / 2. Cesarian section 50 (30) / 3. Adnexal surgery 15 (10) / 4. Abdominal hysterectomy 10 (10) / 5. Vaginal hysterectomy 1 (1) x 6. Cesarian hysterectomy 1 (1) x

53. Question Will the departments accept deficiencies, and still allow residents to graduate? Will the board accept deficiencies and still allow graduates to take the certifying exams?

54. Recommendations 1. Shift FOCUS to process of acquiring and developing operative skills rather than the number performed

55. Recommendations 1. Shift FOCUS to process of acquiring and developing operative skills rather than the number performed 2. Aim for qualitative improvement by providing guidance, supervision and feedback.

56. Recommendations 1. Shift FOCUS to process of acquiring and developing operative skills rather than the number performed 2. Aim for qualitative improvement by providing guidance, supervision and feedback 3. Pay attention to the cognitive factor and the judgment component of performance.

57. Recommendations 1. Shift FOCUS to process of acquiring and developing operative skills rather than the number performed 2. Aim for qualitative improvement by providing guidance, supervision and feedbac 3. Pay attention to the cognitive factor and the judgment component of performance 4. Conduct researches on different components or aspects of residency training

58. Performance-based Evaluation The Objective Structured Clinical Exam (OSCE) Multiple stations Time allotted: 5-25 mins Well-defined clinical task Use of real or standardized patient (SP) Use of raters Rating scales and checklists

59. OSCE MILLER’S PYRAMID LEVEL OF PERFORMANCE MOST EFFECTIVELY MEASURED BY OSCE KNOWS KNOWS HOW +++ SHOWS HOW +++++ DOES

60. OSCE COMPETENCIES LEVEL OF PERFORMANCE EFFECTIVELY MEASURED BY OSCE KNOWLEDGE ++ PROBLEM-SOLVING +++++ CLINICAL DECISION-MAKING +++++ SKILLS, HISTORY-TAKING +++++ SKILLS, PHYSICAL EXAM +++++ SKILLS, PROCEDURES +++++ SKILLS, INTERPERSONAL +++ ATTITUDES +++

61. OSCE Development Planning/preparing needed resources Identification of competencies (test blueprint) Identification/recruitment of raters Training of standardized patients and raters Conducting workshops on standard-setting Constructing rating scales/checklists Gathering diagnostic materials Selecting venue Pilot-testing

62. Summary Educational basis for the acquisition of skills Re-examined PBOGS requirements Research studies on training System of determining requirements and validation System of documentation and reporting Discussion on policies Future implications OSCE

63. END

64. Workshop Activity 1.Determining requirements in 3 columns (Delphi method) Operative Procedure Number of 1 st Assists Number of Supervised operations Number of independently performed operations Round 1 Round 2 Round 3

65. Workshop Output 1.Determining requirements in 3 columns (Delphi method) Caesarian section Number of 1 st Assists Number of Supervised operations Number of independently performed operations Group 18715 Group 2555 Group 310312

66. Workshop Output 1.Determining requirements in 3 columns (Delphi method) TAHBSO Number of 1 st Assists Number of Supervised operations Number of independently performed operations Group 112818 Group 21055 Group 31058

67. Workshop Output 1.Determining requirements in 3 columns (Delphi method) TAHBSO Number of 1 st Assists Number of Supervised operations Number of independently performed operations Group 1 Group 2 Group 3

68. Workshop Activity 2. Discussion: graduates who cannot pass the certifying exams: reasons / courses of action PBOGS Passing: 65-67% (2001-2006) Philhealth: CS done by Diplomates/Fellows (60%) CS done by GP’s with training (30%) CS done by MD’s w/o training (7.4%) Physician Act of 2012: Art. V. Sec.28 (k): Performing… an area of specialization without fulfilling specialization requirements prescribed by the AIPO and the Board of Medicine