Component 2 The Certification of Graduates Using a National Competency-based Examination Tri Hanggono Achmad 2 nd HPEQ Conference Denpasar, 4 December 2011

Outline of presentation Overview of Component 2 Report and key performance indicator Lesson learned Challenges and Future plan

OVERVIEW OF COMPONENT 2

Health Professions Education Quality (HPEQ) Project Rationale: Indonesias current health indicators, the current status of Indonesias health professional education programs, and on the increasing demand for larger numbers of and more highly qualified health professionals. Goals: strengthen quality assurance policies governing the education of health professionals in Indonesia through 1. rationalizing and assuring competency focused accreditation of public and private health professional training institutions, 2. developing national competency standards and testing procedures for certification and licensing of health professionals, and 3. building institutional capacity to employ results based grants for encouraging use of accreditation and certification standards in the development of medical school quality.

Objectives Sub-component 2.1 : Establishing an Independent National Agency for Competence Examination of Health Professionals Sub-component 2.2 : Improving the Methodology and Management of the National Competency- based Examination Sub-component 2.3 : Developing an Item Bank Networking System to Support the National Competence Examination

Milestones for Component 2 Main Program National Agency kick off mid-term evaluation final evaluationtake offsustainable 2.2.a Computer-based Testing (CBT) Paper-based, Preparation and Try out Implementation Paper-based, CBT Preparation and Try Out Implementation Preparation Benchmarking Test Preparation and Try Out Implementation 2.2.b Objective Structured Clinical Examination (OSCE) PreparationTry-outImplementation Preparation Try outImplementation Preparation Try outImplementation 2.3 National Item Bank Networking for Assessment (NIBNA) PreparationEstablishment Enhancement for Pre-clinical Formative Test Enhancement for Clinical Formative Test Enhancement for Progress Test PreparationEstablishment Enhancement for Pre-clinical Formative Test Enhancement for Clinical Formative Test Preparation Establishment Medicine Dentistry Nursing/Midwifery

HPEQ + Pharmacy Nutrition Public Health Other health professionals related to MTKI

Key Performance Indicator Successfull rates on the first take of National Competence Examination Health Professional Baseline (2009) Target Value Medicine 71,67 % 73,25 %75,50 %79,60 %84 % Dental 79 %80 % 81 %82 %83 % Nursing 60 %65 % Midwifery 60 %65 %

ACHIEVEMENT OF COMPONENT 2

Objectives Sub-component 2.1 : Establishing an Independent National Agency for Competence Examination of Health Professionals Sub-component 2.2 : Improving the Methodology and Management of the National Competency- based Examination Sub-component 2.3 : Developing an Item Bank Networking System to Support the National Competence Examination

Subcomponent 2.1 Lembaga Pengembangan Uji Kompetensi (LPUK) An independent national agency that responsible in developing assessment strategy, methodology, and tools to evaluate competency of graduation from health professional education institution (medicine, dentist, nurse, midwife) Function: 1.Assessment service: as resource person and facilitator to help the management of examination for organization in each health professions. 2.Academic: to improve assessment system in health professional education with evidence based practice

Organization Tim Task Force 1300a/E3-HPEQ/SK/08.11 Director: Iwan Dwi Prahasto Secretary: Yulherina Member: Moh. Ghozali (Medicine)Gandes Rahayu (Medicine) Tatong Haryanto (Medicine)Mei Syafriadi (Dentist) Iwan Dewanto (Dentist)Mia Damayanti (Dentist) I Made Kariasa (Nurse)Pramita Iriana (Nurse) Yetty L Irawan (Midwife)Ani Kusumastuti (Midwife) Rahayu Endah Astuti (HPEQ) Advisory board National Regional Examination Center Chariman Examination Division Assess Management Division ICT Division Item Bank Division R & D Division Executive secretary

The Milestone LPUK Before 2010: First phase of development 2010 – 2014: Program Acceleration and Expantion After 2014: Sustainability Phase Establishment of KBUKDI & KDGI Development of KNUKP Various workshop in item development & review, standard setting Implementation of UKDI & UKDGI, and competence examination in some MTKPs Academic paper & various guidelines for workshop & implementation of examination Resource Sharing International Benchmarking Human resource development Regional & International collaboration Recertification & assessment of professional behaviour Research grant & scientific publication Resource & ICT support for assessment Establishment of LPUK & quality standard Collaboration w/ other HP

Subcomponent 2.1 Benchmarking to National Board Examination for medicine, nurse & dentist Technical assistance from Prof. Gordon Page from Univ. British Columbia – Canada Agreement with Ministry of Health MoH Regulation Number 1796/Menkes/Per/VIII/2011 about Registration of health professional Guidelines to conduct various workshop on assessment Item development for CBT IT and CBT management Item development for OSCE OSCE Examiner Trainer for Standardized Patient Item bank administration OSCE Coordinator Research grant on assessment

Objectives Sub-component 2.1 : Establishing an Independent National Agency for Competence Examination of Health Professionals Sub-component 2.2 : Improving the Methodology and Management of the National Competency- based Examination Sub-component 2.3 : Developing an Item Bank Networking System to Support the National Competence Examination

Subcomponent 2.2 Medicine & Dentist: Standard setting with Modified Angoff Try out & CBT Implementation Try out OSCE Nurse & Midwife: Try out CBT in November 2011

Try out & CBT Implementation ActivityNumber of CBT Center MedicineDentistNurseMidwife Try Out (809)13 (610) Try Out 2106 Try Out Try Out Try Out21 CBT 110 CBT 214 CBT 327

Try Out OSCE Medicin e OSCE CenterCandidates Try Out 1114 Try Out DentistOSCE CenterCandidates Try Out 119 Try Out

Standard setting Standard Setting MedicineDentist Number of judgesCut ScoreNumber of judgesCut Score I II III4658 IV4858 V5058 VI6661 Judges: from health professions institutions Implementation of standard setting in some institutions

Total= 32 CBT Center Total workstations= 2892 PC/laptop (842 : 2140) Number of CBT Center & Workstations

Average result of UKDI & UKDGI in 2010 – 2011

Comparison of UKDI result based on Accreditation ANOVA p=0,000

Objectives Sub-component 2.1 : Establishing an Independent National Agency for Competence Examination of Health Professionals Sub-component 2.2 : Improving the Methodology and Management of the National Competency-based Examination Sub-component 2.3 : Developing an Item Bank Networking System to Support the National Competence Examination

Subcomponent 2.3 Establishment of NIBNA Item development for knowledge based assessment – CBT bank masing2 profesi Item development for skill based assessment - OSCE Development of IT support system for student assessment Technical assistance for Item Bank Administration: Prof. Clarke Hazlet from Univ. Alberta – Canada

Number of workshops & participants in 2011 No.WorkshopFrequencyParticipantInstitution 1Item Review and Development (CBT) 6 (4 M, 2 D) 8 (4 N, 4 Mid) 2 (other HP) 389 (M) 45 (D) 168 (N) 132 (Mid) 65 (M) 26 (D) 168 (N) 129(Mid) 2CBT Coordinator226 (M) 8 (D) 26 (M) 8 (D) 3Standard Setting765 (M) 26 (D) 65 (M) 26 (D) 4OSCE Examiner3 (2 M, 1 D)131 (M) 28 (D) 65 (M) 26 (D) 5Item Review and Development (OSCE) 3 (2 M, 1 D)130 (M) 53 (D) 65 (M) 26 (D) 6OSCE Coordinator212 (M) 7Standardized Patient265 (M) 26 (D) 65 (M) 26 (D) M: Medicine; D: Dentist; N: Nurse; Mid: Midwife; HP: Health Professions

Item Bank OSCE: Medicine: 12 stations Dentist: 9 stations

Jumping Hurdles Successfully Developing adequate quantity of high quality items: achievable target If development is collaboratively produced & organized, realistic timelines for achieving a large-enough bank are - 5 yrs for Medicine (given 72 medical schools & 40,000 items reqd) - 10 yrs in Dentistry (if 26 dental schools & 30,000 (?) items reqd) - 4 yrs in Nursing (if 57 nursing schools & 25,000 (?) items reqd) - 3 yrs in Midwifery (if 60 midwifery schools & 20,000 (?) items reqd)

SIPENA Development Road Map

© PadmaNet 2010 – ss IT Modules - interconnectivity Examinee OL- Reg Examinee On Line Registration Information Web Exam Location Adm Local Exam Supervisor Registration National Exam Supervisor Registration Exam Process PBTCBTOSCEBAU Summative - Scoring AbsolutNormatif Formative - Institution Selected Item BankTested Item BankNew Item Bank Summative - Assesment Bank 1 : New ItemBank 2 : Reviewed ItemBank 3 : Tested ItemBank 4 : Selected Item Call for Item Item Writer / Reviewer Registration Peer ReviewPanel Expert Blinding Review Blinding – Review

Lesson learned Building awareness of the role of assessment in improving the quality of life (health) and driving in prioritizing quality Support and Commitment from, and collaboration among various stakeholders More potencies being identified Resources sharing (human, infrastructure, items, system) efficiency & close the gap NCE support the improvement of assessment method & learning process in institutions assessment drives learning (standard of competence) Assessment as a learning / education materials and management process Progress in adopting evidence based protocol security means strong committment

Challenges & Future plan The dynamic of the project continuity of the program based on the road map with possible modification Maintaing quality in bigger scope & quantity of project prioritizing & synchronize & coordination among Project Components & stakeholders resources sharing among health professionals Closing the gap of development & implementation paralel implementation of assessment development & learning response (case : OSCE & Clinical teaching) Customers friendly item banking & evidence based security system Credit for item citation index academic merit system Future research in professional assessment (incl. Publications) Collaboration with other international examination agency

This Project Can be & Will be Collaboration is Projects Mode of Operation Since collaboration among health care educators also involves endorsement & participation of government & professional bodies in health care Successful !

Terima Kasih