1. RANZCP (competency-based) Fellowship Program – WBAs and EPAs Perth 2012-13

2. Presentation Overview Formative & Summative Assessments Workplace-based Assessments – WBAs –Why use WBAs? –When to use WBAs? –WBA tools and how to use them Entrustable Professional Activities – EPAs –What is an EPA? –When to entrust an EPA –Standard of entrustment –How the EPA Handbook and RANZCP Confirmation of Entrustment (COE) work –Entrusting an EPA How WBAs may help gauge readiness to entrust an EPA WBA practice activities

3. Modern educational trends Multi-dimensional competency framework The seven CanMEDS roles

4. Stages of training DEVELOPMENTAL TRAJECTORY STAGE 1STAGE 2STAGE 3 Basic Proficient Advanced

5. Formative & Summative Assessments FormativeSummative Assessment for learningAssessment of learning Immediate feedbackEvaluates trainee’s learning at a particular point Feedback identifies strengths Feedback: Pass/fail Percentage Brief comments Feedback highlights areas for improvement Must pass to progress Feedback typically delayed Ways to improve performance outlined Exams, Psychotherapies, Scholarly Project, ITAs WBAs EPAs

6. Formative Assessment – Why Use WBAs? Competency-based training requires actual demonstration of the knowledge, skills and attitudes needed for safe practice WBAs –measure what doctors actually do in practice –assess competence in an authentic setting –provide mechanism for structured feedback from supervisor to trainee –help assess trainee progress –encourage concrete and effective feedback –formalise and add structure to current supervisor activities

7. Summative Assessments – Why Use EPAs? Entrustable Professional Activities (EPAs) –are representative tasks that are only entrusted to competent professionals –are summative assessment tasks in the competency- based Fellowship program –capture and record supervisor judgements about trainee performance and competence

8. How Many WBAs and When? WBAs are a way to: –Provide feedback for future learning –Help track and assess trainee progress WBAs may occur –At any point during a rotation –Whenever a trainee is seeking to gain formal feedback At least three WBAs required before each EPA can be attained –But, completing three WBAs does not mean an EPA has been achieved

9. WBA Tools Four Board of Education approved WBAs: –Case-based Discussion (CbD) –Mini-Clinical Evaluation Exercise –Observed Clinical Activity (OCA) –Professional Presentation Trainees are responsible for –Initiating the WBA process –Retaining the WBA forms after completion –Updating their learning plans

10. NEW - Workplace-based Assessments (WBAs) CbDCase-based Discussion PPProfessional Presentation Mini-Clinical Evaluation Exercise OCAObserved Clinical Activity Highly recommended that trainees experience all of these tools early in their training

11. WBA Assessment All WBAs are assessed on a 9-point standard rating scale Trainee StageBelow standard for end of Stage (1 2 3) Meets standard for end of Stage (4 5 6) Above standard for end of Stage (7 8 9) Stage 1 – Basic standard Below standard for Basic trainee At Basic level as described on developmental descriptors Above Basic level. Moving towards the standard of a Proficient trainee Stage 2 – Proficient standard Below the standard of a Proficient trainee. Meets standard of a Basic trainee. Meets the standard of a Proficient trainee. Above the standard of Proficient trainee. Moving towards the standard of an Advanced trainee. Stage 3 – Advanced standard Below standard for Advanced trainee. Meets standard of a Proficient level trainee. Meets the standard of an Advanced trainee. Above the standard for an Advanced trainee.

12. WBA Assessment – Developmental Descriptors Developmental Descriptors: –are behavioural descriptors for the Fellowship Competencies –articulate how the Developmental Trajectory applies to the Fellowship Competencies through the training stages –that articulates what is expected at each training stage is provided for each aspect of practice –are used as a guide when assessing a trainee’s performance on a WBA

13. Example: Aspect of Practice: Assessment By the end of each Stage, the trainee’s performance in this aspect of practice can be assessed using the following standards: Use of Developmental Descriptors in WBA Assessment Stage 1 Basic level Stage 2 Proficient level Stage 3 Advanced level Conducts a standard assessment of a patient with a typical psychiatric disorder, but requires supervision to elicit all necessary data and to understand the significance of data obtained. With supervision, performs a detailed and comprehensive assessment of a patient presenting with typical and atypical features Performs a detailed and comprehensive assessment of a patient presenting with complex or multiple problems, or in special groups.

14. NOVICE 2 months BASIC 11 months STAGE 1 Trainee 2 Does not formulate using a BioPsychoSocial model ?Score Trainee 1 Does not formulate using a BioPsychoSocial model ?Score Below standard* for Stage 1 Meets standard* for Stage 1 Above standard* for Stage 1 6. Data synthesis 123456789n/a WBA Assessment example – Case-based discussion

15. Range of WBAs You can choose: Mini-Clinical Briefly observed clinical tasks Evaluation Exercise CbD Detailed case discussions with supervisor PP Professional Presentations OCA A full 50 minutes assessment with structured feedback and clear standards At least 3 of these will determine whether an EPA has been achieved.

16. WBA Tool – Case-based Discussion (CbD) Occurs in regular supervision time Discussion based on existing case notes Used to assess –Clinical reasoning –Decision making –Integration of medical knowledge with case management –Ability to document these skills Supervisor feedback provided to trainee is key

17. CbD Protocol Trainee arranges time with assessor Trainee selects four cases and gives the files to the assessor. Assessor chooses one of the four cases Trainee discusses case with assessor about 20 mins Assessor may prompt trainee on further discussion points – guidance on CbD tool Assessor provides feedback to trainee Assessor rates performance (9-point scale) Learning steps discussed and agreed on Form is signed by assessor and trainee Trainee keeps form and updates learning plan

18. Case-based Discussion

19. WBA Tool – Mini-Clinical Evaluation Exercise Assessor observes a trainee in a clinical encounter with a real patient Feedback provided on the agreed clinical task being assessed Ideally, patient is new to trainee, at least in terms of clinical skill being assessed Assessor should be –Familiar with Mini-Clinical Evaluation Exercise process –Familiar with the patient

20. Mini-Clinical Evaluation Exercise Used to assess –History taking process –History taking content –Mental state exam –Physical exam skills –Communication skills –Data synthesis –Organisation and efficiency Important: identify assessment focus before commencing exercise

21. Mini-Clinical Evaluation Exercise Protocol Trainee arranges with assessor to observe Mini-Clinical Evaluation Exercise Trainee and assessor agree on competencies being assessed Mini-Clinical Evaluation Exercise occurs: –Clinical encounter, observed by assessor (15–20 mins) –Discussion and feedback follows immediately (10–15 mins) –Self-reflection time –Assessor rates performance on WBA form (5–10 mins) Assessor takes no part in encounter unless intervention required for patient safety Trainee and assessor discuss feedback, agree on next steps and sign form Trainee keeps form and updates learning plan

22. WBA Tool– Observed Clinical Activity (OCA) Similar in format to summative Observed Clinical Interview (OCI) Occurs over two sessions Trainee observed for the duration of an initial patient assessment OCA and OCIs are different: OCA is formative but OCI is summative Trainees should be encouraged to use the OCA tool from Stage 1 as this will assist them to prepare for the OCIs. OCAs are good preparation for summative OCI in Stage 3

23. WBA – Observed Clinical Activity (OCA) Assesses –History taking and content –Mental state and relevant physical exam skills –Data synthesis –Management plan skills

24. Observed Clinical Activity (OCA) – Protocol Trainee arranges with assessor to do OCA Trainee nominates a clinical case, ideally new to trainee Full OCA requires two 1-hour sessions OCA (session 1) Clinical encounter50 mins Post-encounter feedback10 mins Thinking/self reflectionTrainee’s own time OCA (session 2) Presentation of Assessment & Viva20 mins Presentation of Plan & Viva20 mins Feedback10 mins Total110 mins

25. Observed Clinical Activity (OCA) – Protocol continued Feedback is given by assessor Trainee and assessor discuss and agree on next steps, and sign the form Trainee keeps the form and updates the learning plan

26. WBA Tool – Professional Presentation Used in situations such as –Journal club –Case presentations –Community education presentations –Clinical audits –Grand rounds –In-service presentations, etc. Assesses aspects of typical presentation skills expected from trainees –Introduction –Context –Analysis and critique –Delivery –Questions –Quality of information

27. Professional Presentation - Protocol Trainee selects forum and makes all necessary arrangements Develops presentation Trainee arranges for assessor to attend Trainee presents presentation of at least 30 mins including questions Assesor observes entire presentation Assessor rates performance Assessor provides feedback to trainee immediately after presentation Next learning steps discussed and agreed on, form signed Trainee retains form in their file

28. EPAs – What is an EPA? Entrustable Professional Activity Specific activities that a trainee must –Demonstrate their ability to perform; and –Perform with sufficient independence with distant (reactive) supervision Supervisor must be confident that the trainee –Knows when to ask for additional help –Will seek timely assistance An EPA should be –A task of high importance (core business) –A high-risk or error prone task –A task that is exemplary of a number of CanMEDS roles

29. EPAs – Stage 1 and 2 General Psychiatry Stage 1 Adult Psychiatry 1.Producing discharge summaries and organising appropriate transfer of care. 2.Initiating an antipsychotic in a patient with schizophrenia. 3.Active contribution to the multidisciplinary team meeting. 4.Communicating with a family about a young adult’s major mental illness. Stage 2 Adult Psychiatry 1.Demonstrating proficiency in all the expected tasks associated with prescription, administration and monitoring of ECT. 2.The application and use of the Mental Health Act. 3.Assessment and management of risk of harm to self and others. 4.The safe and effective use of clozapine in psychiatry. 5.Cultural competence.

30. Which EPAs and how many? In addition, trainees must be entrusted with two EPAs for each 6-month full-time equivalent (FTE) rotation they undertake in Stage 2. The EPAs are area of practice specific, thus trainees must attain TWO EPAs in: Child & Adolescent psychiatry (mandatory rotation) Consultation–Liaison psychiatry (mandatory rotation) Each of their elective 6-month FTE rotations.

31. Which EPAs and how many? Trainees must also be entrusted with: two Addiction Psychiatry EPAs two Psychiatry of Old Age EPAs If the above EPAs are entrusted in their designated area of practice rotations there are no further EPAs a trainee must attain by the end of Stage 2. However, if a trainee completes elective rotations in other areas of practice, they must attain the two EPAs specific to those rotations AND complete the Addiction Psychiatry and/or Psychiatry of Old Age EPAs (ie. in another area of practice rotation).

32. EPAs – Standards of Entrustment BASIC PROFICIENT ADVANCED STAGE 1STAGE 2 STAGE 3 PGY 1/2 LOW INDEPENDENCE DEVELOPMENTAL TRAJECTORY HIGH INDEPENDENCE HIGH LEVELS SUPERVISION LOW LEVELS SUPERVISION JUNIOR CONSULTANT

33. Tasks in Stage 1 training ROTATION 1ROTATION 2 Formative mid rotation Supervisor assessment EPA 1 Summative EPA 1 Summative Formative mid rotation Supervisor assessment WBA Formative WBA Formative Minimum of 3 WBAs for 1 EPA Supervisor assessment Summative WBA Formative WBA Formative WBA Formative WBA Formative WBA Formative EPA 2 Summative EPA 2 Summative EPA 3 Summative EPA 3 Summative EPA 4 Summative EPA 4 Summative WBA Formative WBA Formative WBA Formative WBA Formative WBA Formative Supervisor assessment Summative First 4 weeks

35. EPAs – Standards of Entrustment As with WBAs, EPAs are entrusted at the following levels –Stage 1 – Basic –Stage 2 – Proficient –Stage 3 – Advanced These standards are described by the Developmental Trajectory which traces the development of a trainee through training Developmental Descriptors help to define the standards

36. How do the EPA Handbook and RANZCP Confirmation of Entrustment work? All EPAs are detailed in the EPA Handbook, which is available online The EPA Handbook is essential for supervisors and trainees The online version will always be the most up-to-date Each EPA also has an RANZCP Confirmation of Entrustment (COE) -This is a summary version of full EPA -Sign-off of COE is required to certify entrustment

37. Process of Entrusting an EPA EPA Handbook document contains a full description of the knowledge, skills and attitudes required to gain entrustment of that professional activity May or may not be attained in dedicated supervision time Entrustment decision draws on all data available to supervisor –WBA performance –Observation –Information from other staff/allied health/etc

38. How WBAs Help Gauge Readiness to Entrust EPAs WBAs provide an evidence base The structured feedback provided by a WBA highlights trainee strengths and weaknesses They can measure progress over time WBAs can be used to assess difficult or complex tasks

39. Process of Entrusting an EPA When the supervisor judges the trainee to be competent to perform the EPA with only distant, reactive supervision → entrustment achieved To document entrustment –Trainee prints the RANZCP COE –Supervisor completes RANZCP COE –RANZCP COE signed by supervisor, trainee and DOT –EPA’s attainment will be reported to the College on the In-Training Assessment report at the end of each rotation

40. Overall assessment of a rotation In-Training Assessment (ITA) Form The mid-rotation FORMATIVE Assessment. Will be filed at Postgrad, and may be forwarded to the College as required. (Currently under development.) In-Training Assessment (ITA) Report The end of rotation SUMMATIVE Assessment which must be submitted to the College’s Training Department at the end of a rotation. This informs the Training Department what should be recorded on the trainee’s record for the rotation.

41. Overall assessment of a rotation Each stage of training will have a different ITA Form (mid- rotation report) and Report (end of rotation report). The ITA Report for each Stage will capture the following: The EPAs required for the relevant Stage The WBAs done to support the attainment of each EPA.

42. In-training Assessment (ITA) Form

43. ITA Form-statement of completed EPAs

44. ITA Form-supervisor assessment

45. How it all fits together? An example

46. Survival Guide EPA handbook Supervisors’ Handbook (under development) Developmental Descriptors Know the following; -WBAs -CbD -Mini-Clinical Evaluation Exercise -OCA -PP -ITA -EPA -COE -Your and the trainee’s responsibilities ALL resources available online

47. WBAs in the workplace Sample WBAs

48. CBFP in practice Your trainee approaches you to complete the EPA Communicating with a family about a young adult’s major mental illness. How do you proceed?

49. CBFP in practice Do you think the trainee is ready to do the EPA? Which WBAs would be appropriate?

50. CBFP in practice Consult the EPA handbook (online) (p17) -this will identify relevant Fellowship competencies, and give suggested WBAs Consult CBFP Developmental Descriptors to assist in determining appropriate level (Basic-end of Stage 1) Fellowship Competencies are broad, Developmental Descriptors are specific to stages of training.

51. CBFP in practice You decide to do a mini-Clinical Evaluation Exercise. How do you proceed?

52. WBA Tool – Mini-Clinical Evaluation Exercise Assessor observes a trainee in a clinical encounter with a real patient Feedback provided on the agreed clinical task being assessed Ideally, patient is new to trainee, at least in terms of clinical skill being assessed Assessor should be –Familiar with Mini-Clinical Evaluation Exercise process –Familiar with the patient

53. Mini-Clinical Evaluation Exercise Protocol Trainee arranges with assessor to observe Mini-Clinical Evaluation Exercise Trainee and assessor agree on competencies being assessed Mini-Clinical Evaluation Exercise occurs: –Clinical encounter, observed by assessor (15–20 mins) –Discussion and feedback follows immediately (10–15 mins) –Self-reflection time –Assessor rates performance on WBA form (5–10 mins) Assessor takes no part in encounter unless intervention required for patient safety Trainee and assessor discuss feedback, agree on next steps and sign form Trainee keeps form and updates learning plan

54. CBFP in practice Your trainee approaches you to complete the EPA Assessment and management of risk of harm to self and others. How do you proceed?

55. CBFP in practice Do you think the trainee is ready to do the EPA? Which WBAs would be appropriate?

56. CBFP in practice Consult the EPA handbook (online) (p24) -this will identify relevant Fellowship competencies, and give suggested WBAs Consult CBFP Developmental Descriptors to assist in determining appropriate level (Basic-end of Stage 1) Fellowship Competencies are broad, Developmental Descriptors are specific to stages of training.

57. CBFP in practice You decide to do a CbD. How do you proceed?

58. WBA Tool – Case-based Discussion (CbD) Occurs in regular supervision time Discussion based on existing case notes Used to assess –Clinical reasoning –Decision making –Integration of medical knowledge with case management –Ability to document these skills Feedback provided to trainee is key

59. CbD Protocol Trainee arranges time with assessor Trainee selects four cases and gives the files to the assessor Assessor chooses one of the four cases Trainee discusses case with assessor for about 20 mins Assessor may prompt trainee on further discussion points – guidance on CbD tool Assessor provides feedback to trainee Assessor rates performance (9-point scale) Learning steps discussed and agreed on Form is signed by assessor and trainee Trainee keeps form and updates learning plan

60. Quiz How many WBAs must be documented to attain an EPA? Can they all be the same variety of WBA? Are all WBAs related to an EPA? How many cases does a trainee need to provide for a CbD? Can I sign off EPAs based on another supervisor’s WBAs? How many EPAs can a trainee complete in a stage 1 rotation? Should I keep copies of the WBA documents? Where do I find out how to do a WBA?

61. GOOD LUCK!

62. CBFP in practice You would like to assess your trainee’s knowledge of medication side-effects. How do you proceed?