1. Ramesh Mehay Programme Director (Bradford VTS)
nMRCGP in a nutshell
Originally written 2007, updated Jan 2009

2. Aims and objectives Aims Objectives
Increase our understanding of nMRCGP
Help us feel more prepared for the assessments
And therefore feel better!
Objectives
Provide an overview of nMRCGP
Share understanding
Share concerns (and address them?)
Practise COT

3. Session plan Overview of the MRCGP and its components
Share fears and concerns
Practise some COTs in groups
?modelling
(IS2 – practise some CBDs)

4. Background to nMRCGP nMRCGP replaces both old MRCGP and SA
Based on new GP curriculum
new curriculum developed by reviewing literature, very extensive consultation with doctors and patients, etc
All components of nMRCGP are mapped to the competencies in the curriculum
GP training now overseen by PMETB, like all other medical specialties (JCPTGP is dead)

5. A programme of assessment…
nMRCGP
WPBA
AKT
CSA

6. Components AKT (Applied Knowledge Test)
machine marked test, 3x/year, at various venues
CSA (Clinical Skills Assessment)
OSCE-type exam, 3x/ year, Croydon
WPBA (Workplace Based Assessment)
recorded in e-portfolio held by GP trainee throughout the 3 years

7. Clinical Skills Assessment
‘Integrative assessment’ with 3 domains
Data gathering, technical and assessment
Clinical management
Interpersonal skills
13 stations, 10 mins each, balanced selection of cases
clear pass, marginal pass, marginal fail, clear fail, ‘serious concerns’
significant failure rate
take early enough to have time to retake
‘Integrative assessment’ with 3 domains
Data gathering, technical and assessment
Clinical management
Interpersonal skills
13 stations, 10 mins each, balanced selection of cases
Graded as clear pass, marginal pass, marginal fail, clear fail, ‘serious concerns’
Likely to have significant failure rate
Need to take early enough to have time to retake if necessary

8. Work Place Based Assessment: WBPA
Workplace assessment: the assessment of actual working practices undertaken in the working environment

9. Overview of WBA What the trainee actually does
Competencies demonstrated ‘when ready’
Assessment of developmental progression - guides decisions about future learning
Recorded in an electronic portfolio
Process is learner led - trainee has to ensure their e-portfolio covers the e-curriculum

10. WBA: compulsory components
Case Based Discussion (CBD)
Consultation Observation Tool (COT) or Mini-Clinical Evaluation Exercise (Mini CEX)
Multi Source Feedback (MSF)
Patient Satisfaction Questionnaire (PSQ)
Direct Observation of Procedural Skills (DOPS)

11. WBA: local subunits OOH work booklet
Clinical Supervisor’s Report (CSR)
Naturally Occurring Evidence (NOE)
Significant Event Review (SER)
Referrals analysis
Audit
(Case Review, Personal Learning, Complaints)

12. Who makes judgements?
The Trainer/Clinical Supervisor as (s)he does the assessments
Educational Supervisor as he reviews the ‘whole’ thing with the trainee
ARCP panels who review the whole thing when a trainee is moving up an ST grade

13. Case based Discussion (CBD)
Structured interview designed to explore professional judgement in clinical cases
Professional judgement = ability to make holistic, balanced and justifiable decisions in situations of complexity and uncertainty
Attributes tested:
Application of medical knowledge
Application of ethical frameworks
Ability to prioritise, consider implications, justify decisions
Recognising complexity and uncertainty
Structured interview designed to explore professional judgement in clinical cases
Professional judgement = ability to make holistic, balanced and justifiable decisions in situations of complexity and uncertainty
Attributes tested:
Application of medical knowledge
Application of ethical frameworks
Ability to prioritise, consider implications, justify decisions
Recognising complexity and uncertainty

14. CBD Competency areas CBD looks at 10 of the 12 competencies
Practising holistically
Data gathering and interpretation
Making decisions/diagnoses
Clinical management
Managing medical complexity
Primary Care Administration (IMT)
Working with colleagues
Community orientation
Maintaining an ethical approach
Fitness to practice
(not assessed by CBD: communication skills AND maintaining performance/learning/teaching)

15. CBD - the process
Trainee selects 3 cases, gives material to trainer 1w in advance
Need balance of cases and contexts
Trainer selects 2, and plans structured questions in advance
1h session = cover 2 cases
20mins case, 10mins feedback
Trainer records evidence and judges level of performance
(insuff evid/needs devel/competent/excellent)
Need to do a MINIMUM of 6 per post
All 6 before the ES meeting! (really, within 4m)

16. Key Points on CBD It is a STRUCTURED oral interview
On what the trainee actually did
And why they did that
And if they considered anything else at the time
So, don’t ask “what if” questions like you do in Random Case Analysis
Stick to the ‘here and now’ of the case
Use the question maker framework on (click nMRCGP then click CBD)

17. CBD: What’s the Experience So Far?
Trainees
Initially anxious but less stressful than current SA
Valued feedback
Found it realistic
Some concern re relationship with trainer
Trainers
Time consuming, need extra protected time
Helpful structure
May be more helpful for difficult trainees
Concern re relationship with trainees
From pilots

18. Consultation Observation Tool (COT)
Single consultation per session
Trainee and Trainer view together
Trainer assesses consultation on 4pt rating scale (similar to old MRCGP/SA)
No rule about consultation length
Ideally at least one consultation is assessed by someone other than trainer
Ideally: wide range of contexts required, including at least one child, older person, mental health problem
GPR selects a single consultation per session
GPR and trainer view together
Trainer assesses consultation on 7pt rating scale (criteria fairly similar to old MRCGP/SA)
Wide range of contexts required, including at least one child, older person, mental health problem
No rule about consultation length (though this may be considered in assessment)
Recommended that at least one consultation is assessed by someone other than trainer

19. Why Work Place Based Assessment?
What was wrong with the old MRCGP or Summative Assessment?
Why Work Place Based Assessment?

22. Miller’s Pyramid or Prism of Clinical Competence
Miller’s Prism or Pyramid of Clinical Competence

23. What is Authentic Performance?
“Testing should be as close as possible to the situation in which one attacks the problem.”
“Ill-structured problems are not found in simulated and/or standardized tests.”
“The variation inherent in professional practice will always elude capture by a set of rules.”
Wiggins, Assessing Student Performance: Exploring the Purpose and Limits of Testing, Jossey-Bass, Inc. 1993

24. Relationship between tools and competency areas
This says something about triangulation. So, not reaching a competency in one form of assessment can be searched for in another. Alternatively, you can validate the measures of a certain competency by comparing results in several assessment methods.

25. Good Assessment Instruments have:
Reliability (R)
Validity (V)
Educational impact (E)
Acceptability (A)
Cost (C)
(Mnemonic: CARVE)
Van der Vleuten, The assessment of professional competence: developments, research and practical implications, Adv Health Sci Educ 1 (1996),

26. Why WPBA? High validity = Authenticity High educational impact
Reliability = depends on how many you do; also some built in triangulation
Reconnects assessment with learning and the workplace
Assessment over entire training envelope
Cost Effective and now accepted!
Cost Effective because you, the trainers are doing it all!

27. And it gives continuous feedback
“a process of monitoring student’s progress through an area of learning so that decisions can be made about the best way to facilitate future learning”

28. The Problem With WPBA Inter-observer variation
Intra-observer variation
Case specificity
Inter-observer variation – the tendency of one observer to mark consistently higher or lower than another observer
Intra-observer variation – the variation in one observer’s marks for no apparent reason (good day/bad day phenomenon)
Case specificity – the variation in the candidate’s performance from one challenge to another, even when they seem to test the same attribute

29. Requirements of a high stakes performance assessment
Specification
Calibration
Moderation
Training
Verification and audit
(Baker, O’Neil, Linn 1991)
mmm..... Here are some of the current problems that need sorting out
Specification – of standards, criteria, scoring guides
Calibration – of assessors and moderators
Moderation – of results, particularly those on the borderline
Training – of assessors including retraining
Verification and Audit – through quality assurance measures and collection of reliability data

30. Rough Guide to Rating Scale
Excellent – Smooth and efficient. Able to use knowledge, judgment and skills to adjust management appropriately to the specific patient and operative procedure.
Competent – Lacks smoothness and efficiency but is able to use knowledge, judgment and skills to adjust management appropriately to the specific patient and operative procedure.
NEEDS FURTHER DEVELOPMENT:
Beginner – Lacks smoothness and efficiency. Able to manage the case but exhibits limited use of personal judgment and responsiveness to the specifics of the patient and operative procedure. Requires some limited coaching or attending intervention.
Novice – Can only manage the case with extensive coaching and attending intervention.