1. Understanding reasons for variation by ethnicity in performance of GP specialty trainees in the MRCGP Applied Knowledge Test: cognitive interview study  
Dr Julie Pattinson University of Lincoln
Professor Niroshan Siriwardena, University of Lincoln
Dr Bijoy Sinha, Programme Director, Lincoln VTS
Dr Carol Blow, AKT core group lead, MRCGP

2. Research programme First study exploring:
variations in performance by ethnicity in the MRGCP AKT
employing cognitive interviewing method
Provides valuable insights into the causes of differential attainment
UK graduates (UKGs), BME UKGs, overseas trained (OST).
Importance – high stakes exams with severe consequences for failure
Understand why OST are more likely to underperform compared to UKGs
Concept of ethnicity is sensitive and politically charged
“ethnicity implies one or more of the following: shared origins or social background; shared culture and traditions that are distinctive, maintained between generations, and lead to a sense of identity and group; and a common language or religious tradition”. (Senior, S.A & Bhopal R, 1994).
Introduce the AKT:
The AKT is a 190 minute, 200-item multiple choice test (MCT), which assesses knowledge of clinical medicine (80%), evidence based medicine (10%) and administrative issues (10%) relevant to UK general practice using single best answer, extended modified question, drag and drop and free text formats. The other components of the MRCGP exam include, a Clinical Skills Assessment (CSA) and a Workplace Based Assessment (WBPA), which together assess the curriculum for specialty training for general practice.
Explain relevance – first study employing cog interview method –
Understand why OST candidates are more likely to fail compared to UKGs
Could provide a basis for developing interventions to reduce differential attainment in UK specialty training for general practice.
We appreciate that concept of ethnicity is sensitive and politically charged.
In defining ethnicity, we followed Senior and Bhopal, who wrote……………

3. Background no findings of discrimination based upon examiner judgments
review concluded “the method of the MRCGP assessment is not a reason for the differential outcomes” and noted that “the AKT is a machine marked examination testing applied clinical knowledge” There is a differential pass rate for both BME UKGs and IMGs compared to white UKGs…..(RCGP, 2016).
The fairness of the CSA, a component of the MRCGP examination, was legally challenged in 2014 with a Judicial Review brought by the British Association of Physicians of Indian origin (BAPIO) because of persistent differential attainment.
The review documented the statistics on differential pass rates but made no findings of discrimination based upon examiner judgments; across UKGs, IMGs & BME UKGs.
N.B. A differential pass rate was observed in the AKT for both BME UKGs and IMG when compared to White UKGs it is not possible to attribute this to examiner judgements in comparison to the CSA because the test is computer based. Therefore this poses some potential alternative explanations for differential pass rates that may require further exploration.

4. Aim and Objectives
Investigate reasons for differences in performance in the MRCGP AKT related to ethnicity of candidates.
Explore differences across three subgroups of doctors in GP speciality training by:
Ethnicity
Source of primary medical qualification
Comparative analysis
White British/Irish UK trained (UKGs)
Black and Minority Ethnic (BME UKGs)
Overseas trained (OST)
The Aim of this study is to investigate why there are differences in performance in the MRCGP AKT related to ethnicity of candidates. The objectives include exploring three subgroups of doctors in GP speciality training, by ethnicity and source of primary medical qualification: Next if we take a look at the background – why it is relevant to consider whether the AKT questions are valid and fair, regardless of candidate ethnicity and country of primary medical training.

5. Methods
Qualitative design - semi structured, cognitive interview (Willis, 2005).
15 live AKT questions
understanding questions, wording and answers
thought processes retrieving information and applying it to solving AKT questions
Systematic Grounded Theory (Strauss & Corbin, 1998).
21 Participants
13 OST
1 BME UKG
7 UKGs
Recruitment: East Midlands
GP Vocational Training Centres
Speciality Training Years 1 – 3.
AKT questions chosen on the basis of similar or different performance according to candidate ethnicity. Provided by RCGP. Not informed if their answers were correct.
Talk over methods section what I did and how… Even distribution of males and females and across ages.
GT inductive approach 21 participants reached saturation in the data no new themes coming through. Limitation 1BME UKG.

6. Emergent Theme 1: Theoretical versus real-life clinical experience
“You need theory obviously but the practical exposure makes you remember”. Female, OST
All participants
Classroom versus clinical experience
Sub-theme 1
Exposure
“If you are coming from outside Europe, they don’t see any CT in real life.”, Male, OST
All participants
Sub-theme 2
Clinical Exposure limited minor specialities
Easier to recall AKT information
OST less exposure
“We are taught to memorise things, all the doses and names.” Female OST
All participants - complexities responding to AKT questions from theory alone
Exposure- All Participants - Complexities recalling information and responding to AKT questions from theory alone. Learning methods include textbook, classroom, NICE (guidelines). Compared with questions on topics covered in clinical practice demonstrated in extract 1
Clinical exposure : OST participants were at a greater disadvantage compared to UKGs, OST participants stated they had not been exposed to minor speciality topics during their UK training or at undergraduate level, as demonstrated in Extract 2.
Memorisation Rote learning is fixing information in the memory through sheer repetition learning through memorisation, may account for increased attainment compared to UKGs.
Repeated rote learning at undergraduate level, may result in an enhanced cognitive ability compared to UK and BME UK trained participants. OST participants were more familiar with the term ‘Adie’s pupil’ drawn from theory (86% versus 43%) on a theoretical based question with no reported clinical exposure for all participants. OST participants have demonstrated that prior rote memorisation in undergraduate training may affect AKT attainment scores
OST participants
Memorisation - Theoretical Advantage for OST
Sub-theme 3
Rote Learning

7. Emergent Theme 2. ‘Recency, frequency, opportunity and relevance’
“I haven’t worked in paediatrics or come across this[UK].When I was in basic training seven years ago [overseas], that’s the time I read about vaccinations. I have forgotten.” Male, OST.
All participants
AKT Topics not revisited
Sub-theme 1 Recency
OST – lengthier intervals
All participants
“Getting exposed to many different patients makes it easier to remember.” Male OST
Frequent exposure to patients reinforced learning
Sub-theme 2 Repeated Exposure
All participants
I’m not going to be doing a rotation through that”. Male UKG
Sub-theme 3
Limited Opportunity
Rotation, gender, uncommon presentations
OST participants disadvantaged:
No access in UK or undergraduate training
Recency. Opening statement: All participants reported difficulties answering AKT questions on topics they had not recently studied or covered in clinical practice. Refer to laws of learning:
All participants reported that lengthy intervals between learning negatively affected their ability to recall information to answer AKT questions.
This was more difficult for OST participants as when compared with UKGs they reported lengthier intervals between learning. For example, some OST participant had not recently revisited an AKT topic whilst training in the UK, nor during or since undergraduate training as demonstrated in Extract 1.
Repeated exposure – reinforced learning All participants
Limited Opportunity: All Participants
Not covered in my rotation
Not a common presentation in general practice
Gender Impeding answering AKT question successfully was missed opportunity to learn either through rotation (e.g. gyne and obs, paediatrics - to answer question specific to women's sexual health, children's health).
OST limited opportunity - statistical learning in undergraduate training/UK training
Relevance
AKT topics relevant for minor/specialists (ophthalmology, vaccination in pregnancy).
Conflict applying statistical and mathematical OST training to UK exam systems
Theoretical AKT questions, specific only to an exam setting and not exemplary of real life
1. All participants stated that some AKT topics were more relevant for minor/specialists (e.g. smoking cessation advice, ophthalmology, vaccination in pregnancy).
2. OST participants – limited stats education at undergraduate level – did not understand the relevance of learning stats until training in the UK –outcome - motivation ,confidence, attitude toward learning stats with in OST.
Note - OST candidates were aware(stats) this is something they needed to pay attention to: affect motivation, attitude to learn.
OST Conflict, unfamiliar with lay out of calculation based AKT questions , how it is presented, too technical, and a lack of understanding of the context of writing it out for OST candidates.
All participants said – Irrelevant applying maths in a clinical setting (10 mins)
“overseas, we didn’t have much statistics. When we started studying for AKT, we got to know [UK] that we need to study statistics.” OST Male
All participants
Sub-theme 4
Relevance
AKT topics; speciality, stats, exam scenario
OST participants disadvantaged; statistics

8. Emergent Theme 3. ‘Cultural barriers’
“If you are not familiar with the system, you don’t know what services are available.” Female OST
Sub-theme 1 Unfamiliarity of the process of the NHS system
Transition to UK practice
“even the simplest things. Abbreviations” OST Female
“we read the question, will have got a bit of three way translation” OST Male
Translation in exam - UK training
Sub-theme 2
Abbreviations Language barriers; Exam Format
“guidance are used more
in the UK. So I still have to read a lot of guidance. Back where I trained, I don’t think national guidelines apply.” OST male
Distinct cultural barriers in the UK NHS system regarding unfamiliarity with the system – outcome unaware what services to offer patients.
Language barriers abbreviations - Challenges regarding translation of AKT questions as OST participants reported they think in their own native language, it’s a translation – it takes longer for OST participants to answer AKT questions. This may impact on performance and overall attainment scores.
NICE – Particularly for OST participants, accessing guidelines prior to training in the UK; less experience and limited time to revise and catch up on the guidelines. Applying guidelines in practice – unfamiliar working in a clinical framework.
Sub-theme 3
NICE - working in a clinical framework
Point of accessing UK guidelines

9. Implications Key messages
OST participants compared to UKGs appear to face additional difficulties answering AKT questions
Generated insights into reasons for difficulty in answering AKT questions in all participants
Provided a basis for developing interventions to reduce differential attainment in UK specialty training for general practice.
RCGP Fairness project (2013): Observed for IMG candidates limited evidence of insufficient time
Opened discussions
Issues related to test timing
Need for a robust test of competence to ensure safety for patients
Identified areas in the AKT where risk is more likely to be acute (e.g. calculating patient risk in practice, OST candidates adaption to UK clinical guidance, NICE)
A stronger position/impact
RCGP/BAPIO
Inform the test
Inform training (e.g. limited stats teaching in UK graduate speciality curriculum)
We haver provided a basis for developing interventions to reduce differential attainment in UK specialty training for general practice
because we identified specific types of AKT topics that may affect OST candidates AKT performance; including
Calculations
Process of adaption to NHS system
NICE
Learning patterns
Barriers affecting attainment in specific question types for OST participants.
Created discussions around additional time for OST candidates.
Need for a robust test of competence to ensure safety for patients – identify risk in training where patient care may be compromised in all participants.
We also found reasons for difficulty in answering AKT questions in all participants, therefore, this type of information will be helpful for MRCGP speciality training schemes and inform those that develop AKT test.
Who is interested who will we inform
Identified reasons for difficulty in answering AKT questions in all candidates
BME candidates trained overseas compared with white British candidates
Who is interested in our findings and can make changes RCGP, GMC BAPIO.

10. References
Benjamin, AS, Bjork, RA. (2000). On the relationship between recognition speed and accuracy for words rehearsed via rote versus elaborative rehearsal. J Exp Psychol. Learn Mem Cog, 26, 638.
Gill PS. General practitioners, ethnic diversity and racism. In: Coker N, ed. Racism in medicine: an agenda for change. King’s Fund, 2001:
Royal College of General Practitioners. (2013). AKT "fairness" project. MRCGP AKT Core Group. Internal publication.
Royal College of General Practitioners. (2016). MRCGP Annual Reports 2014 to Retrieved from
Senior PA, Bhopal R. Ethnicity as a variable in epidemiological research. BMJ1994;309:327-30
Strauss A L, Corbin J. (1988). Basics of Qualitative Research: Grounded Theory Procedures and Techniques. 2nd edition. Thousand Oaks: Sage.
Thorndike, E (1932). The Fundamentals of Learning. AMS Press Inc. ISBN

11. Thank you for listening!
and