1. Prescribing Skills Assessment Background, purpose and structure Professor Simon Maxwell Medical Director, Prescribing Skills Assessment s.maxwell@ed.ac.uk Slide compilation © 2011 The British Pharmacological Society & Medical Schools Council. All rights reserved.

2. Background

3. Prescribing medicines the most common intervention (for good or bad) that most doctors make to improve the health of their patients

4. Concerns about FY1 Prescribing Illing et al (2008) – How prepared are medical graduates to begin practice? A comparison of three diverse UK medical schools. – Prescribing considered to be the key problem – Over 80% graduates failed a prescribing assessment Skills for Health Report (2009) – Junior doctors in the NHS: Preparing medical students for employment and post-graduate training – 65% of respondents considered prescribing was an issue EQUIP Study (2009) – An in depth investigation into causes of prescribing errors by foundation trainees in relation to their medical education. – 125,000 prescriptions in North-West England – error rate 9% PROTECT Study (2011) – 45,000 prescriptions in Scotland – error rate 7%

5. An in depth investigation into causes of prescribing errors by foundation trainees in relation to their medical education. EQUIP study. Dornan et al, 2009 Available at http://www.gmc-uk.org/about/research/research_commissioned_4.asp 124,260 medication orders

6. Safe Prescribing Working Group (2008) Agreed learning outcomes of undergraduate medical education in relation to prescribing National e-Learning materials to support medical students Development of an assessment that might demonstrate whether the outcomes have been met National prescription chart Access to the BNF for students Available at www.medschools.ac.uk/AboutUs/Projects/Documents

7. Those competencies (and more) included in the new outcomes-based version of Tomorrow’s Doctors See paras on outcomes: 3, 8, 12, 14, 16, 17, 18 (Appendix 1), 21, 23 See paras on process and assessment: 27, 28, 29, 37, 21, 86, 108, 109, 112, 117, 171 “We realise that meeting these outcomes and standards will be challenging” – Foreword by Prof Peter Rubin

8. Purpose

9. Potential value of a national assessment No validated, reliable and widely accepted measure of prescribing performance currently exists All medical schools are required to demonstrate that their students are competent (Tomorrow’s Doctors para 117) NHS organisations are starting to set up multiple ad hoc assessments and would value a recognised assessment A national prescribing assessment would – pool academic resources – serve to raise and unify standards – be relevant prescribing groups: other grades of doctor, nurses, pharmacists, dentists and other health professionals

10. Why assess prescribing? Clinical governance/patient safety reasons – Prescribing is a fundamental skill of a doctor – Assessment might be a marker of competence to enter into (or continue in) clinical practice – Protection of patients – patient safety Educational reasons – Measure the success of training – Identify the need for improvements in training – Contribution to training (formative assessment) – Foster higher standards of attainment in prescribing

11. Prescribing Skills Assessment Developed jointly by the Medical Schools Council and British Pharmacological Society Summative (pass/fail) assessment of knowledge, judgement and skills related to prescribing medicines

12. Basic principles of the PSA It should be passed before qualification and subsequent assumption of NHS prescribing responsibilities It will be available to be taken during the final year and can be repeated if necessary It will be delivered online Candidates will have access to the British National Formulary It will assess competencies in prescribing relevant to Foundation doctors that map onto the outcomes identified in Tomorrow’s Doctors 2009 Will test skills and deductive powers (as well as knowledge) relevant to early postgraduate practice

13. Development

14. Steering Group Medical Schools Council British Pharmacological Society GMC Postgraduate Deans MSC-Assessment Alliance BMA Medical Students Committee NHS Employers Prescribing Skills Assessment Project Team is responsible for delivering the key work streams Steering Group Work Stream 1 ITEM BANK Work Stream 1 ITEM BANK Project Team Work Stream 2 DELIVERY Work Stream 2 DELIVERY Work Stream 3 GUIDANCE Work Stream 3 GUIDANCE

15. Funding The project has received initial funding from the Department of Health to support the development and piloting of the assessment – Employment of an assessment officer to support development of the item bank – Development of online delivery system – Initial piloting

16. Progress

17. Steering group met 8 times Paper-based pilots with student feedback – 11 medical schools (summer 2010) – Around 1,300 students Blueprint agreed Eight question item styles defined Templates for writing question items developed (Ppt) Department of Health funding secured (February 2011) Assessment officers appointed (summer 2011)

18. Student Feedback (Summer 2010) This assessment was an appropriate test of the prescribing skills expected of a final year medical student (4.1) The material in this assessment was relevant to the task expected of a Foundation Doctor (4.2) I was able to answer most of the questions in this assessment (3.8) The questions were set at the right level of difficulty (4.0) I do not have the knowledge and skills to enable me to pass a Prescribing Skills Assessment of this kind (2.0) Having access to the BNF during the assessment was very helpful (4.5) The layout and presentation of the questions was easy to follow (4.2) Feedback based on Likert scale (1 = strongly disagree to 5 = strongly agree)

19. Structure

20. Make a diagnosis Establish therapeutic goal Choose the therapeutic approach Choose the drug Choose the dose, route and frequency Choose the duration of therapy Write the prescription Inform the patient Monitor drug effects Review/alter prescription PATIENT History - examination - investigations PATIENT History - examination - investigations Knowledge Judgement Skill Knowledge Judgement Skill Knowledge Judgement Skill Knowledge Judgement Skill Knowledge Judgement Skill Knowledge Judgement Skill Knowledge Judgment Skill Knowledge Judgment Skill Knowledge Judgement Skill Knowledge Judgement Skill 1 1 2 2 3 3 4 4 5 5 6 6 7 7 8 8 9 9 10 Prescribing sub- competencies

21. Prescribing Assessment Station 12 Data Interpretation Station 2 Prescribing Station 8 Prescribing Station 9 Prescription Review Station 5 Communicating Information Station 6 Calculation Skills Station 7 Prescribing Station 1 Prescribing 12 stations TOTAL = 60 mins (100 marks) Station 11 Drug Monitoring Station 3 Prescription Review Station 10 Adverse Drug Reactions Station 4 Planning Management

22. Prescribing Skills Assessment Blueprint

23. StationDescriptionMarksComments 1Prescribing 1101 question item of 10 marks 2Prescribing 2101 question item of 10 marks 3Prescription Review 182 question items of 4 marks each 4Planning Management82 question items of 4 marks each 5Communicating Information63 question items of 2 marks each 6Drug Calculation Skills84 question items of 2 marks each 7Prescribing 3101 question item of 10 marks 8Prescribing 4101 question item of 10 marks 9Prescription Review 282 question items of 4 marks each 10Adverse Drug Reactions84 question items of 2 marks each 11Drug Monitoring84 question items of 2 marks each 12Data Interpretation63 question items of 2 marks each TOTAL MARKS100 Prescribing Skills Assessment – Format

24. Summary Prescribing is a key clinical skill for all doctors There is evidence of poor prescribing practice amongst new graduates and others A reliable and valid assessment of competence would be helpful for medical schools and the NHS, and serve to stimulate improved educational experiences for students The Prescribing Skills Assessment will enable graduates to demonstrate that they have achieved a basic level of competence

26. Prescribing Skills Assessment Quality control, peer review and delivery Professor Simon Maxwell Medical Director, Prescribing Skills Assessment s.maxwell@ed.ac.uk

27. Work streams Question item bank development Online delivery platform Guidance and communications

28. Quality control

29. Item Bank and Test Development Aim to deliver around 560 items by April 2012 Question item authors (around 60) currently being recruited – British Pharmacological Society – Medical Schools – Clinical pharmacy Training events (4) planned for September 2011 Items initially collected as Ppt files but will move to an online authoring system (submitted to questions.psa@prescribe.ac.uk)questions.psa@prescribe.ac.uk Peer review event over 2 days in February 2012 Assessment Board first meeting April 2012 Processes for assembling (e.g. standards) and standard setting assessments being developed

30. Unformatted content (scenarios and MCQs) provided by medical schools Steering Group members and volunteers apply content to PSA templates (ppt files) Volunteer item writers write assessment items using the 8 station type PSA templates Project team conducts initial review of items, checking for completeness, face validity, relevance to station type etc. Entered into item log with appropriate metadata. Sent for peer review. Project team collects item performance data based on performance amongst randomly selected students. Peer review workshops to (i) conduct face validity and cover up test of scenario and correct answer, (ii) review and edit the item to remove any ambiguity and eliminate detractors in the answer options, and (iii) decide whether to refer the item for standard setting. Assessment Board standard sets V3 items with reference to performance data and the definition of a borderline student. Project team enters V4 items into live bank V1 V2 V3 V4 Assessment Board monitors and maintains the bank in relation to current prescribing policy and practice, current relevance to the FY1 role, and balance of items across station type, domain, patient group and drug type. PSA Item Bank

31. Delivery

32. Technical capacity and delivery Online delivery system – developed by University of Edinburgh (based on OSCA system) – Authoring and editing tools, delivery of question items, assessment construction tools, marking, reporting, item analysis Piloting strategy – Preliminary online pilots (2) held May 2011 – Further online pilots being offered (Feb-June 2012) - to test the online delivery in a range of schools including those with and without experience of delivering online assessments* – Engaging with schools during 2011/2012 on local issues relating to the capacity to deliver this online assessment – Full scale pilot involving all medical schools (2012/13)

35. Guidance and Communication

36. Guidance and communications Engaging with all medical schools on the development of national policies to guide the implementation – Running of assessment events – Retakes and remediation – Appeals – Candidates with disabilities Development of appropriate contacts – Assessment and IT leads Website development – www.prescribe.ac.uk/psa Medical Schools Medical Students Other stakeholders

37. www.prescribe.ac.uk/psa

38. Timelines Question item author training – Sept 2011 Major peer-review event – Feb 2012 Pilots to be offered to schools – Feb–Jun 2012 Full national pilot – 2013 Implementation - 2014