Presentation on theme: "Audit for Registrars Dr. Ramesh Mehay Course Organiser Bradford VTS"— Presentation transcript:
1 Audit for Registrars Dr. Ramesh Mehay Course Organiser Bradford VTS NOTE : Key points = core points to note for any sytematic approach to auditHandout 1 – resources for auditHand out 2 – a fun auditHand out 3 sample audit paperHandout 4 – COGPED CriteriaTutorial Evaluation FormThis Power Point Presentation
2 DefinitionClinical audit is the systematic and critical analysis of the quality of clinical care.This includes the procedures used for diagnosis and treatment, the associated use of resources and the effect of care on the outcome and quality of life for the patient.Clinical Governance = improving standardsDespite the fact that audit ties in with clinical governance, don’t you think the principles are sound and something we all should be doing anyway? Even before clinical governance came in?
3 Crombie et al. definedAudit as the process of reviewing the delivery of health care to identify deficiencies so that they may be remedied.Marinker (1990)the attempt to improve quality of medical care by measuring the performance in relation to desired standards and by improving on this performance
4 Definition – less formal Taking note of what we doLearning from itChanging it if necessaryWith the aim of improving care
5 Why do It? Development of professional education and self regulation Improvement of quality of patient careIncreasing accountabilityImprovement of motivation and teamworkAiding in the assessment of needsAs a stimulus to researchClinical audit aims to lead to an improvement in the quality of service providing:-improved care of patientsenhanced professionalism of staffefficient use of resourcesaid to continuing educationaid to administrationaccountability to those outside the profession
6 Fundamental Principles All about improving patient careShould be seen as part of day to day practiceDeveloping a critical eye on what we are doingTrying to improve things all the time
7 The Audit Cycle What Should Be Happening What Is Happening? What changes are needed
9 What Audit Is Not Not about: Performance Appraisal of Staff Disciplinary ActionsNeeds AssessmentResearch (which is usually about establishing new knowledge)Computers and StatisticsCompetition between doctors“Never judge good and bad professionals based on audit” – it is about improving care
10 Audit vs Research Research Audit Defines Good Practice Assess extent to which good practice is being obeyed/improvementsMay involve allocating pts to random treatment groupsNever involves thisMay involve placebo RxNever involves placebo RxDisturbs the pt beyond that required for normal clinical managementNever does thisMay involvea completely new treatmentNever involves a completely new treatmentOne off processOngoingOther notesBoth audit and Research are concerned with clinical practice effectivenessAudit can contribute to research – issues that need further exploration
11 When to Use What Method When to use it Why Research Good practice is not defined and comparisons are neededTo define good practiceData Collection or structured observationPractice patterns unknownTo catalogue prevailing practice without making judgementsAuditGood practice is defined but we want to know how much we are sticking to itTo improve current performance
12 Does Audit Lead to Change Hearnshaw et al, BJGP 1998Of 1257 auditsAround 80% on clinical careAround 65% led to change
13 Making Audit Easier – Avoid the Blocks BEFORE YOU STARTTime – big audits can eat up time in an already busy schedule, so :Keep it simple and smallLook at one or two criteriaEngage the whole team – otherwise it will be difficult! Is the team ready? (Enthusiasm, wanting to improve)WHEN YOU STARTDelegate & Share the workload – involve othersMake life easier – use computers to do the laborious stuff (patient searches)Use protocols / standards already laid by others (why re-invent the wheel?)Be careful of data collection – choose a topic which does not entail too much data collection to the extent it becomes exhaustive with subsequent loss of enthusiasmClear Aims – which must then be imparted to the whole practice teamBlocks to getting started1. TIMEEvery general practitioner is busy and so isthe staff. To overcome the time problem: Keep it small and simple, use a small sample of a large topic. or tackle a small topic.Work together in a team.2. RECORDS AND COMPUTERSIt is good to have good records and to have a computer. The topics that are suitable for data collection may be important for the practice. A great deal of information needed for audit is not adequately recorded by computers, so a note search may still be required.3. LACK OF SKILLS AND PROTOCOLSThis can be overcome by searching, preparing and setting protocols by the staff before starting.4. THE PRACTICE TEAMNegative attitudes to audit and general lack of support from the practice team is one of the blocks to starting an audit. To overcome thisproblem, ensure team members understand audit and the purpose of it and consider training in team building. You may even involvethe practice manager.Setting the aimsClear aims must be identified at the outset of any audit project in order to define its purpose explicitly.Self-selected aims should be chosen against the background of other studies where statistically significant results were obtained, i.e. evidence-based. The team should justify its choice of the aims and be prepared to examine those areas where the aims are included in the audit process.
14 Some Ideas You can do an Audit of Structure ie facilities being providedEg waiting times, availability of staff, record keeping (all patient records should have a summary card), equipmentProcess ie what was done to the patient eg referrals, prescribing, investigationsAspirin post MI, BP measurements 5 yearly in those aged 20-65Outcome ie result for the patientEg patient satisfaction, patients with high BP aged between should have a diastolic below 90mmHg within the first year of treatmenthigh risk practices (significant event audits) eg pneumococcal vaccines in splenectomised patients, are significant events being acted upon?The outcome is the ideal indicator for care but the most difficult to measure.Structure of the Audit refers to the physical features of the practice, the premises, the availability of staff and their training, clinic building, practice equipment, and records.Process refers to what the GP actually does, i.e. practice activity, e.g. prescribing habits, referrals, laboratory investigations, etc.Outcome refers to the results of health care, changes in patient’s future health status that can be attributed to health care, e.g. prevention of disease, prevention of premature death and patient satisfaction with the care provided.The outcome is the ideal indicator for care but the most difficult to measure.
15 Choosing a TopicCondition has an important impact on health or of great local concern KEY POINT ie serious consequences otherwiseCondition affects a large number of peopleGood reasons for believing current performance can be improved or improvements are needed KEY POINTConvincing evidence about appropriate care is availableData collection – choose a topic which does not entail too much data collection to the extent it becomes exhaustive with subsequent loss of enthusiasm (? A pilot??)CHOOSE SOMETHING THAT REALLY INTERESTS YOUNO POINT AUDITING SOMETHING YOU THINK THE PRACTICE IS DOING REALLY WELLThen discuss with others – are they interested too?Factor ConsequenceCondition should have important impact on health Likely increase in morbidity and mortality if care is poorCondition should affect a large number of people Improving quality of care in common conditions usually has more impact than in rare conditionsGood reasons for believing that current performance Concentrates effort on optimum elements of carebe improvedConvincing evidence is available about appropriate care Otherwise efforts to change current performance aredifficult to justifyIf others aren’t interested in your chosen topic, how can you affect change?
16 Choosing a Topic Remember, topic should be important : Chronic Disease Management eg referrals or use of lab services (INR’s in warfarin)Preventative Care eg childhood imms, Cervical CytologyPrescribing eg aspirin post MI, PPI’s (cost issue)
17 Examples Ways of spotting audit topics examples Important clinical events admissions for asthma“Significant events” patient died of MI – no record of smoking history or BPPatients' complaints too long to get an appointmentObservation no system for ensuring bag drugs up to dateObservations of staff patient on Warfarin not had INR for 6 monthsNICE subjects post-MI patients on aspirin
18 Criteria= yardsticks“An audit criterion is a specific statement of what should be happening.”A statement whichA) defines a measurable item of health care whichB) can be used to assess qualityKEYPOINT Criteria should be explicit. You must demonstrate evidence for justifying them (literature search, Evidence Based!).Yardstick = measuring toolA criterion describes AND assesses the quality of health careEg All patients with asthma should have their inhaler technique at least once a yearCriterion = A statement of the ideal (in a perfect faultless world). By stating the ideal, we can they focus on being realistic and setting a more achievable level – the so called standard
19 Criteria – KEY POINTS Ensure that the criterion is measurable – · “asthmatics should have had yearly PFs” is difficult to measure (how many years will you go back?);· “asthmatics should have had a PF recorded in the past year” is more practical.Don’t try to audit too many criteria at once – one or two will keep you busy enough.Try filling in the gaps of the following phrase to set your audit criterion:“All patients with xxxxx should have had a xxxxx in the last xxxxx.”
20 Criteria"All eligible women aged should have had a cervical smear in the last 5 years."“All asthmatics should have had a Peak Flow recorded in the past year.”“All drugs in our doctors’ bags should be in-date.”
21 Standards“An audit standard is a minimum level of acceptable performance for that criterion.”Make sure the standard is directly related to the criterion, also :-Should include a suitable timeframeA statement of the minimum level of acceptance on how well you should be achieving the ideal (the ideal = the criteria)
22 Standards→ Examples:"At least 80% of eligible women aged should have had a cervical smear in the last 5 years."“At least 60% of asthmatics should have had a Peak Flow recorded in the past year.”“100% of drugs in our doctors’ bags should be in-date.”The standard should reflect the clinical and medico-legal importance of the criterion.in the example above, 80% of women should have had a cervical smear,But of those who've had an abnormal smear, 100% should have had action taken.
23 Standards How to set standards Look at national guidelines Literature (journals), textbooksLocal guidelinesDiscussion with consultants/GPSI’sDiscussion with trainer/partnersKEY POINT : Standards set should be realistic and attainable. Justifiable reasons for the standard set should be made explicitly clear.A statement of how well you should be achieving the ideal (the ideal = the criteria)If you cant justify your chosen standard (ie the %) with literature, explain why you chose the % that you did.
24 Standards Some criteria are so important that they need 100% standard. However, 100% standards are unusual – patients or circumstances usually conspire against perfection and the standard needs to reflect that.Your literature search should give you an idea of what standards others have managed to reach.Your standard needs to follow on directly from your criterion – for example,“Patients on thyroxine should have had TFTs done in the last year; this should have happened in at least 90% of patients”.
25 TYING IT ALL TOGETHER Examples of Standards & Criteria All children under 2 years should be immunised against tetanus and polio90% of registered patients under the age of 2 years should have been immunised against polio and tetanusAll notes of those patients with an allergy to penicillin should be marked95% of patients with an allergy to penicillin should be clearly markedAll patients in the surgery should wait no longer than 30 minutes before a consultation70% of patients in the surgery should wait no longer than 30 minutes before a consultation
26 Preparation & Planning Must show evidence of teamwork – otherwise you will failTips: Audit will only result in change if you involve all the relevant team-members from the start. This may include the GPs and Nurses who will have to implement any changes, as well as office staff who can help you doing a computer search. Again, you will need to document this
27 Data Collection (1) You can collect information from: computer registersreview of contents of medical recordsquestionnaires – patients, staff or GPsdata collection sheets
28 Data CollectionBe careful of data collection – choose a topic which does not entail too much data collection to the extent it becomes exhaustive with subsequent loss of enthusiasm? Sampling – random or systemicOnly collect essential informationUse computers, ?data collection formsUse other staff & delegate – don’t do all the work yourselfSet a deadlineThe duration of data collection should be decided. It should be the minimum required to collect the essential information to keep the eagerness.Sampling - The number of the patients to be included in any audit can involve all potential patients in the target population. This may be possible if the total numbers are small or the data required is easy to gather.In other situations using a sample of the patients representing the target population can minimize the total number of the patients.Optimal sample size can be calculated by using random or systemic sampling. Random sampling can be done by using a computer or by using a random number table. Systemic sampling requires arranging the itemsto be audited in sequence and sampling every nth item in a predetermined pattern.You may wish to use a data collection form to make the process easier - The form should be self explanatoryand easy to complete.Delegate! You don’t have to collect all the data yourself. Most practices have a member of staff who can help you set up the search or organise questionnaires.
29 Presenting the Results Collect ResultsAnalyse ResultsSummarise ResultsPresent Results to the teamSimple arithmetic calculationsUse percentagesResults of 2nd data collection presented in the same way as the 1stpresenting the results to the team - The aim of this process is to produce evidence, which will certainly influencethe next stage of the audit cycle – implementingPercentages – Represent the results in the form of percentage of patients whose care complies withthe criterion so as to ease the comparison with the agreed standards.THE READER SHOULD NOT HAVE TO DO THE CALCULATIONS/WORK HIM/HERSELF#The results of the second data collection must be presented in the same way as the first to monitor the progress and to ease comparison.
31 Comparing Results to Standards CriterionStandardObserved ResultAll patients should be seen within 15 minutes of their appointment timeMinimum70%45%70%45%All diabetics to have had HbA1C in last 3 monthsMinimum95%90%90%Drug allergies to be marked as “active problem” on computerMinimum100%
32 Discussion – Data Collection (1) KEY POINT (Discussion of Data Collection 1) : You need to explain why you think the practice didn't meet the standard that was set.
33 Discussion – why standards not met Think: What reasons are there for practices not meeting audit standards?For example : reasons have included:Practice reasons:· Results having been put down as free text on computer, rather than coded;· Opportunistic rather than formal recall system in use;Doctor reasons:· Not all GPs were aware of the practice policy;· Not all partners agreed with the policy;Patient reasons:· Patients refusing to have tests done;· Patients on holiday when tests due.
34 Implementing Changes The most challenging stage Audit can tell you whether changes are needed, but it can’t tell you what methods to useChanges to be implemented should be a team discussion and decision – not imposed by the leader (ie you).The team has to discuss the ways through which they have to work, make a change and achieve the agreed standards.It has to discuss the advantages and disadvantages of each one and choose the most appropriateones.It has to consider the most practical methods, and all involved in that stage must be familiar with the aims of doing that and their roles.The team has to decide about the date of the second data collection and whenaudit should be repeated in the future.The audit leader has to monitor the process and get others to share the ideas, which is so importantin effecting change and keeping the enthusiasm.
35 Implementing ChangeThe changes to be implemented should be a team discussion and decision (?a practice meeting)What to do at the Practice Meeting:Emphasise what has been achieved.What are we proud of?What are we not so proud of?How can we correct any deficiencies?
36 Implementing Change Changes must be practical! How are you actually going to make the changes?Simply saying “We’ve got to do better” won’t result in changeYou need to think through in detail· what needs to be done· who’s going to do it· when· and how.If you get very low results, you may consider resetting the standards to a more realistic level (but justify it)
37 Implementing Change KEYPOINT Just telling people to do things better won't result in change. You need to write up in some detail how the changes will take place.FAIL Example: "The GPs agreed to do a serum rhubarb on any patient that they see who is on Viagra" - fail - this wouldn't be likely to pass, as there is no system to help them remember.PASS Example : ”(a) The GPs were given a prompt card that they could stick on their computer screen as a reminder to do a serum rhubarb on any patient that they see who is on Viagra;(b) the secretary will search every three months for patients who are overdue for their serum rhubarb, and flag it as an active problem on the computer system" - pass - as it should result in change.
38 Closing the Loop Ie repeating the cycle Re-evaluate care to ensure that any remedial action has been effective.Audit is a continuous cycle – if you didn’t meet the standard and you’ve planned changes, you’ll need to repeat the audit to make sure the changes have happened.
39 Conclusions from the Audit Summary of main issues learned KEYPOINTS:Comment on any improvements that have resulted.How well did your proposals for change work?If you again didn't reach the standard that you set, why not?If you did, should you be aiming higher next time, or look at something else e.g. whether abnormal serum rhubarbs have actually been acted on?Where should the practice go from here
40 Useful Resources MAAGs – medical audit advisory groups Clinical Governance Advisory GroupsNational/Local GuidelinesRCGP database of simple tested audits for day-to-day useLiterature, BooksThe WWWConsultants, GPSI’s, Trainers, PartnersMAAG’s – available in each PCT – ask your local PCT for contact details. They are really helpful. An increasing number of MAAGs have changed their name to incorporate titles such as ‘Primary Care Audit Group’, ‘Quality Steering Group’, ‘Audit and Education Group’, and ‘Quality Forum’. In some localities, MAAGs have been disbanded and their functions taken over by clinical governance staff of primary care groups and trusts.
41 How To Fail No justification for choice of audit No justification for criteria/standard settingsNot having explicit criteria/standardsSetting unreasonable standardsA general lack of evidence based literature or using material that is not peer referencedNot explicitly displaying teamwork in the “method” – must give specific examplesNumerical errors re: data collectionPresentation of data collection eg no graphs, no percentages (ie the reader has to do the hardwork him/herself)Not giving much thought to “changes to be evaluated” and not being specific enough. Not delegating specific changes to specific people/persons.Poor conclusions and what the process has taught youNo inclusion for possible sources of biasReferences not properly quotedJustification for Choice of AuditIf your audit results from a problem you've noticed in the practice, document it in your write-up; it suggests that there is a potential for change.Justification for Criteria/StandardsYou must state the evidence behind you chosen criteria/standards – and it should be evidence based. Cite journals and other literature. At least four relevant references, preferably more.Quoting ReferencesAuthors, year, journal/book, volume, pages etc
42 IF YOU DON’T WANT TO FAIL Go through the following online tutorialLook at the Marking Schedule – (yes, they provide you with an answer sheet!)You must pass on all 8 criteria.
43 SHO’s doing Audit for Summative Assessment If you are doing the audit while an SHO, you need to choose a topic that looks at the GP-hospital interface. Referrals or discharge letters are possible areas for audit. Again, you need to demonstrate that you've found a problem that needs to be investigated. I suggest that you discuss your proposed audit with your GP Scheme Organiser before you go ahead - your hospital colleagues may not know what's needed for Summative Assessment.
44 Checking GPR Understanding DISCUSS THE FOLLOWING STATEMENTSAn example of the Audit of process is audit of referrals to hospitals.Audit usually consumes an extensive amount of resources (of time, money etc.).Rare conditions should be audited.The higher the standard the practitioner starts with, the stronger is the resulting audit.Maintaining clearly written notes of at least 20% of patients who are sensitive to penicillin is an acceptable standard in generalpractice.The higher the amount of data the practitioner collects, the easier is the decision making process in audit.The most challenging stage in Audit is implementing change.In data collection all in the target population must be included.The agreed standards can be reset at realistic percentages after the first round of data collection.
45 Clinical Audit Association Ltd Clinical Audit Association Ltd Cleethorpes Centre Jackson Place Wilton Road Hunberton Lincolnshire DN36 4ASTel:
46 Clinical Governance Research and Development Unit Dept of General Practice and Primary Health Care University of Leicester Leicester General Hospital Gwendolen Rd Leicester LE5 4PWTel: Fax:
47 Cochrane Database of Systematic Review c/o British Medical Association BMA House Tavistock Square London WC1H 9JP
49 RCGP Effective Clinical Practice Unit School of Health and Related Research Regent Court 30 Regent Street Sheffield S1 4DATel: Fax:
50 RCGP NE Scotland Faculty The Primary Care Resource Centre Foresterhill Road Aberdeen AB25 2ZPTel: Fax:nescot/index.asp
51 UK Cochrane CentreDr Iain Chalmers, Director NHS Research and Development Programme Summertown Pavilion Middle Way Oxford OX2 7LG Tel:
52 Cochrane Collaboration in the field of primary care For information concerning work by the Cochrane Collaboration in the field of primary care, contact: Dr Lorne Becker Professor and Chair Dept of Family Medicine SUNY Upstate Medical University 475 Irving Avenue Syracuse, NY USA Tel: Fax: default.html
53 NHS Centre for Reviews and Dissemination University of York York, YO1 5DDTel: Fax:
54 Sources This power point has been derived from :