A Guide to Clinical Audit Dr. CHANDY ABRAHAM. Chief of Med. Staff. Baptist Hospital. B’lore. MS. DNB. MRCS. ADHA.

How do we ensure a high quality of service How do we ensure a high quality of service? How do we improve our service? One way is to ask questions about how, where and when we deliver the service This is the essence of Clinical Audit – at its simplest it is asking whether we do the right thing, at the right time, to the right patient Clinical Audit isn’t really anything new. Early proponents of clinical audit include Florence Nightingale, who studied infection rates at Scutari Hospital during the Crimean War (1854) and Dr Ernest Codman, who monitored surgical outcomes at his Massachusetts Hospital (1914)

Clinical Audit is………… Clinical Audit has many definitions, this is the one above is used by NICE and the Healthcare Commission “a quality improvement process that seeks to improve patient care and outcomes through systematic review of care against explicit criteria and the implementation of change” (Principles for Best Practice in Clinical Audit 2002)

Clinical Audit “a quality improvement process that seeks to improve patient care and outcomes through systematic review of care against explicit criteria and the implementation of change” From the definition we can extract two key points about Clinical Audit That we need to set explicit criteria That we need to set standards by which to systematically review care And also a third concept – that audit should lead to change

Clinical Audit…….Criteria What do we mean by criteria? The criteria (singular criterion) are the things that we are measuring in our audit. A criterion can cover a single event e.g. “Did Staff sign the medical record in black ink?” or can cover a multitude of separate events within one process e.g. “Was the patient assessed and treated within 2 hours of arrival in clinic?” The main thing is that the criteria used are objective, unambiguous and should generally be answerable either yes or no. Lets have a look at some possible criteria

Clinical Audit…….Criteria Is the patient seen in a timely manner? NOT an audit criterion – timely is ambiguous. Was the patient satisfied with their treatment? NOT an audit criterion – satisfaction is not objective. Could very well be an important area to look at, but it is not strictly an audit. Was the patient seen by a specialist within two weeks of referral? Possibly an audit criterion – but is referral absolutely unambiguous? – does it mean within 2 weeks of the date the referral letter was sent, date referral letter was received or date patient was put on the waiting list? And what exactly does two weeks mean? Was the patient seen by a specialist within 14 working days of their GP appointment? YES – an audit criterion we can know the dates of both appointments and unambiguously measure the time between them.

Clinical Audit…….Standards Once we have decided on our criteria we can define our standards. The standard is “how many of the cases that we look at do we expect to meet our audit criterion”. We could set the standard at 100% e.g. All patients will be seen within 4 hours of arrival at A&E. Or we might consider 100% to be unrealistic and so set the standard at a lower e.g. 95% of children will receive the MMR vaccination. Or we could set more than one standard e.g. 50% of stroke patients should be SALT assessed within 24 hours and 100% within 72 hours.

Clinical Audit…….Standards But how do we decide what the standard should be? Fortunately there are a lot of sources to help us set fair and evidence-based standard for our audits. These include: Nabh guidelines NICE guidelines the Royal Colleges and professional bodies Scientific research papers Local good practice more on this follows

Clinical Audit…….Standards NABH GUIDELINES: The NABH has policies, procedures and guidelines covering many subject areas. It is straight forward to develop an audit based on policy For example let us take Discharge Policy.AAC15: Organisation defines the content of the discharge summary. So the audit tool would be Criterion a) Is the DS in the disch.patient’s notes YES/NO Standard is 100% YES Criterion b) Does the DS have a clear diagnosis YES/NO Standard is 100% YES Criterion c) Does it contain info on urgent care YES/NO Standard is 100% YES

Clinical Audit…….Standards Scientific research Of course, the standards suggested by all of the national bodies are based ultimately on the scientific evidence. The National Library for Health website is an excellent portal to a vast array of information on evidence based healthcare. You can find links to Bandolier, Medline, Clinical knowledge Summaries (formerly PRODIGY), and the Cochrane Library. http://www.library.nhs.uk/Default.aspx

Clinical Audit…….Standards he National Institute for health and Clinical Excellence or NICE is an independent organisation responsible for providing national guidance on promoting good health and preventing and treating ill health (www.nice.org.uk). NICE provide guidance both on treatment methods (known as clinical guidelines) and the use of specific drugs (known as technical appraisal documents). Often these guidelines will come complete with an audit tool to help you monitor your practice. NICE guidance is highly evidence-based and so is regarded as a “Gold Standard” by external appraisal organisations such as Welsh Risk Pool and Healthcare Inspectorate Wales

Clinical Audit…….Standards The Royal Colleges and other professional bodies have a great interest in promoting professional and service development through the use of audit. It is worth looking at the web sites of the Royal Colleges and other professional bodies to see what audit tools they may have developed. http://www.rcplondon.ac.uk/news/news.asp?PR_id=354

Clinical Audit…….Standards Local Good Practice It may be that you wish to audit something that doesn’t fall into any of the other categories. The interest here is seeing whether a situation is static or changing. So for example I could do an audit using the criterion “a reply is to be sent to any written enquiries received by my office within 2 working days” and the standard “all written enquiries to be responded to in this way” However it could be argued that since I have set both the criteria and the standard myself it may mean I am not working to any agreed best practice. Wherever possible try to base your audit on policy or national guidelines

Clinical Audit…….Summary Remember audit is about two things An unambiguous criterion – ideally one that can be answered yes or no A standard – how many of the cases you examine is it reasonable to expect to meet the criterion? This standard should be set using the best available evidence. “Gold Standard” NABH,Scientific evid. NICE etc. “Silver Standard” State Policy based on registry etc. “Bronze Standard” Locally agreed standards Ready? Lets do an audit

Audit example My topic of interest: shiny sports cars

Audit example - Criteria I have decided to audit my car’s fuel efficiency by measuring the miles I get per gallon of fuel Objective audit? Yes, this can be measured by anyone in the same way Unambiguous audit? Yes, as long as I give the results expressed as miles per gallon every one will know what I mean

Audit example - Standard It is important to set a fair and evidence-based criterion and standard, so I do some research and look at several evidence sources Evidence a) The Environmental Protection Agency, an impartial government department, suggests I should get 17.2 MPG Evidence b) The Manufacturers, Lamborghini, suggest 18.1 MPG (under ideal conditions in Tuscany) Evidence c) The Royal College of Sport Car Drivers, a respected professional group, suggest 17.5 MPG I decide that 17 MPG is a fair figure for the criterion and that a fair standard would be that my car always meets this performance target

Audit example Our audit is ready to go We have the subject – car fuel economy We have the Criterion – The car must achieve at least 17 Miles per Gallon We have the Standard – The car must achieve this figure 100% of the time. Finally we need to decide if this is a one–off measurement or a regular audit. In this case lets do a monthly audit

Results Date result standard met? May 07 17.5mpg yes June 07 17.9mpg yes July 07 17.1mpg yes Aug 07 On Holiday n/a Sept 07 16.8mpg no Oct 07 16.4mpg no Nov 07 15.9mpg no

Results Clearly my car is falling below an acceptable standard and I need to take it in for a service. Note that nothing disastrous needs to have happened to my car. The benefit of audit is that I can pick up small problems before they become serious ones. Congratulations, you’ve just done an audit!

How to choose an audit topic This simple checklist may help you decide what to audit A suitable subject for audit: is one with direct importance for patient care a large number of patients are involved a high cost is involved good-practice guidance has been issued by a professional or governmental body there is significant clinical concern over patient health and well-being unsatisfactory outcomes have been reported –either incidents or complaints

How to choose an audit topic What makes a good audit? Clear nationally agreed criteria – such as auditing to NABH, NICE standards or national Royal College audits Audits that involve multidisciplinary teams to get a complete picture of patient care A realistic appraisal of how difficult it will be to complete the audit both in terms of time and in being able to access the data.

How to choose an audit topic Think SMART ! S – have I defined my audit topic well enough for it to be truly Specific M – have I chosen criteria such that they are practical and Measurable A – Have I enough resources/access to make the audit Achievable R – in appraising the difficulty in completing the audit have I been Realistic T – Can I give the audit enough effort to complete it in a Timely way.

OK, lets have a further example Scenario : You wish to undertake an audit around the supply of insulin to diabetic patients who present with a doctors prescription. Is this a good audit subject? We can decide by going down our checklist –remember it’s a guide, not a rulebook, we don’t have to answer ever question yes, but if you answer every question no, is it really a worthwhile audit?

Insulin supply – a good audit? Important for patient health? Yes, Control of diabetes by the taking of insulin or oral hypoglycaemic agents has major health benefits. Large number of patients involved? Yes, Diabetes is a fairly common condition High Cost? Not specifically relating to treatment but costs of error or non-compliance can be high Good practice guidance available? Yes, this area is well researched Is there concern over incidents or complaints? No, none at the moment Is it multidisciplinary? Yes, involves more than one specialty, pharmacists and patients Can the data be obtained easily? Yes, by discussion with patients and examination of prescriptions So overall we can see this will be a worthy audit subject

Let us begin by setting measurable, unambiguous criteria Does the prescription contain instructions as to dose and frequency? Does the patient know how to take the medication? Is the patient compliant? Does the patient suffer side effects? Does the patient know their last blood glucose level? Does the patient smoke? Has the dose been added to the medication label? There could well be others you can think of, but these were the ones actually chosen for this particular audit which was conducted at a NHS Trust in Wales in 2005

Let us then set our standards 1) Does the prescription contain instructions as to dose and frequency? We feel this is important so the standard should be 90% YES 2) Does the patient know how to take the medication? We feel this is very important so the standard should be 100% YES 3) Is the patient compliant? We feel this is important but the standard should be more that 85% YES since we cannot enforce total compliance. 4) Does the patient suffer side effects? We recognise that some people may suffer side effects, however the evidence suggests it should be less than 15% of patients, so the standard should be less than 15% YES

Let us then set our standards 5) Does the patient know their last blood glucose level? We feel this is important so the standard should be 100% YES 6) Does the patient smoke? The evidence available to us suggests that smoking is very harmful to diabetics therefore we feel the standard should be that no one smokes i.e. 0% YES 7) Has the dose been added to the medication label?

Create your audit tool Yes No total % of total that was yes Was there Appropriate instructions? Did the Patient know how to take the medication Was the patient compliant? Did the patient suffer side effects Did the patient know last Blood Glucose? Does the patient Smoke Was the Dose added to label?

Collect your data 36 patients 12 patients 48 patients 75% 44 patients Yes No total % of total that was yes Was there Appropriate instructions? 36 patients 12 patients 48 patients 75% Did the Patient know how to take the medication 44 patients 4 patients 91% Was the patient compliant? 42 patients 6 patients 87% Did the patient suffer side effects? 7 patients 41 patients 14% Did the patient know last Blood Glucose? 26 patients 22 patients 54% Does the patient Smoke? 13 patients 35 patients 27% Was the Dose added to label? 0 patients 0%

Be critical as you go along Remember that you are using an audit tool. Like any tool it must do the job you want it to. If you find that the audit isn’t capturing the data you want or there is some unexpected difficulty don’t be afraid to stop and rethink the tool. For this reason it can be a good idea to have a trial run before doing any large scale audits. In this case the audit staff realise there actually two distinct patient group – those who are prescribed insulin and those who are prescribed oral hypoglycaemic agents. So they modified the audit tool to capture the data slightly differently – presented this way we see that the two patient groups have markedly different results for 4 of the 7 criteria

New audit tool with data for insulin dependent patients and those taking oral hypoglycaemic agents (HGA) recorded separately Insulin YES Insulin NO Insulin % YES HGA YES HGA NO HGA Was there Appropriate instructions? 0 patients 8 patients 0% 36 patients 4 patients 90% Did the Patient know how to take medication 100% Was the patient compliant? 34 patients 6 patients 85% Did the patient suffer side effects? 2 patients 25% 5 patients 35 patients 12.5% know last Blood Glucose? 3 patients 37.5% 23 patients 17 patients 57.5% Does the patient Smoke? 1 patients 7 patients 14% 12 patients 28 patients 30% Was the Dose added to label? 40 patients

How to write up your audit Begin with an introduction giving a brief background as to why you are doing this audit and, if you are using external criteria and standards eg NICE guidance, what these may be. Generally you will want to give enough detail about what you did to allow someone else to repeat your audit just from reading your report Don’t forget to date your report and to put your name or job title – someone might want your advice if they are doing a similar audit.

How to write up your audit Title: Insulin and HGA supply Background: Good medication control is essential in the treatment of diabetes Criteria and Standards used: Audit was based on that of the Pharmaceutical Society of Finland April 2006 Sample: All diabetic patients presenting at the pharmacy with prescriptions in the month of September 2006. 48 patients took part in the audit Report date: January 2007 Author: J G Pharmacist esq. But how do I present the data?

Do I need to do a complex statistical analysis?

Do I need to do a complex statistical analysis? Generally no, unlike research, most audits will not require heavy number crunching In fact, a simple graphical display is often the most effective method of sharing your data. Don’t be tempted to overcomplicate things just because your computer will let you! Lets have a look at those diabetic patients in the audit we just did who are smokers

Presenting the data -Diabetics who smoke This is typical of the many graphs that Excel and other programs can do for you. Very stylish- but also very difficult to understand the eye is drawn all over the graph and finds it hard to make sense of it

Presenting the data -Diabetics who smoke This is exactly the same data but shown in a different way. The reader can quickly grasp that just over a quarter of our diabetic patients are smokers

Of course you will then want to include your audit data Insulin YES Insulin NO Insulin % YES HGA YES HGA NO HGA Was there Appropriate instructions? 0 patients 8 patients 0% 36 patients 4 patients 90% Did the Patient know how to take medication 100% Was the patient compliant? 34 patients 6 patients 85% Did the patient suffer side effects? 2 patients 25% 5 patients 35 patients 12.5% know last Blood Glucose? 3 patients 37.5% 23 patients 17 patients 57.5% Does the patient Smoke? 1 patients 7 patients 14% 12 patients 28 patients 30% Was the Dose added to label? 40 patients

But also consider an easy to read summary Patients taking insulin Patients taking HGA’s criteria Pass or Fail Appropriate instructions Fail Pass Patient knowledge patient compliance patient side effects last Blood Glucose known patient Smoking Dose added to label

Discussing the data A brief discussion will help flesh out the bare bones of the audit perhaps take each point in turn Criterion 1 It was notable that whilst most patients taking the tablets had appropriate documentation with the prescription, none of those taking insulin did Criterion 2 On the other hand, whilst insulin taking patients were very knowledgeable about their treatment, tablet-taking patients were much less sure of what to do. Criterion 3 Again, whilst insulin-taking patients were very compliant with their treatment, tablet-taking patients were much less so. Criterion 4 Whilst overall side effects were within acceptable levels when look at as separate patient groups insulin-taking patients may be suffering an unacceptable level of side effects Criterion 5 Both patient groups fall below the standard we set for knowing their last blood glucose Criterion 6 A higher than desirable number of patients smoke Criterion 7 – A disaster apparently – no labelling at all!

Be critical after you’re done Why did all the patients taking insulin not have detailed prescriptions? To complete your audit maybe some follow up questions are needed. Ask the doctors who prescribe whether there are good reasons for this or not. It might be that all the patients involved are long-standing insulin takers who are no longer thought to require detailed prescriptions –but does this mean the procedures have become casual, even slack as a result? Would a new insulin taking patients also not get a detailed prescription?

Be critical after you’re done Why did some the patients taking tablets not have a good understanding of how to take their medication? Is it perhaps because tablets are thought to be less of a problem than injections and therefore less time is devoted to explaining to patients how the drugs should be taken? Do older patients find it difficult to open packaging and bottles?

Be critical after you’re done Can you cross-check your data? Did the 6 patients who were non-compliant include the 4 patients who did not have a good understanding of how to take their medication? Otherwise you might mistakenly think compliance is the issue when in fact it is due to a lack of patent understanding

Be critical after you’re done Is your data hiding something? When looking at side effects the overall rate was an acceptable 14%. However when broken down into sub-groups we see that the whilst the tablet-taking patients had an acceptable 12.5% side effect rate, the insulin taking patients had an unacceptably high side effect rate of 25% and would merit review.

Be critical after you’re done On the face of it only 54% of patients know their last blood glucose levels, a figure that gives us concern. But talking to audit staff reveals that we have asked a rather weak question. Remember one of our rules for good audit criteria was that they should be unambiguous. What exactly are we asking here: A. That patients know their last blood glucose level and remember it to two decimal places when asked? B. Patients did know their last blood glucose level and approximately remember what it was when asked? C. Patients do regularly check their blood glucose levels and know the appropriate action to take but can’t quite remember what it was last time?

Be critical after you’re done Compare the question on glucose levels to the next question on smoking. We can be confident that this criterion is more robust in that patients will absolutely remember if they smoke or not! However we must be aware that some patients might be reluctant to admit to doing something that is known to be bad for them!

Be critical after you’re done Finally why did we score 0% for dose on label are staff unaware of procedure? are staff aware but unwilling to adopt new practice? has new legislation just implemented? However if you suspect a particularly poor outcome for a given criterion you should explain to staff why you are measuring it. It is often the system that is at fault not the individual

Be critical after you’re done Perhaps the reason for the 0% score was that it was a technical impossibility to score any better. Perhaps we know that we have no printers that can print on the label? So why measure something when you know the answer? a) Identify risks b) highlight concern c) force action d) provide baseline data to measure future improvement

Formulate action plan This in many ways is the most important part of the audit. What changes would you like to see in place to address the issues that your audit has found? Your action plan might look something like this; Criterion 1 Agree with doctors what details should be on all patients’ prescriptions New audit standard – all prescriptions to contain agreed details Criterion 2 and 3 Ask staff to review what information is given to improve patients understanding of their medication. Criterion 4 Ask the doctors to review the side effects suffered by insulin-taking patients New audit subject – specifically investigate insulin-taking diabetics in your area to see if there is generally a higher than expected rate of side effects

Formulate action plan Criterion 5 Change the wording of the audit standard to be less ambiguous Criterion 6 Ask all diabetic patients who present if they are smokers. New audit standard – all diabetic smokers to be given advice on smoking cessation Criterion 7 Put in a request for funding to obtain printers capable of printing labels.

The next steps Audit gives a formal record of what is going on. Audit highlights current strengths as well as possible weaknesses. Audit provides a benchmark against which we can measure change. Audit gives us a mechanism to consider what we do and how we do it. Audit is a process not an event – then end of one audit should provide the information that allows the planning of the next audit. Audit is a spiral that strives to push standards ever higher.

Audit is part of the process not the end point

How audit fits with the big picture Clinical effectiveness: Did the patients recover as well as we expected? Clinical audit: All patients must take 2 red pills per week Patient satisfaction: Are the pills easy to take? Research: Do patients recover better if they take 2 red pills or 3 blue pills per week? Patient experience: Did the patient recover as well as they expected? Patient interface: Do patients find it easy to take the correct dose with blue pills or red pills?