Saudi Diploma in Family Medicine Clinical Audit Saudi Diploma in Family Medicine Center of Post Graduate Studies in Family Medicine Presented by: Dr. Zekeriya Aktürk zekeriya.akturk@gmail.com www.aile.net / 29

Clinical Audit and the Audit Cycle / 29

What we are going to cover today The audit cycle / 29

What we are going to cover today The audit cycle Criteria and standards / 29

What we are going to cover today The audit cycle Criteria and standards Structure, process and outcome / 29

What we are going to cover today The audit cycle Criteria and standards Structure, process and outcome An audit example / 29

What we are going to cover today The audit cycle Criteria and standards Structure, process and outcome An audit example Problems with audit / 29

What we are going to cover today The audit cycle Criteria and standards Structure, process and outcome An audit example Problems with audit How audit fits into ‘Clinical Governance’ / 29

A definition of audit “The systematic critical analysis of the quality of medical care, including the procedures used for diagnosis and treatment, the use of resources and the resulting outcome and quality of life for the patient” Working for Patients 1989 / 29

Reactions to audit Enthusiasm Cynicism Obligation Doubt Exhaustion / 29

Types of audit Managerial / 29

Types of audit Managerial Financial / 29

Types of audit Managerial Financial Clinical / 29

Types of audit Managerial Financial Clinical Multi-disciplinary / 29

Types of audit Managerial Financial Clinical Multi-disciplinary Cross boundary / 29

Types of audit Managerial Financial Clinical Multi-disciplinary Cross boundary Cohort based / 29

Types of audit Managerial Financial Clinical Multi-disciplinary Cross boundary Cohort based Comparative / 29

Types of audit Managerial Financial Clinical Multi-disciplinary Cross boundary Cohort based Comparative Significant event based / 29

Effective audit An educational activity / 29

Effective audit An educational activity Promotes understanding / 29

Effective audit An educational activity Promotes understanding Resource effective / 29

Effective audit An educational activity Promotes understanding Resource effective Raises standards / 29

Effective audit An educational activity Promotes understanding Resource effective Raises standards Promotes change / 29

Effective audit An educational activity Promotes understanding Resource effective Raises standards Promotes change Source of information / 29

Effective audit An educational activity Promotes understanding Resource effective Raises standards Promotes change Source of information Peer led / 29

Effective audit An educational activity Promotes understanding Resource effective Raises standards Promotes change Source of information Peer led Involves patients / 29

Problem or objective identified The audit cycle Problem or objective identified / 29

The audit cycle Problem or objective identified Criteria agreed and standards set / 29

The audit cycle Problem or objective identified Criteria agreed and standards set Audit (Data collected) / 29

The audit cycle Problem or objective identified Criteria agreed and standards set Audit (Data collected) Identify areas for improvement / 29

The audit cycle Problem or objective identified Criteria agreed and standards set Make necessary changes Audit (Data collected) Identify areas for improvement / 29

The audit cycle Problem or objective identified Criteria agreed and standards set Re-audit Make necessary changes Audit (Data collected) Identify areas for improvement / 29

The audit cycle Problem or objective identified Criteria agreed and standards set Re-audit Make necessary changes Audit (Data collected) Identify areas for improvement / 29

Another way of expressing the audit cycle / 29

Another way of expressing the audit cycle Determine which aspects of current work are to be considered / 29

Another way of expressing the audit cycle Determine which aspects of current work are to be considered Describe and measure present performance / 29

Another way of expressing the audit cycle Determine which aspects of current work are to be considered Describe and measure present performance Develop explicit standards / 29

Another way of expressing the audit cycle Determine which aspects of current work are to be considered Describe and measure present performance Develop explicit standards Decide what needs to be changed / 29

Another way of expressing the audit cycle Determine which aspects of current work are to be considered Describe and measure present performance Develop explicit standards Decide what needs to be changed Negotiate change / 29

Another way of expressing the audit cycle Determine which aspects of current work are to be considered Describe and measure present performance Develop explicit standards Decide what needs to be changed Negotiate change Mobilise resources for change / 29

Another way of expressing the audit cycle Determine which aspects of current work are to be considered Describe and measure present performance Develop explicit standards Decide what needs to be changed Negotiate change Mobilise resources for change Review and renew the process / 29

Criteria and standards / 29

Criteria and standards Criteria are those aspects of care that you wish to examine / 29

Criteria and standards Criteria are those aspects of care that you wish to examine Standards are the pre-stated or implicit levels of success that you wish to achieve / 29

Structure, process and outcome / 29

Structure, process and outcome Structure refers to resources you have available (including current knowledge, skills and attitudes) / 29

Structure, process and outcome Structure refers to resources you have available (including current knowledge, skills and attitudes) Process refers to what you actually do, e.g. a protocol / 29

Structure, process and outcome Structure refers to resources you have available (including current knowledge, skills and attitudes) Process refers to what you actually do, e.g. a protocol Outcome refers to the health benefits, cost effectiveness or patient satisfaction / 29

The great coffee audit / 29

The great coffee audit Problem The doctors feel that their coffee isn’t hot enough after slogging through morning surgery / 29

The great coffee audit Problem Criteria The doctors feel that their coffee isn’t hot enough after slogging through morning surgery The coffee shall be hot and satisfying to the hard pressed docs / 29

The great coffee audit Problem Criteria Standards The doctors feel that their coffee isn’t hot enough after slogging through morning surgery The coffee shall be hot and satisfying to the hard pressed docs The coffee shall be served at a temperature of 85-90C on 80% of occasions and there will be 90% satisfaction level expressed by the docs / 29

The great coffee audit Methods The junior receptionist shall check the temperature of the coffee daily for two weeks and circulate a questionnaire to the partners asking them to score a coffee satisfaction level between 1 and 10. The practice manager shall visit Tesco’s and interview the manager about the availability, costs, quality and sell-by dates of the coffee brands available / 29

The great coffee audit Review After a rather tense audit team meeting it was found that the coffee temperature fell below 37C on at least 33% of occasions and reached the standard on only 10% of occasions. The doctors scored the coffee at an average 3/10 and two expressed it undrinkable. The practice manager reported the results of her Tesco’s visit. / 29

The great coffee audit Change It was agreed to replace the aged coffee maker ( after agreeing suitable redundancy terms for the senior receptionist) with a shiny new machine from Argos. As an additional ‘quality initiative’, cream cakes would be served after surgery. The coffee contract would be switched from the corner shop to Tesco PLC Trust / 29

The great coffee audit For a further two weeks it was agreed to measure the coffee temperature and re-circulate the questionnaire. It was gratifying to find 100% correlation with agreed standards with the exception of one partner who didn’t like coffee anyway. Re-audit / 29

The great coffee audit Re-audit Future audit For a further two weeks it was agreed to measure the coffee temperature and re-circulate the questionnaire. It was gratifying to find 100% correlation with agreed standards with the exception of one partner who didn’t like coffee anyway. Cost implications of standard maintenance Cholesterol assays for partners / 29

Contents of an audit / 29

Contents of an audit Background / 29

Contents of an audit Background Literature review / 29

Contents of an audit Background Literature review Criteria and standards / 29

Contents of an audit Background Literature review Criteria and standards Methods or protocol / 29

Contents of an audit Background Literature review Criteria and standards Methods or protocol Results / 29

Contents of an audit Background Literature review Criteria and standards Methods or protocol Results Recommendations for change / 29

Contents of an audit Background Literature review Criteria and standards Methods or protocol Results Recommendations for change Recommendations for further audit / 29

Why audit / 29

Why audit Useful clinically / 29

Why audit Useful clinically Encourages teamwork / 29

Why audit Useful clinically Encourages teamwork Improves patient care / 29

Why audit Useful clinically Encourages teamwork Improves patient care Financial benefits (sometimes!) / 29

Why audit Useful clinically Encourages teamwork Improves patient care Financial benefits (sometimes!) Becoming contracturally an obligation with the arrival of PCG’s / 29

Problems with audit / 29

Problems with audit Audit and research / 29

Problems with audit Audit and research Statistical verification / 29

Problems with audit Audit and research Statistical verification Outcome measures and proxies / 29

Problems with audit Audit and research Statistical verification Outcome measures and proxies Protocols / 29

Problems with audit Audit and research Statistical verification Outcome measures and proxies Protocols Closing the loop - introducing change and re-audit / 29

A word about clinical governance / 29

A word about clinical governance “A framework through which NHS organisations are accountable for continuously improving the quality of their services and safeguarding high standards by creating an environment in which excellence in clinical care will flourish” A First Class Service 1998 / 29

‘Clinical Governance’ / 29

‘Clinical Governance’ / 29

‘Clinical Governance’ Research / 29

‘Clinical Governance’ Research Audit / 29

‘Clinical Governance’ Research Evidence based medicine Audit / 29

‘Clinical Governance’ Research Evidence based medicine Audit Dissemination of guidelines / 29

‘Clinical Governance’ Practice development plans Research Evidence based medicine Audit Dissemination of guidelines / 29

‘Clinical Governance’ Practice development plans Research Evidence based medicine Audit Personal development plans Dissemination of guidelines / 29

‘Clinical Governance’ Practice development plans Research Postgraduate medical education Evidence based medicine Audit Personal development plans Dissemination of guidelines / 29

‘Clinical Governance’ Practice accreditation Practice development plans Research Postgraduate medical education Evidence based medicine Audit Personal development plans Dissemination of guidelines / 29

‘Clinical Governance’ Practice accreditation Practice development plans Research Postgraduate medical education Special interest groups Evidence based medicine Audit Personal development plans Dissemination of guidelines / 29

‘Clinical Governance’ Accountability / 29

Clinical Audit: Tools and Techniques Helen Betts Head of School Chair of CHIRAD / 29

What is Audit? A systematic and critical appraisal of the planning, delivery and evaluation of service/s in terms of efficiency, effectiveness and quality, within given resources. / 29

Research is concerned with discovering the right thing to do; audit with ensuring that it is done right. / 29

Research or Audit into Nutrition? Determination of the population’s consumption of fatty acids identification of actions to reduce fatty acid levels in local population investigation of the interaction between the effects of fatty acid and obesity implementation of actions to reduce coronary heart disease quantification of the level of fatty acid in prepared foods communication exercise to inform “at risk” patients of beneficial lifestyle changes R A R/A / 29

“Clinical audit involves systematically looking at the procedures used for diagnosis, care and treatment, examining how associated resources are used and investigating the effect care has on the outcomes and quality of life for the patient”. Department of Health Clinical Audit: Meeting and Improving Standards in Healthcare (1993). / 29

Care is audited against defined standards derived from research findings, professional expertise and information about patient needs and expectations. / 29

In concurrent audit, care is evaluated at the time it is taking place In concurrent audit, care is evaluated at the time it is taking place. In retrospective audit, care is evaluated after it has been completed. / 29

Reliability refers to the ability of an instrument to measure the area of interest consistently, in the same way across time and with different assessors. Validity refers to the ability of an instrument to measure what it is intended to measure. / 29

Audits of the quality of care are normally undertaken through a process of peer review: the review of a professional’s practice by someone of the same profession, against professionally defined standards. / 29

The main methods used in audit of the quality of care are: Direct observation Checklists Documentation audit Questionnaires Interviews Case review / 29

You are a general practitioner organising an audit of the home care for cardiac rehabilitation patients. List all the professions who contribute to this care, including those from other organisations who input to the holistic programme of home support. How could you receive their observations? / 29

Items that would indicate clinical audit is developing successfully: It is undertaken by multi-professional healthcare teams it is focused on the patient it develops a culture of continuing evaluation and improvement of clinical effectiveness focusing on patient outcomes / 29

Benefits for professionals from a commitment to quality assurance: uphold professional/service standards increased job satisfaction opportunity for continual improvement fewer dissatisfied patients recognition/valuing of achievements productive use of time/effort acquisition of new skills/experience / 29

Clinical Governance & Audit Facilitator Clinical Audit Julie Bone Clinical Governance & Audit Facilitator Telford & Wrekin Primary Care Trust / 29

Introduction What is clinical audit? Audit versus research The audit cycle Five stages of clinical audit Summary / 29

What is clinical audit? A quality improvement process that seeks to improve patient care and outcomes through systematic review of care against explicit criteria and the implementation of change. / 29

What is clinical audit? Aspects of the structure, processes, and outcomes of care are selected and systematically evaluated against explicit criteria. Where indicated, changes are implemented at an individual, team, or service level and further monitoring is used to confirm improvement in healthcare delivery. NICE. Principles for Best Practice in Clinical Audit. Oxford, Radcliffe Medical Press, 2002. This definition is endorsed by the National Institute for Clinical Excellence. / 29

clinical governance and clinical effectiveness What is clinical audit? Clinical audit is at the heart of clinical governance and clinical effectiveness It has been built on a long history of doctors, nurses and other healthcare professionals reviewing case notes and seeking ways to serve their patients better. Audit has evolved from an informal quality of care monitoring back in pre NHS days before 1948 To formal quality studies during the 1950’s-1960’s when systems were first implemented To the introduction of medical audit in 1989 and Multi-professional clinical audit in 1993 1997 The new NHS introduced clinical governance. Now clinical audit sits within the clinical and cost effectiveness domain of the standards for better health produced by the DOH July 2004. / 29

Audit:- are we doing the best thing in the best way? Measures current practice against specific standards Never experimental Uses data in existence by virtue of practice May require ethical approval Aims to improve delivery of patient care Audit helps us to answer the question are we doing the best thing in the best way It uses research evidence to assess existing practice It does not involve experiments It uses data that already exists Sometimes ethical approval is required And ultimately it aims to improve patient care / 29

Research:- What is the best thing to do/the best way to do it? Provides sound basis for clinical audit Involves experimental trials Uses detailed and often sophisticated data collection Needs ethical approval and registration Aims to add to body of scientific knowledge / 29

The audit cycle Clinical audit can be described as a cycle or a spiral Within the cycle there are stages that follow a systematic process of establishing best practice, measuring care against criteria, taking action to improve care, and monitoring to sustain improvement. The spiral suggests that as the process continues, each cycle aspires to a higher level of quality. / 29

Five stages of clinical audit / 29

Stage 1: Preparing for audit Involving users Selecting a topic Defining the purpose Planning / 29

Involving users The focus of any audit project must be those receiving care. Users can be genuine collaborators, rather than merely sources of data (Balogh et al., 1995). The concerns of users can be identified from various sources, including: letters containing comments or complaints critical incident reports individual patients’ stories or feedback from focus groups direct observation of care direct conversations. Involve a user in your audit project team Where users are involved in this way, careful thought needs to be given to issues of access, preparation and support (Kelson, 1998). / 29

Selecting a topic There seems little point in trying to audit a rare condition, with a cheap intervention with a fairly superficial outcome. Which topic? Selecting a topic for audit this is the starting point for many quality improvement initiatives. It requires careful thought and planning, because any clinical audit project needs a significant investment of resources. There seems little point in trying to audit a rare condition, with a cheap intervention with a fairly superficial outcome Do you choose your favourite topic. / 29

Selecting a topic High cost/volume/risk Area of local clinical concern Potential for improvement National priority Organisational priority Evidence based When trying to prioritise audit topics questions to ask include Is the topic concerned of high cost, volume, or risk to staff or users? Is there evidence of a serious quality problem, for example patient complaints or high complication rates? Is the problem concerned amenable to change? Is there potential for involvement in a national audit project or pertinent to national policy initiatives? Is the topic a priority for the organisation? Is good evidence available to inform standards, for example systematic reviews or national clinical guidelines? / 29

Defining the purpose The following series of verbs may be useful in defining the aims of an audit (Buttery, 1998): to improve to enhance to increase to change to ensure A project without clear objectives cannot achieve anything: a clear sense of purpose must be established before appropriate methods for audit can be considered. Once the topic for a clinical audit project has been selected, therefore, the purpose of the project must be defined, so that a suitable audit method can be chosen. The following series of verbs may be useful in defining the aims of an audit (Buttery, 1998): / 29

Planning Involve ALL the right people Time and resources Access the evidence Methodology Pilot Report and Action Re-audit Planning is Paramount During the planning stage of an audit, it is important to consider the mechanisms for project management. The audit methods, including the aims and objectives, criteria and target levels of performance, data requirements, data collection instrument, and agreed terms, these should all be documented. / 29

Stage 2: Selecting criteria Defining criteria Sources of evidence Appraising the evidence / 29

Defining criteria Criteria: are explicit statements that define what is being measured represent elements of care that can be measured objectively. Definition of criteria Different professional groups have used different definitions of ‘criteria Within clinical audit, criteria are used to assess the quality of care provided by an individual, a team, or an organisation. This can include assessment of the process and/or outcome of care. The choice depends on the topic and objectives of the audit. Explicit rather than implicit criteria should be preferred. They should relate to important aspects of care and be measurable. / 29

Sources of evidence NeLH /Cochrane/WISH NICE Official Websites NSFs Local or regional guidelines/policy Royal College or Professional Body Recognised journals Systematic methods should be used to derive criteria from evidence. These include methods for deriving criteria from good-quality guidelines or from reviews of the evidence. This is often the most difficult & critical part. This can be an exasperating time, but it does get easier with practice. Developing such criteria can be time-consuming and requires considerable expertise. An alternative is to use criteria developed by people who are trained in the processes of evaluating evidence from the literature and grading criteria by strength of evidence. Provided that research evidence confirms that clinical care processes have an influence on outcome, measurement of the process of care is generally more sensitive and provides a direct measure of the quality of care. Measurement of outcome can be used to identify problems in care, provided outcomes are clear, influenced by process, and occur within a short period. Adjustment for case mix is generally required for comparing the outcomes of different providers. If the criteria incorporate, or are based on, the views of professionals or other groups, formal consensus methods are preferable. There is insufficient evidence to determine whether it is necessary to set target levels of performance in audit. However, reference to levels achieved in audits undertaken by other professionals is useful. In some audits, benchmarking techniques could help participants in audit to avoid setting unnecessarily low or unrealistically high target levels of performance. Can develop own standards Important to obtain everyone's agreement over the standards selected. / 29

Appraising the evidence Aim /objectives Methodology Results /conclusions Applicable to your patient group Bias/ causes for concern Evidence needs to be evaluated to find out if it is valid, reliable and important Don’t just accept everything that you read. / 29

Stage 3: Measuring level of performance Planning data collection Methods of data collection Handling data / 29

Planning data collection Need to establish the user group to be included, with any exceptions noted the healthcare professionals involved in the users’ care the time period over which the criteria apply. To make sure that the data collected are precise, and that only essential data are collected, certain details about what is to be audited must be established from the outset. the user group to be included, with any exceptions noted the healthcare professionals involved in the users’ care the time period over which the criteria apply. Examples All children under 16 years diagnosed with asthma and registered with the primary healthcare team All people with multiple sclerosis in a Health Authority area All women receiving treatment for breast cancer in England and Wales / 29

Methods of data collection Computer stored data Case notes/Medical Records Surveys Questionnaires Interviews Focus Groups Prospective recording of specific data Do not try and collect too many items – focus on the things that are important and that you are wanting to improve; KEEP IT SIMPLE! Do not try and run before you can walk. Do a pilot and analyse. KEEP IT SHORT A small sample of carefully collected data will usually tell more than a large amount of poor quality data. How will this be done? (computer software or manually) Compare performance against the criteria Keep focused on the objective of the audit Keep it simple / 29

Handling data Health service professionals must be aware of the ethical implications of and their responsibilities under the Data Protection Act (1998) when collecting data and presenting results. Confidentiality applies at all times Caldicott Anthony Rathbone Information Governance Data protection policy / 29

Stage 4: Making improvements Identifying barriers to change Implementing change External relationships / 29

Identifying barriers to change Fear of change Lack of understanding Low morale Poor communication Culture Pushing too hard Consensus not gained Use writing signature example Pushing hand example Time Lack of confidence/interest/motivation Education and training Staffing Management support Feeling out of control Guidelines standards / 29

Implementing Change Discuss the results with those likely to be affected Agree an Action Plan Clearly define – who is doing what Check progress Produce report and disseminate Share findings and changes in practice A systematic approach to implementation appears to be more effective. Such an approach includes the identification of local barriers to change, the support of teamwork, and the use of a variety of specific methods. Agreement reduces barriers people are more willing to change practice if they have been involved from the beginning in agreeing to what ought to be done. Recommendations should be soild Add in Who – Who will change affect – are they involved – if not why? What – What needs to change – what resources are needed- What is expected outcome How- Clear method- communication – motivated staff When- precise timetable- clear deadlines Think SMART Specicific Measruable Acheivable/Appropriate Timetabled Be patient Be flexible Remember what you are trying to achieve / 29

Stage 5: Sustaining improvement Monitoring and evaluation Re-audit Maintaining and reinforcing improvement / 29

Monitoring and evaluation Although improving performance is the primary goal of audit, sustaining that improvement is also essential. Indeed, any systematic approach to changing professional practice should include plans to: monitor and evaluate the change maintain and reinforce the change (NHS Centre for Review and Dissemination, 1999). Re-audit Ongoing monitoring Clinical performance indicators Adverse incidents Significant events audit / 29

Re-audit Close the loop Review evidence Measure effectiveness Decide how often to re-audit Re-audit to confirm improvement new different treatments, options Have the changes worked – made a difference Use the same process and method Leave enough time for your changes to have an impact / 29

Maintaining and reinforcing improvement Common factors: reinforcing or motivating factors built in by the management to support the continual cycle of quality improvement integration of audit into the organisation’s wider quality improvement systems strong leadership. Maintaining and reinforcing improvement over time is a complex process. In UK projects in which improvements have been sustained, some common factors have been identified (Dunning et al., 1999), including: / 29

Summary Defined clinical audit Compared audit and research The audit cycle. Five stages of clinical audit / 29

Celebrate Share learning Publicise results Give credit where credit is due! / 29