2. Competency of the audit programme & transition to ISO 15189 Professor Ian Sharp MS Head of Quality Email: ianrsharp@hotmail.com

3. The presentation will include Some thoughts on auditing and the transition to IS0 15189 Answer some specific questions Likely to lead to further discussion later in the day 2Competency of the audit programme & transition to ISO 15189

4. Answer some questions Checklists or not? Consistency during staff training? Level of detail in findings? What is the Holistic approach? Scope – Is it ok to have non-accredited tests, due to e.g. no EQA scheme? 3Competency of the audit programme & transition to ISO 15189

5. Transition to ISO 15189 CPA standards were reviewed to ensure they were ‘equivalent’ to ISO 15189 Therefore, there are not that many differences between the two standards The main difference is in the way UKAS will assess compliance Competency of the audit programme & transition to ISO 15189 4

6. ISO 15189 has greater emphasis on Control of external documents Monitoring of external services and supplies Calibration traceability Assay verification and validation Uncertainty of measurement - specifically related to quantitative results and qualitative results if there is a measurement step in its production Proficiency testing (EQA) Management of Information Management Systems Staff suggestions The end to end process Competency of the audit programme & transition to ISO 15189 5

7. Why is ISO 15189 more demanding than CPA? When you read the requirements they are very similar but CPA accredits the laboratory More interested in overall competency of the laboratory UKAS accredits the test Assess overall competency of the laboratory Assess competency to perform individual tests Each test is added to the scope of accreditation UKAS require the whole scope of accreditation to be audited UKAS keen to see that the whole QMS is audited 6Competency of the audit programme & transition to ISO 15189

8. Auditing to ISO 15189 Do you have to change your approach to how you audit? NO The principles of audit have not changed Do you throw away all your old checklists and start again? NO I have always taught that you are auditing for yourselves not for CPA The same applies to UKAS Therefore, if your audits were fit for purpose before they remain fit for purpose Don’t audit against the standards Audit against your processes Your processes haven’t changed significantly 7Competency of the audit programme & transition to ISO 15189

9. Auditing the entire scope Horizontal audits cover the whole scope as they cross through the whole system It would be impossible to perform vertical audits on every assay in the laboratory’s repertoire Assays fall within groups Same analyser Same process Consider the assay groups Make sure you cover each group within your audit schedule over a two year period 8Competency of the audit programme & transition to ISO 15189

10. Providing evidence that all requirements are covered Even though checklists should be based on your processes rather than the standards, it is good practice to reference the standard each question is applicable to in the checklist “Does the department hold appropriate QSOPs / local SOPs on equipment? (CPA D1.2 & ISO 9001:2000 7.5 &7.6)” Produce a table of CPA and ISO 165189 standards You may be able to use the table you prepared for your gap analysis 9Competency of the audit programme & transition to ISO 15189 ISO 15189CPAChecklist 5.3.1.1D1.2QW0026, Q1 4.14.4Not ApplicableNeed to include

11. Assumptions Attendees are working in laboratories which operate mature quality management systems Laboratories have been through several cycles of CPA assessments Laboratories have had audit programmes in place for many years Therefore, the system should be very mature 10Competency of the audit programme & transition to ISO 15189

12. So what’s the problem? We should now be very good at auditing and managing audit programmes Are we just looking for further improvement? I believe we are not as good as we should be / could be / must be The transition to ISO 15189 is a good time to review our audit practice 11Competency of the audit programme & transition to ISO 15189

13. Why aren’t audits as good as they might be? CPA have never really challenged the quality of auditing CPA are content for labs to use the checklist published on their website which was designed to be used by CPA assessors I estimate that over 90% of labs I have visited use the CPA checklist Lack of resources put into quality management Time Training If there is no pressure from the assessors to improve then why should we? Lack of available external audit training specifically for pathology laboratories 12Competency of the audit programme & transition to ISO 15189

14. Common situation Many labs are still auditing the same way they did five or even ten years ago With pressure on work load, they are just grateful to be able to get audits performed Little opportunity to reflect and develop audit as an improvement tool Audit should be about effectiveness not just ensuring compliance with standards 13Competency of the audit programme & transition to ISO 15189

15. Who typically does audits? Quality Managers are responsible for managing the audit programme and perform some audits Quality Managers have to rely on other pathology staff to perform audits in order to complete the annual programme This has many benefits it improves the quality culture and ensures that all staff are aware of their quality responsibilities Auditors are not seen as policemen Auditors understand the important issues The disadvantage is that staff seldom prioritise audits over bench work Staff need training but will always lack the expertise of a full time quality professional 14Competency of the audit programme & transition to ISO 15189

16. Training – in house Perfectly acceptable Ensures a consistent approach Lack of external input can result in no development of new ideas 15Competency of the audit programme & transition to ISO 15189

17. Training – externally provided Can be expensive Usually off-site Usually not pathology specific Some training, e.g., ISO 9001 lead assessor courses, is excellent Followed up by in-house (cascade) training Can have external training provided internally 16Competency of the audit programme & transition to ISO 15189

18. Ideal training Focused on pathology Covers all principles of audit Relates the principles to the overarching QMS and the principles of quality assurance Provides realistic workshop activities relating to the pathology labortatory This is only the start 17Competency of the audit programme & transition to ISO 15189

19. Ideal training The most important thing is for trainees to perform a number of audits with a competent auditor and have their competency assessed using objective criteria: Good interpersonal skills Good communication / questioning skills Good report writing skills Good analytical skills – able to interpret findings Enthusiasm Understand the importance of auditing for effectiveness of the system and not just compliance 18Competency of the audit programme & transition to ISO 15189

20. Checklists or not? 19Competency of the audit programme & transition to ISO 15189

21. Checklists – No There are no rules about use of checklists I prefer to audit without a checklist I find a checklist inhibiting I prefer to follow the process as it flows I always seem to be trying to find where to record my findings Questions never seem to be in the right order However, I am very experienced 20Competency of the audit programme & transition to ISO 15189

22. Checklists – Yes Not all auditors have the same level of experience We need to be sure that all relevant questions are asked We need to obtain assurance that all requirements of the standards are assessed We need to ensure a consistent approach is taken We must ensure that the checklists are easy to complete and flow properly 21Competency of the audit programme & transition to ISO 15189

23. Performing the audit It doesn’t matter how good the auditor is, if he or she is using a poor checklist, you are unlikely to achieve the best audits 22Competency of the audit programme & transition to ISO 15189 Specimen Reception 5.1 Are there procedures for sample receipt and booking in? 5.2 Are there procedures for handling urgent samples? 5.3 Are there criteria for the rejection of specimens and recording of rejected specimens? 5.4 Are procedures in place to ensure staff safety?

24. Another example Some checklists are formatted in away which does not encourage or require details of observations to be documented Questions like “Were all reagents dated and suitable for use?” Answer “Yes” Record the name and lot number of reagents you check. What was the evidence that they were suitable for use? The audit and record within the checklist should provide evidence of full traceability I should be able to retrace you audit by following your completed checklist 23Competency of the audit programme & transition to ISO 15189

25. Examples of vertical audit questions 24Competency of the audit programme & transition to ISO 15189

26. What is a holistic approach? It is not about the individual audit but the audit programme overall Audit must be a review of effectiveness, not just compliance Audit should go beyond the requirements of the standards Site security Adverse incident management Confidentiality of patient information Links to H&S Part of the governance framework of the organisation Quality and governance are two sides of the same coin Quality underpins governance 25Competency of the audit programme & transition to ISO 15189

27. A good approach to auditing Regularly review and update checklists Regularly review and update approach to auditing Take a risk based approach Two yearly programme Focus on in depth vertical audits to cover all key aspects of the diagnostic pathway Plus horizontal management type audits not covered by vertical audits If vertical audits identify issues, we consider factoring specific horizontal audits into the programme Overall, this results in less time auditing to obtain satisfactory assurance 26Competency of the audit programme & transition to ISO 15189

28. Review of effectiveness ISO 15189 4.10 Corrective action The laboratory shall take corrective action to eliminate the cause(s) of nonconformities. Corrective actions shall be appropriate to the effects of the nonconformities encountered. The laboratory shall have a documented procedure for: f) reviewing the effectiveness of the corrective action taken 27Competency of the audit programme & transition to ISO 15189

29. How to review effectiveness Re-audit every nonconformance after is has been closed This might be a lot of additional work How long afterwards? What would be the measure of effectiveness? Do we leave the audit open until the review has taken place? Take a more holistic approach We should already be producing regular quality reports Reports should include trends related to nonconformities identified during audits This is a good assessment of effectiveness of the audit programme 28Competency of the audit programme & transition to ISO 15189

30. Trend report showing analysis by fault type Competency of the audit programme & transition to ISO 15189

31. Analysis of CPA F3 Findings Competency of the audit programme & transition to ISO 15189 30

32. Scope – it’s OK to have non-accredited tests due to e.g. no EQA scheme Yes it is OK to have non-accredited tests May be an issue for the commissioners or other organisations you have a contract with Contracts usually require the laboratory to be accredited As CPA did not have a scope then this was easy to fulfil May need to liaise with users of your service so that they understand the changes If use UKAS logo on your reports, any test outside the scope of accreditation must have a statement to make this clear If the test outside your scope of accreditation is the only one being reported then you are not permitted to use the UKAS logo on the report 31Competency of the audit programme & transition to ISO 15189

33. Lack of an EQA scheme does not necessarily prevent a test being accredited ISO 15189 5.6.3.2 Alternative approaches Whenever an interlaboratory comparison is not available, the laboratory shall develop other approaches and provide objective evidence for determining the acceptability of examination results. Whenever possible, this mechanism shall utilize appropriate materials. 32Competency of the audit programme & transition to ISO 15189

34. No EQA scheme NOTE Examples of such materials may include: — certified reference materials; — samples previously examined; — material from cell or tissue repositories; — exchange of samples with other laboratories; — control materials that are tested daily in interlaboratory comparison programmes. Therefore, non availability of an EQA scheme is not reason to exclude a test from your scope of accreditation 33Competency of the audit programme & transition to ISO 15189

35. Summary Audits are the most valuable tool in the Quality Manager’s arsenal Audits ensure compliance with standards Audits ensure effectiveness of the system Audits provide opportunities for improvement This requires provision of sufficient resources to ensure well trained, competent auditors sufficient time is allocated to auditing The checklists and the programme are regularly updated to ensure they are fit for purpose We make take the opportunity while revising our systems for the transition to ISO 15189 to improve the way we audit to achieve the greatest benefit 34Competency of the audit programme & transition to ISO 15189

36. 35Competency of the audit programme & transition to ISO 15189 The competency of the audit programme is as important as the competency of the auditors