Evaluations and Validations post Carter Keith Perry, Head of Evaluations Evaluations & Standards Laboratory BSMT, November 2006

Carter review of NHS Pathology Ways we currently assess Evaluations post-Carter Evaluations & managed introduction of new technology? Today’s presentation

Need for standardised methods Need for co-ordinated system for evaluation managed intro of new technology

Need for standardised methods Need for co-ordinated system for evaluation managed intro of new technology 2006

Little evidence of a strategic programme of investment for new & replacement technology Little automated ‘front end’ & sample tracking NHS too focussed on cost not benefit Difficult to cost pathology services Silos Fragmented pathology services Relatively small investment can make a difference Labs can influence expensive treatments Carter review of NHS Pathology What was found ?

Managed introduction of new technology - Benefits realisation - Competition Appropriate investment Work equipment harder Healthcare closer to home More PoC testing – accreditation & links with labs Common national framework Standardisation of test results CEP – PASA role Evaluations Carter review of NHS Pathology Recommendations

Ways we currently assess But does not cover: Ease of Use Monitoring of internal QCs External QA Use of combinations ie kits/equipment Comparative data Evaluation and validation CE Marking

Focus on device performance Comparative Independent and unbiased Provide informative reports Must be accurate and timely Relevant to standardisation of methods Encourage development Should influence managed introduction of new technology Ways we currently assess Evaluations

= combined antigen-antibody= immunometric = Class specific antibody capture = antiglobulin / indirect Biotest Anti- HIV TETRA ELISA Wellcozy me Anti-HIV *Earliest anti-HIV detection days Ortho Ab-capt. ELISA Vitros ECi anti-HIV 1/2 Murex HIV 1/2 VK84/85 IMx HIV1/2 III plus Murex HIV GE94/95 Vironostika HIV Uni-form II Ag/Ab Enzygnost HIV 1/2 plus Biotest Anti- HIV TETRA ELISA Clonesystems Detect-HIV v1 Innotest HIV-1/-2 Murex ICE HIV 1+2 AxSYM HIV 1/2 gO Abbott 3rd gen Plus Pasteur Genscreen Version 2 Access HIV 1/2 NEW Earliest HIV detection Enzygnost HIV Integral VIDAS HIV DUO Vironostika HIV Uniform II plus O GENSCREEN PLUS HIV Ag- Ab AxSYM HIV Ag/Ab Combo Murex HIV Ag/Ab combination Biotest HIV 1/2 recombinant Timing of detection of primary HIV following seroconversion Ways we currently assess

Molecular Assessments with CVN in collaboration with ESL to reassure users that their in-house assays give comparable results In-house real-time PCR assays Assays assessed to date (DNA viruses: HSV1 and 2; ADV; CMV) Viral targets chosen via CVN committee & members Labs submit SOPs Panel of 25/target, comprising dilutions of strong or tissue-culture grown material extracted using two methods (9), plus clinical specimens (6) sent out for assessment Next RSV, VZV, Norovirus, & Enterovirus Ways we currently assess Assessments

Any new equipment or kit Molecular extraction machines New media Ways we currently assess Quick Look/See

Hospital based study of IDI-MRSA (direct detection of MRSA in nasal swabs by a rapid molecular method ) bioMérieux Vitek Ways we currently assess Clinical Assessments

Validation is the confirmation by examination and provision of objective evidence that the particular requirements for a specific intended use are fulfilled (ISO 17025:2005) Validation is evaluation of the whole process & ability to achieve the correct result (not the kit or reagent in isolation) Ways we currently assess Validation

VALIDATION FILE Ways we currently assess

Ways we currently assess

‘Evaluation’ issues post-Carter National framework

‘Evaluation’ issues post-Carter The CEP-PASA Prioritisation Board meets 3 times per year to consider proposals Projects that are innovative, related to government priorities or purchasing exercises will gain higher scores Next closing date for proposals: 31 st January 2007 Next meeting: 14 th March 2007 website Prioritisation

Extent of NHS use Evaluation Adoption/purchase Time Evaluation project proposals Chair Head of CEP Business Planning Manager Head of Commissioning and Delivery Technical Advisor PASA / Hubs (2 reps) NICE (1 rep) NIIII (1 rep) HTA (1 rep) NPSA (1 rep) MHRA (1 rep) Prioritisation Board National Evaluations Programme for Microbiology Priorities to be aligned to customer requirements & peer-organisations Anyone can submit proposals ‘Evaluation’ issues post-Carter

‘Evaluation’ issues post-Carter High rate of introduction of new diagnostic tests & automation Many subject areas are not covered by evaluations Scope

Products to reduce HAIs Virology Bacteriology Parasitology SerologyMolecular Automation Identification HIV EIAs HCV EIAs HBsAg EIAs PoCTs Chlamydia NAATs Toxoplasma EIAs Extraction, Processors, Closed systems HSV Real-time PCRs Kiestra plate automation Syphilis EIAs Culture media ‘Evaluation’ issues post-Carter Scope

‘Evaluation’ issues post-Carter Evidence & processes Well designed studies Well defined procedures Challenging specimen panel Expert input Technical quality Comprehensive quality system Safeguards / checks in place Project management

‘Evaluation’ issues post-Carter

‘Evaluation’ issues post-Carter Recognised evaluation protocols e.g. for evaluations of lab automation Developed in consultation with experts / users Developed with reference to National Standard Methods Process for review and authorisation

‘Evaluation’ issues post-Carter National Evaluations Register Study title Status (proposal, preparation, technical, report in draft, completed) Expected Output (evaluation report, look/see, literature review, market review) Lead centre Contact name, telephone, Awareness

‘Evaluation’ issues post-Carter Evaluating relevance & utility is time consuming Products are often in use prior to evaluation report Matching evaluations and tender processes Timing

‘Evaluation’ issues post-Carter Mechanism The mechanism for where and how evaluations are done is still insufficiently defined Central front-door through CEP evaluation centres? Evaluation networks co-ordinated through CEP centres? - with expert working groups - professional groups - reference laboratories - clinical user groups

‘Evaluation’ issues post-Carter How large is the cost? How large is the benefit? Is introducing the test/device cost effective? Resources Integral to the work of CEP-PASA CEP has Health Economist in post Cost benefit

Government priority To tackle a specific issue (Chlamydia screening programme) Money made available Implementation of new technology preceded evaluation (used literature reviews) Multi-site evaluation took >2 years Evaluations & managed introduction of new technology? Chlamydia NAATs

Data collation, analysis, & report HPA-ESL Urine 25mls+ Routine Screen Source Labs (3) Retest positives Report 1 RoutineEvaluationSpecimen sourcingTransport Identify discordants HPA-ESL SDA 2b TMA 3b PCR 4b Aliquot, Label & Send Portsmouth Liverpool UCLH / Kings HPA-STBRL Add to TMA tube SDA 2a TMA 3a PCR 4a R-time PCR 5 SDA pouch Aim for 4 day turnaround Chlamydia NAATs evaluation Evaluations & managed introduction of new technology?

Responding to the need for a wide range of microbiological device evaluations How to prioritise Joining device performance with procurement Earlier assessment and usage of innovative technologies Improved access to available evaluation results eg National Evaluations Register Improved sharing of information Impact on lab autonomy to choose equipment/kits Funding of evaluations (eg PASA) Funding for implementation (capital & revenue) Challenges

All collaborators / advisors All in Evaluations & Standards Laboratory, including: - Valerie Bevan, Director ESL - Standards, Quality Control Reagents & Quality System Units - The Evaluations Team: Katrina Barlow Christine Burgess Johanna Curtis Laura Dean Galit Gonen Fu Li Thanks to….