CAESAR Computer Assisted Evaluation – Screening And Reporting The Cervical Screening Wales Focal Point Project Dave Nuttall Network Manager and Scientific Advisor PHW Staff Conference – Oct 15th, 2009

Acknowledgements The staff at Llandudno General Hospital, Ysbyty Glan Clwyd and Ysbyty Maelor, Wrexham Bryan Rose Helen Beer and the CSW information team David Addington-Hall and Karen Winder for advice during the scoping phase Drs Nick Dallimore, Sally Williams and Tom Hockey Wilma Anderson and Luke Nottage of

Background Rapidly changing programme HPV Vaccination New technology (HPV testing, Computer Assisted Screening!!) Modernisation (ScHARR1 and NHS CSP2 guidance, WAG pathology modernisation initiatives3) Option appraisal: Assessment of a seven-day turn-around for the reporting of cervical smear results. School of Health and Related Research, University of Sheffield. Jan 2006 NHS CSP: Achieving a 14 day turnaround for results by 2010. Advice to the NHS. April 2008. Welsh Assembly Government. Getting Results: A Strategy for Diagnostic Services in Wales, 2004. Diagnostic Services Programme: The Future Delivery of Pathology Services in Wales, 2007

Background Pressures on the Laboratory Service Staff leaving Staff recruitment and training Fewer cases being screened – can we maintain our skills? Challenges around service delivery – CAS is expensive technology and if beneficial, will require a laboratory service that is structured to take full advantage of it

Background What is Rapid Quality Assurance screening? Designed to improve the quality of primary screening in the cytology lab Can be either preview or review Also used to detect false negative and positive cases Is therefore a requirement in calculating individual and laboratory sensitivity and specificity

Why CAS? Since the introduction of LBC, screener sensitivity has improved. Calculation of sensitivities is becoming increasingly invalid as the manual QA methods have become unfit for this purpose Anecdotally, Rapid Preview/Review isn't particularly well liked by staff

Some staff perform better at QA screening than others – leads to inconsistencies which will affect the calculation of individual and laboratory sensitivities Automated screening might therefore provide a more sensitive QA approach that is also more consistent in its application

Introducing the Becton Dickenson Focal Point Slide Profiler

Focal Point GS – An Overview The BD FocalPoint™ slide profiler is an automated imaging system designed to assist in the primary screening of SurePath™ and conventional Papanicolaou cervical cytology slides. Designed to detect slides with evidence of squamous carcinoma and adenocarcinoma and their usual precursor conditions. Up to 300 individual features are evaluated by the BD FocalPoint™ by using both morphometric and densitometric algorithms. The BD FocalPoint™ identifies up to 25% of successfully processed slides as requiring No Further Review (NFR). When used in conjunction with a BD FocalPoint™ Guided Screener Review Station the 10 highest scoring Fields Of View (FOV) from each slide are presented to the screener for review via an automated stage fitted to a standard laboratory microscope

Loading FocalPoint LOAD & LEAVE Add barcode labelled SurePath™ slides to a slide tray (8 slides per tray) LOAD & LEAVE Load slide trays into input hopper of BD FocalPoint™ slide profiler (max 36 slide trays at any one time = 288 slides) Racks can be loaded and unloaded at any time Minimum 120 slides per run There is no maximum number of slides per run

BD FocalPoint™ Slide Profiler processing steps 20x Scan (High Resolution) Each FOV scanned twice A high resolution image is acquired for each of the 1000 FOV’s FOV processors - segment all objects in each acquired FOV image (segmentation = separation of meaningful objects from background) Objects classified as single cells, groups and thick groups A single cell, group and thick group score is given for each FOV The FOV scores are accumulated and integrated into a final slide score between 0 and 1

BD FocalPoint™ GS Imaging Workflow: GS Review Station Database BD FocalPoint™ Slide Profiler

BD FocalPoint™ GS Imaging data transfer: GS Review Station Data transfer via Removable hard-drive or VPN connection Data transfer Server PC GS Review Station

Principal Hypothesis The Focal Point GS slide profiler technology is an acceptable replacement for the currently established processes of quality assurance in primary screening

The Project Aims Assessment of the Focal Point Slide Profiler ‘Quintile Ranking’ facility against diagnostically confirmed sample abnormality profiles Assessment of the Focal Point Slide Profiler NFR (No Further Review) category as a reliable indicator of negative samples

The Project Aims Evaluate Focal Point GS as a means of assessing sample quality establishing baseline parameters for assessing adequacy of samples by creating a reproducible parameter for the assessment of sample taker performance Evaluate Focal Point GS across an integrated laboratory network assessing sample identification, tracking and transportation processes, workload allocation patterns and logistics and user friendliness Note: Get project aims from study protocol and then present one by one. Title slide eg Project Aims – 1 Then discuss reasoning on subsequent slides

The Project Aims Monitoring screener performance using the Focal Point Slide Profiler This technology has the capability to monitor operator/observer functions such as screener opinion and action, time taken to perform screen. These data items - gathered using the Location Guided Screener workstation (LGS) can be compared against primary screening and/or final outcome in order to provide screener performance indicators.

Examine Operational Issues Identify sample identification issues e.g. bar codes and maintaining “chain of custody” across a network How does the technology deal with inadequate samples – does this compare with current adequacy criteria? Does the technology compare with manual techniques in terms of operator time?

PROCESS

Process Samples received and processed at laboratory as normal Despatched to Medical Solutions for FP scanning Returned with data disc for Location Guided Screen (LGS) assessment in laboratory Automated Arm) Manually screened independently of the LGS evaluation (Manual Arm) Results correlated before final report issued

The ‘Automated Arm’ Consisted of screener evaluation of 10 fields of view (FOV) on each sample Difficult cases were referred to a checker who also examined the 10 FOV Findings were recorded on the proforma illustrated overleaf Outcome was recorded on LIMS (TelePath)

The ‘Manual’ Arm Slides were primary screened and QA screened as usual following standard laboratory protocols Codes and reports were recorded on TelePath Automated arm report codes were recorded FPQ or FPQQ codes were recorded depending on whether or not a discrepancy was noted between the two arms Cases were authorised in the normal manner CSW retrieved FP/LGS data as part of weekly downloads

CERVICAL SCREENING WALES FOCAL POINT PROJECT - PROCESS MAP AUTOMATED ARM COURIER SAMPLE TO LAB PROCESS SLIDES TO NOTTINGHAM SLIDES ARE READ ON FOCAL POINT INDIVIDUALS CREENERS OPINION OF 10 F.O.V. HARD DRIVE + PRINT OUT Process Review data RESULTS AND SLIDES RETURNED TO LAB CODE RESULTS ON TELEPATH SLIDE WIZARD NETWORK DATA TRANSFER AVAILABLE DESIGNATED PROCEDURE CODES FULL SCREEN Via Manual Arm QC DOUBLE SCREEN/ RAPID REVIEW RESULTS TO CSW PROJECT EVALUATION MANUAL ARM

Data and Morphology Anomaly Checking Automated:manual arm mismatches were highlighted by the CSW Information department as and when they arose Any resultant slide morphology issues were addressed in team assessment sessions at Med Sol in Nottingham and resolved by: - LGS/multi-header microscope sessions involving representatives from Labs / CSW/ Med Sol

RESULTS

Total Slides Processed

QA beyond the Lab - Focal Point GS as a means of assessing sample quality FP reports on the Squamous and Endocervical components detected in the sample Potentially of use to the screening programme as an indicator of smear taker proficiency – but emotive!!! Wales monitors TZ reporting – however generally a wide spread – between 80% - 96% reported by NW labs The study has shown that the FP reported endocervical component detection rates in NW labs only varied between 79% and 85% Possibly screener interpretation – CAS may provide a more uniform assessment Further work is ongoing in this area

Focal Point GS performance across an integrated laboratory network VPN data transfer means that scanned slide information is available anywhere within the combined laboratory service Economies of scale are definite considerations when planning the introduction of this technology – possible consideration for LBC procurement Therefore, in this operational scenario – probably more efficient to combine the sample processing and CAS on one processing site.

Teaching and Case review using the Focal Point GS LGS can mark and record interesting features within a slide and direct the user to these areas on the slide on demand With a system configured using VPN these slides can be viewed anywhere on the network Considerable potential for teaching and MDT use when combined with a multi-head microscope/projector

Assessment of the Focal Point Slide Profiler ‘Quintile Ranking’ facility against diagnostically confirmed sample abnormality profiles

Sorting & Ranking (Illustration) The FocalPoint™ SORTS and RANKS slides between the values of 0 and 1 : based on the likelihood of abnormality being present. 0 is negative : 1 is abnormal x < 25% Archive > 75% Review > 15% QC Negative Abnormal Q5 Q4 Q3 Q2 Q 1

FP Quintile Distribution Note: this category Contains PR as Well as NFR samples FP Quintile Distribution

FP Quintile Distribution

Focal Point Scant Cellularity Category (FPSC) Compared to a Final Report of Inadequate Comparison of the inadequacy levels showed that the manual arm had a similar inadequate rate – not what was expected with Wales operating a 15000 cell threshold Interestingly the majority of cases did not correspond between the manual and automated arms

“NFR” – DOES IT WORK? Assessment of the Focal Point Slide Profiler NFR (No Further Review) category as a reliable indicator of negative samples

Samples Assigned for NFR Compared to Final Result

What was missed by NFR? 1. Cases from all labs, NFR negative, finally reported as abnormal 78 cases in a total of 2842 (Picked up by 10 screener, reviewer or checker) 2.74% of cases missed by NFR – False Negatives

What was missed by NFR? Statistically – Significant difference from 1. In over 50 000 cases manually rapid previewed by 4 Welsh Labs during Jan – September 2009 (Picked up by 10 screener, reviewer or checker) The overall False Negative Rate was 3.5% Statistically – Significant difference from 1. NFR is therefore at least as effective as Manual Rapid Pre-screen as a QA tool for primary screening of cervical samples.

Overall Sensitivity In total, the automated QC screening arm did not “identify” 12 HG cases (histologically confirmed) in a total of 134 reported Sensitivity (134-12)/134 = 91% Compare this to a quoted range of 75-95%* HG sensitivity for conventional Rapid Preview QA methods 202 LG cases were missed or consigned to NFR in 788 reported LG samples Sensitivity (788-202)/788 = 74.4% * See references and CSW data

But……….! On review, morphological abnormalities were present in the majority of 10 FOVs in the HG cases “missed “ by the automated arm. So potentially therefore, 91% is the minimum sensitivity rate that can be expected This is because the FocalPoint GS presented the abnormalities to the LGS user, but not identified by the user for a number of possible reasons This may be due to a number of reasons e.g. unfamiliarity and further analysis currently being performed on this data

In Summary……… Automated QC screen appears to be no less effective than rapid pre-screen Most (161/219) cases not identified by automated arm were borderline HPV cases – FP does not seem to recognise koilocytes! – especially with minimal nuclear abnormalities. Despite this finding, sensitivity of the automated QC screen appears to be appreciably greater in comparison to available data on manual QC methods.

User experience is a large factor in the efficiency of the screen A high proportion of the undetected abnormalities were shown to be present in the FOV’s on review – but not identified by the screener. Staff had only received 6-8months experience of the technology - further work is currently ongoing to see if screener performance improved with time/experience

Monitoring laboratory and screener performance using the Focal Point Slide Profiler

Individual Laboratory Process Review Results Is an indicator of the number of failed slide scans in a given FP run It reflects technical quality, bar code integrity etc. Note the low value for YGC, which has been mainly attributed to tape coverslipping

Screening times for 10 FOV Note the remarkable similarity of the mean screening times. Note also that NFR category accounts for 22% of samples - no FOVs, therefore potential productivity increase.

Manual QC Screening Times (includes data entry on LIMS)

OBSERVATIONS

Observations Thought that FP operates on TBS inadequacy threshold of 5- 6k cells, therefore we hypothesised that the manual arm would identify a correspondingly higher number of inadequate samples, however…………….. Comparison of the inadequacy levels showed that the manual arm had only a slightly higher inadequate rate – not as expected with Wales operating a 15000 cell threshold Interestingly the majority of cases did not correspond between the manual and automated arms – possibly indicative of the fact that the FPSC report is applied to cases that exhibit other criteria, such as:

Observations The slide cannot be focussed on correctly because of bubbles, too thick, too few cells in the centre of the prep. etc. It is therefore likely that the actual number of cases reported as FPSC for reasons related to cellular adequacy are lower than first thought Further work will be undertaken to determine threshold values of the criteria that make up the FP report “Insufficient Reference Cells” to determine if this could be compared with the manual arm report of “Inadequate due to scant cellularity”

Observations Tape coverslipping appreciably increases the technical quality of the scanned slide, thereby reducing the process review rate and decreasing the number of rejected slides. Selecting a large coverslip size on the tape cover slipper extended the FP scan time from approx. 11 slides/min on glass coverslips to 9 slides/min using tape coverslips. Reducing the tape size reduced this to 10 slides/min* Staff used to primary screening on a x20 objective may have initial difficulties in operating the LGS which only displays FOVs at x10 *Note: Updated versions of the FP GS allow faster scan rates

Observations Staff have varying ability to correctly interpret the 10 FOVs and like manual screening, are only as good as the experience they have – both as primary screeners and users of the LGS A small proportion of slides simply cannot be evaluated on 10 FOVs – these should be subjected to a full scan and further checking, if necessary Despite best efforts to blind manual and automated arms interpretational bias may have resulted as staff were aware that this was a study

Observations The study resulted in additional workload for staff which might also have affected interpretation Clearly defined protocols are mandatory for routine use Transport of slides and data via external couriers posed a few challenges! – eventually settled down

RECOMMENDATIONS

Recommendations A number of issues have yet to be fully evaluated, including: Completing comparisons between available manual QC data and project results Full evaluation of NFR – do we need to continue with manual QC for these cases? Subsequent effects on workload/staffing Implementation and logistical issues Full Cost/Benefit analysis

Recommendations Cost/benefit analysis Need to further evaluate the potential of the NFR category With demonstrated > 99% False Negative rate for HG disease in 22% of cases – potentially labour saving whilst improving quality! This is in the region of 1100 hours screener time in Wales with an annual workload of 200 000+ cases

CONCLUSIONS

Conclusions Analysis of the findings of this study are not yet complete. At this stage initial results indicate that the technology is suitable for routine use, with potential quality and service management benefits to the cervical screening programme in Wales

Recommendations for Further Investigation CSW AWMG have approved an extension of the project and in collaboration with Medical Solutions will be working towards full use of FP as a QA system. The network of laboratories will be extended to include a fourth laboratory from South Wales – at the Royal Gwent Hospital, Newport – project commenced July 2009 Define implemention plan Networking issues – the technology seems particularly suited to networks. Wales is currently exploring the introduction of managed networks in cytology Develop protocols Specifically issues include standardisation of manual rapid QA methods across labs to rapid pre-screen IT coding protocols require total standardisation – part of N Wales objectives. Data extraction will become more consistent

Recommendations for Further Investigation Implementation QA use initially – follow on project to implement FP as a replacement for QC screening and further assessment as a primary screening tool, especially in the light of the changing screening programme Move to full adoption subject to satisfactory outcome of project and final approval by AWMG Outcome will inform LBC procurement process

“Any sufficiently advanced technology is indistinguishable from magic “Any sufficiently advanced technology is indistinguishable from magic.” Arthur C. Clarke – The Lost Worlds of 2001 (1972)