1. CAESAR Computer Assisted Evaluation – Screening And Reporting The Cervical Screening Wales Focal Point Project
Dave Nuttall
Network Manager and Scientific Advisor
PHW Staff Conference – Oct 15th, 2009

2. Acknowledgements
The staff at Llandudno General Hospital, Ysbyty Glan Clwyd and Ysbyty Maelor, Wrexham
Bryan Rose
Helen Beer and the CSW information team
David Addington-Hall and Karen Winder for advice during the scoping phase
Drs Nick Dallimore, Sally Williams and Tom Hockey
Wilma Anderson and Luke Nottage of

3. Background Rapidly changing programme HPV Vaccination
New technology (HPV testing, Computer Assisted Screening!!)
Modernisation (ScHARR1 and NHS CSP2 guidance, WAG pathology modernisation initiatives3)
Option appraisal: Assessment of a seven-day turn-around for the reporting of cervical smear results. School of Health and Related Research,
University of Sheffield. Jan 2006
NHS CSP: Achieving a 14 day turnaround for results by Advice to the NHS. April 2008.
Welsh Assembly Government. Getting Results: A Strategy for Diagnostic Services in Wales, 2004.
Diagnostic Services Programme: The Future Delivery of Pathology Services in Wales, 2007

4. Background Pressures on the Laboratory Service Staff leaving
Staff recruitment and training
Fewer cases being screened – can we maintain our skills?
Challenges around service delivery – CAS is expensive technology and if beneficial, will require a laboratory service that is structured to take full advantage of it

5. Background What is Rapid Quality Assurance screening?
Designed to improve the quality of primary screening in the cytology lab
Can be either preview or review
Also used to detect false negative and positive cases
Is therefore a requirement in calculating individual and laboratory sensitivity and specificity

6. Why CAS?
Since the introduction of LBC, screener sensitivity has improved. Calculation of sensitivities is becoming increasingly invalid as the manual QA methods have become unfit for this purpose
Anecdotally, Rapid Preview/Review isn't particularly well liked by staff

7. Some staff perform better at QA screening than others – leads to inconsistencies which will affect the calculation of individual and laboratory sensitivities
Automated screening might therefore provide a more sensitive QA approach that is also more consistent in its application

8. Introducing the Becton Dickenson Focal Point Slide Profiler

9. Focal Point GS – An Overview
The BD FocalPoint™ slide profiler is an automated imaging system designed to assist in the primary screening of SurePath™ and conventional Papanicolaou cervical cytology slides.
Designed to detect slides with evidence of squamous carcinoma and adenocarcinoma and their usual precursor conditions.
Up to 300 individual features are evaluated by the BD FocalPoint™ by using both morphometric and densitometric algorithms.
The BD FocalPoint™ identifies up to 25% of successfully processed slides as requiring No Further Review (NFR).
When used in conjunction with a BD FocalPoint™ Guided Screener Review Station the 10 highest scoring Fields Of View (FOV) from each slide are presented to the screener for review via an automated stage fitted to a standard laboratory microscope

10. Loading FocalPoint LOAD & LEAVE
Add barcode labelled SurePath™ slides to a slide tray (8 slides per tray)
LOAD
&
LEAVE
Load slide trays into input hopper of BD FocalPoint™ slide profiler (max 36 slide trays at any one time = 288 slides)
Racks can be loaded and unloaded at any time
Minimum 120 slides per run
There is no maximum number of slides per run

11. BD FocalPoint™ Slide Profiler processing steps
20x Scan (High Resolution)
Each FOV scanned twice
A high resolution image is acquired for each of the 1000 FOV’s
FOV processors - segment all objects in each acquired FOV image (segmentation = separation of meaningful objects from background)
Objects classified as single cells, groups and thick groups
A single cell, group and thick group score is given for each FOV
The FOV scores are accumulated and integrated into a final slide score between 0 and 1

12. BD FocalPoint™ GS Imaging Workflow:
GS Review Station
Database
BD FocalPoint™
Slide Profiler

13. BD FocalPoint™ GS Imaging data transfer:
GS Review Station
Data transfer via
Removable hard-drive
or VPN connection
Data transfer
Server PC
GS Review Station

14. Principal Hypothesis
The Focal Point GS slide profiler technology is an acceptable replacement for the currently established processes of quality assurance in primary screening

15. The Project Aims
Assessment of the Focal Point Slide Profiler ‘Quintile Ranking’ facility against diagnostically confirmed sample abnormality profiles
Assessment of the Focal Point Slide Profiler NFR (No Further Review) category as a reliable indicator of negative samples

16. The Project Aims
Evaluate Focal Point GS as a means of assessing sample quality
establishing baseline parameters for assessing adequacy of samples by creating a reproducible parameter for the assessment of sample taker performance
Evaluate Focal Point GS across an integrated laboratory network
assessing sample identification, tracking and transportation processes, workload allocation patterns and logistics and user friendliness
Note: Get project aims from study protocol and then present one by one. Title slide eg Project Aims – 1
Then discuss reasoning on subsequent slides

17. The Project Aims
Monitoring screener performance using the Focal Point Slide Profiler
This technology has the capability to monitor operator/observer functions such as screener opinion and action, time taken to perform screen.
These data items - gathered using the Location Guided Screener workstation (LGS) can be compared against primary screening and/or final outcome in order to provide screener performance indicators.

18. Examine Operational Issues
Identify sample identification issues e.g. bar codes and maintaining “chain of custody” across a network
How does the technology deal with inadequate samples – does this compare with current adequacy criteria?
Does the technology compare with manual techniques in terms of operator time?

19. PROCESS

20. Process Samples received and processed at laboratory as normal
Despatched to Medical Solutions for FP scanning
Returned with data disc for Location Guided Screen (LGS) assessment in laboratory Automated Arm)
Manually screened independently of the LGS evaluation (Manual Arm)
Results correlated before final report issued

21. The ‘Automated Arm’
Consisted of screener evaluation of 10 fields of view (FOV) on each sample
Difficult cases were referred to a checker who also examined the 10 FOV
Findings were recorded on the proforma illustrated overleaf
Outcome was recorded on LIMS (TelePath)

22. The ‘Manual’ Arm
Slides were primary screened and QA screened as usual following standard laboratory protocols
Codes and reports were recorded on TelePath
Automated arm report codes were recorded
FPQ or FPQQ codes were recorded depending on whether or not a discrepancy was noted between the two arms
Cases were authorised in the normal manner
CSW retrieved FP/LGS data as part of weekly downloads

23. CERVICAL SCREENING WALES FOCAL POINT PROJECT - PROCESS MAP
AUTOMATED ARM
COURIER
SAMPLE
TO LAB
PROCESS
SLIDES TO
NOTTINGHAM
SLIDES ARE READ
ON FOCAL POINT
INDIVIDUALS CREENERS
OPINION OF 10 F.O.V.
HARD DRIVE +
PRINT OUT
Process Review data
RESULTS AND
SLIDES RETURNED
TO LAB
CODE RESULTS
ON TELEPATH
SLIDE WIZARD
NETWORK
DATA TRANSFER
AVAILABLE
DESIGNATED
PROCEDURE CODES
FULL SCREEN
Via
Manual Arm QC
DOUBLE SCREEN/
RAPID REVIEW
RESULTS TO CSW
PROJECT
EVALUATION
MANUAL ARM

24. Data and Morphology Anomaly Checking
Automated:manual arm mismatches were highlighted by the CSW Information department as and when they arose
Any resultant slide morphology issues were addressed in team assessment sessions at Med Sol in Nottingham and resolved by:
- LGS/multi-header microscope sessions involving representatives from Labs / CSW/ Med Sol

25. RESULTS

26. Total Slides Processed

27. QA beyond the Lab - Focal Point GS as a means of assessing sample quality
FP reports on the Squamous and Endocervical components detected in the sample
Potentially of use to the screening programme as an indicator of smear taker proficiency – but emotive!!!
Wales monitors TZ reporting – however generally a wide spread – between 80% - 96% reported by NW labs
The study has shown that the FP reported endocervical component detection rates in NW labs only varied between 79% and 85%
Possibly screener interpretation – CAS may provide a more uniform assessment
Further work is ongoing in this area

28. Focal Point GS performance across an integrated laboratory network
VPN data transfer means that scanned slide information is available anywhere within the combined laboratory service
Economies of scale are definite considerations when planning the introduction of this technology – possible consideration for LBC procurement
Therefore, in this operational scenario – probably more efficient to combine the sample processing and CAS on one processing site.

29. Teaching and Case review using the Focal Point GS
LGS can mark and record interesting features within a slide and direct the user to these areas on the slide on demand
With a system configured using VPN these slides can be viewed anywhere on the network
Considerable potential for teaching and MDT use when combined with a multi-head microscope/projector

30. Assessment of the Focal Point Slide Profiler ‘Quintile Ranking’ facility against diagnostically confirmed sample abnormality profiles

31. Sorting & Ranking (Illustration)
The FocalPoint™ SORTS and RANKS slides between the values of and 1 : based on the likelihood of abnormality being present is negative : 1 is abnormal x
< 25% Archive
> 75% Review
> 15% QC
Negative
Abnormal Q5 Q4 Q3 Q2
Q 1

32. FP Quintile Distribution
Note: this
category
Contains PR as
Well as NFR
samples
FP Quintile Distribution

33. FP Quintile Distribution

34. Focal Point Scant Cellularity Category (FPSC)
Compared to a Final Report of Inadequate
Comparison of the inadequacy levels showed that the manual arm had a similar inadequate rate – not what was expected with Wales operating a cell threshold
Interestingly the majority of cases did not correspond between the manual and automated arms

35. “NFR” – DOES IT WORK?
Assessment of the Focal Point Slide Profiler NFR (No Further Review) category as a reliable indicator of negative samples

36. Samples Assigned for NFR
Compared to Final Result

37. What was missed by NFR?
1. Cases from all labs, NFR negative, finally reported as abnormal
78 cases in a total of 2842
(Picked up by 10 screener, reviewer or checker)
2.74% of cases missed by NFR – False Negatives

38. What was missed by NFR? Statistically – Significant difference from 1.
In over cases manually rapid previewed by 4 Welsh Labs during Jan – September 2009 (Picked up by 10 screener, reviewer or checker)
The overall False Negative Rate was 3.5%
Statistically – Significant difference from 1.
NFR is therefore at least as effective as Manual Rapid Pre-screen as a QA tool for primary screening of cervical samples.

39. Overall Sensitivity
In total, the automated QC screening arm did not “identify” 12 HG cases (histologically confirmed) in a total of 134 reported
Sensitivity (134-12)/134 = 91%
Compare this to a quoted range of %* HG sensitivity for conventional Rapid Preview QA methods
202 LG cases were missed or consigned to NFR in 788 reported LG samples
Sensitivity ( )/788 = 74.4%
* See references and CSW data

40. But……….!
On review, morphological abnormalities were present in the majority of 10 FOVs in the HG cases “missed “ by the automated arm.
So potentially therefore, 91% is the minimum sensitivity rate that can be expected
This is because the FocalPoint GS presented the abnormalities to the LGS user, but not identified by the user for a number of possible reasons
This may be due to a number of reasons e.g. unfamiliarity and further analysis currently being performed on this data

41. In Summary………
Automated QC screen appears to be no less effective than rapid pre-screen
Most (161/219) cases not identified by automated arm were borderline HPV cases – FP does not seem to recognise koilocytes! – especially with minimal nuclear abnormalities.
Despite this finding, sensitivity of the automated QC screen appears to be appreciably greater in comparison to available data on manual QC methods.

42. User experience is a large factor in the efficiency of the screen
A high proportion of the undetected abnormalities were shown to be present in the FOV’s on review – but not identified by the screener.
Staff had only received 6-8months experience of the technology - further work is currently ongoing to see if screener performance improved with time/experience

43. Monitoring laboratory and screener performance using the Focal Point Slide Profiler

44. Individual Laboratory Process Review Results
Is an indicator of the number of failed slide scans in a given FP run
It reflects technical quality, bar code integrity etc.
Note the low value for YGC, which has been mainly attributed to tape coverslipping

45. Screening times for 10 FOV
Note the remarkable similarity of the mean screening times.
Note also that NFR category accounts for 22% of samples
- no FOVs, therefore potential productivity increase.

46. Manual QC Screening Times (includes data entry on LIMS)

47. OBSERVATIONS

48. Observations
Thought that FP operates on TBS inadequacy threshold of 5- 6k cells, therefore we hypothesised that the manual arm would identify a correspondingly higher number of inadequate samples, however……………..
Comparison of the inadequacy levels showed that the manual arm had only a slightly higher inadequate rate – not as expected with Wales operating a cell threshold
Interestingly the majority of cases did not correspond between the manual and automated arms – possibly indicative of the fact that the FPSC report is applied to cases that exhibit other criteria, such as:

49. Observations
The slide cannot be focussed on correctly because of bubbles, too thick, too few cells in the centre of the prep. etc.
It is therefore likely that the actual number of cases reported as FPSC for reasons related to cellular adequacy are lower than first thought
Further work will be undertaken to determine threshold values of the criteria that make up the FP report “Insufficient Reference Cells” to determine if this could be compared with the manual arm report of “Inadequate due to scant cellularity”

50. Observations
Tape coverslipping appreciably increases the technical quality of the scanned slide, thereby reducing the process review rate and decreasing the number of rejected slides.
Selecting a large coverslip size on the tape cover slipper extended the FP scan time from approx. 11 slides/min on glass coverslips to 9 slides/min using tape coverslips. Reducing the tape size reduced this to 10 slides/min*
Staff used to primary screening on a x20 objective may have initial difficulties in operating the LGS which only displays FOVs at x10
*Note: Updated versions of the FP GS allow faster scan rates

51. Observations
Staff have varying ability to correctly interpret the 10 FOVs and like manual screening, are only as good as the experience they have – both as primary screeners and users of the LGS
A small proportion of slides simply cannot be evaluated on 10 FOVs – these should be subjected to a full scan and further checking, if necessary
Despite best efforts to blind manual and automated arms interpretational bias may have resulted as staff were aware that this was a study

52. Observations
The study resulted in additional workload for staff which might also have affected interpretation
Clearly defined protocols are mandatory for routine use
Transport of slides and data via external couriers posed a few challenges! – eventually settled down

53. RECOMMENDATIONS

54. Recommendations
A number of issues have yet to be fully evaluated, including:
Completing comparisons between available manual QC data and project results
Full evaluation of NFR – do we need to continue with manual QC for these cases?
Subsequent effects on workload/staffing
Implementation and logistical issues
Full Cost/Benefit analysis

55. Recommendations Cost/benefit analysis
Need to further evaluate the potential of the NFR category
With demonstrated > 99% False Negative rate for HG disease in 22% of cases – potentially labour saving whilst improving quality!
This is in the region of 1100 hours screener time in Wales with an annual workload of cases

56. CONCLUSIONS

57. Conclusions
Analysis of the findings of this study are not yet complete.
At this stage initial results indicate that the technology is suitable for routine use, with potential quality and service management benefits to the cervical screening programme in Wales

58. Recommendations for Further Investigation
CSW AWMG have approved an extension of the project and in collaboration with Medical Solutions will be working towards full use of FP as a QA system. The network of laboratories will be extended to include a fourth laboratory from South Wales – at the Royal Gwent Hospital, Newport – project commenced July 2009
Define implemention plan
Networking issues – the technology seems particularly suited to networks. Wales is currently exploring the introduction of managed networks in cytology
Develop protocols
Specifically issues include standardisation of manual rapid QA methods across labs to rapid pre-screen
IT coding protocols require total standardisation – part of N Wales objectives. Data extraction will become more consistent

59. Recommendations for Further Investigation
Implementation
QA use initially – follow on project to implement FP as a replacement for QC screening and further assessment as a primary screening tool, especially in the light of the changing screening programme
Move to full adoption subject to satisfactory outcome of project and final approval by AWMG
Outcome will inform LBC procurement process

60. “Any sufficiently advanced technology is indistinguishable from magic
“Any sufficiently advanced technology is indistinguishable from magic.” Arthur C. Clarke – The Lost Worlds of 2001 (1972)