Presentation on theme: "The validation of Optscore, a software to evaluate cornea photographs Christine M. Toutain-Kidd, Travis Porco, Eric M. Kidd, Thomas M. Lietman, Michael."— Presentation transcript:
The validation of Optscore, a software to evaluate cornea photographs Christine M. Toutain-Kidd, Travis Porco, Eric M. Kidd, Thomas M. Lietman, Michael E. Zegans Dartmouth Medical School Hanover, NH - USA The authors have no financial interests.
Purpose Digital photo-grading systems for corneal photographs are underdeveloped compared to systems for retinal photographs. Nevertheless these systems could be an important help to clinicians. The Optscore software was created to attempt to fill that void. Its validation study is presented here. Validation of Optscore, a software to evaluate cornea pictures
Methods Optscore is based on the Ruby on Rails web technology. One hundred cornea pictures from the Mycotic Ulcer Treatment Trial (MUTT) pilot study (Prajna et al, in press), were looked at by 6 graders (5 medical students plus an ophthalmologist used as Gold Standard). After going through a training module, they evaluated, twice at least 2 weeks apart, several variables commonly used in the diagnosis of a corneal infection: size, location, opacity, neovascularization, conjunctival inflammation and presence of a hypopyon (see list of questions on following slide). The Interclass Correlation Coefficient (ICC) was calculated to assess the intra and inter-grader variability. Validation of Optscore, a software to evaluate cornea pictures
Question 1: Is the ulcer/scar visible on this picture? 1.Yes 2.No Question 2 (if yes in question 1): Circle the lesion on the photo Question 3 (if yes in question 1): Location of the lesion. 1.Entirely in periphery 2.Overlaps central 4mm circle and periphery without filling center 3.Entirely in central circle 4.Completely fills 4mm circle and extends into periphery Question 4 (if yes in question 1): Opacity of the lesion (worst 50% of the lesion). 1.Mild, iris details visible 2.Moderate, some iris visible 3.Complete, no iris visible Question 5: Corneal neovascularization (in clock hours). 0.0 1.1 2.2 3.3 4.4 5.5 6.6 7.7 8.8 9.9 10.10 11.11 12.12 Question 6: Conjunctival inflammation (in clock hours) 0.0 1.1 2.2 3.3 4.4 5.5 6.6 7.7 8.8 9.9 10.10 11.11 12.12 Question 7: Hypopyon. 0.No hypopyon 1.Hypopyon but non-hemorrhagic 2.Hemorrhagic Question 8: Quality of the picture. 1.Good 2.Poor but gradable 3.Ungradable Questions asked during the evaluation of all pictures in this study Validation of Optscore, a software to evaluate cornea pictures
Snapshot of the software screen presenting a picture ready to be evaluated (with the 4 mm central circle and the 12 mm outside circle overlapping the limbus) as well as the first question asked to the graders. Once the grader has answered the question, the software automatically goes to the next question.
Snapshot of the screen summarizing the questions and answers provided for a picture. Until the grader clicks on “next image” (not visible on this snapshot), all answers can be revised and changed. Validation of Optscore, a software to evaluate cornea pictures
The Steroids for Corneal Ulcers Trial (SCUT) and Mycotic Ulcer Treatment Trials (MUTT) I and II are collaborative studies involving the Aravind Eye Hospital in India, the F.I. Proctor Foundation at UCSF and Dartmouth Medical School. We used some of the pictures obtained during the MUTT pilot study in this work. The following tables present the Interclass Correlation Coefficient (ICC) calculated for all variables, which allows to compare the consistency of each grader and between the graders and the gold standard (GS). Validation of Optscore, a software to evaluate cornea pictures Results and Conclusions
VariableAreaLocationOpacityNeovascula rization Conj. Inflam. Hypopyon Overall ICC0.89420.79740.63220.63130.86320.8425 Interclass Correlation Coefficient (ICC) calculated for all variables evaluated in this study. Table A shows the overall ICC for each variable. A. Validation of Optscore, a software to evaluate cornea pictures - The measure of the area and the evaluation of the presence of a hypopyon are the two variables with the highest ICC. - More subjective variables like level of opacity or estimation of neovascularization have a low ICC.
Intergrade r variability Grader A/ GS1 Grader B/ GS1 Grader C/ GS1 Grader D/ GS1 Grader E/ GS1 Area0.92280.94610.89660.87220.9365 Location0.78770.89960.80970.84960.7794 Opacity0.65090.53380.62750.58850.6877 Neovascul arization 0.63540.64150.51610.57570.6454 Conj. Inflam 0.87220.83420.88260.82870.7903 Hypopyon0.82410.83290.94160.75420.8390 B. Table B presents the intergrader variability calculated between each grader and the Gold Standard’s first grading. Validation of Optscore, a software to evaluate cornea pictures - When comparing the evaluations of each grader with the gold standard (GS), the variables with the highest ICC here are also the ones with the highest overall ICC, such as area, location and hypopyon. - The evaluation of the level conjunctival inflammation also shows in both cases, tables A and B, a high ICC. This variable is continuous but also quite subjective like neovascularization, the subjectivity has to be taken into account. There were many cases where the number of clock hours of inflammation estimated by the graders was different by one or two clock hours (which is not much on the calculation of the ICC but still leads to a lack of consistency).
Validation of Optscore, a software to evaluate cornea pictures Table C shows the intragrader variability calculated using the two evaluations done by each grader at least 2 weeks apart. Intragrader variability GS1/GS2Grader AGrader BGrader CGrader DGrader E Area0.98230.97720.98360.97620.90860.9771 Location0.85330.76550.94810.67360.74190.8132 Opacity0.69180.39150.85340.61940.64020.8164 Neovascula rization 0.75680.88360.90260.93390.88950.9019 Conj. Inflam 0.95100.97540.97930.94690.95290.9364 Hypopyon0.88790.93530.85310.94160.77720.8883 C. - It should be noted that all graders went through the same training module composed of two parts, a power point presentation teaching them about the cornea and the variables that they would have to evaluate, and a training set of pictures to teach them how to use the software. The time required by each grader to perform the evaluations was recorded as well. - Grader D, who overall has a relatively low intragrader ICC, including for non subjective variables such as area size and presence or absence of hypopyon, did the evaluations the fastest.
- This study demonstrates that Optscore can be reliably used to evaluate several variables required for the assessment of corneal infections. - The training of the graders is essential and the importance of spending enough time per picture should be emphasized. - The least subjective variables show the highest correlation amongst graders. - Optscore is a very flexible software that can be modified to fit any requirements. - We believe that Optscore could be developed as a diagnostic help and an educational tool. Validation of Optscore, a software to evaluate cornea pictures