Assessment of Computational Visual Attention Models on Medical Images Varun Jampani 1, Ujjwal 1, Jayanthi Sivaswamy 1 and Vivek Vaidya 2 1 CVIT, IIIT Hyderabad, India 2 GE Global Research, Bangalore, India Center for Visual Information Technology International Institute of Information Technology, Hyderabad, INDIA Contact OBJECTIVE: To investigate the performance of computational saliency models in medical images in the context of abnormality detection. Visual Search in Medical Images Computational Visual Attention Models Study on Chest X-rays Results Conclusion and Future Work Feature Integration Theory [Treisman and Gelade 1980] Visual search paradigm involves identifying targets from surrounding distractors Disjunctive Target Conjunctive Target Bottom-up and Top-Down Visual attention models Produces saliency maps which predict salient regions Bottom-up models used in present study – Itti-Koch Model (IK) [Itti and Koch 2001] – Spectral residual model (SR) [Hou and Zhang 2007] – Graph based visual saliency model (GBVS) [Harel et al. 2007] Despite the importance of top-down knowledge, bottom-up knowledge plays considerable role Including top-down knowledge of anatomical regions resulted in marginal improvement in accuracy Saliency models can be used in the development of CAD tools for medical images Need to incorporate other types of top-down knowledge Visual Attention: Process of selectively attending to an area of visual field while ignoring the surrounding visual areas Influenced by image-dependent bottom-up features and task- dependent top-down features Plays important role in finding abnormalities in medical images [Matsumoto et al. 2011]: Showed that bottom-up mechanisms also play a significant role in guiding the eye movements of neurologists looking for stroke lesions on brain CT images Global-Focal model of visual search for tumor detection in chest x-rays [Nodine and Kundel 1987] – Overall Pattern recognition – Focal attention – Decision making Eye fixations of an observer on a sample chest x-ray We evaluated the role of bottom-up saliency in chest x-rays of pneumoconiosis by comparing the saliency maps against the eye fixations of observers of different expertize levels. Eye movement recordings were done using a remote head free eye tracker (SR Research - Eyelink 1000) See [Jampani et al. 2011] for experimental details ROC analysis is done by using saliency maps as classifiers and considering eye fixations as ground truth Sample X-ray segments showing different stages of Pneumoconiosis (Conjunctive Targets) A sample chest x-ray and the corresponding saliency maps computed using different saliency models Study on Retinal Images Saliency maps extracted from 2 sample retinal images 4 Sample retinal images showing hard exudates (Disjunctive Targets) A sample retinal image and ground truth markings by four medical experts. Taken from DIARETDB1 dataset [Kauppi et al. 2007] We evaluated the bottom-up saliency models by comparing the saliency maps against the ground truth markings by medical experts Publicly available DIARETDB1 dataset [Kauppi et al. 2007] ROC analysis is done to compare saliency maps with ground truth markings In Chest X-ray Study GBVS model (Mdn AUC = 0.77) outperforms (Z=0.0, p <.001) SR model (Mdn AUC = 0.67) and IK model (Mdn AUC = 0.67 ) in predicting eye fixations of observers AUC value of 0.77 suggests that GBVS saliency model can be used to a reasonably good accuracy to predict the fixations of the observers Extended GBVS model by giving more importance to lung regions AUCs for EGBVS (Mdn = 0.81) are significantly higher (Z = 2.0, p <.001) than those of GBVS (Mdn = 0.77) Mean AUCs corresponding to different saliency models for all the observers in chest x-ray study In Retinal Image Study AUC values found to be: 0.72 for IK, 0.70 for GBVS and 0.73 for SR All three models perform roughly the same AUC of SR model rose to 0.74 with optic disk suppression Extended SR model by computing saliency maps at multiple scales and combining them ESR model (Mean AUC = 0.94) performed significantly better (28.76% increase) than SR model (Mean AUC = 0.74) Average ROC curves for different saliency models in retinal image study Different steps in attracting EGBVS saliency map from a sample chest x-ray Steps for deriving ESR saliency maps Input Image Itti- Koch GBVS SR