Image Retrieval Using Eye Movements Fred Stentiford & Wole Oyekoya University College London

Outline 1. Eye Movement Behaviour 2. Image Identification 3. Image Search 4. Conclusions & Future Work

Eye Gaze Computer (Eye Image Processing) Client Computer (Application Program) Chinrest Eye Monitor Real-Time Gazepoint Display Application Display Eye Tracking System

Eye Movement Behaviour saliency map image fixation and saccade map no ROI

Eye Movement Behaviour clear ROI saliency map image fixation and saccade map

Eye Movement Behaviour – no ROI participant Aparticipant B participant D participant C

Eye Movement Behaviour – clear ROI participant Aparticipant B participant D participant C

Variance of Attention Measure Image Image Image4 Obvious ROI Image Image Image1 Unclear ROI DCBA Participants Image Variance

Time Fixating Salient Regions (ms) Obvious ROI Unclear ROI DCBA Participants Images

Findings  No special fixation sequence although many look at salient regions first  Very salient regions inspected frequently and compared with background

Eye vs Mouse for Image Identification target image 1.Mouse click 2.Fixation > 40ms

Screen Display Sequence D = distractor Tn = target image

Eye vs Mouse Response Times 12 participants

Eye vs Mouse Response Times 6 participants in each group

Image Search Task target image steps to target 1000 images 13 participants

Image Selection Gaze selection of an image is determined by the sum of all fixations of 80ms or more on that image exceeding a threshold. Two thresholds 400ms and 800ms Successive sets of 15 images are retrieved based on their similarity with selected image. Performance compared with images randomly retrieved Participants not told what determines screen changes

Target Images easy to findhard to find

Similarity Links

Results 13 participants 8 sessions Main effect: Eye gaze 18 steps Random 22 steps p < 0.037

Results – Easy vs Hard Images

Other Selection Criteria 24 participants 8 sessions Main effect: fixation threshold not significant

Results - Lower Fixation Thresholds 6 participants 3 sessions Significant differences between random and 200ms + 300ms.

Results - Lower Fixation Thresholds

Conclusions  Eye tracking can be faster than tactile interfaces for visual tasks  Eye tracking interfaces are feasible for fast image search  Pre-attentive vision plays a part in very rapid search

Future Work  Further study of human visual behaviour  Use of higher performance similarity measures  Application to browsing large collections of photos/videos  Shared interaction