1. Predicting Visual Search Targets via Eye Tracking Data
Dr. Ali Borji
University of Central Florida
Fawad Ahmed
University of Central Florida
Manisha Gupta
University of Texas at Austin
I. Purpose & Applications
III. Gaze Information
V. Euclidean Distances
Participants’ gaze information was plotted against the collages
Size of the fixation points corresponds to the duration
Predict visual search targets in closed and open world settings
Develop a learning mechanism over an unknown set of targets
Applications: Implemented in cameras to scan the environment and pinpoint where a lost object (ex: set of keys) is located
II. Dataset
Extracted book covers
Search target (top left)
6 participants searched for a target within a collage
20 variations of each collage
5 search targets (shown next to each collage)
Yields 600 search tasks
*Generated for all 600 search tasks
*Repeated for the 20 variations of each collage per search target
*Generated for 5 search tasks for each of the three types of collages
Lower distances demonstrates higher similarities between the search target and the fixation point
IV. Convolutional Neural Network
Pre-trained Siamese model
Extracts 2x512 vector features from linear layer
Compares and outputs image similarities
Pipeline:
Architecture:
Search Target
ConvNet
Decision Network
Image Similarity
Extracted Book Cover 1
Amazon book covers: Distinct structure/Distinct color
Extracted Book Cover 2
----
Extracted Book Cover N
*Repeated for all six users
The value of average Euclidean distances and lingering percentages imply that the set of Amazon collages was the most difficult search task.
O’Reilly book covers: Low structure/Distinct color
Decision Network
Decision Network
Branch Network 1
Image 1
Conv + ReLu
Max pooling
Max Pooling
VI. Future Work
VII. Acknowledgements
Send feature vector and search targets to a binary SVM
Add weights to fixation points to account for gaze duration
Thank you to Dr. Mubarak Shah and Dr. Niels Lobo for their guidance during the program. The research project presented is funded by NSF REU Summer Program 2016.
Un-shared
(pseudo-siamese)
Shared
(siamese)
Image 2
Conv + ReLu
Max pooling
Max Pooling
Branch Network 2
Mugshots: Low structure/Low color