Why important? Current status Methods: numerous / 8 categories (Borji and Itti, PAMI, 2012) Databases: Measures: scan-path analysis correlation based measures ROC analysis Visual Saliency How good my method works?
Benchmarks Judd et al. http://people.csail.mit.edu/tjudd/SaliencyBenchmark/ http://people.csail.mit.edu/tjudd/SaliencyBenchmark/ Borji and Itti https://sites.google.com/site/saliencyevaluation/ https://sites.google.com/site/saliencyevaluation/ Yet another benchmark!!!?
Dataset Challenge Dataset bias : Center-Bias (CB), Border effect Metrics are affected by these phenomena. MIT Le Meur Toronto
Tricking the metric Solution ? sAUC Best smoothing factor More than one metric
The Feature Crises intensity orientation color size depth Low level people symmetry car text signs High level Features Does it capture any semantic scene property or affective stimuli? Challenge of performance on stimulus categories & affective stimuli Challenge of performance on stimulus categories & affective stimuli
The Benchmark Image categories and affective data
Lessons learned We recommend using shuffled AUC score for model evaluation. Stimuli affects the performance. Combination of saliency and eye movement statistics can be used in category recognition. There seems the gap between models and IO is small (though statistically significant). It somehow alerts the need for new dataset. The challenge of task decoding using eye statistics is open yet. Saliency evaluation scores can still be introduced