1. Spatio-temporal saliency model to predict eye movements in video free viewing Gipsa-lab, Grenoble Département Images et Signal CNRS, UMR 5216 S. Marat, T. Ho Phuoc, L. Granjon, N. Guyader, D. Pellerin, A. Guérin-Dugué GDR-vision 12/06/2008

2. Plan Introduction Model Experiment and results Conclusion 2/24

3. Salient region attracts attention and so the eyes Saliency depends mainly on two factors: Bottom-up : task-independent, depending on intrinsic features of the stimuli Top-down : task-dependant, integrating high-level processes (cognitive state,...) 3/24 Introduction

4. Spatio-temporal saliency model Achromatic stimuli Simulates some parts of the human visual system: retina, primary visual cortex (V1) Two pathways : static and dynamic 4/24 Model MsMs MdMd

5. 5/24 Model MsMs MdMd Spatio-temporal saliency model Achromatic stimuli Simulates some parts of the human visual system: retina, primary visual cortex (V1) Two pathways : static and dynamic

6. 6/24 Model MsMs MdMd Spatio-temporal saliency model Achromatic stimuli Simulates some parts of the human visual system: retina, primary visual cortex (V1) Two pathways : static and dynamic

7. 7/24 Model Spatio-temporal saliency model Achromatic stimuli Simulates some parts of the human visual system: retina, primary visual cortex (V1) Two pathways : static and dynamic

8. Two outputs : Magnocellular-like:  Low spatial frequencies, band pass filter, whitens spectrum, provides global information Parvocellular-like:  High spatial frequencies, high pass filter, whitens spectrum, enhances frame contrast 8/24 Retina model Model _ retina model

9. 9/24 Retina model Model _ retina model « Parvocellular-like » « Magnocellular-like » MsMs MdMd

10. Visual stimuli are processed in different frequency bands and orientation in V1 Static: 6 orientations, 4 frequency bands Dynamic: 6 orientations, 3 frequency bands (lower) 10/24 Cortical-like filters Model _ cortical-like filters MsMs MdMd

11. 11/24 Static pathway Model _ static pathway Static pathway: Interactions: strengthens the contours  Short: between cells of overlapping receptive field  Long: between collinear cells Normalization Summation in all orientation and frequency bands: static saliency MsMs MdMd

12. 12/24 Dynamic pathway Model _ dynamic pathway Dynamic pathway: 2 motion estimation steps:  Dominant motion compensation  Local motion estimation using the same bank of cortical filters as static pathway Temporal filtering Dynamic saliency: module of motion vector MsMs MdMd

13. 13/24 Dynamic pathway Model _ dynamic pathway Dynamic pathway: 2 motion estimation steps:  Dominant motion compensation  Local motion estimation using the same bank of cortical filters as static pathway Temporal filtering Dynamic saliency: module of motion vector MsMs MdMd

14. Multiplicative fusion 14/24 Fusion and example of saliency maps Model _ fusion and example of saliency maps Original video MdMd M and MsMs

15. Purpose : compare model results with human eye positions Free viewing, eye positions recorded by Eyetracker Eyelink II 15 subjects 20 clips of 30s composed of different snippets strung together Stimulus size = 720x576 pixels, 40°x30° field of view 15/24 Experiment and results Experiment and result Snippet 1Snippet 2 Snippet k-1Snippet kSnippet k-2 [Itti] : R. Carmi and L. Itti, « Visual causes versus correlates of attentional selection in dynamic scenes », Vision Research, vol.46, 2008 MhMh

16. Criterion : Normalized Scanpath Saliency (NSS) [Itti] 16/24 Global analysis Experiment _ global analysis 0.540.440.54Real eye movements M dn M snSD M snH Naives saliency maps M s : static M d : dynamic M and : fusion M snH : entropy M snSD : standard-deviation M dn : absolute difference [Itti] : R. J. peters and L. Itti, « Applying computational tools to predict gaze direction in interactive visual environments », ACM Trans. On Applied Perception, vol.5, 2008 Saliency mapsMsMs MdMd M and Real eye movements0.680.870.96

17. NSS as a function of frame 17/24 Temporal analysis Experiment _ temporal analysis Snippet 1Snippet 2 Snippet N Average on the k th frame of each snippet 1…k … Frame rate = 25 fps

18. NSS as a function of frame 18/24 Temporal analysis Experiment _ temporal analysis Frame rate = 25 fps Snippet 1Snippet 2 Snippet N Average on the k th frame of each snippet 1…k …

19. NSS as a function of frame 19/24 Temporal analysis Experiment _ temporal analysis Frame rate = 25 fps Snippet 1Snippet 2 Snippet N Average on the k th frame of each snippet 1…k …

20. NSS as a function of frame 20/24 Temporal analysis Experiment _ temporal analysis Frame rate = 25 fps Snippet 1Snippet 2 Snippet N Average on the k th frame of each snippet 1…k …

21. NSS as a function of frame 21/24 Temporal analysis Experiment _ temporal analysis Dispersion of eye positions as a function of frame Frame rate = 25 fps

22. NSS as a function of frame 22/24 Temporal analysis Experiment _ temporal analysis Dispersion of eye positions as a function of frame Frame rate = 25 fps 10-13 th frame ≈ 400-520 ms

23. NSS as a function of frame 23/24 Temporal analysis Experiment _ temporal analysis Dispersion of eye positions as a function of frame Frame rate = 25 fps

24. New model of spatio-temporal saliency, biologically inspired Retina filter with two outputs Interactions Same bank of cortical-like filters for static and dynamic pathways This model is reliable to predict the first fixations references : S. Marat, T. Ho Phuoc, L. Granjon, N. Guyader, D. Pellerin, A. Guérin-Dugué, « Spatio-temporal saliency model to predict eye movements in video free viewing », Proc. Eusipco 2008 S. Marat, T. Ho Phuoc, L. Granjon, N. Guyader, D. Pellerin, A. Guérin-Dugué, « Modelling spatio-temporal saliency to predict gaze direction for short videos », submitted in International Journal of Computer vision 24/24 Conclusion

25. Thanks for your attention !