1. Visual Attention Derek Hoiem March 14, 2007 Misc Reading Group

2. The Eye 120 million rods (intensity) 7 million cones (color) Fovea: 2 degrees of cones

3. Saccades and Fixations Scope: 2 deg (poor spatial res beyond this) Duration: 50-500 ms (mean 250 ms) Length: 0.5 to 50 degrees (mean 4 to 12) Various types (e.g., regular, tracking, micro)

4. Saccades and Fixations Free Examine What are the material circumstances of the family? What are their ages? What were they doing before arrival? Remember the clothes Remember object and person positions How long has the unexpected visitor been away? [Yarbus 1967]

5. Visual Phenomena Fast scene recognition (100-150 ms) Fast “contains animal” (100-150 ms) Pop-out Attentional blindness Change blindness

6. Pop-out (texture) + V V V V V V V V V V V T V V V V V

7. Pop-out (more texture) + V V V V V V V V V V V V V V V V V V V V V V V V V V V V V V V V T

8. Pop-out (harder texture) + V V V V V V V V V V V V V V V V V V V V V V V V V V V V V V V V Y

9. Pop-out (color) + V V V V V V V V V V V V V V V V V V V V V V V V V V V V V V V

10. Pop-out (color + texture) + T V V V V V V V V V T V V V V V T V V T V T V V V T T T V T V

11. Pop-out (layout) + TTTTTTT T TTTTT TTTTTTT T

12. Pop-out Performance vs. Distractors

13. Demos http://viscog.beckman.uiuc.edu/djs_lab/demos.html

14. Model of Vision Pre-Attentive Stage [Rensink 2000] (figure from Itti 2002)

15. Purpose of attention Warning (animals, flashes, sudden motion) Exploration (find objects, verification) Inspection

16. Bottom-up Attention Models [Itti Koch Niebur 1998] Gabor Pyramid + Orientation Filters Subtract low-res (3-4 octaves) from higher res Normalize (0..1) + map * ( 1 – max ave ) 2 + add maps Average Maps Inhibition + Excitation

17. Bottom-Up: Normalization [Itti Koch Niebur 1998] Normalize map values to fixed range [0..1] Compute average local maximum m Multiply map by (1-m) 2

18. Bottom-Up: Predicted Fixations [Itti Koch Niebur 1998]

19. Updates to Bottom-Up Model [Peters Iyer Itti Koch 2005] Cross-orientation suppression Long-range contour interactions Eccentricity-dependent processing (e -x ) – Goal: better prediction of subsequent fixations

20. Experiments with Newer Model [Peters Iyer Itti Koch 2005]

21. Experiments with Newer Model [Peters Iyer Itti Koch 2005] Normalized Scanpath SalienceInter-observer Salience

22. Almost No Benefit to More Complicated Models [Peters Iyer Itti Koch 2005]

23. “Eccentricity-Dependent Filtering” Helps [Peters Iyer Itti Koch 2005] No EDF EDF

24. Other Bottom-Up Issues Real viewing vs. images ( Gajewski et al. 2005) – Longer saccades (12 deg vs. 4 deg) – Short saccades may be due to density of images, rather than movement cost Saliency map? – Evidence for multi-saliency representations (not clear there is a single map) – Capability to ignore predictable motions is difficult in map formulation ImageSaliency Map

25. Alternative Bottom-up Models Itti-Koch accounts for some pop-out effects – Is it biologically plausible? (peak normalization) – Is it biologically reasonable? – No reasonable mechanism for next fixation Top-down bias only (Wolfe’s guided search) – Does not account for free viewing behavior Surprise or explanation seeking (expectation-based saliency) – No saliency map required – May provide better prediction of next fixation, account for motion prediction

26. Top-Down Attention Models

27. Feature weighting – Verbal – Visual Location prior – From memory of scene (direct or indirect) – From scene information and semantics

28. Verbal Cueing  Feature Weighting Faster search if cued as to color or texture Faster yet if exemplar is shown Searching for mid-level cues (e.g., intensity, size, saturation) is harder – But may still be cued [Navalpakkam Itti 2006]

29. Verbal Cueing  Feature Weighting [Navalpakkam Itti 2006]

30. Verbal Cueing  Feature Weighting [Navalpakkam Itti 2006]

31. Verbal Cueing  Feature Weighting [Navalpakkam Itti 2006]

32. Role of Memory People can remember hundreds or thousands of scenes from single exposure (Shephard 1967) After seeing repeated scenes (in random order) – Faster finding of target (Brockmole and Henderson 2006) When mirrored after learning – First look at original location, then quickly go to new location (still faster) Learning of upside-down scenes takes twice as long

33. Role of Memory [Brockmole and Henderson 2006]

34. Scene Context Scene-constrained targets detected faster, with fewer eye movements Strategy 1st: check target-consistent regions 2nd: check target-inconsistent regions [Neider Zelinsky 2005]

35. Scene Context [Neider Zelinsky 2005] Target Presence Target Absence

36. Scene Context “Gist” can provide image height prior [Torralba et al. 2006] Saliency = inverse probability ^(0.05) * gaussian

37. Scene Context [Torralba et al. 2006] Gist

38. Scene Context [Torralba et al. 2006]

39. Scene Context [Torralba et al. 2006]

40. Scene Context [Torralba et al. 2006]

41. Scene Context [Torralba et al. 2006]

42. Scene Context: People Search [Torralba et al. 2006]

43. Scene Context: Object Search [Torralba et al. 2006]

44. Bottom-up + Top-down Attention Method 1: Weight individual features Method 2: Saliency.* Bias

45. Conclusions Artificial static scenes and pop-out well- explained by existing models Little recent progress in bottom-up models (stuck with Itti-Koch model) Only simplistic scene information modeled

46. Sources Saliency – Itti, Koch, Niebur (1998). A model of saliency-based visual attention for rapid scene analysis. – Itti, Koch (2001). Computational Modelling of Visual Attention. – Itti (2002). Modeling Primate Visual Attention. – Itti (2002). Visual Attention. – Navalpakkam, Arbib, Itti (2004). Attention and Scene Understanding. – Peters, Iyer, Itti, Koch (2005). Components of bottom-up gaze allocation in natural images. Role of memory – Chun, Jiang (1998). Contextual cueing: implicit learning and memory of visual context guides spatial attention. – Chun, Jiang (2003). Implicit, long-term spatial contextual memory. – Brockmole, Henderson (2006). Recognition and attention guidance during contextual cueing in real-world scenes: Evidence from eye movements. Top-Down Attention – Niedur, Zelinksy (2005). Scene context guides eye movements during visual search. – Navalpakkam, Itti (2006). Top-down attention selection is fine grained. – Torralba, Oliva, Castelhano, Henderson (2006). Contextual guidance of eye movements and attention in real-world scenes: the role of global features on object search.

47. Sources Others (used) – Rensink, O’Regan, Clark (1997). To see or not to see: the need for attention to perceive changes in scenes. – Liversedge, Findlay (2000). Saccadic eye movements and cognition. – Rensink (2000). The dynamic representation of scenes. – Delorme, Rousselet, Mace, Fabre-Thorpe (2004). Interaction of top-down and bottom-up processing in the fast visual analysis of natural scenes. – Gajewski, Pearson, Mack, Bartlett, Henderson (2005). Human gaze control in real world search. – http://www.diku.dk/~panic/eyegaze/node13.html Others (not used but potentially interesting) – Itti, Koch, Braun (2000). Revisiting spatial vision: toward a unifying model. – Epstein (2005). The cortical basis of visual scene processing. – Baldi, Itti (2005). Attention: Bits versus Wows.