1. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 1 Computational Architectures in Biological Vision, USC Lecture 12. Visual Attention Reading Assignments: None

2. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 2

3. 3

4. 4 Several Forms of Attention Attention and eye movements: - overt attention (with eye movements) - covert attention (without eye movements) Bottom-up and top-down control: - bottom-up control based on image features very fast (up to 20 shifts/s) involuntary / automatic - top-down control may target inconspicuous locations in visual scene slower (5 shifts/s or fewer; like eye movements) volitional Control and modulation: - direct attention towards specific visual locations - attention modulates early visual processing at attended location

5. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 5 What is attention then? Attention is often described as an information processing bottleneck. Controls access to higher levels of processing, short-term memory and consciousness. Hence, the strategy nature has developed to cope with information overload is to break down the problem of analyzing a visual scene: from a massively parallel approach to a rapid sequence of circumscribed recognitions.

6. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 6

7. 7

8. 8

9. 9

10. 10

11. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 11

12. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 12

13. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 13

14. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 14

15. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 15

16. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 16 First Computational Model Koch & Ullman, Hum. Neurobiol., 1895 Introduce concept of a single topo- graphic saliency map. Most salient location selected by a winner-take-all network.

17. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 17

18. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 18 Shifter Circuits Anderson & van Essen, PNAS, 1987 Information dynamically routed through cortical hierarchy. Yields rotation- and scale-independent representation.

19. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 19 Shifter Circuits (cont.) Olshausen et al., J Neurosci, 1993 Implemented shifter circuits and demonstrated proof of concept. Control neurons in the pulvinar send the (attention-based) control signals that will determine the “passing” region of the circuit, through a modulation of intracortical connection weights. Perform recognition using associative memory at top level.

20. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 20 only attended item reaches output layer

21. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 21 Selective Tuning Model Tsotsos et al., Artificial Intelligence, 1995 - attention modulates neurons to earliest levels; wherever there is a many-to-one mapping many-to-one mapping - signal interference controlled by surround inhibition throughout processing network throughout processing network -task knowledge biases computations throughout processing network - attentional control is local, distributed and internal - competition is based on WTA (different form than previous models) (different form than previous models) - pyramid representation with reciprocal convergence and divergence neuron ‘sees’ this receptive field subject ‘attends’ to single item

22. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 22 The basic idea (BBS 1990)

23. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 23 Selective Tuning Model processing pyramid inhibited pathways pass pathways unit of interest at top input Caputo & Guerra 1998 Bahcall & Kowler 1999 Vanduffel, Tootell, Orban 2000 Smith et al. 2000 Kastner, De Weerd, Desimone, Ungerleider, 1998

24. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 24

25. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 25 Guided Search Wolfe, Psychonomic Bull. & Rev., 1994 How can we combine information from several modalities? Use top-down (task-dependent) weighting.

26. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 26

27. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 27

28. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 28

29. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 29

30. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 30

31. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 31

32. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 32

33. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 33

34. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 34

35. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 35

36. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 36

37. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 37

38. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 38

39. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 39

40. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 40

41. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 41

42. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 42

43. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 43 Image Compression

44. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 44 Evaluation of Advertising

45. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 45

46. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 46

47. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 47 Brefczynski & DeYoe, Nature Neuroscience 1999

48. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 48

49. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 49 Treue & Martinez-Trujillo, Nature 1999

50. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 50

51. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 51 Attentional Modulation in Humans Gandhi et al, 1999

52. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 52

53. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 53

54. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 54

55. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 55

56. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 56 Attentional Modulation Hernandez et al. Picture naming by bi-lingual persons. Increased attention/concentration due to increased difficulty when non-native tongue? No, same patterns of activation. But increased activation when Switching between languages, Probably reflecting increased Attentional load. Broca: speech generation Supramarginal: articulation & phonology processing Cingulate: emotion, memory, vigilance, attention?

57. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 57

58. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 58

59. Laurent Itti: CS599 – Computational Architectures in Biological Vision, USC. Lecture 12: Visual Attention 59