Computer Vision Lecture 2: Vision, Attention, and Eye Movements January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

Computer Vision Lecture 2: Vision, Attention, and Eye Movements Stimuli in receptive field of neuron January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

Computer Vision Lecture 2: Vision, Attention, and Eye Movements Cat V1 (striate cortex) Orientation preference map Ocular dominance map January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

Computer Vision Lecture 2: Vision, Attention, and Eye Movements January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

Structure of NNs (and some ANNs) In biological systems, neurons of similar functionality are usually organized in separate areas (or layers). Often, there is a hierarchy of interconnected layers with the lowest layer receiving sensory input and neurons in higher layers computing more complex functions. For example, neurons in macaque visual cortex have been identified that are activated only when there is a face (monkey, human, or drawing) in the macaque’s visual field. January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

Computer Vision Lecture 2: Vision, Attention, and Eye Movements “Data Flow Diagram” of Visual Areas in Macaque Brain Blue: motion perception pathway Green: object recognition pathway January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

Receptive Fields in Hierarchical Neural Networks neuron A Complexity tells us to use pyramids - but new problem... Routing is one of those problems that was little considered before Anderson & Van Essen pointed it out (Koch & Ullman did say selected features routed to high level, but not how). They used shifts in space (a giant multiplexor) controlled by outside signals to route a signal from anywhere int he input field to the appropriate portions of the output layer. Claimed that stereo,motion can also be handled analogously. Never implemented or tested the idea; in Olshausen’s thesis. connectivity was prohibitive. ‘Patchy’ connectivity solved this. Burt used a Laplacian pyramid: each level is determined as the difference of the previous two levels of the Guassian pyramid. Fovea imposed on this with 1/theta acuity solves routing. Only attended signal reaches output (rest is discarded at each level) Key to my routing is observation that stimulus feeds a forward cone and that any output ayer unit is root of a sub-pyramid. Potential for cross-talk is large Single attentional fixation at a time - can’t route two without interference! receptive field of A January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

Receptive Fields in Hierarchical Neural Networks neuron A in top layer Complexity tells us to use pyramids - but new problem... Routing is one of those problems that was little considered before Anderson & Van Essen pointed it out (Koch & Ullman did say selected features routed to high level, but not how). They used shifts in space (a giant multiplexor) controlled by outside signals to route a signal from anywhere int he input field to the appropriate portions of the output layer. Claimed that stereo,motion can also be handled analogously. Never implemented or tested the idea; in Olshausen’s thesis. connectivity was prohibitive. ‘Patchy’ connectivity solved this. Burt used a Laplacian pyramid: each level is determined as the difference of the previous two levels of the Guassian pyramid. Fovea imposed on this with 1/theta acuity solves routing. Only attended signal reaches output (rest is discarded at each level) Key to my routing is observation that stimulus feeds a forward cone and that any output ayer unit is root of a sub-pyramid. Potential for cross-talk is large Single attentional fixation at a time - can’t route two without interference! receptive field of A in input layer January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

Computer Vision Lecture 2: Vision, Attention, and Eye Movements January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

Face aftereffect – Thanks to Arash Afraz for the slides! January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

Computer Vision Lecture 2: Vision, Attention, and Eye Movements Visual Illusions Visual Illusions demonstrate how we perceive an “interpreted version” of the incoming light pattern rather that the exact pattern itself. January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

Computer Vision Lecture 2: Vision, Attention, and Eye Movements Visual Illusions He we see that the squares A and B from the previous image actually have the same luminance (but in their visual context are interpreted differently). January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

Computer Vision Lecture 2: Vision, Attention, and Eye Movements Visual Attention Visual attention is the selective allocation of visual processing resources. For example, we can focus our attention on a particular object of interest in the visual field. Visual processing of that object is enhanced while being rather shallow for other objects. Also, we can respond more quickly and accurately to changes in an attended region. This prioritization is necessary due to our limited processing resources. January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

Computer Vision Lecture 2: Vision, Attention, and Eye Movements Visual Attention The attentional cueing task introduced by Michael Posner gives insight into the dynamics of visual attention. Subjects are instructed to fixate on the central cross. One of two boxes (left or right) flashes to capture the subject’s attention (an automatic, involuntary response). After some a short delay (stimulus onset asynchrony - SOA) an asterisk appears in one of the boxes. The subject has to report as quickly as possible in which box the asterisk appeared. January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

The Posner Attention Task x January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

The Posner Attention Task x January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

The Posner Attention Task x January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

The Posner Attention Task * x January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

The Posner Attention Task x January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

The Posner Attention Task For short SOAs (< 200 ms), subjects respond faster if flash and asterisk appear on the same side. Cueing of attention to relevant location allows faster response. For longer SOAs, subjects respond more slowly if flash and asterisk appear on the same side. Inhibition-of-Return mechanism makes attention less likely to remain on the side of the flash until the asterisk appears. January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

Computer Vision Lecture 2: Vision, Attention, and Eye Movements Eye Muscles January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

Computer Vision Lecture 2: Vision, Attention, and Eye Movements Types of Eye Movement Fixations: The eye is almost motionless, for example, while reading a single, short word. The information from the scene is almost entirely acquired during fixation. Duration varies from 100-1000 ms, typically between 200-600 ms. Typical fixation frequency is about 3 Hz. Fixations are interspersed with saccades. January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

Computer Vision Lecture 2: Vision, Attention, and Eye Movements Types of Eye Movement Saccades: Quick “jumps” that connect fixations Duration is typically between 30 and 120 ms Very fast (up to 700 degrees/second) Saccades are ballistic, i.e., the target of a saccade cannot be changed during the movement. Vision is suppressed during saccades to allow stable perception of surroundings. Saccades are used to move the fovea to the next object/region of interest. January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

Computer Vision Lecture 2: Vision, Attention, and Eye Movements Types of Eye Movement Smooth Pursuit Eye Movements: Smooth movement of the eyes for visually tracking a moving object Cannot be performed in static scenes (fixation/saccade behavior instead) January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

Why Eye-Movement Research? About eye movements and visual attention: Usually, saccades follow shifts of attention to provide high acuity at the attended position. It is possible to look at an object without paying attention to it (staring). It is possible to shift attention without eye movement (covert shifts of attention). It is impossible to perform a saccade while not shifting attention. During specific, natural tasks it is reasonable to assume that saccades follow shifts of attention. January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

Why Eye-Movement Research? The investigation of visual attention, in turn, is at the core of cognitive science. Studying visual attention yields insight into general attentional mechanisms. It can provide information on a person’s stream of conscious and unconscious processing while solving a task. Attention is closely linked to the concept of consciousness. Attentional mechanisms could improve artificial vision systems. January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

Computer Vision Lecture 2: Vision, Attention, and Eye Movements Eye-Movement Studies Eye movements while watching a girl’s face (early study by Yarbus, 1967) January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

Computer Vision Lecture 2: Vision, Attention, and Eye Movements Eye-Movement Studies Eye movements as indicators of cognitive processes (Yarbus): trace 1: examine at will trace 2: estimate wealth trace 3: estimate ages trace 4: guess previous activity trace 5: remember clothing trace 6: remember position trace 7: time since last visit January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

Computer Vision Lecture 2: Vision, Attention, and Eye Movements Eye-Movement Studies Visual scan paths on instruments/dashboards – studies for the improvement of human-computer interfaces January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

Eye-Movement Studies Gaze trajectory measurement for the optimization of web page layout January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

Eye-Movement Studies Improving advertisements with eye-movement studies January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

Selectivity in Complex Scenes January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements

Computer Vision Lecture 2: Vision, Attention, and Eye Movements Face Recognition Gaze-contingent window deteriorates face recognition, allows to identify relevant visual information. January 25, 2018 Computer Vision Lecture 2: Vision, Attention, and Eye Movements