1. Control of Attention and Gaze in the Natural World.

2. Selecting information from visual scenes What controls the selection process?

3. Humans must select a limited subset of the available information in the environment. Fundamental Constraints Acuity is restricted. Attention is limited. Visual Working Memory is limited. Only a limited amount of information can be retained. What controls these processes?

8. Image properties eg brightness, edges, color, can account for some fixations when viewing images of scenes. “Saliency”

9. Can we rely on image properties to guide where we look?

10. Important information may not be salient eg Stop signs in a cluttered environment. Salient information may not be important - eg retinal image transients from eye/body movements. Doesn’t account for many observed fixations, especially in natural behavior (eg Land etc).

11. Natural vision is not the same as viewing pictures. Behavioral goals determine what information is needed. Why Study Natural Behavior?

12. Viewing pictures of scenes is different from acting within scenes.

13. Foot placement Obstacle avoidance Heading Viewing pictures of scenes is different from acting within scenes. “Top-down” factors Top-down versus “bottom-up”

14. Where do top down factors come from? 1. Behavioral goals. 2. Need to learn the locations in the world that need our attention.

15. Gaze Patterns in Driving

16. When instructed to obey normal traffic rules, subjects spend a lot of time looking in the neighborhood of intersections. Time fixating Intersection. Follow Obey Traffic Rules

17. Subjects learn where to get the information they need from the world.

18. target selection signals to muscles inhibits SC saccade decision saccade command planning movements Neural Circuitry for Saccades Substantia nigra pc (Dopamine) Visual input

19. Basal Ganglia play a role in learning gaze patterns Striatum = caudate+putamen

20. Neural Substrate for Learning Neurons in substantia nigra pc (pars compacta) in basal ganglia release dopamine. These neurons signal expected reward. Neurons at all levels of saccadic eye movement circuitry are sensitive to reward. This provides the neural basis for learning gaze patterns in natural behavior.

21. Dopaminergic neurons in basal ganglia signal expected reward. (Schultz, 2000) Response to unexpected reward Increased firing for earlier or later reward Expected reward is absent. SNpc time

23. target selection signals to muscles inhibits SC saccade decision saccade command planning movements Neural Circuitry for Saccades Substantia nigra pc modulates caudate SNpr controls activity in SC

24. Neurons at all levels of saccadic eye movement circuitry are sensitive to reward. LIP: lateral intra-parietal cortex. Neurons involved in initiating a saccade to a particular location have a bigger response if reward is bigger or more likely SEF: supplementary eye fields FEF: frontal eye fields Caudate nucleus in basal ganglia SC: superior colliculus

25. Monkey makes a saccade to a stimulus - some directions are rewarded. Cells in caudate signal both saccade direction and expected reward. Hikosaka et al, 2000

26. To make a top-down system work, Subjects need to learn statistics of environmental events and distribute gaze/attention based on these expectations.

27. Walking -Real World Experimental question: Do subjects learn to deploy gaze in response to the probability of environmental events? General design: Subjects walked on an oval path and avoided pedestrians

28. Experimental Setup System components: Head mounted optics (76g), Color scene camera, Modified DVCR recorder, Eye Vision Software, PC Pentium 4, 2.8GHz processor A subject wearing the ASL Mobile Eye

29. Occasionally some pedestrians veered on a collision course with the subject (for approx. 1 sec) 3 types of pedestrians: Trial 1: Rogue pedestrian - always collides Safe pedestrian - never collides Unpredictable pedestrian - collides 50% of time Trail 2: Rogue Safe Safe Rogue Unpredictable - remains same Experimental Design (ctd)

30. Fixation on Collider

31. Effect of Collision Probability Probability of fixating increased with higher collision probability.

32. Learning to Adjust Gaze Changes in fixation behavior fairly fast, happen over 4-5 encounters (Fixations on Rogue get longer, on Safe shorter)

33. Detecting Collisions: pro-active or reactive? Probability of fixating risky pedestrian similar, whether or not he/she actually collides on that trial.

34. Shorter Latencies for Rogue Fixations Rogues are fixated earlier after they appear in the field of view. This change is also rapid.

35. Learning to Adjust Gaze Changes in fixation behavior fairly fast, happen over 4-5 encounters (Fixations on Rogue get longer, on Safe shorter)

36. Shorter Latencies for Rogue Fixations Rogues are fixated earlier after they appear in the field of view. This change is also rapid.

37. Effect of Behavioral Relevance Fixations on all pedestrians go down when pedestrians STOP instead of COLLIDING. STOPPING and COLLIDING should have comparable salience. Note the the Safe pedestrians behave identically in both conditions - only the Rogue changes behavior.

38. Fixation probability increases with probability of a collision. Fixation probability similar whether or not the pedestrian collides on that encounter. Changes in fixation behavior fairly rapid (fixations on Rogue get longer, and earlier, and on Safe shorter, and later)

39. Our Experiment: Allocation of gaze when walking. Where do we look when walking around a normal environment? Do we look where we are heading? Do we look at obstacles? Does the gaze pattern change if we repeat the path? Are we learning where things are, and does that change how much we have to look? More detailed questions: timing of gaze relative to avoidance. nature of obstacle