1. Control of Attention and Gaze in Natural Environments

2. Selecting information from visual scenes What controls the selection process?

3. Humans must select a limited subset of the available information in the environment. Fundamental Constraints Acuity is spatially restricted. Attention is limited. Visual Working Memory is limited. Only a limited amount of information can be retained. What controls these processes?

8. Image properties eg contrast, edges, chromatic saliency can account for some fixations when viewing images of scenes. Saliency - bottom-up

9. Limitations of Saliency Models Important information may not be salient eg Stop signs in a cluttered environment. Salient information may not be important - eg retinal image transients from eye/body movements. Doesn’t account for many observed fixations, especially in natural behavior (eg Land etc).

10. Natural vision is not the same as viewing pictures. Behavioral goals determine what information is needed. Task structure (often) allows interpretation of role of fixations. Need to Study Natural Behavior

11. Foot placement Obstacle avoidance Heading Viewing pictures of scenes is different from acting within scenes. Top-down factors

12. To what extent is the selection of information from scenes determined by cognitive goals (ie top-down) and how much by the stimulus itself (ie salient regions - bottom-up effects)?

13. Modeling Top Down Control Virtual Humanoid has a small library of simple visual behaviors: –Sidewalk Following –Picking Up Blocks –Avoiding Obstacles Each behavior uses a limited, task-relevant selection of visual information from scene. This is computationally efficient. Walter the Virtual Humanoid Sprague & Ballard (2003)

14. Walter learns where/when to direct gaze using reinforcement learning algorithm. Walter’s sequence of fixations obstacles sidewalk litter

15. Walter the Virtual Humanoid Sprague & Ballard (VSS 2004) What about unexpected events?

16. Dynamic Environments

17. Computational load Unexpected events Bottom-up ExpensiveCan handle unexpected salient events Top-down EfficientHow to deal with unexpected events?

19. Reward weights estimated from human behavior using Inverse Reinforcement Learning - Rothkopf 2008. Human path Avatar path

20. Driving Simulator

21. Gaze distribution is very different for different tasks Time fixating Intersection.

22. The Problem Any selective perceptual system must choose the right visual computations, and when to carry them out. How do we deal with the unpredictability of the natural world? Answer - it’s not all that unpredictable and we’re really good at learning it.

23. Human Gaze Distribution when Walking Experimental Question: How sensitive are subjects to unexpected salient events? General Design: Subjects walked along a footpath in a virtual environment while avoiding pedestrians. Do subjects detect unexpected potential collisions?

24. Virtual Walking Environment Virtual Research V8 Head Mounted Display with 3 rd Tech HiBall Wide Area motion tracker V8 optics with ASL501 Video Based Eye Tracker (Left) and ASL 210 Limbus Tracker (Right) D&c emily Video Based Tracker Limbus Tracker

25. Virtual Environment Bird’s Eye view of the virtual walking environment. Monument

26. 1 - Normal Walking: Avoid the pedestrians while walking at a normal pace and staying on the sidewalk. 2 - Added Task: Identical to condition 1. However, the additional instruction of following a yellow pedestrian was given Normal walking Follow leader Experimental Protocol

27. Pedestrians’ paths Colliding pedestrian path What Happens to Gaze in Response to an Unexpected Salient Event? The Unexpected Event: Pedestrians on a non-colliding path changed onto a collision course for 1 second (10% frequency). Change occurs during a saccade. Does a potential collision evoke a fixation?

28. Fixation on Collider

29. No Fixation During Collider Period

30. Probability of Fixation During Collision Period Pedestrians’ paths Colliding pedestrian path More fixations on colliders in normal walking. No effect in Leader condition Controls Colliders Normal Walking Follow Leader

31. Small increase in probability of fixating the collider. Failure of collider to attract attention with an added task (following) suggests that detections result from top-down monitoring. Why are colliders fixated?

32. Detecting a Collider Changes Fixation Strategy Longer fixation on pedestrians following a detection of a collider “Miss”“Hit” Time fixating normal pedestrians following detection of a collider Normal Walking Follow Leader

33. To make a top-down system work, Subjects need to learn statistics of environmental events and distribute gaze/attention based on these expectations. Subjects rely on active search to detect potentially hazardous events like collisions, rather than reacting to bottom-up, looming signals.

34. Possible reservations… Perhaps looming robots not similar enough to real pedestrians to evoke a bottom-up response.

35. Walking -Real World Experimental question: Do subjects learn to deploy gaze in response to the probability of environmental events? General design: Subjects walked on an oval path and avoided pedestrians

36. Experimental Setup System components: Head mounted optics (76g), Color scene camera, Modified DVCR recorder, Eye Vision Software, PC Pentium 4, 2.8GHz processor A subject wearing the ASL Mobile Eye

37. Occasionally some pedestrians veered on a collision course with the subject (for approx. 1 sec) 3 types of pedestrians: Trial 1: Rogue pedestrian - always collides Safe pedestrian - never collides Unpredictable pedestrian - collides 50% of time Trail 2: Rogue Safe Safe Rogue Unpredictable - remains same Experimental Design (ctd)

38. Fixation on Collider

39. Effect of Collision Probability Probability of fixating increased with higher collision probability.

40. Detecting Collisions: pro-active or reactive? Probability of fixating risky pedestrian similar, whether or not he/she actually collides on that trial.

41. Learning to Adjust Gaze Changes in fixation behavior fairly fast, happen over 4-5 encounters (Fixations on Rogue get longer, on Safe shorter)

42. Shorter Latencies for Rogue Fixations Rogues are fixated earlier after they appear in the field of view. This change is also rapid.

43. Effect of Behavioral Relevance Fixations on all pedestrians go down when pedestrians STOP instead of COLLIDING. STOPPING and COLLIDING should have comparable salience. Note the the Safe pedestrians behave identically in both conditions - only the Rogue changes behavior.

44. Fixation probability increases with probability of a collision. Fixation probability similar whether or not the pedestrian collides on that encounter. Changes in fixation behavior fairly rapid (fixations on Rogue get longer, and earlier, and on Safe shorter, and later)

45. Our Experiment: Allocation of gaze when driving. Effect of task on gaze allocation. Does task affect ability to detect unexpected events? Drive along street with other cars and pedestrians. 2 instructions - drive normally or follow a lead car. Measure fixation patterns in the two conditions.

47. Subjects must learn the probabilistic structure of the world and allocate gaze accordingly. That is, gaze control is model-based. Subjects behave very similarly despite unconstrained environment and absence of instructions. Control of gaze is proactive, not reactive, and thus is model based. Anticipatory use of gaze is probably necessary for much visually guided behavior. Conclusions

48. Behaviors Compete for Gaze/ Attentional Resources The probability of fixation is lower for both Safe and Rogue pedestrians in both the Leader conditions than in the baseline condition. Note that all pedestrians are allocated fewer fixations, even the Safe ones.

49. Conclusions Data consistent with task-driven sampling of visual information rather than bottom up capture of attention - No effect of increased salience of collision event. - Colliders fail to attract gaze in the leader condition, suggesting the extra task interferes with detection. Observers rapidly learn to deploy visual attention based on environmental probabilities. Such learning is necessary in order to deploy gaze and attention effectively.

50. Certain stimuli thought to capture attention bottom-up (eg Theeuwes et al, 2001 etc ) Looming stimuli seem like good candidates for bottom-up attentional capture (Regan & Gray, 200; Franceroni & Simons,2003).

51. No Leader Normal WalkingFollow Leader Greater saliency of the unexpected event does not increase fixations. No effect of increased collider speed.

52. Other evidence for detection of colliders? Do subjects slow down during collider period? Subjects slow down, but only when they fixate collider. Implies fixation measures “detection”. Slowing is greater if not previously fixated. Consistent with peripheral monitoring of previously fixated pedestrians.

53. Conclusions Subjects learn the probabilities of events in the environment and distribute gaze accordingly The findings from the Leader manipulation support the claim that different tasks compete for attention

54. Effect of Context Probability of fixating Safe pedestrian higher in a context of a riskier environment

56. Summary Direct comparison between real and virtual collisions is difficult, but colliders are still not reliably fixated. Subjects appear to be sensitive to several parameters of the environment: –Experience Experience with the Rogue pedestrian elevated fixation probabilities of the Safe pedestrian to 70% (50% wto. exp.) Experience with the Safe lead to 80% fixation probability of the Rogue (89% wto. exp.) Experience of Safe carries less weight than the experience of Rogue

57. Time fixating Intersection. “Follow the car.” or “Follow the car and obey traffic rules.” CarRoadsideRoadIntersection Shinoda et al. (2001) Detection of signs at intersection results from frequent looks.

58. Intersection P = 1.0 Mid-block P = 0.3 Greater probability of detection in probable locations Suggests Ss learn where to attend/look. How well do human subjects detect unexpected events? Shinoda et al. (2001) Detection of briefly presented Stop signs.

59. What do Humans Do? Shinoda et al. (2001) found better detection of unexpected stop signs in a virtual driving task.