1. Dynamic attention and predictive tracking Todd S. Horowitz Visual Attention Laboratory Brigham & Womens Hospital Harvard Medical School Lomonosov Moscow State University Cognitive Seminar, 6/10/2004

2. lab photo Jeremy Wolfe David Fencsik George Alvarez Sarah Klieger Randy Birnkrant Jennifer DiMase Helga Arsenio Linda Tran (not pictured)

3. Multi-element visual tracking task (MVT) Devised by Pylyshyn & Storm (1988) Method for studying attention to dynamic objects

4. Multi-element visual tracking task (MVT) Present several (8-10) identical objects Cue a subset (4-5) as targets All objects move independently for several seconds Observers asked to indicate which objects were cued

5. Demo mvt4 demo

6. Interesting facts about MVT Can track 4-5 objects (Pylyshyn & Storm, 1988) Tracking survives occlusion (Scholl & Pylyshyn, 1999) Involves parietal cortex (Culham, et al, 1998) Clues to objecthood - Scholl

7. Accounts of MVT performance FINSTs (Pylyshyn, 1989) Virtual polygons (Yantis, 1992) Object files (Kahneman & Treisman, 1984) Object-based attention

8. These are all (partially) wrong FINSTs (Pylyshyn, 1989) Virtual polygons (Yantis, 1992) Object files (Kahneman & Treisman, 1984) Object-based attention

9. Common assumptions Low level (1st order) motion system updates higher-level representation –FINST –Object file –Virtual polygon Continuous computation in the present

10. Overview MVT and attention Tracking across the gap Tracking trajectories

11. MVT and attention Clearly a limited-capacity resource Attentional priority to tracked items (Sears & Pylyshyn) Hypothesis: MVT is mutually exclusive with other attentional tasks George Alvarez, Helga Arsenio, Jennifer DiMase, Jeremy Wolfe

12. MVT and attention Clearly a limited-capacity resource Attentional priority to tracked items (Sears & Pylyshyn) Hypothesis: MVT is mutually exclusive with visual search

13. MVT and attention Clearly a limited-capacity resource Attentional priority to tracked items (Sears & Pylyshyn) Hypothesis: MVT is mutually exclusive with visual search Method: Attentional Operating Characteristic (AOC)

14. AOC Theory

15. General methods - normalization Single task = 100 Chance = 0 Dual task performance scaled to distance between single task performance and chance

16. General methods - staircases Up step (following error) = 2 x down step Asymptote = 66.7% accuracy Staircase runs until 20 reversals Asymptote computed on last 10 reversals

17. General methods - tracking 10 disks 5 disks cued Speed = 9°/s

18. AOC Theory

19. AOC reality Tasks can interfere at multiple levels Interference can occur even when resource of interest (here visual attention) is not shared How independent are two attention- demanding tasks which do not share visual attention resources?

20. Gold standard: tracking vs. tone detection

21. Gold standard method Tracking –Duration = 6 s Tone duration –10 600 Hz tones –Onset t = 1 s –ITI = 400 ms –Distractor duration = 200 ms –Task: target tone longer or shorter? –Target duration staircased ( 31 ms) –Dual task priority varied N = 10

22. Gold standard AOC

23. Tracking + search method Tracking –Duration = 5 s Search –2AFC E vs. N –Distractors = rest of alphabet –Set size = 5 –Duration staircased (mean = 156 ms) –Onset = 2 s N = 9

24. Tracking + search method

25. Tracking + search AOC

27. Does tracked status matter? T L L L T L

28. method Tracking –Duration = 3 s Search –2AFC left- or right-pointing T –Distractors = rotated Ls –Set size = 5 –Duration staircased (mean = 218 ms) –Onset = 1 s N = 9

29. search inside tracked set T L T L L L L

30. search outside tracked set T L T L L L L

31. mixedblocked search inside tracked set search outside tracked set

32. inside vs. outside AOC

33. Does spatial separation matter? E F V H P

34. method Tracking –Duration = 5 s Search –2AFC E vs. N –Distractors = rest of alphabet –Set size = 5 –Duration = 200 ms –Onset = 2 s N = 9

35. spatial separation AOC

36. search v track summary

37. MVT and search Clearly not mutually exclusive Not pure independence Close to gold standard MVT and search use independent resources?

38. Two explanations Separate attention mechanisms Time sharing

39. Predictions of time sharing hypothesis Should be able to leave tracking task for significant periods with no loss of performance Should be able to do something in that interval

40. Track across the gap method

41. Track 4 of 8 disks Speed = 6°/s Blank interval onset = 1, 2, or 3 s Trajectory variability: 0°, 15°, 30°, or 45° every 20 ms Blank interval duration staircased (dv) N = 11

42. track across the gap asymptotes

43. Predictions of time sharing hypothesis Should be able to leave tracking task for significant periods with no loss of performance (see also Yin & Thornton, 1999) - confirmed Should be able to do something (e.g. search) in that interval

44. search during gap method AOC method Tracking task same as before Search task in blank interval –Target = rotated T –Distractors = rotated Ls –Set size = 8 –4AFC: Report orientation of T Duration of search task staircased (326 ms)

45. search during gap AOC

47. Predictions of time sharing hypothesis Should be able to leave tracking task for significant periods of time with no loss of performance (see also Yin & Thornton, 1999) - confirmed Should be able to do something (e.g. search) in that interval - confirmed

48. Summary MVT and visual search can be performed independently in the same trial May support independent visual attention mechanisms May support time-sharing

49. Summary Tracking across the gap data support time sharing Tracking across the gap data raise new questions

50. What is the mechanism? Not a continuous computation in the present Not first order motion mechanisms Not apparent motion Randall Birnkrant, Jennifer DiMase, Sarah Klieger, Linda Tran, Jeremy Wolfe

51. None of these theories fit FINSTs (Pylyshyn, 1989) Virtual polygons (Yantis, 1992) Object files (Kahneman & Treisman, 1984)

52. What is the mechanism? Some sort of amodal perception? (e.g. tracking behind occluders, Scholl & Pylyshyn, 1999) … but there are no occlusion cues!

55. Scholl & Pylyshyn, 1999

56. Maybe the gap is just an impoverished occlusion stimulus No occlusion/disocclusion cues Synchronous disappearance

57. Predictions of impoverished occlusion hypothesis Occlusion cues will improve performance Asynchronous disappearance will improve performance

58. Method Track for 5 s Speed = 12°/s Track 4 of 10 disks Independent variables (blocked) –Gap duration:107 ms, 307 ms, 507 ms –Occlusion cues absent, present –Disappearances synchronous, asynchronous N = 15

59. synchronous disappearance all items reappear simultaneously items invisible but continue to move

60. synchronous disappearance + occlusion occlusion begins disocclusion begins

61. Occlusion/Disocclusion

62. asynchronous disappearance item reappears one item at a time disappears but continues to move

63. asynchronous disappearance + occlusion... moves while invisible...... then disoccludes one item at a time begins to be occluded...

64. comparing cue types

65. Occlusion hypothesis fails Occlusion cues dont help Asynchronous disappearance doesnt help

66. Method Track for 5 s Speed = 12°/s Synchronous condition only Independent variables (blocked) –Gap duration:107 ms, 307 ms, 507 ms –Occlusion cues absent, present –Track 4, 5, or 6 of 10 disks N = 11

67. comparing cue types

68. Occlusion hypothesis fails Occlusion cues dont help Occlusion cues can actually harm performance Asynchronous disappearance doesnt help

69. What is the mechanism? Not a continuous computation in the present Not first order motion mechanisms Not apparent motion Not amodal perception (occlusion)

70. How do we reacquire targets? remember last location (backward) store trajectory (forward) David Fencsik, Sarah Klieger, Jeremy Wolfe

71. location-matching account Memorized pre-gap target location. Nearest to memorized location: identified as target. First Post-Gap Frame

72. trajectory-matching account Memorized pre-gap target trajectory. On target trajectory: identified as target. First Post-Gap Frame

73. Shifting post-gap location 0 Last visible pre-gap location opposite of expected location Expected post-gap location +1 = Stimulus trajectory

74. shifting post-gap location predictions

75. Shifting post-gap location methods track for 5 s speed = 8°/s track 5 of 10 disks gap duration = 300 ms post-gap location condition blocked stimuli continue to move after gap

76. shifting post-gap location

77. Location vs. trajectory-matching support for location-matching –see also Keane & Pylyshyn 2003; 2004 but advantage for -1 is suspicious

78. Location vs. trajectory-matching time +1.0 + time +2.0 + + time +1.5

79. shift & stop methods track for 4-6 s speed = 9°/s track 2 or 5 of 10 disks gap duration = 300 ms post-gap location condition blocked stimuli stop after gap

80. moving vs. static after gap

82. 2 vs. 5 targets

83. Location vs. trajectory-matching support for location-matching However... –conditions are blocked –observers might see their task not as tracking across the gap, but learning which condition theyre in –might not tell us about normal target recovery

84. Location vs. trajectory-matching can subjects use trajectory information? always have items move during gap vary whether trajectory information is available or not

85. moving condition invisible motion

86. static condition invisible motion

87. manipulate pre-gap information methods track for 4 s speed = 9°/s track 1 to 4 of 10 disks gap duration = 300 ms

88. manipulate pre-gap information

90. Location vs. trajectory-matching observers can use trajectory information unlimited (or at least > 4) capacity for locations smaller (1 or 2) capacity for trajectories

91. Conclusions Flexible attention system allows rapid switching between MVT and other attention-demanding tasks Some representation allows recovery of tracked targets after 300-400 ms gaps This representation includes location and trajectory information

92. Speculation MVT reveals two mechanisms, rather than just one Frequently (but perhaps not continuously) updated location store Attention to trajectories