1. "Theoretical and Experimental Approaches to Auditory and Visual Attention" Cold Spring Harbor April 20, 2008 Ervin Hafter and Anne-Marie Bonnel Department of Psychology U.C.Berkeley “A role for memory in Shared Attention”

2. Simplest definition of attention: a process inferred when responses on one task are affected by responding simultaneously to another

3. S(+) S(0) S(+) S( ) DetectionIdentification Energy detection model: compare stimuli in the signal epochs Two paradigms S(0) is the level of the standard.

4. Area under the ROC 1.00 Signal Levels in z B.50.70.60.90.80 1.52.01.00.5 0 2.5 A % correct in 2AFC Ideal detection based on energy in the signal epochs. Probability density A B S(0) S(+) S(0) λ λ z Predicts d’ B > d’ A

5. But, Macmillan (w/tonal pedestal) and Bonnel et al. (w/visual pededestal) found d’ detection > d’ identification

6. Stock Market National Debt University salaries Audio demo comparing detection to identification.

7. Both attributed this to use of Transients

8. Thinking that, unlike the case with energy, responses to transients might be pre-attentive, Bonnel et al. (1992) tested in a dual task with independent stimuli on side-by-side LEDs.

9. Thinking that, unlike the case with energy, responses to transients might be pre-attentive, Bonnel et al. (1992) tested in a dual task with independent stimuli on side-by-side LEDs. Detection was comparable to performance found with S instructed to attend to only one LED; identification showed a tradeoff indicative of a shared attentional resource.

10. Thinking that, unlike the case with energy, responses to transients might be pre-attentive, Bonnel et al. (1992) tested in a dual task with independent stimuli on side-by-side LEDs. Detection was comparable to performance found with S instructed to attend to only one LED; identification showed a tradeoff indicative of a shared attentional resource. Caveat: “Perceptual grouping” in identification

11. S(0) S(+) S(-) Visual Auditory 17 ms CRT 500 Hz Some changes in Berkeley

12. S(+) S(0) S(+) S( ) DetectionIdentification S( )

13. Ideal energy detection Probability density z λ λ A B S(0) S(+) S(0) S(+) S(0) S(-) 0 +1 +2-2 λ λ λ C C 1.00 Area under the ROC Signal Levels in z B.50.70.60.90.80 1.52.01.00.5 0 2.5 A % correct in 2AFC C C

14. Use the strictest possible criterion for asserting that there is no cost of shared attention: compare performance in the dual-task to that found in the separate single tasks.

15. Support for transient hypotheses: 1) detection > identification 2) No cost in detection 3) Tradeoff in accord with instructions in identification Attention-Operating Characteristic d’ audition d’ vision 3 2 1 3 21 0 50%,50% 20%,80% 50%,50% 20%,80% 80%,20% are single tasksand

16. SIAOC

17. These data clearly imply that detection of transience put no demand on shared attention, unlike discrimination of energy. SIAOC Sharing Index (Audition) Sharing Index (Vision).2.4.6.8 1.0.8.6.4.2

18. A more direct test of the idea that detecting transience (change per se) doesn’t require shared attention simply removes transients as information.

20. Often called a reminder Another classic ΔI/I A classic ΔI/I

21. d’ Same signal levels 1.0 2.0 3.0 4.0 200 4006008001000 1200 140016001800 Gap duration (ms) no gap Energy detection Auditory single task no gap gap

22. Dashes suggest absolute identification, i.e. comparisons to long-term, context-coded memory, rather than to a sensory trace of the reminder. 1.0 2.0 3.0 4.0 200 4006008001000 1200 140016001800 d’ Gap duration (ms) Same signal levels

23. 1.0.8.6.4.2.4.6.8 1.0 Sharing Index (Vision) SI-AOC “larger” “smaller” “different” “same” represent 500-msec signals.

24. Test Duration {D} Integration Time {IT} Single task D 1 = {D} {D} < {IT} d’ 1 A or V Impact of increased duration. e.g.,, on apparent cost of attention.

25. Test Duration {D} Integration Time {IT} Single task D 1 = {D} {D} < {IT} d’ 1 Dual task D 2 = ½ {D} A V d’ 2 =.707d’ 1 {D} < {IT} A or V

26. . d’ 1 Single task D 1 = {IT} {D} > {IT} Test Duration {D} Integration Time {IT} A or V

27. . d’ 1 Single task D 1 = {IT} {D} > {IT} Test Duration {D} Integration Time {IT}.707d’ 1 < d’ 2 ≤ d’ 1 Dual task ½{D} < {D 2 } < {IT} {D} > {IT} A or V

28. . d’ 1 Single task D 1 = {IT} {D} > {2IT} Test Duration {D} Integration Time {IT} A or V

29. . d’ 1 Single task D 1 = {IT} {D} > {2IT} Test Duration {D} Integration Time {IT} d’ 2 Dual task {D 2 } = {IT} {D} > {IT} d’ 2 *Here, time sharing produces the same result as no sharing. A or V

30. A still stronger test forces the subject to use context-coded memory by simply removing the reminder. “other” “standard” “large” “small”

31. It seems clear that responses based on context-coded memory were limited by sharing of an attentional resource..2.4.6.8 1.0.8.6.4.2 Sharing Index (Vision) Sharing Index (Audition)

32. Perhaps the reason that use of change per se did not provoke a cost of sharing is that it was done in sensory trace (?rehearsal?) memory?

33. Subjects can be forced to use sensory-trace memory by roving the standard from trial to trial. StandardTest ‘louder’ ‘softer’ Rove 1 3 2 4 Trial Correct response

34. dada Unlike the case with context-coding, performance fell as the ephemeral sensory-trace faded over time. Roved levels Fixed levels Auditory Identification 0.5 1.0 1.5 2.0 2.5 3.0 GAP Duration (sec) 12 3 4 56 7 8 50-ms reminders and signals *Need higher signals in roving

35. Most intriguing is that despite very poor performance, especially with long delays, there was no cost of sharing. 1.0.8.6.4.2.4.6.8 1.0 Sharing Index (Vision) Sharing Index (Audition) 8350 Gap (ms) 1022 510 310 rove

36. loud, dim, green, hot, salty, etc. Compare Context-coded, long-term memory Experience In typical, everyday life, we label sensory stimuli on the basis of comparisons to long-term memory. Test Sensory/ Neural

37. Compare Context-coded, long-term memory Experience Sensory/ Neural T1T1 T2T2 SOA Reminder When presented with an adjacent standard, the response may be to Test Sensory/ Neural loud, dim, green, hot, salty, etc.

38. Compare Context-coded, long-term memory Experience Test Sensory/ Neural Reminder Sensory/ Neural When presented with an adjacent standard, the response may be to simply ignore it, labeling the test in accord with long term memory. T1T1 T2T2 SOA loud, dim, green, hot, salty, etc.

39. Compare Context-coded, long-term memory Experience Test Sensory/ Neural Reminder Compare Sensory/ Neural Rehearsal Memory Without a reliable context-coded memory, S must compare the test to the reminder the in sensory trace memory. T1T1 T2T2 SOA loud, dim, green, hot, salty, etc. louder, dimmer, greener, hotter, saltier, etc..

40. Compare Context-coded, long-term memory Experience Test Sensory/ Neural “loud, dim, green, hot, salty, etc.” T1T1 T2T2 SOA Reminder Compare Our audio/video dual-task shows these comparisons to be independent, i.e., no cost of sharing. Conversely, these comparisons were limited by a shared attentional resource. Sensory/ Neural Rehearsal Memory louder, dimmer, greener, hotter, saltier, etc..

41. Okay, so comparisons to the sensory trace memory produced no cost of shared attention. What in the world is trace memory?

42. Okay, so comparisons to the sensory trace memory produced no cost of shared attention. What in the world is trace memory? Recently, we’ve approached this in terms of Weber’s Law. Bringing the lab up to 1834.

43. I = k I In signal detection terms, k can be treated as a multiplicative noise Weber’s Law

44. 3 1 -3 -5 -7 Threshold 10 log (ΔI/I) Pedestal (dB) 55 60657075 Identification Ped = Sig = 50 ms Gap = 1s Without Roving: performance based on based on long-term, labeled memory produce a constant Weber fraction

45. What happens when comparisons are to a roved standard? Thresholds go up. But in what way? To answer this, we parse the data in terms of the individual standards, analyzing performance separately for each pedestal level.

46. 3 1 -3-3 -5 -7 Threshold 10 log (ΔI/I ) Pedestal 55 6065 70 75 No Rove Labeled memory Rove Trace memory

47. 3 1 -3 -5 -7 Threshold (dB) Pedestal (dB) 55 60657075 multiplicative noise multiplicative + additive noise I = k I + c The change in slope implies a second, additive noise, c.

48. 6 0 -4 -6 -2 4 2 5560 657075

49. ΔI = kI + c What is c? Perhaps it is a decision noise associated with responding to the stimulus in trace memory? Maybe it is simply the result of decay in the trace that makes that adds noise to the remembered amplitude code.

50. Our next plan is to go into fMRI in search of sensory rehearsal. Wish us luck.