2. Visually-induced auditory spatial adaptation in monkeys and humans Norbert Kopčo, I-Fan Lin, Barbara Shinn-Cunningham, Jennifer Groh Center for Cognitive Neuroscience, Duke University Hearing Research Center, Boston University Technical University, Košice, Slovakia

3. 2Nov 6, 2007SFN 07 San Diego Introduction Visual stimuli can affect the perception of sound location e.g. the Ventriloquism Effect Way to go Red Sox! Way to go Red Sox! But does effect persist?

4. 3Nov 6, 2007SFN 07 San Diego Introduction Visual stimuli can affect the perception of sound location e.g. the Ventriloquist Effect Way to go Red Sox! But does effect persist? - barn owls: prism adaptation (Knudsen et al.) - monkeys: “ventriloquism aftereffect” (Woods and Recanzone, Curr. Biol. 2004)

5. 4Nov 6, 2007SFN 07 San Diego GOALS 1. Ventriloquism “aftereffect” in saccade task, in monkeys and humans? 1. - well-defined sensory-motor paradigm 2. - bridge to barn owl prism adaptation studies (on different time scale) 2. Reference frame of plasticity? 1. - Visual, auditory, or oculomotor reference frame?

6. 5Nov 6, 2007SFN 07 San Diego Methods Basic idea: 1. Pre-adaptation baseline: Measure auditory saccade accuracy 2. Adaptation phase: Present combined visual-auditory stimuli, with visual location shifted 3. Compare auditory saccade accuracy pre- and post-adaptation

7. 6Nov 6, 2007SFN 07 San Diego Methods Initial experiment: Does it work? Design: Monkey Pre-adaptation baseline – ~100 Auditory-only trials Adaptation phase – 80% V-A stimuli, visual stimulus shifted 6 deg. Left or Right 20% Auditory-only Compare Auditory-only trials from adaptation phase to pre- adaptation phase Sounds: Loudspeakers Visual stimuli: LEDs

8. 7Nov 6, 2007SFN 07 San Diego RESULTS

9. 8Nov 6, 2007SFN 07 San Diego RESULTS

10. 9Nov 6, 2007SFN 07 San Diego RESULTS

11. 10Nov 6, 2007SFN 07 San Diego RESULTS

12. 11Nov 6, 2007SFN 07 San Diego Methods: reference frame Eye-centered? Head (ear) -centered? Oculomotor? ? ?

13. 12Nov 6, 2007SFN 07 San Diego Method Audiovisual display Expected behavior Stimulus Location (°) Magnitude (°) Induce shift: - in only one region of space - from a single fixation point Test to see if shift generalizes to the same sub-region in: - head-centered space - eye-centered space Experiment divided into 1-hour blocks: (12 for humans,16 monkeys) Within a block, 3 types of randomly interleaved trials: - Training AV stim: 50% - Test A-only stimuli: 50% (25% from trained, 25% shifted FP)

14. 13Nov 6, 2007SFN 07 San Diego Results: Humans Audiovisual display Expected Responses FP LEDs Speakers Stimulus Location (°) Magnitude of Induced Shift (°) or Trained FP A-only responses: - Shift induced in trained sub-region - Generalization to untrained regions (asymmetrical) Shifted FP A-only responses: - Shift reduced in center region Head-centered representation, modulated by eye position

15. 14Nov 6, 2007SFN 07 San Diego Results: Monkeys Audiovisual display Expected Responses FP LEDs Speakers Stimulus Location (°) Magnitude of Induced Shift (°) or Trained FP A-only responses: - Shift in trained sub- region weaker - Generalization to untrained regions stronger (asymmetry oppo- site to humans) Shifted FP A-only responses: - Shift decreases on the right - Shift increases on the left Humans: Representation more mixed than in humans

16. 15Nov 6, 2007SFN 07 San Diego Summary The main results are consistent across species: Locally induced ventriloquist effect results in short-term adaptation, causing shifts in responses to A-only stimuli from trained sub-region. The induced shift generalizes outside the trained sub-region, with gradually decreasing strength (However, the pattern of generalization differs across the species) The pattern of induced shift changes as the eyes move. But, overall, it appears to be in a representation frame that is more head-centered than eye-centered.

17. 16Nov 6, 2007SFN 07 San Diego Discussion Posterior Parietal Cortex Neural adaptation could have been induced at several stages along the pathway (IC, MGB, AC, PPX, MC, SC). In humans, multiple effects observed at different temporal scales  likely adaptation at multiple stages Future work Extend Examine temporal and spatial factors influencing the eye-centered modulation. Look at other trained sub-regions. Midbrain Pons Cerebrum Thalamus Midbrain Pons Thalamus (Kandel, Schwartz, Jessel) and (Purves)

18. 17Nov 6, 2007SFN 07 San Diego Summary and discussion the main results are consistent across species: Locally induced ventriloquist effect results in short-term adaptation, causing shifts in responses to A-only stimuli from trained sub-region. The pattern of induced shift is modified as the eyes move. Bad news – there is a lot of differences between species: HumansMonkeys Representationhead-centered, eye-modulatedeye-centered Generalization to untrained sub-regionsmore on the side away from FPopposite Difference between hyper- and hypometric shiftsnoyes Representation when shift induced on side (data not shown)head-centered, no eye modulationeye-centered

19. 18Nov 6, 2007SFN 07 San Diego Results: Humans Audiovisual display Expected Responses FP LEDs Speakers Human Behavior Mean+SE Stimulus Location (°) Magnitude of Induced Shift (°)

20. 19Nov 6, 2007SFN 07 San Diego Results: Humans Stimulus Location (°) Audiovisual display Expected Responses FP LEDs Speakers Human Behavior Data collapsed across direction of induced shift Trained FP A-only responses: - Shift induced in trained sub-region - Generalization to untrained regions (asymmetrical) Shifted FP A-only responses: - Shift reduced in center region Head-centered repre- sentation, modulated by eye position Mean+SE

21. 20Nov 6, 2007SFN 07 San Diego Results: Humans vs. Monkeys Audiovisual display FP LEDs Speakers Human Behavior Monkey data (only hypometric) AV responses: - as expected Trained FP A-only responses: - Shift in trained sub- region weaker - Generalization to untrained regions stronger (asymmetry opposite to humans) Shifted FP A-only responses: - Shifted with eyes Representation more eye-centered Mean+SE Monkey Behavior Mean 6 6 -24 0 24 Mean+individuals Mean+SE Mean+individuals