Connecting Sound with the Mind’s Eye: Multisensory Interactions in Music Conductors W. David Hairston, Ph.D Advanced Neuroscience Imaging Research Lab Department of Radiology; Wake Forest University School of Medicine, Winston-Salem, NC 27157

Multisensory enhancement within a number of paradigms, including: Simple reaction times Saccadic response latencies Signal detection Orientation/localization behavior in cat General conditions for enhancements to be observed -spatial alignment -temporal congruence -minimal efficacy (“inverse effectiveness”)

Stimuli: Vis (LED) Aud (broadband) Vis-Aud 50 ms, 30 reps What about human localization ability Hairston et al, J. Cog. Neuro, 2003

Evaluating Precision Precision ~ Std Dev

Hmmmm…. Why? The total amount of gain/ enhancement observed is determined by the relative contribution of each sense, based upon its own perceptual acuity How do you increase auditory acuity?

Conductors as auditory localizers Daily, career experience requires rapid and accurate assessment of auditory scene Also requires coordination of multisensory information – “who” played “what” wrong note, etc 20 conductors, min 7 years “podium experience” (ave 10.2) Matched on age, education, sex, etc (Hodges, Hairston & Burdette ’06) A#A# A

Visual and multisensory performance very similar Non-musicians

Multisensory performance enhanced Improved auditory performance

Conclusions from this… While untrained subjects gain little from an additional auditory cue, music conductors appear to benefit from additional auditory signals, for which their spatial acuity is much better. The degree of multisensory gain in this unique population appears to be tied to their improved auditory performance These results suggest that the specialized training and experience of these individuals has a profound affect within both the auditory and multisensory realms

But what if… Multisensory “enhancements” are beneficial when the task can make use of additional information BUT - When do not match or are not relevant – can be detrimental or inhibiting…. -Slower RT -Biased localization -Illusions and misperceptions Many times the task at hand required focusing on one sense and ignoring other.

“Tuning out” irrelevant information…

Current studies… Physiological effects of “tuning in” to one sense over another “Cross-modal deactivation”: Decreased activity within one sensory cortex related to stimulation of another E.g., decreased BOLD signal in occipital cortex during auditory task Some evidence for relation to selective attention Unclear whether the extent of deactivation observed is related to the difficulty of the task at hand

Methods Goals: Does changing task difficulty affect cross-modal deactivation? Does extreme acuity within one modality, and unique multisensory training affect this process? Subjects - Non-musicians – various careers, no formal musical training, y/o - Conductors – min. 5 yrs podium experience Matched on age, gender, education, etc Why conductors? - Heightened auditory acuity - Conducting activities require NOT inhibiting other (e.g., visual) information

Methods: Tasks Pitch Discrimination 440 Hz 660 Hz 440 Hz660 Hz Easier Difficult 60 ms 20 ms Temporal Discrimination 440 Hz456 Hz 440 Hz443 Hz Easier Difficult (Time) BaseTest Subjects’ thresholds for each task acquired prior to fMRI scanning

Methods: Thresholding Acquired threshold Allows compensation for variability in perceptual acuity between subjects and groups 2 down/1 up rule

Perceptual acuity, non-musicians vs. conductors A A# Base comparison Conductors Non-Musicians Non-musicians need a significantly larger difference to discriminate tones than conductors.

Methods: fMRI Each task performed at 2 levels - at threshold (“difficult”) - above threshold (“moderate”) Also performed visual temporal discrimination (2 circles) Visual Temporal discrimination silenceTR (Scanner ON) (Scanner OFF) Trial Sparse sampling (10 s pause), block design

Schmuel et al, Nature Neuroscience, 2006 Positive BOLD

Analysis Activity in task (ON) blocks contrasted against resting baseline (OFF) Baseline: eyes open on fixation, no task ROI of visual-responsive occipital cortex Used to generate summary stats

Non-musicians: Moderate Deactivation of visual cortex +/ /-8.0 Vis ROI

Non-musicians: Difficult Robust deactivation of visual cortex +/ /-8.0 Vis ROI

Non-musicians: Difficult vs. Moderate Easier Difficult

Non-musicians: stats ROI A.Cort. Post. Cing. V Cort. ROI A.Cort. Post. Cing. V Cort. # significant voxels Total Signal Magnitude Mean signal

Conductors DifficultModerate Vis ROI +/ /-8.0 Only slight deactivation of visual cortex in both cases

Conductors: Difficult vs. Moderate Easier Difficult ROI A.Cort. Post. Cing. V Cort. # significant voxels Total Signal Magnitude

Non-musicians vs. Conductors Difference seen when task is difficult Non- Musicians Conductors n.s. * * Difficult Task

Conclusions The degree and extent of cross-modal deactivation observed specifically depend of the task difficulty; when the same task is easier, cross-modal deactivations are attenuated. Contrary to non-musicians, conductors show only minor cross-modal deactivation, even when the task is very difficult to perform. This suggests that the role of functional deactivations is dynamic, and adapts to fit the needs of the individual and situation at hand.

Conductors Conductors are highly-trained individuals with unique daily activties and specialization This experience leads to not only high auditory acuity, but altered interactions when dealing with multiple senses Two theories: - High auditory acuity negates the need for visual suppression. - Daily experience has led to familiarity with completing complex auditory tasks while also monitoring visual information.