2. Visual-auditory interactions assessed with ABR Dr. W. David Hairston, Visual & Auditory Processes Branch Human Research and Engineering Directorate Army Research Laboratory

3. Overview Background - Who am I, ARL, etc - Background story The Plan - Paradigm - Analysis

4. Who is this guy? My background…. Exp Psych…. BS App State, MA Wake Forest U Multisensory target localization Neuroscience…. PhD (Neurobiology) Wake Forest U (Mark Wallace) Multisensory perceptual interactions in time and space Brain imaging…. Post doc (Radiology) Wake Forest U (Laurienti, Burdette, Maldjian) Cross-modal interactions with attention (music conductors) and timing (dyslexia) Now….Research fellowship w/ Army Research Lab (Letowski, McDowell) Cross-modal workload influence on auditory processing No experience in audiology

5. ARL? US Army Research, Development, and engineering Command Human Research and Development Visual and Auditory Processes Branch Auditory Research Team Officially – Oak Ridge Associated Universities Army Research Lab

6. Interests… My Interests: Visual-Auditory integration Multisensory interactions – how processing in one sense affects others… not just in good ways… (e.g. dark ugly downside to cross-modal interactions) Role of experience – can focused training not only hone sense(s), but also affect how they interact with each other? (e.g. can detrimental interactions be avoided?) ARL’s interests… Application – how can we use multisensory research to enhance US Soldier? (e.g., can specific types of training enhance sensory awareness? Or, can detrimental affects be overcome?) Programatic – How can neuroscientific methods be easily integrated into other ARL studies of human perception (e.g., sound localization, human factors, etc)

7. Research Background Multisensory interactions…. Many studied effects from multisensory stimulation Classically, examined as V+A vs V or A…. “Is 2 better than 1?” Lots of good stuff - Behaviorally: faster response times, higher accuracy, better memory - Neurophysiologically: bigger responses, decreased latency, lower S/N, etc. (even in this lab – enhanced FFR, cortical response slope) But what about cases where you don’t WANT integration? Not everything we do is multisensory…

8. Research Background Cross-modal inhibition…. Sometimes you really need to focus on just one sense… Reading a book in a noisy room Talking on phone (not while driving!) “Cross-modal attention” ( hate that word ) shift mental resources from one sense to another 2.78 7.0 -2.78 -7.0 Deactivation of auditory cortex during visual task

9. Research Background Own recent work… role of task difficulty 440 Hz 660 Hz 440 Hz660 Hz Easier Difficult 60 ms 20 ms Temporal Discrimination Subjects’ thresholds for each task acquired prior to fMRI scanning

10. Non-musicians: Moderate Deactivation of visual cortex +/-3.95 +/-8.0 Vis Response

11. Non-musicians: Difficult Robust deactivation of visual cortex +/-3.95 +/-8.0 Vis Response

12. Non-musicians vs. Conductors Difference seen when task is difficult n.s. * * Highly trained concert conductors (>7 yrs experience) do NOT show this trend Suggests the inhibition can be mediated by experience/training

13. Research Background So…. What about the brainstem? Begs the question… does cross-modal inhibition affect most basic, incoming auditory processing? We know that the ABR is affected by multisensory processing – (enhancement) and this is tied to cortical enhancement (Thanks Auditory Neuroscience Lab!) … but what about suppression? Increased FFR amplitude w/ Auditory attention vs visual (Galbraith) -but unclear if this is aud enhancement or visual suppression (no no-stim baseline)

14. Question at hand… Does cross-modal task difficulty directly affect ABR-level processing? What are the implications at the level of both the brainstem and cortical response?

15. The Plan Use ABR-based recording techniques while subjects perform visual, auditory, and baseline (no) tasks at different levels of difficulty

16. The Plan ABR Stim Oddball Stim: 220 Hz tone, 100 ms duration 400 ms ISI Infrequent oddball (5%?) 235 Hz ABR Task Jittered, 4 sec ave ISI between stim

17. The Plan Visual Auditory X 2 = Easy + Hard No task

18. Tasks? Tasks…? Temporal discrimination N-Back Motion detection Must be similar in both modalities, and manipulated to “easy” and “hard” levels 440 Hz 660 Hz 440 Hz660 Hz Easier Difficult 60 ms 20 ms

19. Analysis Interested in FFR of ABR – shown to be most affected by VA-interactions, cognitive influence etc Also cortical (MMN) responses associated with oddballs, and correlation with FFR Ideally limit # of trials to minimum – save time, more practical

20. Analysis Typical FFR… Trial 1Trial XMean Signal += FFT of Mean Advtantage: Proven method, have full data, derive timing (onset), signal amplitude and FFT amplitude Disadvantage: requires many trials for waveform to average out, requires pefect time-lock response; time consuming

21. Analysis Fourier-Average FFR…. Trial 1Trial X + = Mean FFT Advantage: Many fewer trials needed; no need for perfect phase locking – saves time! Disadvantage: Loss of temporal structure; computationally intensive Option 1 – FFT every trial, average over 1000 trials Option 2 – averge clusters (10?), average over clusters (100) MMN/cortical – standard averaging technique

22. Expectations… 1.(I learn how to do ABR) 2.Larger FFR response in A- task than V 3.Decreased V (suppression) relative to baseline 4.Effect (3) most pronounce in difficult task (e.g., minimal/no diff in easy task) 5.Latency of MMN correlated with FFR modulation 6.(eventually) – Effects (4,5) diminished with training, or specific kind of expertise Alternative: cross-modal effect may be in baseline noise shift (SNR) rather than amplitude change Equally enticing!

23. Done! Open to ideas! Unresolved: Ideal tasks Ideal ABR stim (pure vs complex tone) How many trials Individual vs cluster FFT Min effective sample rate (data reduction)