1. The auditory system Romain Brette (romain.brette@ens.fr) Romain Brette Ecole Normale Supérieure

2. What is sound?

3. Hearing vs. seeing HearingSeeing Acoustical waves, 20 – 20,000 Hz = 1.7 cm – 17 m Electromagnetic waves, 380-740 nm

4. Hearing vs. seeing HearingSeeing Acoustical waves, 20 – 20,000 Hz = 1.7 cm – 17 m Electromagnetic waves, 380-740 nm

5. Hearing vs. seeing HearingSeeing Acoustical waves, 20 – 20,000 Hz = 1.7 cm – 17 m Electromagnetic waves, 380-740 nm Information about volumesInformation about surfaces

6. Hearing vs. seeing HearingSeeing Acoustical waves, 20 – 20,000 Hz = 1.7 cm – 17 m Electromagnetic waves, 380-740 nm Information about volumesInformation about surfaces Sounds are produced by sourcesLight is reflected by sources

7. Hearing vs. seeing HearingSeeing Acoustical waves, 20 – 20,000 Hz = 1.7 cm – 17 m Electromagnetic waves, 380-740 nm Information about volumesInformation about surfaces Sounds are produced by sourcesLight is reflected by sources The source is transient, sounds are « events » The source is persistent, one can « look around » a visual object

8. Hearing vs. seeing HearingSeeing Sounds from different locations are mixed at the ear Light rays from different locations are separated in the eye

9. The information in sound Spatial location Vision: 1)Direction of an object is mapped to place on the retina. 2)Place on the retina varies systematically with self-generated movements. Hearing: 1)Direction is mapped to relationships between binaural signals, among other cues 2)Relationships vary systematically with self-generated movements, 3)but only if sounds are repeated More about this: http://briansimulator.org/category/romains-blog/what-is-sound/

10. The information in sound Shape Vision: the way the visual field changes with viewpoint determines the visual shape Hearing: the sound does not change with viewpoint. But: there is information about shape in the spectrum. Larger object => smaller frequencies (= change of space units). M. Kac (1966) Can one hear the shape of a drum? Am. Math. Monthly 73 (4) W.W. Gaver (1993) What in the world do we hear? Ecological Psychology 5(1) In speech: shape of the vocal tract is linguistic information

11. The information in sound Pitch In voiced vowels, the glottis opens and closes at a fast rate, producing a periodic sound (typically about 100 Hz for men, 200 Hz for women). Vowel ‘o’ Repetition rate contains information about intonation and speaker (used for grouping)

12. The information in sound Summary: what the auditory system needs to process - Precise temporal and intensity relationships between binaural signals - Frequency spectrum - Temporal information - More generally: spectro- temporal information at different scales  t*  f>1/2 (Gabor) The time-frequency trade-off:

13. Anatomy and physiology of the auditory system

14. The ear cochlea inner ear vestibular system (head movements) cochlea (hearing) outer ear middle ear inner ear

15. The basilar membrane

17. Hair cells outer hair cells inner hair cells auditory nerve tectorial membrane basilar membrane

18. Hair cells K+ channels open when the stereocilia is deflected

19. Auditory nerve fibers Tuning curves (threshold) Response curves

20. Phase locking Response to a tone (multiple trials): Time (ms) « Phase locking »: neurons fire at preferred phases of the input tone Phase

21. Phase locking (barn owl) Response to a tone (multiple trials): Time (ms) « Phase locking »: neurons fire at preferred phases of the input tone Phase Vector strength

22. Reverse correlation

23. A simple model of auditory nerve fibers bank of filters sound NB: does not capture nonlinear effects half-wave rectification (+ possibly low-pass filtering for decrease of phase-locking) + random spikes (Poisson)

24. MNTB ICC DNLL INLL VNLL DNLL INLL VNLL LNTB LSO MSO SPN MNTB LSO MSO DCN PVCNAVCN DCN AVCNPVCN DC SC SC LN MMGB DMGB VMGB SGN PFInsCAIIAIPFInsCAIIAI MNTB NCAT N.VIII MMGBDMGBVMGBSGN The rest of the auditory system

25. Sound localization: acoustical cues

26. 3D localization  = azimuth  = elevation (azimuth)

27. Acoustical cues for sound localization or head related impulse responses (time domain; HRIRs) Other cues for distance: level is distance-dependent high frequencies are more filtered with distance reverberation correlates with distance Other cues for elevation: pinna filters out specific frequencies depending on elevation (convolution)

28. HRTFs and HRIRs in the rabbit Kim et al., JARO (2010) HRIR HRTF

29. Interaural time differences (ITDs) distant sound source = plane wave Path length difference with spherical head: r(sin θ + θ ) This is valid when wavelength << head width Low frequencies: (3r/c)*sin θ Kuhn, JASA 62(1), 157-167 (1977) ITD: (r/c)(sin θ + θ ) (c=340 m/s) (Woodworth formula)

30. Frequency-dependence of ITDs

31. relevant range for ITDs different directions Maximum human ITD: about 700 µs in HF, up to 1000 µs in LF

32. ILDs for sinusoidal stimuli Adapted from Feddersen et al. (1957) Very small ILDs in low frequencies (for distant sources) Large ILDs at high frequencies and sources on the side (head shadowing)

33. Duplex theory  For low frequencies, ILDs are very small  For high frequencies, ITDs (for pure tones) are ambiguous, i.e., when wavelength<max. ITD  Duplex theory (Lord Rayleigh, 1907): ITDs are used at low frequencies, ILDs at high frequencies (threshold around 1500 Hz)  Confirmed with psychophysical experiments (using conflicting cues; Wightman & Kistler, 1992)

34. Monaural spectral cues Elevation (deg) The pinna introduces elevation-dependent spectral notches Hofman et al., Nature (1998)

35. Sound localization: anatomy and physiology

36. The first binaural structures The lateral superior olive Golgi stainings in cat by Ramon y Cajal, 1907 The medial superior olive In the superior olivary complex (SOC) in the brainstem: ILD-sensitive neurons ITD-sensitive neurons

37. ITD and ILD pathways (mammals)

38. Cochlear nucleus Bushy cells are more precise than auditory nerve fibers! Likely reason: averaging (several AN inputs/cell) + perhaps gap junctions

39. The medial superior olive (MSO) left right ITD Neuron responses consistent with cross-correlation of monaural inputs « best delay »

40. Cross-correlation, ITD and coincidence detection Two monaural signals:S L (t) S R (t) = a*S L (t-ITD) Cross-correlation: C(s) = Max. when s=ITD Coincidence rate between two Poisson processes = cross-correlation (at s=0)

41. The Jeffress model

42. ITD is encoded by the activation pattern of neurons with heterogeneous tunings (Movie by Tom Yin)

43. The Jeffress model Rate (Hz) « Best delay » = difference between monaural delays ITD is mapped to a pattern of neural activation delay lines

44. Theoretical appeal ITD =-0.3 ms Firing rate of cross-correlator neurons: best delaystimulus ITD Rate is max. when d = ITD, for any sound S: Estimators based on the Jeffress model: Peak coding Centroid estimator (Colburn/Stern)

45. Origin of internal delays Observed: greater delays at LFs

46. The hemispheric model

47. Testing the Jeffress model in small mammals Gerbil MSO (Day & Semple 2011) « natural » ITDs Observations in many species: 1)Contralateral bias 2)Best delay is inversely correlated with best frequency. 3)A number of large best delays « Best delay » = 400 µs For each neuron, one measures firing rate vs. ITD This looks like a contradiction of the place code hypothesis!

48. The hemispheric model of ITD processing Guinea pig In small mammals: best delay around ± π /4 Two-channel model: in each frequency band, 2 neural populations tuned at symmetrical best delays outside physiological range of ITDs. The relative activity indicates the ITD (ratio of activities, for level independence). (McAlpine et al., 2001; Harper & McAlpine, 2004)

49. Conceptual problems with the hemispheric model  ITD code is ambiguous at high frequency  ITD estimation is not robust to noise  ITD estimation is not robust to sound spectrum  Many BDs within the physiological range Sub-optimality of the hemispheric model: Brette R (2010) On the interpretation of sensitivity analyses of neural responses, JASA 128(5), 2965-2972.

50. The synchrony field model

51. Puzzling observations Gerbil MSO (Day & Semple 2011) For some cells, the « best delay » depends on input frequency. PUT A CELL CP CD Frequency Best phase For a pure delay: best phase (BP) = best delay (BD) * frequency (f) Linear regression: BP=CP+CD*f CP (cat IC) CD (ms) Not a pure delay! Not a pure phase!

52. ITDs in real life F R,F L = location-dependent acoustical filters (HRTFs/HRIRs) Delay: low frequency high frequency ITDs: FRONTBACK Frequency ITD (ms)

53. Binaural structure and synchrony receptive fields F R,F L = HRTFs/HRIRs (location-dependent) N A, N B = neural filters (e.g. basilar membrane filtering) input to neuron A: N A *F R *S (convolution) input to neuron B: N B *F L *S Synchrony when: N A *F R = N B *F L SRF(A,B) = set of filter pairs (F L,F R ) = set of source locations = spatial receptive field Independent of source signal S « Synchrony receptive field of (A,B) » Brette (2012), Computing with neural synchrony. PLOS Comp Biol

54. The hypothesis F R *SF L *S N A *F R *S N B *F L *S Each binaural neuron encodes an element of binaural structure

55. Experimental prediction Cells (cat IC) HRTFs Cells (IC) HRTFs Best phase of a neuron vs. frequency = Interaural phase difference vs. frequency for preferred source location PUT A CELL CP CD Best phase Input frequency (Hz)

56. Dendrites and coincidence detection

57. Dendrites of binaural neurons Mammalian MSO neurons Avian NL neurons

58. Coincidence detection with dendrites The problem: the neuron responds to both monaural and binaural coincidences With dendrites: the neuron is more ITD selective because it responds better to binaural coincidences. (Agmon-Snir et al., Nature 1998)

59. Mechanism E syn second spike less effective (current to proportional (E syn - V)) dendrite soma left dendrite right dendrite Monaural coincidence Binaural coincidence soma summation nonlinear effect

60. Binaural conductance threshold g Th = monaural conductance threshold

61. A simplified model With 3 compartments: Soma Dendrites (left, right)