1. Hearing

2. Human ear as it is. It is quite similar to ears of other mammals
Human ear as it is. It is quite similar to ears of other mammals. Its outside part is called the outer ear, which consists of pinna (auricle), the auditory canal (meatus) and ends at the eardrum (tympanum). The middle ear consistye of a set of three bones 1) the hammer (malleus), 2) the advil (incus), and 3) the stirrup (stapes). The inner ear starts at the oval window that touches the stirrup, and consist of an intricate structure calld the cochlea.

3. outer ear eardrum middle ear stirrup anvil hammer inner ear
basilar membrane
tectorial membrane
hairs
oval
window
round
For engineers, it may be easier to look at this “engineering schematic form” of the ear.

4. Mechanical protection of the middle ear
outer ear
eardrum
middle
ear
inner ear
Outer ear:
Mechanical protection of the middle ear
Diffracts and focuses sound waves (pinna)
The ear canal acts as a resonator (3-5 kHz enhancement)
The end of the canal has an eardrum which vibrates with sound
The outer ear protects the rest of the ear, and focuses the sound. It also amplifies mid frequencies (since it functions as a so called quarter-wave resonator). It ends at the eardrum.

5. Characteristic acoustic impedance of a tube filled with gas or fluid
Z0= rc/A,
r-the density of the medium
c-the velocity of sound
A-cross-sectional area of the tube
air outside
salty liquid (cochlear fluid) inside
inner ear
middle
ear
anvil
hammer
outer ear
eardrum
stirrup
Middle ear:
Converts impedance of the air to the impedance of the cochlear liquid
ZAIR:ZLIQ = 1: % loss of energy if no impedance match
Protects inner ear
Reactions to intense sounds (but rather slow ms reaction time)
Low-pass filter 15 dB/oct from 1 kHz
The inner side of the eardrum connects to a set of three bones that form a mechanical system, which converts movements of the eardrum (that reflects changes in acoustic pressure, which represent the sound) to movements of the cochlear liquid that fills the inner ear. However, since the acoustic impedance of the air is very different from the acoustic impedance of the cochlear fluid (the impedance is directly proportional to the density of the medium and the speech of the sound propagation in the medium, both of which are much higher in the liquid), the three-bone mechanical system together with the eardrum and the oval window forms a mechanical transformer, which does the appropriate match between the two acoustic impedances. Without this, most of the energy would get reflected at the air-liquid boundary and would never make it into the inner ear. As a result of all of this, the liquid in the inner ear moves with the same frequency as the air outside.
The middle ear has yet another function – it can protect the fine mechanism of the inner ear from sounds with very high intensities by reducing the efficiency of the acoustic transformation in the middle ear. However, this reflex, which quite adequate to protect from most of naturally-occurring sounds, is relatively slow for many artificially made intense sounds (such as in amplified music).

6. Mechanical frequency analysis of the incoming sound
inner ear
middle
ear
outer ear
Cochlea
oval window
0rgan of Corti
basilar membrane
tectorial membrane
hairs
round window
Inner ear:
Mechanical frequency analysis of the incoming sound
Converts mechanical movements to electrical pulses
The function of the inner ear is to analyze the sound in a way that is quite consistent with the Fourier analysis, which we have been discussing earlier. Further, it also converts the mechanical movements that carry the sound to a code that would be understood by the rest of the perceptual system. Without going into too many details, it happens in the following way: Inside the cochlea (which is actually curled as shown in the left) is several membranes, the most important from our point of view being the basilar membrane. Movements of the cochlear fluid cause the basilar membrane to move with the frequency of the incoming sound.
Changes in acoustic pressure => movement of bones in middle ear
=> movement of membrane on oval window => vibrations in the cochlear liquid
=> vibrations of basilar membrane

7. Basilar membrane as a mechanical frequency analyzer
0.05 mm
0.5 mm
stiff
basal end
pliable
apical
end
500 Hz
100 Hz
The basilar membrane is much stiffer and narrower at one end ( at the entrance to the cochlea near the oval window) and much more pliable and wider at its other end. As it vibrates, it forms a traveling wave. Due to mechanical properties of the basilar membrane, this wave forms a maximum at a certain point along the length of the membrane and then it dies of. The position of this maximum depends on a frequency of the incoming sound. For low frequencies, whole basilar membrane is active, for high frequencies only the stiff part of the membrane is active.

8. Cochlea as frequency analyzer

9. How selective is the basilar membrane ?
Frequency response
input
system
output
Movement of the basilar membrane in dead animal observed by a microscope
von Bekesy 1960
Ratio of output to input
output/input
frequency
It is possible to find out how selective if the basilar membrane (von Bekesy got a Nobel Prize for this). He observed by a microscope the amplitude of a displacement of a point on a basilar membrane (in dead ear) while changing a frequency of the incoming sound. Each point on the basilar membrane acts as a band-pass filter. The filters re asymmetric, the low frequency slopes are less steep than the high frequency slopes. This is because for low frequencies, whole basilar membrane is active, for high frequencies only the stiff part of the membrane is active.

10. Selectivity very different after the death of the animal!
Cochlea is most likely an active system with a positive feedback loop that accounts for the high cochlear sensitivity.
“fresh” animal
“tired” animal
dead animal
(von Bekesy)
Von Bekesy was not exactly correct. New methods of measurement of basilar membrane movements allow for measuring the basilar membrane action in live animal. Selectivity of the basilar membrane is much higher in the live animal.

11. small piece of radioactive material glued on basilar membrane
Doppler shift in emitted g-rays indicates amplitude of the membrane vibrations
Also, the selectivity significantly changes with the intensity of the incoming sound. For high intensities, the selectivity is lower than for the weaker sounds. This suggests a nonlinear system of the active hearing, which is capable to adjust its properties depending on the situation. It seems likely that some feedback from higher levels of hearing is in the action. We will talk about this later.
Nonlinear system!
(curves vary with intensity)

12. Code for the brain Sensory neurons produce spikes
Spike rate increases with an increase in the stimulus intensity (here it was a weight on a muscle)
Adaptation: after a while, the firing rate decreases even when the stimulus intensity stays the same
So we have shown (unfortunately von Bekesy did it first ) that the mechanical properties of a cochlea allow for frequency analysis of an incoming sound (because the cochlea behaves like a band of active nonlinear band-pass filters). How do we let a brain know about results of this mechanical frequency analysis? To understand this, we need to go back to early days of electronics (Adrian 1926). The invention of electronic amplifier allowed for an amplification if weak signals from the neural system. An electrod, inserted in a sensory neuron picked electric pulses, whose rate dependend on the intensity of a stimulus (higher the intensity, higher the spiking rate).

13. Action potential in a brain cell of a fly exposed to visual scenes
time [ms]
150
Shapes of five individual action potential (spikes)
Coding of the information in a perceptual system is done through spikes of called “action potentials”. Cells throughout the neural system generate actions potentials in response to outside stimuli. This picture shows what is happening in a cell of fly’s brain when it is exposed to an outside scene. The spike’s details are shown in the right part of the picture. The spikes are very similar to each other.

14. Stimulus at t=0 (sudden change of the scene that fly sees)
Spikes happen in a response to outside stimuli. The upper figure shows firing of spikes in the fly’s brain when the scene which fly sees suddenly change (at the time t=0). Each line represents one trial. The change in the scene induces rapid spike firing, which gradually goes back to the normal (background) steady low-density firing (after several hundreds of ms), as the fly gets used to the new scene. This illustrates one very important principle, that we will see throughout the perceptual system – change in the stimulus generates most vigorous response, static stimuli are not of too much interest to perception.

15. From movements to electrical pulses
inner ear
middle
ear
outer ear
organ of Corti
The basilar membrane contains ~15,000-20,000 hair cells (sensory cells)
Inner hair cells transduce vibration into electrical signal and send them to the brain
Outer hair cells receive signals from the brain, which could change mechanical properties of the organ of Corti
Now we are ready to discuss how the ear tells the brain what it hears. The picture shows a cut through so called “organ of Corti”. This organ consists of two membranes, the basilar membrane (which is frequency selective as we have discussed earlier), and the tectorial membrane. Hairs are growing from the basilar membrane and into the tectorial membrane. As the basilar membrane moves, the hairs bend. The inner hair cells emit spikes of action potential whenever the hair bends of one direction. There is no spike when the hair moves in the other direction – the hair cells do one-way rectification of the acoustic signal. These spikes then go into higher levels of the hearing system and after further processing the information from the ear reaches the brain.
An interesting thing is that the brain also appear to have means to talk to the ear! The so called “outer hair cells” receive spikes from the brain. They appear to change mechanical properties of the organ of Corti in response to the information received from the brain. Thus, the hearing seems to provide for a feedback that could adapt the ear to a changing acoustic input. There is a strong evidence for this top-down flow of information in a form of so called “otoacoustic emissions”. When a sensitive microphone is placed in a vicinity of the eardrum, it can pick up a delayed echo (by several tens of ms) of the sound that entered the ear.

16. basilar membrane movements => bending of hair cells => electrical pulses
~ 40 hairs/cell
~ 140 hairs/cell
tectorial
membrane
tunnel of corti
basilar
membrane
inner
hair cells
outer hair cells
This a schematic summary of what is happening in the organ of Corti.
auditory nerve
fiber
auditory nerve
fiber
inner hair cells – information
outer hair cells – govern cochlear mechanics ?

17. in out one-way rectifierin
out
one-way
rectifier
Intracellular voltage as a function of stimulus pressure (600 Hz sinusoid)
inner hair cell
outer hair cell
electrode
It is possible to insert an electrode into a haircell and measure changes in electric potential within the haircell. Inner haircells are more sensitive to positive excursions of the pressure, outer haircells are more sensitive to the negative excursions. Both haircells are more sensitive to one direction of their bending, than the opposite one.

18. Intracellular voltage changes in an inner hair cell for different frequencies of stimulation
electrode
electrode ?
When measuring intracellular voltage within the hair cell while the system is stimulated by sinusoidal signals, for low frequencies the voltage appears to copy the signal. As the frequency of the stimulation increases, the AC signal starts to be superposed on a DC component. For frequencies of the stimulus above 3-5 kHz, the AC component more or less disappears and the response is mainly DC.
This is true for the intracellular recordings. The situation changes when the recording is done from the auditory nerve. In this case, the response is dominated by spikes, which are for frequencies below 5 kHz, in phase with the signal. The spikes are emitted only during one ha;f-cycle of the signal, thus the system effectively performs a one-way rectification.

19. Spikes on the auditory nerve are in phase with the signal
Only in one half of the cycle
One-way rectification
Period histogram
where the spike appears with respect to the waveform
The one-way rectification character of the spiking on the auditory nerve is clearly demonstrated on a period histogram. Such histogram is collected in such a way, that at every cycle of the stimulus, the time axis is re-set at a constant point on the stimulus waveform (e.g. at the positive zero crossing). The period histogram follows a half-wave rectified version of the stimulus. This in effect corresponds to deflection of the basilar membrane in the effective direction. deflections in the opposite direction actually suppress the spontaneous firing activity of the nerve. Firing of the nerve increases (there is more spikes in the histogram) with the increasing intensity of the stimulus.

20. Coding of the stimulus intensity
sound level [dB]
In an absence of any stimulus, spikes are still being emitted but only at some low rate (spontaneous firing). From a certain intensity level, the firing rate starts increasing (threshold of firing). As the intensity of the stimulus increases further, there is more and more spikes being emitted, and for some time, the increase is close to linear. After further intensity increase, the firing rate starts to saturate, and any further increase does not induce any further increase in the firing rate. A typical range between the spontaneous rate and the saturation for a particular neuron is about dB.
threshold of firing

21. Tuning curves
When being able to access a particular auditory fibre that is connected to a particular haircell, and measuring thresholds when the firing on this fibre starts increasing while changing the frequency of the stimulus, it is possible to derive tuning curve of the fibre. As shown, there is always a frequency at which the fibre is the most sensitive (the characteristic frequency of the fibre). At frequencies that are different from the best frequency, the fibre is less sensitive. That means the fibre (auditory neuron) behaves like a band-pass filter. Different fibres have different tuning curves. The curves are typically much steeper towards higher frequencies than towards the lower ones (remember the mechanical frequency selectivity of the basilar membrane which is very similar). Also, notice that the curves are approximately similar on a logarithmic scale, that means that in absolute terms, they are getting broader as their best frequency increases.

22. Reverse correlation technique
There is an elegant technique for deriving shapes of auditory tuning curves. Random stimulus (e.g. noise) are applied and parts of the stimulus that preceed the spikes on the nerve are collected. Assuming that the part of the system that preceeds the spiking is linear, the average of all effective stimuli yields the stimulus that most easily excites the nerve (since the irrelevant elements of the stimuli average out and the relevant part (which is always present in the stimulus that excites the neuron) remains. This triggering wavelet is a time domain signal; but its Fourier transform yields the tuning curve of the neuron.

23. Bandwidths of tuning curves increase with frequency
(frequency resolution decreases with frequency)
Here we show how the bandwidth of the tuning curves increases with frequency. The higher the characteristic (best) frequency, the broader is the frequency bandwidth. This means that at lower frequencies, the spectral resolution of the analysis is higher than at high frequencies. The bandwidth increases approximately linearly with the center frequency of the filter (electric circuit people call this the “constant Q – for quality – filter). If we agree that one of the important functions of the ear is to perform frequency analysis of the incoming signal, the decrease of spectral resolution with frequency has important consequences on how is the acoustic stimulus presented by the ear to higher levels of hearing. Spectral details at lower frequencies are being resolved by the ear, while the higher frequency details are nor resolved (are smoothed out). This is an important difference between frequency analysis that is done in the ear and frequency analysis done by a short-term Fourier transform, which is constant in frequency, and is determined by the length and the shape of its single analysis window!

24. Place Theory of Hearing
Tones of certain frequencies excite certain areas of the cochlea that are connected to certain auditory fibres.
the fibres are distributed tonotopically (by their best frequencies) in the auditory nerve
this tonotopical organization is preserved throughout the higher areas of hearing all the way to the brain
The Place Theory of Hearing says exactly that. The theory is strongly supported by the fact that the fibres connected to low frequency parts of the cochlea are in different parts of the auditory nerve than the fibres connected to high frequency part of the cochlea (so called “tonotopical” organization) and this tonotopical organization is preserved all the way to the brain – the low frequencies excite different parts of the brain than the high frequencies do.
However, the things are not all that easy.

25. Place theory of peripheral auditory processing
signal
BP1
BP2
BPn
BRAIN
bank of cochlear band-pass filters
firing rate depends on sound intensity
sound level [dB]
Firing of the auditory nerve
Bandwidths of tuning curves increase with frequency
(frequency resolution decreases with frequency)
So far, what we have heard indicates a relatively straightforward mechanism of the auditory analysis that is done in the ear. The ear could act as a bank of ban-pass filters (which can be approximately emulated by properly modified short-term Fourier analysis), firing rates on the individual fibres in the auditory nerve (which are connected to hair cells distributed along the length of the cochlea) indicate spectral energy of the signal (which would be an equivalent of the spectral energy computed by the engineering analysis).
bandwidth
characteristic frequency

26. time [s]
1.2
frequency [kHz] 5
Emulating the constant Q property in engineering is possible. One way is to design a band of proper real band-pass filters with their bandwidths following the bandwidths observed in hearing, and do the frequency analysis using this band of filter instead of using the short-term Fourier transform. Yet another way that we may understand by now is to use many short-term Fourier analyses in parallel, each using the appropriate analysis window (this is actually very similar to using the bank of band-pass filters, and can be made exactly equivalent by proper choices of windows). A practical low-cost way is to sum the short-term spectrum magnitude spectrum from the Fourier analysis using proper weighting functions (this is done in most of engineering analysis used e.g. in recognition of speech).

27. Response in brain of fly to a change of the scene
Response of hearing periphery to a change in acoustic scene (switching on and off a tone)
As you remember, this vigorous reaction to change in the stimulus is a universal property of sensory neural system. We have seen the same in a brain of a fly when its visual system was exposed to a scene change, now we see it in an auditory periphery of a mammal (I believe this was a cat). Here, the change in the stimulus is introduced in time.

28. stimulus rectify & compress (log?) remove mean differentiate combine
We have seen earlier (even though this was for a fly ) that neural systems like a change. Here we show how the firing rate of a neuron in a auditory periphery changes when a signal is turned on and off. At a moment the signal goes on, the firing rate almost instantaneously increases from the spontaneous rate. However, it quickly starts decreasing even when the signal stays on the same level, and after about ms it saturates on much lower level. When the tone is switched off, the firing rate decreases below the spontaneous rate and settles at this spontaneous rate only after another ms.
We speculate that the reaction of the system might be approximately emulated by a system shown at the bottom of the figure. Such a system compresses the stimulus, removes its mean and differentiates it (band-pass filtering) in one branch and passes the compressed stimulus in the another branch, recombines the two branches (superposition) and expands the combined output by an inverse of the compressive function at its input.
stimulus
rectify &
compress
(log?)
remove
mean
differentiate
combine
expand
(exp?)
firing
rate

29. Response of horseshoe crab’s visual neuron to change in light
This increase of neural activity at a point of change in the stimulus was already observed in early experiments with vision of a horseshoe crab (Hartline and colleagues. (Hartline was a Fellow and later a Professor at the Johns Hopkins a eventually got the Nobel Prize in 1967 for his research at Hopkins). Hartline and colleagues showed that visual perception introduces a peak at the point of the change, even though the change in the light at this point is monotonous. Here, the change is not in time but in space, but as we will learn to appreciate, to a neural system, the change is a change, no matter where. Indeed, you may see a lighter line at the point when the intensity of the light starts decreasing and a darker line at the point when it stops decreasing, as illustrated in the bottom graph. Such a response is consistent with a presence of a differentiating element in the processing.

30. Two-tone suppression (lateral inhibition)
tone elicits certain response (firing rate)
intensity
frequency
second tone in the + area increases the firing rate
Interestingly, firing rate of the auditory nerve does not necessarily need to increase with the increasing intensity of the sound. As long as a weak stimulus is close to the best frequency of the auditory fibre and above the fiber threshold, it indeed elicits an increase in the firing rate on the fibre. When the second stimulus with slightly different frequency and amplitude is added, the firing rate further increases – as long as the second stimulus is with the tuning curve of the fibre. However, when the second stimulus is close but not with the tuning curve area (within the blue areas), the firing rate of the fibre decreases, This phenomenon is called a two-tone suppression.
second tone in the – area decreases the firing rate

31. responds to increase in light intensity
“on center”
(“off surround”)
responds to increase in light intensity
Sensitivity of visual neuron (retinal ganglion cell)
of a frog to changing size of a dot
bright dot
dark dot
“off center”
(“on surround”)
responds to decrease in light intensity
The two-tone suppression observed in hearing is an example of more general phenomenon of lateral suppression, observed throughout the perceptual system. The excitation by a stimulus ocuring in one area of the sensor is accompanied by suppression of the activity when the stimulus is applied somewhere close to this first stimulus. Here we show a similar phenomenon has been observed already long time ago in a visual system. In the visual system, there are to basic types of retinal cells. One that increases its firing rate with increase of the light intensity in its center, and another one which firing rate decreases with the increasing intensity of the stimulus that is applied in its center. Both types exhibits firing rate decrease when the stimulus of the opposite character is applied in a close vicinity of the cell center. This phenomenon is called a lateral inhibition.

32. 2-dimensional “receptive field” in vision
This is a 3-D picture of the visual receptive field. The excitatory and the inhibitory parts of the field are clearly visible.

33. Receptive field on your skin
Experiment that you can try: apply first a pressure by a sharp object (e,g, a tip of a pen) to a point on your arm. You feel a certain sensation. Then apply a pressure by another tip of a pen close to the first one. When the second point is very close to the first one, it feels like one object. When you increase distance sufficiently, it feels like two objects. However, at a certain distance, it still feels like two objects but the pressure feels weaker. The lateral inhibition on your skin is in action.