1. Low Level Visual Processing

2. Information Maximization in the Retina Hypothesis: ganglion cells try to transmit as much information as possible about the image. What kind of receptive field maximizes information transfer?

3. Information Maximization in the Retina In this particular context, information is maximized for a factorial code: For a factorial code, the mutual information is 0 (there are no redundancies):

4. Information Maximization in the Retina Independence is hard to achieve. Instead, we can look for a code which decorrelates the activity of the ganglion cells. This is a lot easier because decorrelation can be achieved with a simple linear transformation.

5. Information Maximization in the Retina We assume that ganglion cells are linear: The goal is to find a receptive field profile, D s (x), for which the ganglion cells are decorrelated (i.e., a whitening filter).

6. Information Maximization in the Retina Correlations are captured by the crosscorrelogram (all signals are assumed to be zero mean): The crosscorrelogram is also a convolution

7. Fourier Transform The Fourier transform of a convolution is equal to the product of the individual spectra: The spectrum of a Dirac function is flat.

8. Information Maximization in the Retina To decorrelate, we need to ensure that the crosscorrelogram is a Dirac function, i.e., its Fourier transform should be as flat as possible.

9. Information Maximization in the Retina

11. If we assume that the retina adds noise on top of the signal to be transmitted to the brain, the previous filter is a bad idea because it amplifies the noise. Solution: use a noise filter first, :

12. Information Maximization in the Retina

13. The shape of the whitening filter depends on the noise level. For high contrast/low noise: bandpass filter. Center-surround RF. For low contrast/high noise: low pass filter. Gaussian RF

14. Information Maximization in the Retina j +

15. 00 0 0 5 5 10 15 1 1 2 2 33 4 4 temporal frequency (Hz) temporal frequency (H

16. Information Maximization beyond the Retina The bottleneck argument can only work once… The whitening filter only decorrelates. To find independent components, use ICA: predicts oriented filter Use other constrained beside infomax, such as sparseness.

17. Center Surround Receptive Fields The center surround receptive fields are decent edge detectors +

18. Center Surround Receptive Fields

19. Feature extraction: Energy Filters

20. 2D Fourier Transform Frequency Orientation

21. 2D Fourier Transform

23. Motion Energy Filters Space Time

24. Motion Energy Filters In a space time diagram 1 st order motion shows up as diagonal lines. The slope of the line indicates the velocity A space-time Fourier transform can therefore recover the speed of motion

25. Motion Energy Filters

28. 1 st order motion Time Space

29. Motion Energy Filters 2 nd order motion

30. Motion Energy Filters In a space time diagram, 2 nd order motion does not show up as a diagonal line… Methods based on linear filtering of the image followed by a nonlinearity cannot work You need to apply a nonlinearity to the image first

31. Motion Energy Filters A Fourier transform returns a set of complex coefficients:

32. Motion Energy Filters The power spectrum is given by

33. Motion Energy Filters

34. cos sin + ajaj bjbj cjcj x2x2 x2x2 I

35. Motion Energy Filters cos sin + ajaj bjbj cjcj x2x2 x2x2 I

36. tt xx

37. Motion Energy Filters Therefore, taking a Fourier transform in space time is sufficient to compute motion. To compute velocity, just look at where the power is and compute the angle. Better still, use the Fourier spectrum as your observation and design an optimal estimator of velocity (tricky because the noise is poorly defined)…

38. Motion Energy Filters How do you compute a Fourier transform with neurons? Use neurons with spatio- temporal filters looking like oriented sine and cosine functions. Problem: the receptive fields are non local and would have a hard time dealing with multiple objects in space and multiple events in time…

39. Motion Energy Filters Solution: use oriented Gabor-like filters or causal version of Gabor-like filters. To recover the spectrum, take quadrature pairs, square them and add them: this is what is called an Energy Filter.

40. Motion Energy Filters x2x2 x2x2 +

41. From V1 to MT V1 cells are tuned to velocity but they are also tuned to spatial and temporal frequencies tt xx

42. From V1 to MT MT cells are tuned to velocity across a wide range of spatial and temporal frequencies tt xx

43. MT Cells

44. Pooling across Filters Motion opponency: it is not possible to perceive transparent motion within the same spatial bandwidth. This suggests that the neural read out mechanism for speed computes the difference between filters tuned to different spatial frequencies within the same temporal bandwidth.

45. Pooling across Filters + Flicker

46. Energy Filters For second order motion, apply a nonlinearity to the image and then run a motion energy filter.

47. Motion Processing: Bayesian Approach The energy filter approach is not the only game in town… Bayesian integration provides a better account of psychophysical results

48. Energy Filters: Generalization The same technique can be used to compute orientation, disparity, … etc.

49. Energy Filters: Generalization The case of stereopsis: constant disparity correspond to oriented line in righ/left RF diagram.

50. Energy Filters: Generalization

52. Orientation Selectivity At first sight: a simple, if not downright stupid, problem. Use an orientation energy filter The only challenge: finding out exactly how the brain does it… Two classes of models: feedforward and lateral connection models.

53. Orientation Selectivity: Feedforward (Hubel and Wiesel)  LGN CTX

54. Orientation Selectivity: The Lateral Connection Model  LGN CTX - +

55. Orientation Selectivity: Feedforward (Hubel and Wiesel) -- +

56. Orientation Selectivity Take a quadrature pair, rectifiy and square their outputs, sum and you get a complex cell tuned to orientation.

57. Orientation Selectivity Most people think this model is wrong, yet the evidence in its favor are overwhelming. –No further tuning over time –Aspect ratio consistent with this model –LGN input to layer 4 cells as tuned as output –LGN/Cortex connectivity as predicted by feedforward model Why are there lateral connections?