1. 1 Nonlinear models for Natural Image Statistics Urs Köster & Aapo Hyvärinen University of Helsinki

2. 2 Overview  A two-layer model learns complex cell pooling  A horizontal product model for Contrast Gain Control  A Markov Random Field generalizes ICA to large images (“convolutional ICA”)  Things I would really like to know 2

3. Limitations of linear image models Natural images have complex structure Linear models ignore much of the rich interactions between units Interactions related to modelling the dependencies of linear components Variance dependencies are a particularly obvious nonlinear dependency Here, we consider Models with (complex cell) pooling of linear filter outputs - hierarchical models Model dependencies by gain control on the pixel level Schwartz & Simoncelli 2001

4. 4 A two-layer model estimated with score matching learns complex cell -like receptive fields

5. 5 Two-layer model estimated with score matching Define an energy based model of the form Squaring the outputs of linear filters Second layer linear transform v are free, as opposed to topographic ICA (Hyvärinen and Hoyer 2001) Non-normalized model: needs either score matching or MCMC methods Related to Osindero & Hinton (2006), Karklin and Lewicki (2005)

6. 6 Two-layer model: Results Some pooling patterns The second layer learns to pool over units with similar location and orientation, but different spatial phase Learns the energy model of Complex Cells without any assumptions on the pooling (but constrained non-negative) Estimating W and V simultaneously leads to a better optimum and more phase invariance of the higher order units

7. 7 Learning to perform local gain control with a horizontal product model

8. Gain Control in Physiology Gain control is common throughout the brain The divisive normalization model for primary visual cortex Previous work has analyzed the effect of gain control on the cortical level We use a statistical model for gain control on the LGN level Important effects on subsequent cortical processing Normalization model Carandini and Heeger,1994

9. 9 Multiplicative interactions Data is described by element-wise multiplying outputs of sub-models Combine aspects of a stimulus generated by separate mechanisms Horizontal layers Two parallel streams or layers on one level of the hierarchy Unrelated aspects of the stimulus are generated separately Observed data is generated by combining all the sub-models A horizontal network model:

10. 10 The model Definition of the model: Likelihood: Constraints: B and t are non-negative, low dimension, W invertible g(.) is log cosh (logistic distribution) t has an L1 sparseness prior 10

11. Horizontal product model: results First layer W 4 units in B First layer W 16 units in B

12. Emerging Contrast Gain Control Reconstruction from As only True image patches Modulation from Bt Emergence of a contrast map in the second layer It performs Contrast Gain Control on the LGN level (rather than on filter outputs) Effectively performing some kind of divisive normalization locally The model can be written as

13. 13 The “big” picture: A Markov Random Field generalizes ICA to images of arbitrary size

14. 14 Markov Random Field Goal: Define probabilities for whole images rather than small patches A MRF uses a convolution to analyze large images with small filters Estimating the optimal filters difficult: the model cannot be normalized Estimation using score matching

15. 15 MRF: Model estimation The energy (neg. log pdf) is We can rewrite the convolution where x i are all possible patches from the image, w k are the different filters Non-normalized model: We can use score matching The MRF is roughly equivalent to overcomplete ICA with filters copied in all possible locations.

16. 16 MRF: Results We can estimate MRF filters of size 12x12 pixels (much larger than previous work, e.g. 5x5) This is possible from 23x23 pixel ‘images’, but the filters generalize to images of arbitrary size This is possible because all possible overlaps are accounted for Filters similar to ICA, but less localized (since they need to explain more of the surrounding patch?) Possible applications in denoising and filling-in 16

17. 17 Things I would like to know

18. 18 Things I would like to know (1) What is the role of unsupervised learning from passively observed data (“Pure cognition”)? What about unsupervised learning in an agent (e.g. Mahadevan et al 2007) Action and perception coupled Cf. reinforcement learning

19. 19 Things I would like to know (2) If we admit pure cognition, what is the goal of statistical modelling in the brain? What is the proper theoretical framework? Coding (information theory) or Bayesian inference Hardly sparsity (statistically) or independence, I think Or is it just about reducing metabolic costs? Sparse coding Minimum wiring length

20. 20 More things I would like to know What is V2 doing? What are meaningful nonlinearities to use? Squaring in the first layer, perhaps.. Can we just use the same logistic everywhere? [This space for sale]