1. Estimating the Likelihood of Statistical Models of Natural Image Patches Daniel Zoran ICNC – The Hebrew University of Jerusalem Advisor: Yair Weiss CifAR NCAP Summer School

2. Natural Image Statistics Natural scenes and images exhibit very distinctive statistics A lot of research has been made in this field since the 1950s Important in image processing, computer vision, computational neuroscience and more…

3. Natural Image Statistics Properties The space of all possible images is huge – For a 256 gray levels, NxN sized matrix, there are possible images – Natural Images occupy a tiny fraction of this space Some statistical properties of natural images: – Translation invariance – Power law spectrum – – Scale invariance – Non-Gaussianity of marginal statistics - (more on that later)

4. Estimating the Likelihood of different statistical models During the years, a lot of models for natural image distributions have been proposed It is hard to test the validity of such models, especially when comparing one model to the other A step towards this – estimating the (log) likelihood of a given model and comparing the results with other models

5. Estimating the Likelihood of different models Variable sized patches were extracted from natural images Different models assumed A training set was used to estimate various parameters of the model Likelihood was calculated over a test set 5000 patches in each set Source images are mostly JPEGs from a Panasonic digital camera, portraying outdoor scenes Also tested on standard images (Lena, Boat and Barbara – PNG format)

7. The models – 1D Gaussian A 1D Gaussian distribution for every pixel – Mean and Variance estimated directly from the sample – The likelihood of an image x under this model is: – Where: This model captures nothing about natural images

8. Results – 1D Gaussian Test Set 4 (Noise) Test Set 3 (PNG)Test Set 2 (JPG)Test Set 1 (JPG) Patch Size -26.918.4-19.620.310x10 -38.826.2-28.029.412x12 -54.535.8-38.441.414x14 -70.547.7-44.652.116x16 -89.160.5-54.768.318x18 -106.472.9-6784.520x20

9. The models – Multidimensional Gaussian with PCA Using the covariance matrix, rotate the images in the image space towards directions of maximum variance (PCA) A Multidimensional Gaussian distribution for the components: Where the covariance matrix is estimated from the training set: This captures the Power-Law spectrum property

10. Results – Multidimensional Gaussian Test Set 4 (Noise) Test Set 3 (PNG)Test Set 2 (JPG)Test Set 1 (JPG) Patch Size 2694646540043810x10 4034569657763112x12 5780894579288614x14 7902012851039117716x16 10477716611334150218x18 13583020071702191620x20

11. The models – Gaussian Mixture Model with PCA Using the same rotation scheme (PCA), now assume a Gaussian Mixture Model for the marginal filter response distributions Under this model: Where W’s rows are the eigenvectors of the covariance matrix The GMM parameters were found using EM This captures both the Power-Law spectrum and the sparseness properties

12. Results – GMM with PCA Test Set 4 (Noise) Test Set 3 (PNG)Test Set 2 (JPG)Test Set 1 (JPG) Patch Size -485218121020010x10 -813824429528512x12 31240037614x14 40451147216x16 48864659518x18 55776769820x20

13. The models – Generalized Gaussian with PCA Finally, instead of using a GMM, we now use a Generalized Gaussian This has the advantage of having less parameters, while still capturing Sparseness: Parameters were obtained directly from the training set

14. Results – Generalized Gaussian with PCA Test Set 4 (Noise) Test Set 3 (PNG)Test Set 2 (JPG)Test Set 1 (JPG) Patch Size -189419121220810x10 -309927330529112x12 -524235641138614x14 -787545652949716x16 -1196654366361518x18 -1678366879673920x20

17. The GG shape parameter During the analysis of the data we have encountered a strange phenomena Marginal distributions get wider as we go measure higher frequency filter responses This is not due to increase in variance (which drops as we go to high frequencies) We modeled this using the shape parameter obtained from the samples

18. Shape parameter for test set 1

19. Shape parameter for test set 2

20. Shape parameter for test set 3

21. Shape parameter for test set 4 - PNG

22. Shape parameter for noise test set

23. Conclusion This is (very) early work, still in progress A lot of things left to do: – Try more models and filter (ICA is in progress) – Actually compare the different models – Try to make some sense out of the shape of the distributions – Look into higher order dependencies and correlations A lot more…

24. Thank you! Questions?