1. Non-Parametric Learning Prof. A.L. Yuille Stat 231. Fall 2004. Chp 4.1 – 4.3.

2. Parametric versus Non-Parametric Previous lectures on MLE learning assumed a functional form for the probability distribution. We now consider an alternative non- parametric method based on window function.

3. Non-Parametric It is hard to develop probability models for some data. Example: estimate the distribution of annual rainfall in the U.S.A. Want to model p(x,y) – probability that a raindrop hits a position (x,y). Problems: (i) multi-modal density is difficult for parametric models, (ii) difficult/impossible to collect enough data at each point (x,y).

4. Intuition Assume that the probability density is locally smooth. Goal: estimate the class density model p(x) from data Method 1: Windows based on points x in space.

5. Windows For each point x, form a window centred at x with volume Count the number of samples that fall in the window. Probability density is estimated as:

6. Non-Parametric Goal: to design a sequence of windows so that at each point x (f(x) is the true density). Conditions for window design: (i)increasing spatial resolution. (ii) many samples at each point (iii)

7. Two Design Methods Parzen Window: Fix window size: K-NN: Fix no. samples in window:

8. Parzen Window Parzen window uses a window function Example: (i) Unit hypercube: and 0 otherwise. (ii) Gaussian in d-dimensions.

9. Parzen Windows No. of samples in the hypercube is Volume The estimate of the distribution is: More generally, the window interpolates the data.

10. Parzen Window Example Estimate a density with five modes using Gaussian windows at scales h=1,0.5, 0.2.

11. Convergence Proof. We will show that the Parzen window estimator converges to the true density at each point x with increasing number of samples.

12. Proof Strategy. Parzen distribution is a random variable which depends on the samples used to estimate it. We have to take the expectation of the distribution with respect to the samples. We show that the expected value of the Parzen distribution will be the true distribution. And the expected variance of the Parzen distribution will tend to 0 as no. samples gets large.

13. Convergence of the Mean Result follows.

14. Convergence of Variance Variance:

15. Example of Parzen Window Underlying density is Gaussian. Window volume decreases as

16. Example of Parzen Window Underlying Density is bi-modal.

17. Parzen Window and Interpolation. In practice, we do not have an infinite number of samples. The choice of window shape is important. This effectively interpolates the data. If the window shape fits the local structure of the density, then Parzen windows are effective.