1. Ch8: Nonparametric Methods
8.1 Introduction
Nonparametric methods do not assume any a priori parametric form.
Nonparametric methods (or memory-based, case-based, or instance-based learning algorithms):
Store the training examples. Later, when given an x,
find a small number of closest training examples and
interpolate from these.
The result is locally affected only by nearby examples.

2. 8.2 Nonparametric Density Estimation
Univariate case:
Given the training set drawn iid from
unknown probability density p(x)
To estimate p(x), i.e.,
Cumulative distribution function (cdf) F(x),

3. Density function (df): the derivative of the cdf
8.2.1 Histogram Estimator
Histogram:

4. smooth
spiky

5. Naive estimator:
: region of influence

6. Histogram estimator needs to know the origin and h.
Naive estimator needs to know only h.
is discrete, so is
8.2.2 Kernel Estimator
Kernel function: a smooth weight function,
e.g., Gaussian kernel:
Kernel estimator (Parzen windows)

7. h larger  smoother
Adaptive methods tailor h as a function of the
density around x.

8. 8.2.3 k-Nearest Neighbor Estimator
Instead of fixing h, fix # nearby examples k
where
the distance to the
kth closest example to x
* Density high, bins small; density low, bins large.
The k-nn estimator is not continuous
To get a smooth estimator, a kernel function is used

10. 8.3 Generalization to Multivariate Data
Kernel density estimator:
and
Multivariate Gaussian kernel
spheric
ellipsoid
(Euclidean form)
(Mahalanobis form takes 1. different scales of dimensions, and 2. correlation into account).

11. 8.4 Nonparametric Classification
Objective: Estimate the class-conditional density
Given by
where
The ML estimate of the prior density:
The discriminant function:

12. k-NN estimator:
where
# neighbors out of the k nearest
The volume of hypersphere centered at x
with radius , where is
the kth nearest neighbor of x:
with as the volume of the unit sphere,
e.g.,
From Bayes’ rule:

13. k-NN classifier assigns the input to the class having most examples among the k neighbors of the input.
Example: 1-NN classifier, i.e., the nearest neighbor
classifier. It divides the space in the form
of a Voronoi tessellation.

14. 8.5 Condensed Nearest Neighbor
Time/space complexity of k-NN is O(N).
Idea: Find a subset Z of X that is small and is accurate
in classifying X.
Algorithm of CNN:
The algorithm depends on the order of the examples being examined; different subsets may be found.

15. The black line segments are the Voronoi tessellation,
The red line segments form the class discriminant.
In the CNN, the examples that do not participate in
defining the discriminant are removed.

16. 8.6 Distance-based Classification
Find a distance function such that if belong to the same class, distance is small and if they belong to different classes, distance is large.
Consider the nearest mean classifier:
Parametric approach:
Euclidean distance:
Mahalanobis distance:

17. Semiparametric approach: each class is written as a
mixture of densities
Idea of distance learning:
Assume a parametric model and learn
its parameters .
Example: Mahalanobis distance
M: the parameter matrix to be estimated

18. If the input dimensionality d is high, factor M as
, where M is d x d and L is k x d, k < d.
Learn L instead of M.

19. 8.7 Outlier Detection
Outliers may indicate (a) abnormal behaviors of systems or (b) record errors.
Difficulties of outlier detection:
Outliers are (a) very few, (b) of many types, and (c)
seldom labeled
Nonparametric approaches are often used:
Idea -- find instances far away from other instances.
Example: Local Outlier Factor (LOF)
where

20. The distance between x and its kth
nearest neighbor,
the set of examples that are in the
neighborhood of x,
LOF(x)  1, P(x not outlier) increases;
 larger, P(x outlier) increases.

21. 8.8 Nonparametric Regression:
Given a training sample
assume
Nonparametric regression:
Given x, i) find its neighborhood N(x) and
ii) average the r values in N(x) to
calculate
There are various ways to defining the neighborhood and averaging in the neighborhood
Consider univariate x first.

22. 8.8.1 Running mean smoother
Regressogram: Define an origin and a bin width, average the r values in the bin

24. Running mean smoother: Define a bin symmetric around x and average in there

26. 8.8.2 Kernel smoother -- Give less weights to further points.
where K( ) is Gaussian additive models

28. K-nn smoother: Fix the number of neighbors k.
8.8.3 Running line smoother
-- Use the data points in the neighborhood to fit a line.
Locally weighted running line smoother (loess):
Usde kernel weighting such that distant points have
less effect on error.

30. 8.9 How to Choose the Smoothing Parameters
h: bin width, kernel spread, k: the number of neighbors.
When k or h is small, single instances matter; bias is
small, variance is large (undersmoothing).
As k or h increases, average over more instances and
bias increases but variance decreases (oversmoothing).
Smoothing splines:

31. Cross-validation is used to fine tune k or h.