1. Debrup Chakraborty Non Parametric Methods Pattern Recognition and Machine Learning

2. Nearest Neighbor classification Given: Given a labeled sample of n feature vectors ( call X) A distance measure (say the Euclidian Distance) To find: The class label of a given feature vector x which is not in X

3. Nearest Neighbor classification (contd.) The NN rule: Find the point y in X which is nearest to x Assign the label of y to x

4. Nearest Neighbor classification (contd.) This rule allows us to partition the feature space into cells consisting of all points closer to a given training point x All points in such cells are labeled by the class of the training point. This partitioning is called a Voronoi Tesselation

5. Voronoi Cells in 2d Nearest Neighbor classification (contd.)

6. Nearest Neighbor classification Complexity of the NN rule Distance calculation Finding the minimum distance

7. Nearest Neighbor classification Nearest Neighbor Editing X= Data set, n= no of training points, j=0 Construct the full Voronoi diagram for X Do j=j+1, for each point x j in X find Voronoi neighbors of x j If any neighbor is not from the same class as x j then mark x j Until j==n Discard all points that are not marked.

8. k nearest neighbor classification Given: Given a labeled sample of N feature vectors ( call X) A distance measure (say the Euclidian Distance) An integer k (generally odd) To find: The class label of a given feature vector x which is not in X

9. k-NN classification (contd.) Algorithm: Find out the k nearest neighbors of x in X Call them Out of the k samples, let k i of them belong to class c i. Choose that c i to be the class of x for which k i is maximum

10. K-nn Classification Class 1Class 2Class 3 z

11. k-NN classification (contd.) Distance weighted nearest neighbor Training set Given an instance x to be classified Let be the nearest neighbors of x Return In case x=x i, return f(x i )

12. Remarks on k-NN classification The distance weighted kNN is robust to noisy training data and is quite effective when it is provided a sufficiently large set of training examples. One drawbak of kNN method is that, it defers all computation till a new querry point is presented. Various methods have been developed to index the training examples so that the nearest neighbor can be found with less search time. One such indexing method is kd-tree developed by Bently 1975 kNN is a lazy learner

13. Locally Weighted Regression In the linear regression problem, to find h(x) at a point x we would do the following: 1.Minimize 2.Output

14. Locally Weighted Regression In the llocally weighted regression problem we would do the following 1.Minimize 2.Output A standard choice of weights is is called the bandwidth parameter

15. Clustering Is different from Classification Classification is partitioning the feature space whereas Clustering is partitioning the data into “homogeneous groups” Clustering is Unsupervised!!

16. K-means Clustering Given: A data set Fix the number of clusters K Let represent the i-th cluster center (prototype) at the k-th iteration Let represent the j-th cluster at the k-th iteration

17. K-means Clustering Steps 1.Choose the initial cluster centers 2.At the k-th iterative step distribute the points in X in K cluster using: 3.Compute 4.If then the procedure has converged else repeat from 2.