Presentation is loading. Please wait.

Presentation is loading. Please wait.

Lecture 03: K-nearest neighbors

Published byAntonio Rodríguez Modified over 5 years ago

Similar presentations

Presentation on theme: "Lecture 03: K-nearest neighbors"— Presentation transcript:

1 Lecture 03: K-nearest neighbors
CS480/680: Intro to ML Lecture 03: K-nearest neighbors 08/04/2019 Yao-Liang Yu

2 Outline Announcements Algorithm Theory Application 08/04/2019
Yao-Liang Yu

3 Outline Announcements Algorithm Theory Application 08/04/2019
Yao-Liang Yu

4 Classification revisited
ŷ = sign( xTw + b ) Decision boundary: xTw + b = 0 Parametric: finite-dim w Non-parametric: no specific form (or inf-dim w) 08/04/2019 Yao-Liang Yu

5 1-Nearest Neighbour Store training set (X, y)
For query (test point) x’ find nearest point x in X predict y’ = y(x) 08/04/2019 Yao-Liang Yu

6 What do you mean “nearest”
Need to measure distance or similarity d: X x X  R+ such that symmetry: d(x, x’) = d(x, x’) definite: d(x, x’) = 0 iff x = x’ triangle inequality: d(a, b) <= d(a,c) + d(c,b) Lp distance: dp(x, x’) = || x – x’ ||p p=2: Euclidean distance p=1: Manhattan distance p=inf: Chebyshev distance 08/04/2019 Yao-Liang Yu

7 Complexity of 1-NN Training: 0… but O(nd) space
Testing: O(nd) for each query point n: # of training samples d: # of features Can we do better? 08/04/2019 Yao-Liang Yu

8 Voronoi diagram In 2D, can construct in
O(n logn) time and O(n) space (Fortune, 1989) In 2D, query costs O(log n) Large d? nO(d) space with O(d log n) query time  approximate NN 08/04/2019 Yao-Liang Yu

9 Normalization Usually, for each feature, subtract the mean and divide by standard deviation Or, equivalently, use a different distance metric 08/04/2019 Yao-Liang Yu

10 Learning the metric Mahalanobias distance Or equivalently let
First perform linear transformation Then use the usual Euclidean distance is small iff such that 08/04/2019 Yao-Liang Yu

11 k-NN Store training set (X, y) For query (test point) x’
find k nearest points x1, x2, … , xk in X predict y’ = y(x1, x2, … , xk) usually a majority vote among the labels of x1, x2, … , xk say y1=1, y2=1, y3=-1, y4=1, y5=-1  y’= Test complexity: O(nd) 08/04/2019 Yao-Liang Yu

12 Effect of k 08/04/2019 Yao-Liang Yu

13 How to select k? 08/04/2019 Yao-Liang Yu

14 Does k-NN work? MNIST: 60k train, 10k test 08/04/2019 Yao-Liang Yu

15 Outline Announcements Algorithm Theory Application 08/04/2019
Yao-Liang Yu

16 Bayes rule Bayes error Bayes rule f*(X) = 1 iff η(X) ≥ ½ 08/04/2019
Yao-Liang Yu

17 Derivation 08/04/2019 Yao-Liang Yu

18 Multi-class This is the best we can do even when we know the distribution of (X,Y) How big can P* be? 08/04/2019 Yao-Liang Yu

19 At most twice worse (Cover & Hart’67)
limninf P(Y1(n)≠Y) ≤ 2P* - c/(c-1) (P*)2 Asymptotically! # of classes 1 / (max P*) 1-NN error Bayes error If P* close to 0, then 1-NN error close to 2P* How big does n have to be? 08/04/2019 Yao-Liang Yu

20 The picture Both assume we have infinite amount of training data!
08/04/2019 Yao-Liang Yu

21 1-NN vs k-NN error(1NN) = 1/2n error(kNN) for k = 2t+1: 08/04/2019
Yao-Liang Yu

22 Curse of dimensionality
Theorem (SSBD, p224). For any c > 1 and any learning algorithm L, there exists a distribution over [0,1]d x {0,1} such that the Bayes error is 0 but for sample size n ≤ (c+1)d/2 , the error of the rule L is greater than ¼. k-NN is effective when have many training samples Dimensionality reduction may be helpful 08/04/2019 Yao-Liang Yu

23 Outline Announcements Algorithm Theory Application 08/04/2019
Yao-Liang Yu

24 Locally linear embedding (Saul & Roweis’00)
08/04/2019 Yao-Liang Yu

25 k-NN for regression Training: store training set (X, y) Query x’
Find k-nns x1, x2, … , xk in X output y’ = y(x1, x2, … , xk) = (y1+y2+…+yk)/k 08/04/2019 Yao-Liang Yu

26 Questions? 08/04/2019 Yao-Liang Yu

Download ppt "Lecture 03: K-nearest neighbors"

Similar presentations

Principles of Density Estimation

Principles of Density Estimation
Data Mining Classification: Alternative Techniques

Data Mining Classification: Alternative Techniques
Indian Statistical Institute Kolkata

Indian Statistical Institute Kolkata
Instance Based Learning

Instance Based Learning
Lazy vs. Eager Learning Lazy vs. eager learning

Lazy vs. Eager Learning Lazy vs. eager learning
Instance based learning K-Nearest Neighbor Locally weighted regression Radial basis functions.

Instance based learning K-Nearest Neighbor Locally weighted regression Radial basis functions.
1-NN Rule: Given an unknown sample X decide if for That is, assign X to category if the closest neighbor of X is from category i.

1-NN Rule: Given an unknown sample X decide if for That is, assign X to category if the closest neighbor of X is from category i.
Prénom Nom Document Analysis: Non Parametric Methods for Pattern Recognition Prof. Rolf Ingold, University of Fribourg Master course, spring semester 2008.

Prénom Nom Document Analysis: Non Parametric Methods for Pattern Recognition Prof. Rolf Ingold, University of Fribourg Master course, spring semester 2008.
Pattern Classification All materials in these slides were taken from Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John Wiley.

Pattern Classification All materials in these slides were taken from Pattern Classification (2nd ed) by R. O. Duda, P. E. Hart and D. G. Stork, John Wiley.
1 Nearest Neighbor Learning Greg Grudic (Notes borrowed from Thomas G. Dietterich and Tom Mitchell) Intro AI.

1 Nearest Neighbor Learning Greg Grudic (Notes borrowed from Thomas G. Dietterich and Tom Mitchell) Intro AI.
Aprendizagem baseada em instâncias (K vizinhos mais próximos)

Aprendizagem baseada em instâncias (K vizinhos mais próximos)
KNN, LVQ, SOM. Instance Based Learning K-Nearest Neighbor Algorithm (LVQ) Learning Vector Quantization (SOM) Self Organizing Maps.

KNN, LVQ, SOM. Instance Based Learning K-Nearest Neighbor Algorithm (LVQ) Learning Vector Quantization (SOM) Self Organizing Maps.
Nearest Neighbour Condensing and Editing David Claus February 27, 2004 Computer Vision Reading Group Oxford.

Nearest Neighbour Condensing and Editing David Claus February 27, 2004 Computer Vision Reading Group Oxford.
INSTANCE-BASE LEARNING

INSTANCE-BASE LEARNING
CS 478 - Instance Based Learning1 Instance Based Learning.

CS Instance Based Learning1 Instance Based Learning.
Nearest-Neighbor Classifiers Sec 4.7. 5 minutes of math... Definition: a metric function is a function that obeys the following properties: Identity:

Nearest-Neighbor Classifiers Sec minutes of math... Definition: a metric function is a function that obeys the following properties: Identity:
Nearest Neighbor (NN) Rule & k-Nearest Neighbor (k-NN) Rule Non-parametric : Can be used with arbitrary distributions, No need to assume that the form.

Nearest Neighbor (NN) Rule & k-Nearest Neighbor (k-NN) Rule Non-parametric : Can be used with arbitrary distributions, No need to assume that the form.
Non-Parameter Estimation 主講人：虞台文. Contents Introduction Parzen Windows k n -Nearest-Neighbor Estimation Classification Techiques – The Nearest-Neighbor.

Non-Parameter Estimation 主講人：虞台文. Contents Introduction Parzen Windows k n -Nearest-Neighbor Estimation Classification Techiques – The Nearest-Neighbor.
Overview of Supervised Learning. 2015-10-23Overview of Supervised Learning2 Outline Linear Regression and Nearest Neighbors method Statistical Decision.

Overview of Supervised Learning Overview of Supervised Learning2 Outline Linear Regression and Nearest Neighbors method Statistical Decision.
David Claus and Christoph F. Eick: Nearest Neighbor Editing and Condensing Techniques Nearest Neighbor Editing and Condensing Techniques 1.Nearest Neighbor.

David Claus and Christoph F. Eick: Nearest Neighbor Editing and Condensing Techniques Nearest Neighbor Editing and Condensing Techniques 1.Nearest Neighbor.

Similar presentations

Ads by Google