1. K Nearest Neighborhood (KNNs)
Lecture 13

2. Outline of Rule-Based Classification
Overview of KNN
Parameter and distance Choices
Characteristics of Instance-based learning
4. Example of KNN Classifiers

3. 1. Nearest Neighbor Classifiers
Basic idea:
If it walks like a duck, quacks like a duck, then it’s probably a duck
Training Records
Test Record
Compute Distance
Choose k of the “nearest” records

4. Nearest-Neighbor Classifiers
Requires three things
The set of stored records
Distance Metric to compute distance between records
The value of k, the number of nearest neighbors to retrieve
To classify an unknown record:
Compute distance to other training records
Identify k nearest neighbors
Use class labels of nearest neighbors to determine the class label of unknown record (e.g., by taking majority vote)

5. Definition of Nearest Neighbor
K-nearest neighbors of a record x are data points that have the k smallest distance to x

6. 1. Overview of KNN Classifier
Instance based learning
K-NN = k nearest neighbors algorithm
Training phase consists only of storing the feature vectors and class labels of the training samples
K is a user-defined parameter
A test point is selected and distances from the test point to every point in the training set is calculated
Class label is assigned via majority vote of the k nearest neighbors to the test point

7. 2. KNN Parameter Selection
Best choice for k value
Depends upon the data
Larger values generally reduce noise effects
But larger values make class boundaries less distinct
Can be selected via heuristic techniques
Can also use evolutionary algorithms to optimize k value

8. KNN Classifier: Parameter Selection
K-NN = k nearest neighbors algorithm
Training phase consists only of storing the feature vectors and class labels of the training samples
K is a user-defined parameter
A test point is selected and distances from the test point to every point in the training set is calculated
Class label is assigned via majority vote of the k nearest neighbors to the test point

9. 1 nearest-neighbor
Voronoi Diagram

10. Nearest Neighbor Classification…
Choosing the value of k:
If k is too small, sensitive to noise points
If k is too large, neighborhood may include points from other classes

11. 2. KNN Distance Measure Selection
Choice of distance measure is important
Nearest neighbors can produce incorrect predictions unless appropriate proximity measures used and pre-processing steps taken
Scale of attributes must be taken into consideration to avoid domination by less important attributes

12. Nearest Neighbor Classification
Compute distance between two points:
Euclidean distance
Determine the class from nearest neighbor list
take the majority vote of class labels among the k-nearest neighbors
Weigh the vote according to distance
weight factor, w = 1/d2

13. Nearest Neighbor Classification…
Scaling issues
Attributes may have to be scaled to prevent distance measures from being dominated by one of the attributes
Example:
height of a person may vary from 1.5m to 1.8m
weight of a person may vary from 90lb to 300lb
income of a person may vary from $10K to $1M

14. Nearest Neighbor Classification…
Problem with Euclidean measure:
High dimensional data
curse of dimensionality
Can produce counter-intuitive results vs
d =
d =
Solution: Normalize the vectors to unit length

15. 3. Characteristics of INSTANCE-BASED LEARNING:
No model built
Classifying test points can be expensive
Short training time
Uses local information to make predictions
Requires distance measure
Requires classification function
Decision boundaries can be of arbitrary shape = more flexible model representation
Decision boundaries also highly variable

16. Instance-Based Classifiers
Store the training records
Use training records to predict the class label of unseen cases

17. Instance Based Classifiers
Examples:
Rote-learner
Memorizes entire training data and performs classification only if attributes of record match one of the training examples exactly
Nearest neighbor
Uses k “closest” points (nearest neighbors) for performing classification

18. Nearest neighbor Classification…
k-NN classifiers are lazy learners
It does not build models explicitly
Unlike eager learners such as decision tree induction and rule-based systems
Classifying unknown records are relatively expensive

19. EAGER LEARNERS: Characteristics
Global model of entire input space
Long training time to develop model
Short time classifying test points
Decision tree and rule-based classifiers restricted to rectilinear decision boundaries

20. 4. Example: PEBLS
PEBLS: Parallel Example-Based Learning System (Cost & Salzberg)
Works with both continuous and nominal features
For nominal features, distance between two nominal values is computed using modified value difference metric (MVDM)
Each record is assigned a weight factor
Number of nearest neighbor, k = 1

21. Example: PEBLS Distance between nominal attribute values:
d(Single,Married)
= | 2/4 – 0/4 | + | 2/4 – 4/4 | = 1
d(Single,Divorced)
= | 2/4 – 1/2 | + | 2/4 – 1/2 | = 0
d(Married,Divorced)
= | 0/4 – 1/2 | + | 4/4 – 1/2 | = 1
d(Refund=Yes,Refund=No)
= | 0/3 – 3/7 | + | 3/3 – 4/7 | = 6/7
Class
Marital Status
Single
Married
Divorced
Yes 2 1 No 4
Class
Refund
Yes No 3 4

22. Example: PEBLS Distance between record X and record Y:
where:
wX  1 if X makes accurate prediction most of the time
wX > 1 if X is not reliable for making predictions

23. Example 2 Distance of two documents Word Touchdown Capital Murder
Juror
profit
Score
Doc A 2 1 3
Doc B 5
Doc C 4
Doc E
Doc F
Doc G

24. App in natural language processing
Use term frequencies to compare documents and classify documents in a corpora
Problem:
Normalizing the word counts to avoid the closed-class word like ‘the’, ‘to’, ‘in’, etc.
Solution: use the tf.idf to normalize word ‘Importance’ when counting the term frequencies

25. Significance tf: Term Frequencies Idf: inverse document frequency
DF.IDF
the
1.00
Chile
0.167
Pinochet
1.0 of
0.703
his
0.159
Spanish
0.377 in
0.536
have
0.152
0.254 a
0.504
has
0.149
Chilean
0.245
and
0.489
British
0.127
Garzon
0.238 to
0.482
The
Argentina
0.176
0.380
arrest
0.120
diplomatic
0.163
that
0.308
who
0.154
for
0.250
0.105
immunity
0.141
said
0.214
Argent.
0.098
dictator
0.133
was
0.203 by
0.123 he
0.196
from
military
0.117
0.185 as
0.094
genocide
0.114 ‘s
0.174
with
Augusto on
0.170
Surgery
0.090
crime
Significance
tf: Term Frequencies
Idf: inverse document frequency
= log(N/df)
N: #of docs in a training corpus
df: #of docs in which term i appear
Ft.idf =
Tf*log(N/df)