1. A Fast and Scalable Nearest Neighbor Based Classification
Taufik Abidin and William Perrizo
Department of Computer Science
North Dakota State University

2. Outline Nearest Neighbors Classification Problems
SMART TV (SMall Absolute diffeRence of ToTal Variation): A Fast and Scalable Nearest Neighbors Classification Algorithm
SMART TV in Image Classification

3. Search for the K-Nearest Neighbors
Classification
Given a (large) TRAINING SET, R(A1,…,An, C), with C=CLASSES and {A1…An}=FEATURES
Classification is: labeling unclassified objects based on the class label assignment pattern of objects in the training set
kNN classification goes as follows:
Training Set
Search for the K-Nearest Neighbors
Vote the class
Unclassified Object

4. Problems with KNN
Finding k-Nearest Neighbor Set can be expensive when the training set contains millions of objects (very large training set)
linear to the size of the training set
Can we make it faster (more scalable)?

5. But it is pure (pure0) so this branch ends
A file, R(A1..An), containing horizontal structures (records) is
Predicate trees: vertically partition; compress each vertical bit slice into a basic Ptree;
horizontally process these basic Ptrees using one multi-operand logical AND.
processed vertically (vertical scans)
R( A1 A2 A3 A4)
R[A1] R[A2] R[A3] R[A4]
Horizontal structures
(records)
Scanned vertically
R11 1
R11 R12 R13 R21 R22 R23 R31 R32 R33 R41 R42 R43
1-Dimensional Ptrees are built by recording the truth of the predicate “pure 1” recursively on halves, until there is purity, P11:
1. Whole file is not pure1 0
2. 1st half is not pure1  0
3. 2nd half is not pure1  0
0 0
P11 P12 P13 P21 P22 P23 P31 P32 P33 P41 P42 P43
0 0
0 1 10
1 0 01
1 0
0 1
5. 2nd half of 2nd half is  1
0 0
0 1
6. 1st half of 1st of 2nd is  1
0 0
0 1 1
7. 2nd half of 1st of 2nd not 0
0 0
0 1 10
4. 1st half of 2nd half not  0
0 0
But it is pure (pure0) so this branch ends
Eg, to count, s, use “pure ”: level
P11^P12^P13^P’21^P’22^P’23^P’31^P’32^P33^P41^P’42^P’43 = level =2
level

6. Total Variation
The Total Variation of a set X about (the mean), , measures total squared separation of objects in X about , defined as follows:
We will use the concept of functional contours in this presentation to determine small pruned superset of the nearest neighbor set (which will then be scanned)
First we will discuss functional contours in general then consider the specific TV contours.

7. Given f:R(A1..An)Y (any range) and SY (any subset of the range) , define contour(f,S)  f-1(S).
A1 A An : :
YS
A1..An space Y S
graph(f) =
{ (a1,...,an,f(a1.an)) | (a1..an)R }
contour(f,S)
There is a DUALITY between functions, f:R(A1..An)Y and derived attributes, Af of R given by
x.Af  f(x)
where Dom(Af)=Y
A1 A An
x1 x xn :
. . .
Y f(x) f
A1 A An Af
x1 x xn f(x) R R*
From the derived attribute point of view,
Contour(Af,S) = SELECT A1..An FROM R* WHERE R*.Af  S.
If S={a}, f-1({a}) is Isobar(f, a)

8. = xRd=1..n(xd2 - 2adxd + ad2) i,j,k bit slices indexes
R(A1..An) TV(a)=xR(x-a)o(x-a) If we use d for a index variable over the dimensions,
= xRd=1..n(xd adxd ad2) i,j,k bit slices indexes
= xRd=1..n(k2kxdk)2 -
2xRd=1..nad(k2kxdk) +
|R||a|2
= xd(i2ixdi)(j2jxdj) -
2xRd=1..nad(k2kxdk) +
|R||a|2
= xdi,j 2i+jxdixdj
2 x,d,k2k ad xdk
|R||a|2
= x,d,i,j 2i+j xdixdj
|R||a|2
2 dad x,k2kxdk
= x,d,i,j 2i+j xdixdj
|R|dadad
2|R| dadd +
= x,d,i,j 2i+j xdixdj
dadad )
|R|( -2dadd +
TV(a) = i,j,d 2i+j |Pdi^dj| -
|R||a|2
k2k+1 dad |Pdk| +
The first term does not depend upon a, thus, the simpler derived attribute, TV-TV() does not have that
term but has with identical contours as TV (just lowered the graph by the constant, TV() ). We also
find it useful to apply a log to this simpler Total Variation function (to reduce the number of bit slices. The
resulting functional is called the High-Dimension-ready Total Variation or HDTV(a).

9. TV(a) = x,d,i,j 2i+j xdixdj + |R| ( -2dadd + dadad )
From equation 7,
f(a)=TV(a)-TV()
TV(a) = x,d,i,j 2i+j xdixdj
+ |R| ( -2dadd +
dadad )
= |R| ( -2d(add-dd) +
d(adad- dd) )
+ dd2 )
= |R|( dad2
- 2ddad
= |R| |a-|2 so
f()=0 and letting g(a) HDTV(a) = ln( f(a) )=
ln|R| + ln|a-|2
Taking  g / ad (a) =
| a- |2 2( a
-)d
The Gradient of g at a =
2/| a- |2 (a
-)
The gradient =0 iff a= and gradient length depends only on the length of a- so isobars are hyper-circles
The gradient function is has the form, h(r) = 2/r in along any ray from , Integrating, we get that g(a) has the form, 2ln|a-| along any coordinate direction (in fact any radial direction from ), so the shape of graph(g) is a funnel:
To get an -contour, we move in and out along a- by  to
inner point, b=(1-/|a-|)(a-) and
outer point c=(1+/|a-|)(a-).
Then take f(b) and f(c) as lower and upper endpoints of the red vertical interval. Then we use formulas on that interval to get a P-tree for the -contour (which is a well-pruned superset of the -nbrhd of a 
-contour
(radius  about a)
What inteval
endpts gives an exact -contour
in feature space? a
f(b)
f(c) b c

10. 2. Calculate b and c (which depend upon a and )
For additional vertical pruning we can use any other functional contours that are can easily computed (e.g., the dimension projection functionals).
To classify a, then
1. Calculate basic P-trees for the derived attribute column of each training point
2. Calculate b and c (which depend upon a and )
3. Get the feature space P-tree for those points with derived attribute value in [f(b),f(c)] (Note, when the camera ready paper was submitted we were still doing this step by sorting TV(a) values and then forming the predicate tree. Now we use the contour approach which speeds up that step considerably).
4. User that P-tree to prune out the candidate NNS.
5. If the root count of the candidate set is now small, proceed to scan and assign votes using Gaussian vote weights, else look for another pruning functional (e.g., dimension projection function for the major a- dimensions). 
-contour
(radius  about a) a
f(b)
f(c) b c

11. Graph of TV, TV-TV() and HDTV
TV()=TV(x33)
TV(x15) 1 2 3 4 5 X Y TV 
HDTV 1 2 3
TV(x15)-TV() 4 5 X Y
TV-TV()

12. Dataset KDDCUP-99 Dataset (Network Intrusion Dataset)
4.8 millions records, 32 numerical attributes
6 classes, each contains >10,000 records
Class distribution:
Testing set: 120 records, 20 per class
4 synthetic datasets (randomly generated):
10,000 records (SS-I)
100,000 records (SS-II)
1,000,000 records (SS-III)
2,000,000 records (SS-IV)
Normal
972,780
IP sweep
12,481
Neptune
1,072,017
Port sweep
10,413
Satan
15,892
Smurf
2,807,886

13. Dataset (Cont.) OPTICS dataset
8,000 points, 8 classes (CL-1, CL-2,…,CL-8)
2 numerical attributes
Training set: 7,920 points
Testing set: 80 points, 10 per class

14. Dataset (Cont.) IRIS dataset 150 samples
3 classes (iris-setosa, iris-versicolor, and iris-virginica)
4 numerical attributes
Training set: 120 samples
Testing set: 30 samples, 10 per class

15. Speed and Scalability Speed and Scalability Comparison (k=5, hs=25)
Algorithm
x 1000 cardinality 10
100
1000
2000
4891
SMART-TV
0.14
0.33
2.01
3.88
9.27
P-KNN
0.89
1.06
3.94
12.44
30.79
KNN
0.39
2.34
23.47
49.28
　NA
Machine used:
Intel Pentium 4 CPU 2.6 GHz machine, 3.8GB RAM, running Red Hat Linux

16. Classification Accuracy (Cont.)
Classification Accuracy Comparison (SS-III), k=5, hs=25
Algorithm
Class TP FP P R F
SMART-TV
normal 18
1.00
0.90
0.95
ipsweep 20 1
0.98
neptune
portsweep
satan 17 2
0.85
0.87
smurf 4
0.83
0.91
P-KNN 15
0.75
0.86 14
0.93
0.70
0.80 5
0.89
KNN 3
0.94

17. Overall Classification Accuracy Comparison
Overall Accuracy
Overall Classification Accuracy Comparison
Datasets
SMART-TV
PKNN
KNN
IRIS
0.97
0.71
OPTICS
0.96
0.99
SS-I
0.72
0.89
SS-II
0.92
0.91
SS-III
0.94
SS-IV NI
0.93 NA

18. Summary
A nearest-based classification algorithm that starts its classification steps by approximating a number of candidates of nearest neighbors
The absolute difference of total variation between data points in the training set and the unclassified point is used to approximate the candidates
The algorithm is fast, and it scales well in very large dataset. The classification accuracy is very comparable to that of KNN algorithm.

19. Appendix: Image Preprocessing
We extracted color and texture features from the original pixel of the images
Color features: We used HVS color space and quantized the images into 52 bins i.e. (6 x 3 x 3) bins
Texture features: we used multi-resolutions Gabor filter with two scales and four orientation (see B.S. Manjunath, IEEE Trans. on Pattern Analysis and Machine Intelligence, 1996)

20. Image Dataset - 54 from color features - 16 from texture features
Corel images (
10 categories
Originally, each category has 100 images
Number of feature attributes:
- 54 from color features
- 16 from texture features
We randomly generated several bigger size datasets to evaluate the speed and scalability of the algorithms.
50 images for testing set, 5 for each category

21. Image Dataset

22. Example on Corel Dataset

23. Results Class SMART-TV KNN k=3 k=5 k=7 hs=15 hs=25 hs=35 C1 0.69 0.72
0.75
0.74
0.73
0.78
0.81
0.77
0.79 C2
0.64
0.60
0.59
0.62
0.68
0.63
0.66 C3
0.65
0.67
0.76
0.57
0.70 C4
0.84
0.87
0.90
0.88 C5
0.91
0.92
0.93
0.89
0.94 C6
0.61
0.71 C7
0.85 C8
0.96 C9
0.52
0.43
0.45
0.54
C10
0.82

24. Results
Classification Time

25. Results
Preprocessing Time

26. Appendix:Overview of SMART-TV
Compute Root Count
Measure HDTV of each object
Large
Training Set
Store the root count
and HDTV values
Preprocessing Phase
Unclassified Object
Approximate the candidate set of NNs
Search the K-nearest neighbors for the candidate set
Vote
Classifying Phase

27. Preprocessing Phase
Compute the root counts of each class Cj,  j  number of classes. Store the results.
Complexity: O(kdb2) where k is the number of classes, d is the total of dimensions, and b is the bit-width.
Compute , 1 j  number of classes.
Complexity: O(n) where n is the cardinality of the training set. Also, retain the results.

28. Classifying Phase Stored values of root count and TV Classifying Phase
Unclassified Object
Approximate the candidate set of NNs
Search the K-nearest neighbors from the candidate set
Vote
Classifying Phase

29. Classifying Phase
For each class Cj with nj objects, 1  j  number of classes, do the followings:
a. Compute , where is the unclassified object
Find hs objects in Cj such that the absolute difference between the total variation of the objects in Cj and the total variation of about Cj are the smallest, i.e.
Let A be an array and , where
Store all objectIDs in A into TVGapList

30. Classifying Phase (Cont.)
For each objectIDt, 1 t  Len(TVGapList) where Len(TVGapList) is equal to hs times the total number of classes, retrieve the corresponding object features from the training set and measure the pair-wise Euclidian distance between and , i.e.
and determine the k nearest neighbors of
Vote the class label for using the k nearest neighbors