1. K-means method for Signal Compression: Vector Quantization

2. Voronoi Region Blocks of signals: A sequence of audio.
A block of image pixels.
Formally: vector example: (0.2, 0.3, 0.5, 0.1)
A vector quantizer maps k-dimensional vectors in the vector space Rk into a finite set of vectors
Y = {yi: i = 1, 2, ..., N}. 
Each vector yi is called a code vector or a codeword. and the set of all the codewords is called a codebook.  Associated with each codeword, yi, is a nearest neighbor region called Voronoi region, and it is defined by:
The set of Voronoi regions partition the entire space Rk .

3. Two Dimensional Voronoi Diagram
Codewords in 2-dimensional space.  Input vectors are marked with an x, codewords are marked with red circles, and the Voronoi regions are separated with boundary lines.

4. The Schematic of a Vector Quantizer (signal compression)

5. Compression Formula Amount of compression:
Codebook size is K, input vector of dimension L
In order to inform the decoder of which code vector is selected, we need to use bits.
E.g. need 8 bits to represent 256 code vectors.
Rate: each code vector contains the reconstruction value of L source output samples, the number of bits per vector component would be:
K is called “level of vector quantizer”.

6. Vector Quantizer Algorithm
Determine the number of codewords, N,  or the size of the codebook.
Select N codewords at random, and let that be the initial codebook.  The initial codewords can be randomly chosen from the set of input vectors.
Using the Euclidean distance measure clusterize the vectors around each codeword.  This is done by taking each input vector and finding the Euclidean distance between it and each codeword.  The input vector belongs to the cluster of the codeword that yields the minimum distance.

7. Vector Quantizer Algorithm (contd.)
4. Compute the new set of codewords.  This is done by obtaining the average of each cluster.  Add the component of each vector and divide by the number of vectors in the cluster.
where i is the component of each vector (x, y, z, ... directions), m is the number of vectors in the cluster.
5. Repeat steps 2 and 3 until the either the codewords don't change or the change in the codewords is small.

8. Other Algorithms
Problem: k-means is a greedy algorithm, may fall into Local minimum.
Four methods selecting initial vectors:
Random
Splitting (with perturbation vector) Animation
Train with different subset
PNN (pairwise nearest neighbor)
Empty cell problem:
No input corresponds to am output vector
Solution: give to other clusters, e.g. most populate cluster.

9. VQ for image compression
Taking blocks of images as vector L=NM.
If K vectors in code book:
need to use bits.
Rate:
The higher the value K, the better quality, but lower compression ratio.
Overhead to transmit code book:
Train with a set of images.

10. K-Nearest Neighbor Learning
22c:145
University of Iowa

11. Different Learning Methods
Parametric Learning
The target function is described by a set of parameters (examples are forgotten)
E.g., structure and weights of a neural network
Instance-based Learning
Learning=storing all training instances
Classification=assigning target function to a new instance
Referred to as “Lazy” learning

12. Instance-based Learning
Its very similar to a
Desktop!!

13. General Idea of Instance-based Learning
Learning: store all the data instances
Performance:
when a new query instance is encountered
retrieve a similar set of related instances from memory
use to classify the new query

14. Pros and Cons of Instance Based Learning
Can construct a different approximation to the target function for each distinct query instance to be classified
Can use more complex, symbolic representations
Cons
Cost of classification can be high
Uses all attributes (do not learn which are most important)

15. Instance-based Learning
K-Nearest Neighbor Algorithm
Weighted Regression
Case-based reasoning

16. k-nearest neighbor (knn) learning
Most basic type of instance learning
Assumes all instances are points in n-dimensional space
A distance measure is needed to determine the “closeness” of instances
Classify an instance by finding its nearest neighbors and picking the most popular class among the neighbors

17. 1-Nearest Neighbor

18. 3-Nearest Neighbor

19. Important Decisions Distance measure Value of k (usually odd)
Voting mechanism
Memory indexing

20. Euclidean Distance Typically used for real valued attributes
Instance x (often called a feature vector)
Distance between two instances xi and xj

21. Discrete Valued Target Function
Training algorithm:
For each training example <x, f(x)>, add the example to the list training_examples
Classification algorithm:
Given a query instance xq to be classified.
Let x1…xk be the k training examples nearest to xq
Return

22. Continuous valued target function
Algorithm computes the mean value of the k nearest training examples rather than the most common value
Replace fine line in previous algorithm with

23. Training dataset Customer ID Debt Income Marital Status Risk Abel High
Married
Good
Ben
Low
Doubtful
Candy
Medium
Very low
Unmarried
Poor
Dale
Very high
Ellen
Fred
George
Harry
Igor
Very Low
Very High
Jack

24. k-nn
K = 3
Distance
Score for an attribute is 1 for a match and 0 otherwise
Distance is sum of scores for each attribute
Voting scheme: proportionate voting in case of ties

25. Query: Zeb High Medium Married ? Customer ID Debt Income
Marital Status
Risk
Abel
High
Married
Good
Ben
Low
Doubtful
Candy
Medium
Very low
Unmarried
Poor
Dale
Very high
Ellen
Fred
George
Harry
Igor
Very Low
Very High
Jack

26. Query: Yong Low High Married ? Customer ID Debt Income Marital Status
Risk
Abel
High
Married
Good
Ben
Low
Doubtful
Candy
Medium
Very low
Unmarried
Poor
Dale
Very high
Ellen
Fred
George
Harry
Igor
Very Low
Very High
Jack

27. Query: Vasco High Low Married ? Customer ID Debt Income Marital Status
Risk
Abel
High
Married
Good
Ben
Low
Doubtful
Candy
Medium
Very low
Unmarried
Poor
Dale
Very high
Ellen
Fred
George
Harry
Igor
Very Low
Very High
Jack

28. Voronoi Diagram
Decision surface formed by the training examples of two attributes

29. Examples of one attribute

30. Distance-Weighted Nearest Neighbor Algorithm
Assign weights to the neighbors based on their ‘distance’ from the query point
Weight ‘may’ be inverse square of the distances
All training points may influence a particular instance
Shepard’s method

31. Kernel function for Distance-Weighted Nearest Neighbor

32. Examples of one attribute

33. Remarks
+Highly effective inductive inference method for noisy training data and complex target functions
+Target function for a whole space may be described as a combination of less complex local approximations
+Learning is very simple
- Classification is time consuming

34. Curse of Dimensionality
- When the dimensionality increases, the volume of the space increases so fast that the available data becomes sparse. This sparsity is problematic for any method that requires statistical significance. 

35. Curse of Dimensionality
Suppose there are N data points of dimension n in the space [-1/2, 1/2]n.
The k-neighborhood of a point is defined to be the smallest hypercube containing the k-nearest neighbor.
Let l be the average side length of a k-neighborhood. Then the volume of an average hypercube is dn.
So dn/1n = k/N, or d = (k/N)1/n

36. d = (k/N)1/n N k n d
1,000,000 10 2
0.003 3
0.02 17
0.5
200
0.94
When n is big, all the points are outliers.

37. - Curse of Dimensionality

38. - Curse of Dimensionality