1. Classification of unlabeled data:
Learning Theory
Reza Shadmehr
Classification of unlabeled data:
Mixture models, K-means algorithm, and Expectation Maximization (EM)

2. The problem of finding labels for unlabeled data
So far we have solved “supervised” classification problems where a teacher told us the label of each example. In nature, items often do not come with labels. How can we learn labels without a teacher?
Unlabeled data
Labeled data
-10 -8 -6 -4 -2 2 4 6
-10 -8 -6 -4 -2 2 4 6

3. Example: image segmentation
Identify pixels that are white matter, gray matter, or outside of the brain.
Outside the brain
0.2
0.4
0.6
0.8 1
250
500
750
1000
1250
1500
Gray matter
White matter
Pixel value (normalized)

4. If our data is not labeled, we can hypothesize that:
Mixtures
If our data is not labeled, we can hypothesize that:
There are exactly m classes in the data:
Each class y occurs with a specific frequency:
Examples of class y are governed by a specific distribution:
According to our hypothesis, each example x(i) must have been generated from a specific “mixture” distribution:
We might hypothesize that the distributions are Gaussian:
Parameters of the distributions
Mixing proportions
Normal distribution

5. Each data point is generated by a single Gaussian
Mixture densities
Suppose there are only three classes. The data generation process will be something like this:
Each data point is generated by a single Gaussian
Multinomial random variable that represents the class x belongs to
If we had a classifier that could sort our data, we would get labels like this. Here, the idea is that if x is assigned to class i, we set the i-th component of vector z equal to one and keep the remaining components of z at zero.

6. Hidden variable
Measured variable

7. Step 2: Re-estimate the center of each class.
Mixture densities
Our imagined classifier
If our data were labeled, we could estimate the mean and variance of each Gaussian and compute the mixing proportion.
Number of times a particular class was present
Mixing proportion
Mean of each class
Variance of each class
The K-means algorithm begins by making an initial guess about class center:
Step 1: Assign each data point to the class that has the closest center.
Step 2: Re-estimate the center of each class.

8. K-means algorithm on an easy problem (first 5 iterations - converged)
classification
True labels
-10 -8 -6 -4 -2 2 4 6
-10 -8 -6 -4 -2 2 4 6
-10 -8 -6 -4 -2 2 4 6

9. K-means algorithm on a harder problem (first 10 iterations-converged)
-12
-10 -8 -6 -4 -2 2 4 6
K-means
classification
True labels
-12
-10 -8 -6 -4 -2 2
-12
-10 -8 -6 -4 -2 2

10. The cost function for the K-means algorithm
Step 1: assuming that class centers are known, to minimize cost we should classify a point based on the closest center.
Step 2: assuming that class memberships are known, the derivative of cost with respect to centers dictates that centers should be in the middle of the class.
Number of points in class j

11. Types of classification data
Data set is complete.
K-means: Data set is incomplete, but we complete it by assigning ‘hard’ memberships.
EM: data set is incomplete, but we complete it using posterior probabilities (a “soft” class membership).

12. EM algorithm
Instead of using a “hard” class assignment, we could use a “soft” assignment that represents the posterior probability of class membership for each data point:
This gives us a “soft” class membership. Using this classifier, we can now re-estimate the parameters of our distribution for each class:
“soft” class membership
“Number of times” a particular class was present (no longer an integer!)

13. EM algorithm: iteration 0
We assume that there are exactly m classes in the data. We begin with a guess about the initial setting of the parameters:
We select m random samples from the dataset and assume that these are the mean of each class.
We set the variance of each class to the sampled covariance of the whole data.
We could set the mixing coefficients to be equal.

14. EM algorithm: iteration k+1
We have a guess about the mixture parameters:
The “E” step: Complete the incomplete data with the posterior probabilities.
For each data point, we assign a class membership by computing the posterior probability for all m classes:
The “M” step: estimate new mixture parameters based on the new class assignments:

15. Example: classification with EM (easy problem)
Initial guess
5th iteration
10th iteration
15th iteration
-10 -8 -6 -4 -2 2 4 6
-10 -8 -6 -4 -2 2 4 6
-10 -8 -6 -4 -2 2 4 6
-10 -8 -6 -4 -2 2 4 6
To plot each distribution, I draw an ellipse centered at the mean with an area that covers the expected location of the 1st, 2nd, and 3rd quartile of the data.
-10 -8 -6 -4 -2 2 4 6
-10 -8 -6 -4 -2 2 4 6
Colors indicate probability of belonging to one class or another.
True labeled data
Posterior probabilities

16. EM algorithm: log-likelihood increases with each step10 20 30 40
-930
-920
-910
-900
-890
-880
-870

17. Example: classification with EM (hard problem)
Initial guess
30th iteration
60th iteration
-12
-10 -8 -6 -4 -2 2
-12
-10 -8 -6 -4 -2 2
-12
-10 -8 -6 -4 -2 2
-12
-10 -8 -6 -4 -2 2
Posterior probabilities
True labeled data 20 40 60 80
-858
-856
-854
-852
-850
iteration

18. Some observations about EM:
EM takes many more iterations to reach convergence than K-means.
Each cycles requires significantly more computation.
It is common to run K-means first to find a reasonable initialization for EM.
Covariance matrices can be set to cluster covariance found for K-means.
Mixing proportions can be set to the fraction of data assigned to each cluster in K-means.

19. On the “M” step, EM maximizes the log-likelihood of the data
Under the assumption that pi is constant

20. EM and maximizing log-likelihood of the data
This is the same as the rule that we had come up with “soft” classification.
The rules are the same for estimates of class variance and mixture probabilities.
Note that these equation for the parameters are not solutions because the posterior probabilities are themselves function of parameters.
What we have done with the algorithm is to try and solve for the parameters iteratively: we made a guess about the parameters, computed the posteriors, and then found the parameters that maximized the likelihood of the data. We iterated until we found convergence.

21. EM algorithm: iteration k+1
We have a guess about the mixture parameters:
The “E” step: Compute the posterior probabilities for all data points:
The “M” step: under the assumption that the posterior probabilities are constant, estimate new parameter values to maximize log-likelihood: