1. Concave Minimization for Support Vector Machine Classifiers
Unlabeled Data Classification & Data Selection
Glenn Fung
O. L. Mangasarian

2. Part 1: Unlabeled Data Classification
Given a large unlabeled dataset
Use a k-Median clustering algorithm to select a small (5% to 10%) representative sample.
Representative sample is labeled by expert or oracle.
Combined labeled-unlabeled dataset is classified by a Semi-supervised Support Vector Machine.
Test set correctness within 5.2% of a linear support vector machine trained on the entire dataset labeled by an expert.

3. Part 2: Data Selection for Support Vector
Machines Classifiers
Extract a minimal set of data points from a given dataset.
Minimal set used to generate a Minimal Support Vector Machine (MSVM) classifier.
MSVM classifier as good or better than that obtained by training on entire dataset.
Feature selection is incorporated into procedure to obtain a minimal set of input features.
Data reduction as high as 81% and averaged 66% over seven public datasets.

4. SVM: Linear Support Vector Machine

5. 1-norm Linear SVM

6. Unlabeled Data Classification
Given a completely unlabeled large data set.
Costly to label points by an expert or an oracle.
Two Question arise:
How to choose a small subset for labeling?
How to combine labeled and unlabeled data?
Answers:
Use k-median clustering for selecting “representative” points to be labeled.
Use semi-supervised SVM to obtain a classifier based on labeled and unlabeled data.

7. Unlabeled Data Classification
Unlabeled Data Set
k-Median
clustering
Chosen
Data
Remaining
Data
Expert
Labeled
Data
Semi-supervised
SVM
Separating Plane

8. K-Median Clustering Algorithm
Given m data points. Find k clusters of these points such that the sum of the 1-norm distances from each point to the closest cluster center is minimized.

9. K-Median Clustering Algorithm*

10. K-Median Clustering Algorithm

11. Unlabeled Data Classification
Unlabeled Data Set
k-Median
clustering
Chosen
Data
Remaining
Data
Expert
Labeled
Data
Semi-supervised
SVM
Separating Plane

12. Semi-supervised SVM (S3VM)
Given a dataset consisting of:
labeled (+1,-1) points represented by:
unlabeled points represented by:
Classify the data into two classes as follows:
Assign each unlabeled point in to a class (+1,-1) so as to maximize the distance between the bounding planes obtained by a linear SVM1 applied to entire dataset.

13. Formulation

14. :A concave approach
The term in the objective function is concave because it is the minimum of two linear functions.
A local solution to this problem is obtained solving a succession of linear programs (4 to 7) .

15. S3VM: Graphical Example Separate Triangles & Circles
Hollow shapes represent
labeled data
Solid shapes represent
unlabeled data
SVM
S3VM

18. Numerical Tests

19. Part 2: Data Selection for Support Vector
Machines Classifiers
Labeled dataset
1-norm SVM
feature selection
Smaller dimension
dataset
Support vector
suppression
MSVM
Separating surface

20. Support Vectors

21. Feature Selection using 1-norm Linear SVM ( small.)

22. Motivation for the Minimal Support Vector Machine (MSVM)

23. Motivation for the Minimal Support Vector Machine (MSVM)
Suppression of error term y:
Minimizes the number of misclassified points.
Works remarkably well computationally.
Reduces positive components of multiplier u and hence number of support vectors.

24. MSVM Formulation

25. MSVM Formulation

27. Numerical Tests

28. Unlabeled data classification:
Conclusions
Unlabeled data classification:
A fast finite linear programming based approach for Semi-supervised Support Vector Machines was proposed for classifying large datasets that are mostly unlabeled.
Totally unlabeled datasets were classified by:
Labeling a small percentage of clusters by an expert
Classification by a semi-supervised SVM
Test set correctness within 5.2% of a linear SVM trained on the entire dataset labeled by an expert.

29. Data selection for SVM classifiers:
Conclusions
Data selection for SVM classifiers:
Minimal SVM (MSVM) extracts a minimal subset used to classify the entire dataset.
MSVM maintains or improves generalization over other classifiers that use the entire dataset.
Data reduction as high as 81%, and averaged 66% over seven public datasets.
Future work
MSVM: Promising tool for incremental algorithms.
Improve chunking algorithms with MSVM.
Nonlinear MSVM: strong potential for time & storage reduction.