Extreme Re-balancing for SVMs and other classifiers Presenter: Cui, Shuoyang 2005/03/02 Authors: Bhavani Raskutti & Adam Kowalczyk Telstra Croporation Victoria, Austalia

Imbalance makes the minority-classes samples farther from the true boundary than the majority-class samples. Majority-class samples dominate the penalty introduced by soft margin. ideal Majority Minority

Data Balancing up/down samplings No convincing evidence for how the balanced data sampled Imbalance-free algorithm design Objective function should not be accuracy any longer Reference: Machine Learning from Imbalanced Data Sets 101

In this paper Exploring the characters of two class learning and analyses situations with supervised learning. In the experiments offered later, comparing one-class learning with two class learning and list different forms of imbalance compensation.

Two class discrimination to take examples from these two classes generate a model for discriminating them for many machine learning algorithms, the training data should include the example form two classes.

When the data has heavily unbalanced representatives of these two class. design re-balancing ignore the large pool of negative examples learn from positive examples only

Why extreme re-balancing Extreme imbalance in very high dimensional input spaces Minority class consisting of 1-3% of the total data Learning sample size is much below the dimensionality of the input space Data site has more than 10,000 features

The kernel machine The kernel machine is solved iteratively using the conjugate gradient method. Designing a kernel machine is to take a standard algorithm and massage it so that all references to the original data vectors x appear only in dot products ( xi; xj). Given a training sequence(xi,yi) of binary n— vectors and bipolar labels

Two different cases of kernel machines used here

Two forms of imbalance compensation Sample balancing Weight balancing

Sample balancing 1: the case of 1-class learner using all of the negative examples 1: the case of 2-class learner using all training examples 0: the case of 1-class learner using all of the positive examples

Weight balancing Using different values of the regularisation of the regulation constants for both the minority and majority class data B is a parameter called a balance factor

Experiments Real world data collections AHR-data Reuters data

Combined training and test data set Each training instance labeled with “control”, “change” or “nc” Convert all of the info from different files to a sparse matrix containing features AHR-data

Reuters data A collection of documents Each document has been converted to a vector of dimensional word-presence feature space

AROC is used as performance measure AROC is the Area under the ROC Receiver operating characteristic (ROC) curves are used to describe and compare the performance of diagnostic technology and diagnostic algorithms.

Experiments with Real World Data Impact of regularisation constant Experiment with sample balancing Experiments with weight balancing

Impact of regularisation constant

Experiment with sample balancing The AROC with 2-class learners is close to 1 for all categories indicating that this categorization problem is easy to learn

Experiments with weight balancing 1.Test on AHRdata

Experiments with weight balancing 2.Test on Reuters To observe the performance ouf 1-class and 2-class SVMs when the most features are moved

The characters of test on Reuters The accuracy of all classifiers is very high SVM models start degenerating, the drop in performance for 2-class SVM is larger. 1-class SVM models start outperforming 2-class models Similar trends AROC is always bigger than 0.5