Incremental Support Vector Machine Classification Second SIAM International Conference on Data Mining Arlington, Virginia, April 11-13, 2002 Glenn Fung & Olvi Mangasarian Data Mining Institute University of Wisconsin - Madison

Key Contributions  Fast incremental classifier based on PSVM  Proximal Support Vector Machine  Capable of modifying an existing linear classifier by both adding and retiring data  Extremely simple to implement  Small memory requirement  Even for huge problems (1 billion)  NO optimization packages (LP,QP) needed

Outline of Talk  (Standard) Support vector machines (SVM)  Classification by halfspaces  Proximal linear support vector machines (PSVM)  Classification by proximity to planes  The incremental and decremental algorithm  Option of keeping or retiring old data  Numerical results  1 Billion points in 10 dimensional space classified in less than 3 hours!  Numerical results confirm that algorithm time is linear in the number of data points

Support Vector Machines Maximizing the Margin between Bounding Planes A+ A-

Proximal Support Vector Machines Fitting the Data using two parallel Bounding Planes A+ A-

Standard Support Vector Machine Algebra of 2-Category Linearly Separable Case  Given m points in n dimensional space  Represented by an m-by-n matrix A  Membership of each in class +1 or –1 specified by:  An m-by-m diagonal matrix D with +1 & -1 entries  More succinctly: where e is a vector of ones.  Separate by two bounding planes,

Standard Support Vector Machine Formulation  Margin is maximized by minimizing  Solve the quadratic program for some : min s. t. (QP),, denotes where or membership.

PSVM Formulation We have from the standard QP SVM formulation: (QP) min s. t. This simple, but critical modification, changes the nature of the optimization problem tremendously!! Solving for in terms of and gives: min

Advantages of New Formulation  Objective function remains strongly convex.  An explicit exact solution can be written in terms of the problem data.  PSVM classifier is obtained by solving a single system of linear equations in the usually small dimensional input space.  Exact leave-one-out-correctness can be obtained in terms of problem data.

Linear PSVM  We want to solve: min  Setting the gradient equal to zero, gives a nonsingular system of linear equations.  Solution of the system gives the desired PSVM classifier.

Linear PSVM Solution Here,  The linear system to solve depends on: which is of size  is usually much smaller than

Linear Proximal SVM Algorithm Classifier: Input Define Solve Calculate

Linear & Nonlinear PSVM MATLAB Code function [w, gamma] = psvm(A,d,nu) % PSVM: linear and nonlinear classification % INPUT: A, d=diag(D), nu. OUTPUT: w, gamma % [w, gamma] = psvm(A,d,nu); [m,n]=size(A);e=ones(m,1);H=[A -e]; v=(d’*H)’ %v=H’*D*e; r=(speye(n+1)/nu+H’*H)\v % solve (I/nu+H’*H)r=v w=r(1:n);gamma=r(n+1); % getting w,gamma from r

Incremental PSVM Classification  The linear system to solve depends on the compressed blocks: which are of the size and  Suppose we have two “blocks” of data

Linear Incremental Proximal SVM Algorithm Initialization Read from disk Compute and Store in memory Yes Compute output Update in memory No Discard: Keep:

Linear Incremental Proximal SVM Adding – Retiring Data  Capable of modifying an existing linear classifier by both adding and retiring data  Option of retiring old data is similar to adding new data  Financial Data: old data is obsolete  Option of keeping old data and merging it with the new data:  Medical Data: old data does not obsolesce.

Numerical experiments One-Billion Two-Class Dataset  Synthetic dataset consisting of 1 billion points in 10- dimensional input space  Generated by NDC (Normally Distributed Clustered) dataset generator  Dataset divided into 500 blocks of 2 million points each.  Solution obtained in less than 2 hours and 26 minutes  About 30% of the time was spent reading data from disk.  Testing set Correctness 90.79%

Numerical Experiments Simulation of Two-month 60-Million Dataset  Synthetic dataset consisting of 60 million points (1 million per day) in 10- dimensional input space  Generated using NDC  At the beginning, we only have data corresponding to the first month  Every day:  The oldest block of data is retired (1 Million)  A new block is added (1 Million)  A new linear classifier is calculated daily  Only an 11 by 11 matrix is kept in memory at the end of each day. All other data is purged.

Numerical experiments Separator changing through time

Numerical experiments Normals to the separating hyperplanes Corresponding to 5 day intervals

Conclusion  Proposed algorithm is an extremely simple procedure for generating linear classifiers in an incremental fashion for huge datasets.  The linear classifier is obtained by solving a single system of linear equations in the small dimensional input space.  The proposed algorithm has the ability to retire old data and add new data in a very simple manner.  Only a matrix of the size of the input space is kept in memory at any time

Future Work  Extension to nonlinear classification  Parallel formulation and implementation on remotely located servers for massive datasets  Real time on-line application, e.g. fraud detection