Lecture 07: Soft-margin SVM CS489/698: Intro to ML Lecture 07: Soft-margin SVM 10/5/17 Yao-Liang Yu
Announcement A2 due tonight Proposal due next Tuesday ICLR 2018 Reproducibility Challenge select a paper from 2018 ICLR submissions, and aim to replicate the experiments described in the paper produce a Reproducibility report, describing the target questions, experimental methodology, implementation details, analysis and discussion of findings, conclusions on reproducibility of the paper 10/5/17 Yao-Liang Yu
Outline Formulation Dual Optimization Extension 10/5/17 Yao-Liang Yu
Hard-margin SVM Primal hard constraint Dual 10/5/17 Yao-Liang Yu
What if inseparable? 10/5/17 Yao-Liang Yu
Soft-margin (Cortes & Vapnik’95) Primal hard constraint propto 1/margin hyper-parameter training error Primal soft constraint prediction (no sign) 10/5/17 Yao-Liang Yu
Zero-one loss your prediction Find prediction rule f so that on an unseen random X, our prediction sign(f(X)) has small chance to be different from the true label Y 10/5/17 Yao-Liang Yu
The hinge loss upper bound zero-one Squared hinge exponential loss logistic loss exponential loss Squared hinge zero-one still suffer loss! 10/5/17 Yao-Liang Yu
Classification-calibration Want to minimize zero-one loss End up with minimizing some other loss Theorem (Bartlett, Jordan, McAuLiffe’06). Any convex margin loss ℓ is classification-calibrated iff ℓ is differentiable at 0 and ℓ’(0) < 0. Classification calibration. has the same sign as , i.e., the Bayes rule. 10/5/17 Yao-Liang Yu
Outline Formulation Dual Optimization Extension 10/5/17 Yao-Liang Yu
Important optimization trick joint over x and t 10/5/17 Yao-Liang Yu
Slack for “wrong” prediction 10/5/17 Yao-Liang Yu
Lagrangian 10/5/17 Yao-Liang Yu
Dual problem only dot product is needed! 10/5/17 Yao-Liang Yu
The effect of C RdxR C 0? C inf? Rn 10/5/17 Yao-Liang Yu
Karush-Kuhn-Tucker conditions Primal constraints on w, b and ξ: Dual constraints on α and β: Complementary slackness Stationarity 10/5/17 Yao-Liang Yu
Parsing the equations 10/5/17 Yao-Liang Yu
Support Vectors 10/5/17 Yao-Liang Yu
Recover b Take any i such that Then xi is on the hyperplane: How to recover ξ ? What if there is no such i ? 10/5/17 Yao-Liang Yu
More examples 10/5/17 Yao-Liang Yu
Just in case 10/5/17 Yao-Liang Yu
Outline Formulation Dual Optimization Extension 10/5/17 Yao-Liang Yu
Gradient Descent Step size (learning rate) const., if L is smooth diminishing, otherwise (Generalized) gradient O(nd) ! 10/5/17 Yao-Liang Yu
Stochastic Gradient Descent (SGD) average over n samples a random sample suffices O(d) diminishing step size, e.g., 1/sqrt{t} or 1/t averaging, momentum, variance-reduction, etc. sample w/o replacement; cycle; permute in each pass 10/5/17 Yao-Liang Yu
The derivative What about zero-one loss? All other losses are diff. What about perceptron? 10/5/17 Yao-Liang Yu
Solving the dual O(n*n) Can choose constant step size ηt = η Faster algorithms exist: e.g., choose a pair of αp and αq and derive a closed-form update 10/5/17 Yao-Liang Yu
A little history on optimization Gradient descent mentioned first in (Cauchy, 1847) First rigorous convergence proof (Curry, 1944) SGD proposed and analyzed (Robbins & Monro, 1951) 10/5/17 Yao-Liang Yu
Herbert Robbins (1915 – 2001) 10/5/17 Yao-Liang Yu
Outline Formulation Dual Optimization Extension 10/5/17 Yao-Liang Yu
Multiclass (Crammer & Singer’01) separate by a “safety margin” Prediction for correct class Prediction for wrong classes Soft-margin is similar Many other variants Calibration theory is more involved 10/5/17 Yao-Liang Yu
Regression (Drucker et al.’97) 10/5/17 Yao-Liang Yu
Large-scale training (You, Demmel, et al.’17) Randomly partition training data evenly into p nodes Train SVM independently on each node Compute center on each node For a test sample Find the nearest center (node / SVM) Predict using the corresponding node / SVM 10/5/17 Yao-Liang Yu
Questions? 10/5/17 Yao-Liang Yu