Support Vector Machines Joseph Gonzalez TexPoint fonts used in EMF. Read the TexPoint manual before you delete this box.: AAAAAA

From a linear classifier to ... *One of the most famous slides you will see, ever!

*One of the most famous slides you will see, ever! Maximum margin Maximum possible separation between positive and negative training examples *One of the most famous slides you will see, ever!

The Big Idea X X X X X X O O X O O O O O O O - how many people are scared of svms? maybe no one... anyway... it’s a really really simple classifier... - set of points, what do we want? - how about “as separating as possible”, “maximum margin” - note how if we do this, only the “boundary” points determine the decision boundary. those will be called support vectors! X X O O X O O O O O O O

Geometric Intuition SUPPORT VECTORS X X X O O X O O

Geometric Intuition SUPPORT VECTORS X X X O O X X O O

Primal Version min ||w||2 +C ∑ξ s.t. (w.x + b)y ≥ 1-ξ ξ ≥ 0 look at that tiny vector w.x is greater than 0

max ∑α -1/2 ∑αiαjyiyjxixj DUAL Version max ∑α -1/2 ∑αiαjyiyjxixj s.t. ∑αiyi = 0 C ≥ αi ≥ 0 Where did this come from? Remember Lagrange Multipliers Let us “incorporate” constraints into objective Then solve the problem in the “dual” space of lagrange multipliers - Lagrange: useful tool -- so here’s a summary - Uh, but WHY did we go to the dual form??

max ∑α -1/2 ∑αiαjyiyjxixj Primal vs Dual min ||w||2 +C ∑ξ s.t. (w.x + b)y ≥ 1-ξ ξ ≥ 0 max ∑α -1/2 ∑αiαjyiyjxixj s.t. ∑αiyi = 0 C ≥ αi ≥ 0 Number of parameters? large # features? large # examples? for large # features, DUAL preferred many αi can go to zero! - kernel trick... yeah... - some solvers are optimized for this (also said Carlos)

DUAL: the “Support vector” version max ∑α - 1/2 ∑αiαjyiyjxixj s.t. ∑αiyi = 0 C ≥ αi ≥ 0 Wait... how do we predict y for a new point x?? How do we find w? How do we find b? How do we find α? Quadratic programming How do we find C? Cross-validation! w: we know that lagrangian derivative is 0 if at a minimum. Comes from setting the derivative of lagrangian w/r.t. w to zero y = sign(w.x+b) w = Σi αi yi xi y = sign(Σi αi yi xi xj + b)

max ∑α - 1/2 ∑αiαjyiyjxixj “Support Vector”s? max ∑α - 1/2 ∑αiαjyiyjxixj s.t. ∑αiyi = 0 C ≥ αi ≥ 0 y=w.x+b b = y-w.x x1: b = 1- .4 [-2 -1][0 1] = 1+.4 =1.4 b α2 max ∑α - α1α2(-1)(0+2) - 1/2 α12(1)(0+1) - 1/2 α22(1)(4+4) X well... it’s the ratio between the alphas that matter, right... Did everyone get what “support vectors” are? Tell me! . decision boundary? . support vectors? (2,2) α1 O max α1 + α2 + 2α1α2 - α12/2 - 4α22 s.t. α1-α2 = 0 C ≥ αi ≥ 0 (0,1) 4/5 α1=α2=α max 2α -5/2α2 max 5/2α(4/5-α) 2/5 α1=α2=2/5 w = Σi αi yi xi w = .4([0 1]-[2 2]) =.4[-2 -1 ]

max ∑α - 1/2 ∑αiαjyiyjxixj “Support Vector”s? max ∑α - 1/2 ∑αiαjyiyjxixj s.t. ∑αiyi = 0 C ≥ αi ≥ 0 α2 X “power” (alpha) for positive is spread now (2,2) What is α3? Try this at home α1 O (0,1) O α3

Playing With SVMS http://www.csie.ntu.edu.tw/~cjlin/libsvm/

More on Kernels Kernels represent inner products K(a,b) = a.b K(a,b) = φ(a) . φ(b) Kernel trick is allows extremely complex φ( ) while keeping K(a,b) simple Goal: Avoid having to directly construct φ( ) at any point in the algorithm

Kernels Complexity of the optimization problem remains only dependent on the dimensionality of the input space and not of the feature space!

Can we used Kernels to Measure Distances? Can we measure distance between φ(a) and φ(b) using K(a,b)?

Continued:

Popular Kernel Methods Gaussian Processes Kernel Regression (Smoothing) Nadarayan-Watson Kernel Regression