# 3.6 Support Vector Machines

## Presentation on theme: "3.6 Support Vector Machines"— Presentation transcript:

3.6 Support Vector Machines
K. M. Koo

Goal of SVM Find Maximum Margin
find a separating hyperplane with maximum margin margin minimum distance between a separating hyperplane and the sets of or

Goal of SVM Find Maximum Margin
assume that are linearly separable margin find separating hyperplane with maximum margin

Calculate margin separating hyperplane and are not uniquely determined
under the constraint , and are uniquely determine

Calculate margin distance between a point x and is given by
thus, the margin is given by

Optimization of margin
maximization of margin

Optimization of margin
separating hyperplane with maximal margin separating hyperplane with minimum therefore, we want to This is an optimization-problem with inequality constraints

optimization with constraints
cost function min-value min-value constraint constraint optimization with equality constraints optimization with inequality constraints

Lagrange Multiplier optimization problem under constraints can be solved by the method of Lagrange Multipliers let be real valued functions, let and ,and let , the level set for with value . assume if has a local minimum or maximum on at , which is called a critical point of ,then there is a real number ,called a Lagrange multiplier, such that 추가하자

The Method of Lagrange Multiplier

Lagrange Multiplier Lagrangian is obtained as follows: In our case
for equality constraints for inequality constraints In our case Inequality constraints

Convex a subset is convex iff for any , the line segment joining and is also a subset of , i.e. for any , a real-valued function on is convex iff for any two points and for any ,

Convex convex set concave set neither convex nor concave
convex function concave function

Convex Optimization an optimization problem is said to be convex iff the cost function as well as the constraints are convex the optimization problem for SVM is convex the solution to a convex problem, if it exist, is unique. that is, there is no local optimum! for convex optimization problem, KKT(Karush-Kuhn-Tucker) condition is necessary and sufficient for the solution

KKT(Karush-Kuhn-Tucker) condition
KKT condition The gradient of the Lagrangian with respect to the original variable is 0 The original constraints are satisfied Multipliers for inequality constraints (Complementary KKT) product of multiplier and constraints equal to 0 for convex optimize problems,1-4 are necessary and sufficient for the solution

KKT condition for the optimization of margin
recall KKT condition (3.66) (3.62) (3.63) (3.64) (3.65)

KKT condition for the optimization of margin
Combining (3.66) with (3.62) (3.67) (3.68)

Remarks-support vector
of the optimal solution is a linear combination of feature vectors which are associated with support vectors are associated with

Remarks-support vector
The resulting hyperplane classifier is insensitive to the number and position of non-support vector

Remark-computation w0 can be implicitly obtaines by any of the condition satisfying strict complement (i.e ) In practice, is computed as an average value obtained using all conditions of the type

Remark-optimal hyperplane is unique
the optimal hyperplane classifier of a support vector machine is unique under two condition the cost function is convex the inequality constraints consist of linear functions constraints are convex an optimization problem is said to be convex iff the target(or cost) function as well as the constraints are convex (the optimization problem for SVM is convex) the solution to a convex problem, if it exist, is unique. that is, there is no local optimum!

Computation optimal Lagrange multiplier
optimization problem belongs to the convex programming family (convex optimization problem) of problems It can be solved by considering the so called Lagrangian duality and can be stated equivalently by its Wolfe dual representation form Lagrangian duality Wolfe dual representation

Wolfe dual representation form

Computation optimal Lagrange multiplier
once the optimal Lagrangian multipliers have been computed, the optimal hyperplane is obtained (3.75) (3.76)

Remarks the cost function does not depend explicitly on the dimensionality of the input space this allows for efficient generalizations in the case of nonlinearly separable classes although the resulting optimal hyperplane is unique, there is no guarantee about Lagrange multipliers

Simple example consider the two classification task that consists of the following points its Lagrangian function KKT condition

Simple example Lagrangian duality optimize with equality constraint
result more then one solution

SVM for Non-separable Classes
in the case of non-separable, the training feature vector belong to one of the following three categories

SVM for Non-separable Classes
All three cases can be treated under a single type constraints

SVM for Non-separable Classes
goal is make the margin as large as possible keep the number of points with as small as possible (3.79) is intractable because of discontinuous function (3.79)

SVM for Non-separable Classes
as common case, we choose to optimize a closely related cost function

SVM for Non-separable Classes
to Lagrangian

SVM for Non-separable Classes
The corresponding KKT condition (3.85) (3.86) (3.87) (3.90) (3.88) (3.89)

SVM for Non-separable Classes
The associated Wolfe dual representation now becomes

SVM for Non-separable Classes
equivalent to

Remarks-difference with the linearly separable case
Lagrange multipliers( ) need to be bounded by C the slack variables, , and their associated Lagrange multipliers, , do not enter into the problem explicitly reflected indirectly though C

Remarks M-class problem
SVM for M-class problem design M separating hyperplanes so that separate class from all the others assign