Download presentation

Presentation is loading. Please wait.

1
**3.6 Support Vector Machines**

K. M. Koo

2
**Goal of SVM Find Maximum Margin**

find a separating hyperplane with maximum margin margin minimum distance between a separating hyperplane and the sets of or

3
**Goal of SVM Find Maximum Margin**

assume that are linearly separable margin find separating hyperplane with maximum margin

4
**Calculate margin separating hyperplane and are not uniquely determined**

under the constraint , and are uniquely determine

5
**Calculate margin distance between a point x and is given by**

thus, the margin is given by

6
**Optimization of margin**

maximization of margin

7
**Optimization of margin**

separating hyperplane with maximal margin separating hyperplane with minimum therefore, we want to This is an optimization-problem with inequality constraints

8
**optimization with constraints**

cost function min-value min-value constraint constraint optimization with equality constraints optimization with inequality constraints

9
Lagrange Multiplier optimization problem under constraints can be solved by the method of Lagrange Multipliers let be real valued functions, let and ,and let , the level set for with value . assume if has a local minimum or maximum on at , which is called a critical point of ,then there is a real number ,called a Lagrange multiplier, such that 추가하자

10
**The Method of Lagrange Multiplier**

11
**Lagrange Multiplier Lagrangian is obtained as follows: In our case**

for equality constraints for inequality constraints In our case Inequality constraints

12
Convex a subset is convex iff for any , the line segment joining and is also a subset of , i.e. for any , a real-valued function on is convex iff for any two points and for any ,

13
**Convex convex set concave set neither convex nor concave**

convex function concave function

14
Convex Optimization an optimization problem is said to be convex iff the cost function as well as the constraints are convex the optimization problem for SVM is convex the solution to a convex problem, if it exist, is unique. that is, there is no local optimum! for convex optimization problem, KKT(Karush-Kuhn-Tucker) condition is necessary and sufficient for the solution

15
**KKT(Karush-Kuhn-Tucker) condition**

KKT condition The gradient of the Lagrangian with respect to the original variable is 0 The original constraints are satisfied Multipliers for inequality constraints (Complementary KKT) product of multiplier and constraints equal to 0 for convex optimize problems,1-4 are necessary and sufficient for the solution

16
**KKT condition for the optimization of margin**

recall KKT condition (3.66) (3.62) (3.63) (3.64) (3.65)

17
**KKT condition for the optimization of margin**

Combining (3.66) with (3.62) (3.67) (3.68)

18
**Remarks-support vector**

of the optimal solution is a linear combination of feature vectors which are associated with support vectors are associated with

19
**Remarks-support vector**

The resulting hyperplane classifier is insensitive to the number and position of non-support vector

20
Remark-computation w0 can be implicitly obtaines by any of the condition satisfying strict complement (i.e ) In practice, is computed as an average value obtained using all conditions of the type

21
**Remark-optimal hyperplane is unique**

the optimal hyperplane classifier of a support vector machine is unique under two condition the cost function is convex the inequality constraints consist of linear functions constraints are convex an optimization problem is said to be convex iff the target(or cost) function as well as the constraints are convex (the optimization problem for SVM is convex) the solution to a convex problem, if it exist, is unique. that is, there is no local optimum!

22
**Computation optimal Lagrange multiplier**

optimization problem belongs to the convex programming family (convex optimization problem) of problems It can be solved by considering the so called Lagrangian duality and can be stated equivalently by its Wolfe dual representation form Lagrangian duality Wolfe dual representation

23
**Wolfe dual representation form**

24
**Computation optimal Lagrange multiplier**

once the optimal Lagrangian multipliers have been computed, the optimal hyperplane is obtained (3.75) (3.76)

25
Remarks the cost function does not depend explicitly on the dimensionality of the input space this allows for efficient generalizations in the case of nonlinearly separable classes although the resulting optimal hyperplane is unique, there is no guarantee about Lagrange multipliers

26
Simple example consider the two classification task that consists of the following points its Lagrangian function KKT condition

27
**Simple example Lagrangian duality optimize with equality constraint**

result more then one solution

28
**SVM for Non-separable Classes**

in the case of non-separable, the training feature vector belong to one of the following three categories

29
**SVM for Non-separable Classes**

All three cases can be treated under a single type constraints

30
**SVM for Non-separable Classes**

goal is make the margin as large as possible keep the number of points with as small as possible (3.79) is intractable because of discontinuous function (3.79)

31
**SVM for Non-separable Classes**

as common case, we choose to optimize a closely related cost function

32
**SVM for Non-separable Classes**

to Lagrangian

33
**SVM for Non-separable Classes**

The corresponding KKT condition (3.85) (3.86) (3.87) (3.90) (3.88) (3.89)

34
**SVM for Non-separable Classes**

The associated Wolfe dual representation now becomes

35
**SVM for Non-separable Classes**

equivalent to

36
**Remarks-difference with the linearly separable case**

Lagrange multipliers( ) need to be bounded by C the slack variables, , and their associated Lagrange multipliers, , do not enter into the problem explicitly reflected indirectly though C

37
**Remarks M-class problem**

SVM for M-class problem design M separating hyperplanes so that separate class from all the others assign

Similar presentations

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google