1. CS 2750: Machine Learning Support Vector Machines Prof. Adriana Kovashka University of Pittsburgh February 17, 2016

2. Announcement Homework 2 deadline is now 2/29 – We’ll have covered everything you need today or at the latest on Monday Project proposal due tonight on CourseWeb How many of you want me to print handouts for next time?

3. Plan for today Linear Support Vector Machines Non-linear SVMs and the “kernel trick” Soft-margin SVMs Example use of SVMs Advanced topics (very briefly) – Structured SVMs – Latent variables How to solve the SVM problem (next class)

4. Lines in R 2 Let Kristen Grauman

5. Lines in R 2 Let Kristen Grauman

6. Lines in R 2 Let Kristen Grauman

7. Lines in R 2 Let distance from point to line Kristen Grauman

8. Lines in R 2 Let distance from point to line Kristen Grauman

9. Linear classifiers Find linear function to separate positive and negative examples Which line is best? C. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining and Knowledge Discovery, 1998A Tutorial on Support Vector Machines for Pattern Recognition

10. Support vector machines Discriminative classifier based on optimal separating line (for 2d case) Maximize the margin between the positive and negative training examples C. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining and Knowledge Discovery, 1998A Tutorial on Support Vector Machines for Pattern Recognition

11. Support vector machines Want line that maximizes the margin. Margin Support vectors C. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining and Knowledge Discovery, 1998A Tutorial on Support Vector Machines for Pattern Recognition For support, vectors, wx+b=-1 wx+b=0 wx+b=1

12. Support vector machines Want line that maximizes the margin. Support vectors For support, vectors, wx+b=-1 wx+b=0 wx+b=1 Distance between point and line: For support vectors: C. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining and Knowledge Discovery, 1998A Tutorial on Support Vector Machines for Pattern Recognition Margin

13. Support vector machines Want line that maximizes the margin. Margin Support vectors For support, vectors, wx+b=-1 wx+b=0 wx+b=1 Distance between point and line: Therefore, the margin is 2 / ||w|| C. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining and Knowledge Discovery, 1998A Tutorial on Support Vector Machines for Pattern Recognition

14. Finding the maximum margin line 1.Maximize margin 2/||w|| 2.Correctly classify all training data points: Quadratic optimization problem: Minimize Subject to y i (w·x i +b) ≥ 1 One constraint for each training point. Note sign trick. C. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining and Knowledge Discovery, 1998A Tutorial on Support Vector Machines for Pattern Recognition

15. Finding the maximum margin line Solution: Support vector Learned weight C. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining and Knowledge Discovery, 1998A Tutorial on Support Vector Machines for Pattern Recognition

16. Finding the maximum margin line Solution: b = y i – w·x i (for any support vector) Classification function: Notice that it relies on an inner product between the test point x and the support vectors x i (Solving the optimization problem also involves computing the inner products x i · x j between all pairs of training points) If f(x) < 0, classify as negative, otherwise classify as positive. C. Burges, A Tutorial on Support Vector Machines for Pattern Recognition, Data Mining and Knowledge Discovery, 1998A Tutorial on Support Vector Machines for Pattern Recognition MORE DETAILS NEXT TIME

17. Inner product Adapted from Milos Hauskrecht

18. Plan for today Linear Support Vector Machines Non-linear SVMs and the “kernel trick” Soft-margin SVMs Example use of SVMs Advanced topics (very briefly) – Structured SVMs – Latent variables How to solve the SVM problem (next class)

19. Datasets that are linearly separable work out great: But what if the dataset is just too hard? We can map it to a higher-dimensional space: 0x 0 x 0 x x2x2 Andrew Moore Nonlinear SVMs

20. Φ: x → φ(x) General idea: the original input space can always be mapped to some higher-dimensional feature space where the training set is separable: Andrew Moore Nonlinear SVMs

21. Nonlinear kernel: Example Consider the mapping x2x2 Svetlana Lazebnik

22. The “Kernel Trick” The linear classifier relies on dot product between vectors K(x i, x j ) = x i · x j If every data point is mapped into high-dimensional space via some transformation Φ: x i → φ(x i ), the dot product becomes: K(x i, x j ) = φ(x i ) · φ(x j ) A kernel function is similarity function that corresponds to an inner product in some expanded feature space The kernel trick: instead of explicitly computing the lifting transformation φ(x), define a kernel function K such that: K(x i, x j ) = φ(x i ) · φ(x j ) Andrew Moore

23. Examples of kernel functions Linear: Polynomials of degree up to d: Gaussian RBF: Histogram intersection: Andrew Moore / Carlos Guestrin

24. (C) Dhruv Batra24 Slide Credit: Blaschko & Lampert

25. (C) Dhruv Batra25 Slide Credit: Blaschko & Lampert

26. Plan for today Linear Support Vector Machines Non-linear SVMs and the “kernel trick” Soft-margin SVMs Example use of SVMs Advanced topics (very briefly) – Structured SVMs – Latent variables How to solve the SVM problem (next class)

27. Assuming data separable Maximize margin The w that minimizes…

28. Allowing misclassifications Maximize marginMinimize misclassification Slack variable The w that minimizes… Misclassification cost # data samples BOARD

29. What about multi-class SVMs? In practice, we obtain a multi-class SVM by combining two-class SVMs One vs. others –Training: learn an SVM for each class vs. the others –Testing: apply each SVM to the test example, and assign it to the class of the SVM that returns the highest decision value One vs. one –Training: learn an SVM for each pair of classes –Testing: each learned SVM “votes” for a class to assign to the test example There are also “natively multi-class” formulations –Crammer and Singer, JMLR 2001 Svetlana Lazebnik / Carlos Guestrin

30. SVMs for recognition 1.Define your representation for each example. 2.Select a kernel function. 3.Compute pairwise kernel values between labeled examples 4.Use this “kernel matrix” to solve for SVM support vectors & weights. 5.To classify a new example: compute kernel values between new input and support vectors, apply weights, check sign of output. Kristen Grauman

31. Example: learning gender with SVMs Moghaddam and Yang, Learning Gender with Support Faces, TPAMI 2002. Moghaddam and Yang, Face & Gesture 2000. Kristen Grauman

32. Support Faces Moghaddam and Yang, Learning Gender with Support Faces, TPAMI 2002.

33. Human vs. Machine SVMs performed better than any single human test subject, at either resolution Kristen Grauman

34. Plan for today Linear Support Vector Machines Non-linear SVMs and the “kernel trick” Soft-margin SVMs Example use of SVMs Advanced topics (very briefly) – Structured SVMs – Latent variables How to solve the SVM problem (next class)

35. Structured SVMs y is a vector Tsochantaridis et al., Large Margin Methods for Structured and Interdependent Output Variables, JMLR 2005.Large Margin Methods for Structured and Interdependent Output Variables

36. Latent Variables Adapted from S. Nowozin and C. Lampert

37. Application: Detecting objects (people) We can have a confident person prediction if the features in a person-sized window match a global template (root filter), the parts (e.g. head) match the part templates (filters), and the parts are not too far from where we’re used to seeing them (the lower the deformation weights, the better) P. Felzenszwalb, R. Girshick, D. McAllester, D. Ramanan, Object Detection with Discriminatively Trained Part Based Models, PAMI 32(9), 2010Object Detection with Discriminatively Trained Part Based Models Root filter Part filters Deformation weights Adapted from Lana Lazebnik

38. Scoring an object hypothesis The score of a hypothesis is the sum of appearance scores (features times weights) minus the sum of deformation costs Locations of parts are latent (unobserved) Appearance weights Subwindow features Deformation weights Displacements Adapted from Lana Lazebnik

39. Training with latent variables Our classifier has the form w are model parameters, z are latent hypotheses h is a feature representation that depends on part locations Latent SVM training: Initialize w and iterate: Fix w and find the best z for each training example Fix z and solve for w (standard SVM training) Lana Lazebnik

42. SVMs: Pros and cons Pros –Kernel-based framework is very powerful, flexible –Often a sparse set of support vectors – compact at test time –Work very well in practice, even with very small training sample sizes –Solution can be formulated as a quadratic program (next time) –Many publicly available SVM packages: e.g. LIBSVM, LIBLINEAR, SVMLight (or use built-in Matlab version but slower) Cons –Can be tricky to select best kernel function for a problem –Computation, memory At training time, must compute kernel values for all example pairs Learning can take a very long time for large-scale problems Adapted from Lana Lazebnik

43. SVM Solution Equations from Andrew Ng’s notes

44. SVM Solution Equations from Andrew Ng’s notes

56. Sequential Minimal Optimization Convergence: all α i ’s satisfy Karush-Kuhn-Tucker (KKT) conditions used to determine if at optimal solution Repeat until convergence: –Pick α i that violates the conditions –Pick another α j –Recompute new values for α i and α j Proposed by John Platt in 1998: “Fast Training of Support Vector Machines using Sequential Minimal Optimization” Further reading: –http://research.microsoft.com/pubs/68391/smo-book.pdfhttp://research.microsoft.com/pubs/68391/smo-book.pdf –http://cs229.stanford.edu/materials/smo.pdfhttp://cs229.stanford.edu/materials/smo.pdf