1. Support Vector Machines Exercise solutions Ata Kaban The University of Birmingham

2. a)What is the main idea behind linear Support Vector Machines (SVM)? Illustrate your explanation by drawing a figure. ANSWER The figure should show e.g. two linearly separable clusters of points, each cluster corresponding to a different class. Even though there are many possible separating lines, we pick the one that has maximal minimum distance from the closest points of each class. This choice is supported by a theorem in learning theory that bounds generalization error in terms of separation margins.

3. b) How can the linear SVM be made non-linear? ANSWER The non-linearity comes from using the `Kernel Trick': Instead of the dot product in the input space, use a kernel function K(x 1,x 2 ). For a proper kernel K (cf. Mercer Theorem) there is another (usually) high-dimensional feature space F and a feature map, such that K(x 1,x 2 ) can be interpreted as a dot product between images F(x i ) in F.

4. c) Decide which of the following formulae define proper kernels and explain why. ANSWER K 3 and K 4 are proper kernels, since given a proper kernel K, then aK, a > 0, is a proper kernel. Also given proper kernels K' and K'', then K' K'' is a proper kernel. K 5 is not a proper kernel, since the Gram matrix will be negative definite for all training sets.

5. d) Consider the 2-dimensional inputs. Is the following a proper kernel? Explain why. ANSWER It is a proper kernel, since for any real valued function over the input space, is a proper kernel.

6. Remember to master the worked questions / exercises How do we know if a kernel is proper? - - given a proper kernel K, then aK, a > 0, is a proper kernel - - given proper kernels K' and K'', then K' K'' is a proper kernel - - if K is a proper kernel, for any real valued function over the input space, is a proper kernel.