1. Cross Validation of SVMs for Acoustic Feature Classification using Entire Regularization Path Tianyu Tom Wang T. Hastie et al. 2004 WS04

2. Acoustic Feature Detection Manner Features: e.g. +sonorant vs. – sonorant Place of Articulation Feature: e.g. +lips vs. –lips Vectorize spectrograms as feature vectors of +/- manner/place feature Hasegawa-Johnson, 2004

3. Binary Linear SVM Classifiers n training pairs: [x i, y i ] where x i 0ú p ; p = number of attributes of vector; y i 0 {-1, +1} Linear SVM: h(x i ) = sgn[f(x i ) = $ 0 + $ T x i ] $ T x i : dot product Finding f( x i ): subject to y i C f(x i ) $ 1 for each i Hastie, Zhu, Tibshirani, Rosset, 2004

4. Binary Linear SVM Classifiers

5. Inseparable Case, Binary Linear SVM Classifiers Zhu,Hastie, 2001 Slack Variables 1 - y i C f(x i ) = > i Outside Boundary (correct): > i < 0 Inside Boundary (correct/incorrect): > i > 0 On Boundary (correct): > i = 1

6. Binary Linear SVM Classifiers, Slack Variables Finding f(x): subject to y i C f(x i ) $ 1 - > i for each i Rewrite as: C = cost parameter = 1/ 8 Note: [arg] + indicates max(0, arg) 1 - y i C f(x i ) = > i

7. Generalizing to Non-Linear Classifiers Kernel SVM: h(x i ) = sgn[f(x i ) = $ 0 +g(x i )] g(x i ) = 3" i y i K(x i, x) = 3" i y i M (x i ) CM (x) Value picked for C = 1/ 8 is crucial to error. Hastie, Zhu, Tibshirani, Rosset, 2004

8. Example – mixture Gaussians Hastie, Zhu, Tibshirani, Rosset, 2004

9. Tracing Path of SVMs w.r.t. C One condition for SVM solution - C = 1/ 8 controls width of margin (1/|| $ ||) 0 # " i # 1 y i C f(x i ) > 0, " i = 1, incorrectly classified y i C f(x i ) > 1, > = 0, " i = 0 y i C f(x i ) = 1, > = 0, 0 < " i < 1, on margin, support vector Hastie, Zhu, Tibshirani, Rosset, 2004

10. Tracing Path of SVMs w.r.t. C Algorithm: Set 8 to be large (margin very wide) All " i consequently = 1 Decrease 8 while keeping track of the following sets: Theoretical Findings: " i values are piecewise linear w.r.t. C Can find “breakpoints” of " i interpolate in between Hastie, Zhu, Tibshirani, Rosset, 2004

11. Tracing Path of SVMs w.r.t. C Breakpoint: " i reaches 0 or 1 (boundary), or y i C f(x i ) = 1 for a point in set O or I Find corresponding 8 and store " i ‘s Terminate when either: Set I is empty (classes separable) 8 ~ 0 Hastie, Zhu, Tibshirani, Rosset, 2004

12. Linear Example Hastie, Zhu, Tibshirani, Rosset, 2004

13. Finding Optimal C in Cross Validation Stored " i generate all possible SVMs for a given training set. Test all possible SVMs on new test points. Find SVM with minimum error rate and corresponding C Tested method on +/- continuant feature of NTIMIT corpus. Resultant C values were close to those found by training individual SVMs.

16. Conclusions Can use entire regularization path of SVM to find optimal C value for cross validation Faster than training individual SVM’s for each C value Finer traversal of C values