Download presentation

Presentation is loading. Please wait.

Published byLarry Wolford Modified over 2 years ago

1
Support Vector Machine Classification Computation & Informatics in Biology & Medicine Madison Retreat, November 15, 2002 Olvi L. Mangasarian with G. M. Fung, Y.-J. Lee, J.W. Shavlik, W. H. Wolberg & Collaborators at ExonHit – Paris Data Mining Institute University of Wisconsin - Madison

2
What is a Support Vector Machine? An optimally defined surface Linear or nonlinear in the input space Linear in a higher dimensional feature space Implicitly defined by a kernel function K(A,B) C

3
What are Support Vector Machines Used For? Classification Regression & Data Fitting Supervised & Unsupervised Learning

4
Principal Topics Proximal support vector machine classification Classify by proximity to planes instead of halfspaces Massive incremental classification Classify by retiring old data & adding new data Knowledge-based classification Incorporate expert knowledge into a classifier Fast Newton method classifier Finitely terminating fast algorithm for classification Breast cancer prognosis & chemotherapy Classify patients on basis of distinct survival curves Isolate a class of patients that may benefit from chemotherapy

5
Principal Topics Proximal support vector machine classification

6
Support Vector Machines Maximize the Margin between Bounding Planes A+ A-

7
Proximal Support Vector Machines Maximize the Margin between Proximal Planes A+ A-

8
Standard Support Vector Machine Algebra of 2-Category Linearly Separable Case Given m points in n dimensional space Represented by an m-by-n matrix A Membership of each in class +1 or –1 specified by: An m-by-m diagonal matrix D with +1 & -1 entries More succinctly: where e is a vector of ones. Separate by two bounding planes,

9
Standard Support Vector Machine Formulation Margin is maximized by minimizing Solve the quadratic program for some : min s. t. (QP),, denotes where or membership.

10
Proximal SVM Formulation (PSVM) Standard SVM formulation: (QP) min s. t. This simple, but critical modification, changes the nature of the optimization problem tremendously!! (Regularized Least Squares or Ridge Regression) Solving for in terms of and gives: min

11
Advantages of New Formulation Objective function remains strongly convex. An explicit exact solution can be written in terms of the problem data. PSVM classifier is obtained by solving a single system of linear equations in the usually small dimensional input space. Exact leave-one-out-correctness can be obtained in terms of problem data.

12
Linear PSVM We want to solve: min Setting the gradient equal to zero, gives a nonsingular system of linear equations. Solution of the system gives the desired PSVM classifier.

13
Linear PSVM Solution Here, The linear system to solve depends on: which is of size is usually much smaller than

14
Linear & Nonlinear PSVM MATLAB Code function [w, gamma] = psvm(A,d,nu) % PSVM: linear and nonlinear classification % INPUT: A, d=diag(D), nu. OUTPUT: w, gamma % [w, gamma] = psvm(A,d,nu); [m,n]=size(A);e=ones(m,1);H=[A -e]; v=(d’*H)’ %v=H’*D*e; r=(speye(n+1)/nu+H’*H)\v % solve (I/nu+H’*H)r=v w=r(1:n);gamma=r(n+1); % getting w,gamma from r

15
Numerical experiments One-Billion Two-Class Dataset Synthetic dataset consisting of 1 billion points in 10- dimensional input space Generated by NDC (Normally Distributed Clustered) dataset generator Dataset divided into 500 blocks of 2 million points each. Solution obtained in less than 2 hours and 26 minutes on a 400Mhz About 30% of the time was spent reading data from disk. Testing set Correctness 90.79%

16
Principal Topics Knowledge-based classification (NIPS*2002)

17
Conventional Data-Based SVM

18
Knowledge-Based SVM via Polyhedral Knowledge Sets

19
Incoporating Knowledge Sets Into an SVM Classifier This implication is equivalent to a set of constraints that can be imposed on the classification problem. Suppose that the knowledge set: belongs to the class A+. Hence it must lie in the halfspace : We therefore have the implication:

20
Numerical Testing The Promoter Recognition Dataset Promoter: Short DNA sequence that precedes a gene sequence. A promoter consists of 57 consecutive DNA nucleotides belonging to {A,G,C,T}. Important to distinguish between promoters and nonpromoters This distinction identifies starting locations of genes in long uncharacterized DNA sequences.

21
The Promoter Recognition Dataset Numerical Representation Simple “1 of N” mapping scheme for converting nominal attributes into a real valued representation: Not most economical representation, but commonly used.

22
The Promoter Recognition Dataset Numerical Representation Feature space mapped from 57-dimensional nominal space to a real valued 57 x 4=228 dimensional space. 57 nominal values 57 x 4 =228 binary values

23
Promoter Recognition Dataset Prior Knowledge Rules Prior knowledge consist of the following 64 rules:

24
Promoter Recognition Dataset Sample Rules where denotes position of a nucleotide, with respect to a meaningful reference point starting at position and ending at position Then:

25
The Promoter Recognition Dataset Comparative Algorithms KBANN Knowledge-based artificial neural network [Shavlik et al] BP: Standard back propagation for neural networks [Rumelhart et al] O’Neill’s Method Empirical method suggested by biologist O’Neill [O’Neill] NN: Nearest neighbor with k=3 [Cost et al] ID3: Quinlan’s decision tree builder[Quinlan] SVM1: Standard 1-norm SVM [Bradley et al]

26
The Promoter Recognition Dataset Comparative Test Results

27
Wisconsin Breast Cancer Prognosis Dataset Description of the data 110 instances corresponding to 41 patients whose cancer had recurred and 69 patients whose cancer had not recurred 32 numerical features The domain theory: two simple rules used by doctors:

28
Wisconsin Breast Cancer Prognosis Dataset Numerical Testing Results Doctor’s rules applicable to only 32 out of 110 patients. Only 22 of 32 patients are classified correctly by this rule (20% Correctness). KSVM linear classifier applicable to all patients with correctness of 66.4%. Correctness comparable to best available results using conventional SVMs. KSVM can get classifiers based on knowledge without using any data.

29
Principal Topics Fast Newton method classifier

30
Fast Newton Algorithm for Classification Standard quadratic programming (QP) formulation of SVM:

31
Newton Algorithm Newton algorithm terminates in a finite number of steps Termination at global minimum Error rate decreases linearly Can generate complex nonlinear classifiers By using nonlinear kernels: K(x,y)

32
Nonlinear Spiral Dataset 94 Red Dots & 94 White Dots

33
Principal Topics Breast cancer prognosis & chemotherapy

34
Kaplan-Meier Curves for Overall Patients: With & Without Chemotherapy

35
Breast Cancer Prognosis & Chemotherapy Good, Intermediate & Poor Patient Groupings (6 Input Features : 5 Cytological, 1 Histological) (Grouping: Utilizes 2 Histological Features &Chemotherapy)

36
Kaplan-Meier Survival Curves for Good, Intermediate & Poor Patients 82.7% Classifier Correctness via 3 SVMs

37
Kaplan-Meier Survival Curves for Intermediate Group Note Reversed Role of Chemotherapy

38
Conclusion New methods for classification All based on rigorous mathematical foundation Fast computational algorithms capable of classifying massive datasets Classifiers based on both abstract prior knowledge as well as conventional datasets Identification of breast cancer patients that can benefit from chemotherapy

39
Future Work Extend proposed methods to broader optimization problems Linear & quadratic programming Preliminary results beat state-of-the-art software Incorporate abstract concepts into optimization problems as constraints Develop fast online algorithms for intrusion and fraud detection Classify the effectiveness of new drug cocktails in combating various forms of cancer Encouraging preliminary results for breast cancer

40
Breast Cancer Treatment Response Joint with ExonHit ( French BioTech) 35 patients treated by a drug cocktail 9 partial responders; 26 nonresponders 25 gene expression measurements made on each patient 1-Norm SVM classifier selected: 12 out of 25 genes Combinatorially selected 6 genes out of 12 Separating plane obtained: 2.7915 T11 + 0.13436 S24 -1.0269 U23 -2.8108 Z23 -1.8668 A19 -1.5177 X05 +2899.1 = 0. Leave-one-out-error: 1 out of 35 (97.1% correctness)

41
E1I1E2I2E3E4E5I3I4 DNA 3'5' E1E2E3I1I2E4E5I3I4 pre-mRNA (m=messenger) Transcription Alternative RNA splicing E1E2E4E5 E1E2E4E5E3 (A) n mRNA Chemo-Sensitive NH 2 COOH Chemo-Resistant NH 2 COOH Proteins Translation Detection of Alternative RNA Isoforms via DATAS (Levels of mRNA that Correlate with Senitivity to Chemotherapy) DATAS E3 DATAS: Differential Analysis of Transcripts with Alternative Splicing

42
Talk Available www.cs.wisc.edu/~olvi

Similar presentations

OK

Proximal Plane Classification KDD 2001 San Francisco August 26-29, 2001 Glenn Fung & Olvi Mangasarian Second Annual Review June 1, 2001 Data Mining Institute.

Proximal Plane Classification KDD 2001 San Francisco August 26-29, 2001 Glenn Fung & Olvi Mangasarian Second Annual Review June 1, 2001 Data Mining Institute.

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google