Presentation is loading. Please wait.

Presentation is loading. Please wait.

Using SVM Weight-Based Methods to Identify Causally Relevant and Non-Causally Relevant Variables Alexander Statnikov 1, Douglas Hardin 1,2, Constantin.

Similar presentations


Presentation on theme: "Using SVM Weight-Based Methods to Identify Causally Relevant and Non-Causally Relevant Variables Alexander Statnikov 1, Douglas Hardin 1,2, Constantin."— Presentation transcript:

1 Using SVM Weight-Based Methods to Identify Causally Relevant and Non-Causally Relevant Variables Alexander Statnikov 1, Douglas Hardin 1,2, Constantin Aliferis 1,3 1 Department of Biomedical Informatics, 2 Department of Mathematics, 3 Department of Cancer Biology, Vanderbilt University, Nashville, TN, USA NIPS 2006 Workshop on Causality and Feature Selection

2 Major Goals of Variable Selection Construct faster and more cost-effective classifiers. Improve the prediction performance of the classifiers. Get insights in the underlying data- generating process.

3 Taxonomy of Variables Variables RelevantIrrelevant Causally relevant Non-causally relevant F CD E T J A K L M B Response

4 Support Vector Machine (SVM) Weight-Based Variable Selection Methods Scale up to datasets with many thousands of variables and as few as dozens of samples Often yield variables that are more predictive than the ones output by other variable selection techniques or the full (unreduced) variable set (Guyon et al, 2002; Rakotomamonjy 2003) Currently unknown: Do we get insights on the causal structure ? (Hardin et al, 2004): Irrelevant variables will be given a 0 weight by a linear SVM in the sample limit; Linear SVM may assign 0 weight to strongly relevant variables and nonzero weight to weakly relevant variables.

5 Simulation Experiments X1X1 X2X2 XNXN T … Z1Z1 Z2Z2 ZMZM … Relevant variables Irrelevant variables (hidden from the learner) Y Network structure 1 Response Causally relevant P(Y=0) = ½ and P(Y=1) = ½. Y is hidden from the learner; {X i } i=1,…,N are binary variables with P(X i =0|Y=0) = q and P(X i =1|Y=1) = q. {Z i } i=1,..,M are independent binary variables with P(Z i =0) = ½ and P(Z i =1) = ½. T is a binary response variable with P(T=0|X 1 =0) = 0.95 and P(T=1|X 1 =1) = q = 0.95 Network 1a q = 0.99 Network 1b

6 Simulation Experiments Network structure 1 in real-world distributions Adrenal gland cancer pathway produced by Ariadne Genomics PathwayStudio software version 4.0 (http://www.ariadnegenomics.com/).http://www.ariadnegenomics.com/ Disease and its putative causes (except for kras)

7 Simulation Experiments Network structure 2 {X i } i=1,..,N are independent binary variables with P(X i =0) = ½ and P(X i =1) = ½. {Z i } i=1,..,M are independent binary variables with P(Z i =0) = ½ and P(Z i =1) = ½. Y is a synthesis variable with the following function: X1X1 XNXN T … Y Relevant variables X2X2 Z1Z1 Z2Z2 ZMZM … Irrelevant variables Response Causally relevantT is a binary response variable defined as where v i s are generated from the uniform random U(0,1) distribution and are fixed for all experiments.

8 Simulation Experiments Network structure 2 in real-world distributions Putative causes of the disease Targets of putative causes of the disease

9 Data Generation Generated 30 training samples of sizes = {100, 200, 500, 1000} for different values of N (number of all relevant variables) = {10, 100} and M (number of irrelevant variables) = {10,100,1000}. Generated testing samples of size 5000 for different values of N and M. Added noise to simulate random measurement errors: replace {0%, 1%, 10%} of each variable values with values randomly sampled from the distribution of that variable in simulated data.

10 Overview of Experiments with SVM Weight-Based Methods Variable selection by SVM weights & classification - Used C = {0.001, 0.01, 0.1, 1, 10, 100, 1000} - Classified 10%, 20%,…,90%, 100% top-ranked variables Also classified baselines (causally relevant, non-causally relevant, all relevant, and irrelevant). Variable selection by SVM-RFE & classification - Removed one variable at a time - Used C = {0.001, 0.01, 0.1, 1, 10, 100, 1000} - 75% training/25% testing

11 SVM Formulation Used

12 Results

13 I. SVMs Can Assign Higher Weights to the Irrelevant Variables than to the Non-Causally Relevant Ones Average ranks of variables (by SVM weights) over 30 random training samples of size 100 (w/o noise) from network 1a with 100 relevant and irrelevant variables C is small (0.01)C is large (0.1)

14 I. SVMs Can Assign Higher Weights to the Irrelevant Variables than to the Non-Causally Relevant Ones AUC analysis for discrimination between groups of all relevant and irrelevant variables based on SVM weights AUC classification performance obtained on the 5,000-sample independent testing set: results for variable ranking based on SVM weights

15 II. SVMs Can Select Irrelevant Variables More Frequently than Non-Causally Relevant Ones Probability of selecting variables (by SVM-RFE) estimated over 30 random training samples of size 100 (w/o noise) from network 1a with 100 relevant and irrelevant variables C is small (0.01)C is large (0.1)

16 II. SVMs Can Select Irrelevant Variables More Frequently than Non-Causally Relevant Ones AUC classification performance obtained on the 5,000-sample independent testing set: results for variable selection by SVM-RFE

17 III. SVMs Can Assign Higher Weights to the Non-Causally Relevant Variables Than to the Causally Relevant Ones Average ranks of variables (by SVM weights) over 30 random training samples of size 500 (w/o noise) from network 2 with 100 relevant and irrelevant variables AUC analysis for discrimination between groups of causally relevant and non-causally relevant variables based on SVM weights

18 IV. SVMs Can Select Non-Causally Relevant Variables More Frequently Than the Causally Relevant Ones Probability of selecting variables (by SVM-RFE) estimated over 30 random training samples of size 500 (w/o noise) from network 2 with 100 relevant and irrelevant variables

19 V. SVMs Can Assign Higher Weights to the Irrelevant Variables Than to the Causally Relevant Ones Average ranks of variables (by SVM weights) over 30 random training samples of size 100 (w/o noise) from network 2 with 100 relevant and irrelevant variables AUC analysis for discrimination between groups of causally relevant and non-causally relevant variables based on SVM weights

20 VI. SVMs Can Select Irrelevant Variables More Frequently Than the Causally Relevant Ones Probability of selecting variables (by SVM-RFE) estimated over 30 random training samples of size 100 (w/o noise) from network 2 with 100 relevant and irrelevant variables

21 Theoretical Example 1 X1X1 T X2X2 Y (Network structure 2) P(X 1 =-1) = ½, P(X 1 =1) = ½, P(X 2 =-1) = ½, and P(X 2 =1) = ½. Y is a synthesis variable with the following function: T is a binary response variable defined as: Variables X 1, X 2, and Y have expected value 0 and variance 1. The application of linear SVMs results in the following weights: 1/2 for X 1, 1/2 for X 2, and for Y. Therefore, the non-causally relevant variable Y receives higher SVM weight than the causally relevant ones.

22 Theoretical Example 2 X Y T = + T = - G1 G2 X T X Y Y T Y T | X X Y | T X T Y The maximum-gap inductive bias is inconsistent with local causal discovery.

23 Discussion 1.Using nonlinear SVM weight-based methods Preliminary experiment: When polynomial SVM-RFE is used, non- causally relevant variable is never selected in network structure 2. However, the performance of polynomial SVM-RFE is similar to linear SVM-RFE. 2.The framework of formal causal discovery (Spirtes et al, 2000) provides algorithms that can solve these problems, e.g. HITON (Aliferis et al, 2003) or MMPC & MMMB (Tsamardinos et al, 2003; Tsamardinos et al, 2006). 3.Methods based on modified SVM formulations, e.g. 0- norm and 1-norm penalties (Weston et al, 2003; Zhu et al, 2004). 4.Extend empirical evaluation to different distributions

24 Conclusion Causal interpretation of the current SVM weight-based variable selection techniques must be conducted with great caution by practitioners The inductive bias employed by SVMs is locally causally inconsistent. New SVM methods may be needed to address this issue and this is an exciting and challenging area of research.


Download ppt "Using SVM Weight-Based Methods to Identify Causally Relevant and Non-Causally Relevant Variables Alexander Statnikov 1, Douglas Hardin 1,2, Constantin."

Similar presentations


Ads by Google