A Comparative Study on Variable Selection for Nonlinear Classifiers C. Lu 1, T. Van Gestel 1, J. A. K. Suykens 1, S. Van Huffel 1, I. Vergote 2, D. Timmerman 2 1 Department of Electrical Engineering, Katholieke Universiteit Leuven, Leuven, Belgium, 2 Department of Obstetrics and Gynecology, University Hospitals Leuven, Leuven, Belgium address:

1. Introduction  Variable selection refers to the problem of selecting input variables that are relevant for a given task. In pattern recognition, variable selection can have an impact on the economics of data acquisition and on the accuracy and complexity of the classifiers.  This study aims at input variable selection for nonlinear blackbox classifiers, particularly multi-layer perceptrons (MLP) and least squares support vector machines (LS-SVMs). 2. Feature extraction Variable Selection Variable (feature) measure Heuristic search: forward, backward, stepwise, hill-climbing, branch and bound… Filter approach: filter out irrelevant attributes before induction occurs Wrapper approaches : focus on finding attributes that are useful for performance for a specific type of model, rather than necessarily finding the relevant ones. Feature Extraction Feature selection or variable selection Feature transformation e.g. PCA, not desirable for maintaining data, difficulty in interpretation, and not immune from distortion under transformation Variable measure Correlation Mutual information (MI) Evidence (or Bayes factor) in Bayesian framework Classification performance Sensitivity analysis: change in the objective function J by removing variable i: DJ(i) Statistical partial F test (Chi- square value) Pattern recognition: feature extraction -> classification

3. Considered nonlinear classifiers: MLPs and LS-SVMs LS-SVM Classifier Note: by integrating the MLP(Mackay 1992) or LS-SVM (VanGestel, Suykens 2002) with the Bayesian evidence framework, the tuning of hyperparameters and computation of posterior class probabilities can be done in a unified way. Variable selection can also be done based on the model evidence. solved in dual space Model evidence Bayesian Evidence Framework Inferences are divided into distinct levels. MLP Classifiers

4. Considered variable selection methods MethodVariable measure SearchPredefined parameters (Dis) advantages Mutual information feature selection under uniform information distrib. (MIFS-U) [8] Mutual information I(X;Y) Greedy search: begin from no variables, repeat selecting the feature until predifined k variables are selected Density function estimation (parametric or nonparametric), here the simple discretization method is used. Linear/Nolinear, easy to compute; computational problems increase with k, for very high dimensional data. Information lost due to discretization. Bayesian LS-SVM variable forward selection (LSSVMB-FFS) [1] Model evidence P(D|H) Greedy search, select each time a variable that gives the highest increase in model evidence, until no more increase. Kernel type.(Non)linear. Automatically select a certain number of variables that max the evidence. Gaussian assumption. Computationally expensive for high dimensional data. LS-SVM recursive feature elimination (LSSVM-RFE) [7] For linear kernel, use(w i ) 2 Recursively remove the variable(s) that have the smallest DJ(i). Kernel type, regularization and kernel parameters. Suitable for very high dimensional data. Computationally expensive for large sample size, and nonlinear kernels. Stepwise logistic regression (SLR) Chi-square (statistical partial F-test). Stepwise: recursively add or remove a variable at each step. P-values for determining addition or removal of variables in models. Linear, easy to compute. Troubles in case of multicolinearity.

5. Experimental results on benchmark data sets Table 2. Accuracy on Test set with different number of variables II. Biomedical real life data set (1)Gene selection for leukemia classification [] #variables: 7129, Classes: ALL, AML, #Training data: 38; #test data: 34 I. Synthetic data: noisy XOR problem linearly inseparable. 50 random generated input data, X1, X2: {0,1} random Y: XOR(x1, x2). X3, X4: noise~N(0, 0.3) was added to X1 X2 X5, X6: noise~N(0, 0.5) was added to X1 X2 X7~X16: noise~N(0, 2) Table 1. LssvmRFE (using a polynomial kernel with degree 2) selected correctly the top2 variables 25 times from the 30 random trials based on the 50 noisy training data; Averaged performance on a test set of 100 examples over 30 random trials using. Notes: Linear classifier and selection method can’t solve the XOR problem which is nonlinear. MIFSU: entropy for the first 2 binary variable smaller than the other continuous variables Bayesian LSSVM FFS: evidence for the first 2 binary variables is smaller than other continuous variables; however backward Bayesian LSSVM can always remove the other noisy variables.. *linear kernels are used for lssvm-RFE and lssvmB-FFS - MLP has 1 hidden layer with 2 hidden neurons, using Baysian MLP to determine the regularization parameter. - the LSSVM classifier uses a polynomial kernel with degree 2.

 Good variable selection can improve the performance of the classifiers both in accuracy and computation.  LSSVM-RFE can be suitable for both linear and nonlinear classification problems. And can deal with the very high dimensional data.  Bayesian LSSVM forward selection can identify the important variables in some cases, however should be used with more care in the satisfaction of the assumptions.  A strategy which combines variable ranking and the wrapper methods should give more confidence in the selected variables. References 1. C. Lu, T. Van Gestel, et al. Preoperative prediction of malignancy of ovarian tumors using Least Squares Support Vector Machines (2002), submitted paper. 2. D. Timmerman, H. Verrelst, et al., Artificial neural network models for the preoperative discrimination between malignant and benign adnexal masses. Ultrasound Obstet Gynecol (1999). 3. J.A.K. Suykens, J. Vandewalle, Least Squares support vector machine classifiers, Neural Processing Letters (1999), 9(3). 4. T. Van Gestel, J.A.K. Suykens, et al., Bayesian framework for least squares support vector machine classifiers, Neural Computation (2002), 15(5). 5. D.J.C. MacKay, The evidence framework applied to classification networks, Neural Computation (1992), 4(5). 6. R. Kohavi and G. John, Wrappers for feature subset selection, Artificial intelligence, special issue on relevance 97 (1-2): I. Guyon, J. Weston, et al. Gene selection for cancer classification using support vector machines, Machine learning (2000). 8. N. Kwak and C.H. Choi Input feature selection for classification problems, IEEE Transactions on neural networks (2002) 13 (1). Table 3. Accuracy on test set with different number of variables 6. Conclusions (2) Ovarian tumor classification # variables 27, classes: benign, malignant # training data: 265, #test data: 160