1. School of Computer Science & Engineering
Artificial Intelligence
Feature Selection
Dae-Won Kim
School of Computer Science & Engineering
Chung-Ang University

2. What is feature selection?

3. We predict the grade of ‘AI’ of a new student (age, sex,
We predict the grade of ‘AI’ of a new student (age, sex, .. smoke) using the following training data.
Name
Age
Sex
Alg.
C++
Military
Height
Weight C
Smoke
Class
label
Student-1 21 M A B Y
178 80
Yes
Student-2 23 F N
165 50 No
Student-3 22
160 45
Student-4
180 70

4. Definition: Given a set of features, select a subset that performs best.

5. Q: Advantages ? (4 viewpoints)

6. Benefits-1: simpler and faster

7. It reduces the number of features.

8. Benefits-2: cost-effect

9. It provides lower computational cost and cost of testing, etc.

10. Benefits-3: a better accuracy

11. It may provide a better accuracy by removing useless features.

12. Benefits-4: a deeper insight

13. Knowledge of good features gives insights into the underlying structure.

14. Caution: Feature Selection
vs.
Feature Transformation

15. Related to scaling and normalization.

16. Certain transformation of features may lead to the discovery of structures that were not obvious on the original scale.

18. Y = 2X or Y = X
Given X, the result Y is predictable

20. For scaling, we usually take square roots, reciprocals, and logarithms.

21. We also handle the question of normalization.

22. Features are often normalized to lie in a fixed range, from zero to one.

23. 1. By dividing all values by the maximum value encountered.

24. 2. By subtracting the minimum value and dividing by the range between the max. and the min.

25. 3. By calculating the standard mean and standard deviation of the feature, subtract the mean from each value, and divide the result by the standard deviation.
Q: Try these three options.

26. This is also called standardization. (mean-zero and s.d.-one)

27. However, we may sacrifice the way it represents the underlying data.

28. Caution: Feature Selection
vs.
Feature Extraction

29. Definition: It creates new good features using combinations or transformation of the original feature. (dimension reduction)

30. 1. By using linear combinations that are simple to compute & tractable.

31. 2. By projecting high dimensional data onto a lower dimensional space.

32. PCA (principal component analysis)
SVD (singular value composition)
LDA (linear discriminant analysis)
MDA (multiple discriminant analysis)
ICA (independent component analysis)
MDS (multi-dimensional scaling)
NMF (nonnegative matrix factorization) …

33. PCA extracts principal components calculated by the covariance.

34. New features are obtained by a linear combination of PCs.

35. Have you heard about eigen vector, eigen value in linear algebra?

38. PCA (Principal Component Analysis)
Given four data where each data point has three attributes, reduce the dimension of attributes to ‘two’
Step 1. Find the eigen vectors () and corresponding eigen values () through the covariance matrix
Step 2. Sort the eigen vectors according to their corresponding eigen values in descending order
1 =  1 = [ ]T (referred to as the first principal component)
2 =  2 = [ ]T (referred to as the second principal component)
3 =  3 = [ ]T (referred to as the third principal component)
Step 3. Create new attributes using the top k eigen vectors where k = 2.
Original attributes x eigenvectorT
= [1 2 1] x 1T
= [1 2 1] x [ ]T
= 0.41
Original attributes x eigenvectorT
= [1 2 1] x 2T
= [1 2 1] x [ ]T
= -1.42

39. SVD (Singular Value Decomposition)
Given the relation matrix of documents and terms, recalculate the relation matrix with top 2 singular values
Step 1. Decompose the original matrix A into three matrices (i.e., using MATLAB)
A = [1/singular values x A x eigen vectors] [singular values] [eigen vectors]T, singular value = (eigen value)1/2
1st eigen vector
2nd eigen vector
2nd singular value
1st singular value
Step 2. Create new matrix Anew using the top k entries of the decomposed matrices where k = 2.

40. PCA is useful for representing data.

41. LDA is useful for discriminating data.

43. PCA seeks for the orthogonal directions.

44. ICA seeks for the independent directions.

45. ICA is useful for blind source separation.

46. Let us go back to feature selection.

47. How can we select good features?

48. Quiz: propose your idea on feature selection using examples

49. We can find a set of good features.

50. We can find a set of bad features.

51. We need definitions on the good and bad feature.

52. redundant, relevant, inconsistent,…

53. Feature selection is a special case of search problem (optimization prob).

54. We can use all search techniques.

55. Among many approaches, we learn techniques using Filter vs. Wrapper.

56. The difference: learning algorithm.

57. Wrapper uses learning algorithms.

58. Filter: greedy by univariate ranking
score each feature
sort them
select top-ranked features

59. Q: pros and cons of univariate filter

60. Filter: multivariate correlation issue

61. Wrapper:
Search + Learning Evaluation
Form subsets by search
Evaluate them using classifers
Expand or modify subsets

62. Wrapper:
Revisit your solution to TSP

63. Wrapper:
1. Search by (Greedy, B&B, HC, GA)
2. Evaluation by learning accuracy (Bayesian or K-NN)

64. Q: pros and cons of wrapper

65. Q: which feature handling technique is appropriate to your project?