1. Feature Selection for Pattern Recognition
2018/9/11
Machine Learning Feature Selection
Feature Selection for Pattern Recognition
J.-S. Roger Jang ( 張智星 )
CSIE Dept., National Taiwan University
( 台灣大學 資訊工程系 )
...
In this talk, we are going to apply two neural network controller design techniques to fuzzy controllers, and construct the so-called on-line adaptive neuro-fuzzy controllers for nonlinear control systems. We are going to use MATLAB, SIMULINK and Handle Graphics to demonstrate the concept. So you can also get a preview of some of the features of the Fuzzy Logic Toolbox, or FLT, version 2.

2. Feature Selection: Goal & Benefits
2018/9/11
Feature Selection: Goal & Benefits
Feature selection
Also known as input selection
Goal
To select a subset out of the original feature set for better prediction accuracy
Benefits
Improve recognition rate
Reduce computation load
Explain relationships between features and classes
Specifically, this is the outline of the talk. Wel start from the basics, introduce the concepts of fuzzy sets and membership functions. By using fuzzy sets, we can formulate fuzzy if-then rules, which are commonly used in our daily expressions. We can use a collection of fuzzy rules to describe a system behavior; this forms the fuzzy inference system, or fuzzy controller if used in control systems. In particular, we can can apply neural networks?learning method in a fuzzy inference system. A fuzzy inference system with learning capability is called ANFIS, stands for adaptive neuro-fuzzy inference system. Actually, ANFIS is already available in the current version of FLT, but it has certain restrictions. We are going to remove some of these restrictions in the next version of FLT. Most of all, we are going to have an on-line ANFIS block for SIMULINK; this block has on-line learning capability and it ideal for on-line adaptive neuro-fuzzy control applications. We will use this block in our demos; one is inverse learning and the other is feedback linearization.

3. Exhaustive Search Steps for direct exhaustive search Drawback
2018/9/11
Exhaustive Search
Steps for direct exhaustive search
Use KNNC as the classifier, LOO for RR estimate
Generate all combinations of features and evaluate them one-by-one
Select the feature combination that has the best RR.
Drawback
d = 10  1023 CV for evaluation  Time consuming!
Advantage
The optimal feature set can be identified.

4. Exhaustive Search Direct exhaustive search 1 input . 2 inputs .
2018/9/11
Exhaustive Search
Direct exhaustive search x1 x2 x3 x4 x5
1 input .
x1, x2
x1, x3
x1, x4
x1, x5
x2, x3
2 inputs .
x1, x2, x3
x1, x2, x4
x1, x2, x5
x1, x3, x4
3 inputs
x1, x2, x3, x4
x1, x2, x3, x5
x1, x2, x4, x5 .
4 inputs .

5. 2018/9/11
Exhaustive Search
Characteristics of exhaustive search for feature selection
The process is time consuming, but the identified feature set is optimum.
It’s possible to use classifiers other than KNNC.
It’s possible to use performance indices other than LOO.

6. Heuristic Search Heuristic search for input selection One-pass ranking
2018/9/11
Heuristic Search
Heuristic search for input selection
One-pass ranking
Sequential forward selection
Generalized sequential forward selection
Sequential backward selection
Generalized sequential backward selection
‘Add m, remove n’ selection
Generalized ‘add m, remove n’ selection

7. Sequential Forward Selection
2018/9/11
Sequential Forward Selection
Steps for sequential forward selection
Use KNNC as the classifier, LOO for RR estimate
Select the first feature that has the best RR.
Select the next feature (among all unselected features) that, together with the selected features, gives the best RR.
Repeat the previous step until all features are selected.
Advantage
If we have d features, we need to perform d(d+1)/2 CV  A lot more efficient.
Drawback
The selected features are not always optimal.

8. Sequential Forward Selection
2018/9/11
Sequential Forward Selection
Sequential forward selection (SFS) x1 x2 x3 x4 x5
1 input
x2, x1
x2, x3
x2, x4
x2, x5
2 inputs
x2, x4, x1
x2, x4, x3
x2, x4, x5
3 inputs
x2, x4, x3, x1
x2, x4, x3, x5
4 inputs .

9. Example: Iris Dataset
Sequential forward selection
Exhaustive search

10. Example: Wine Dataset SFS SFS with input normalization
3 selected features, LOO RR=93.8%
6 selected features, LOO RR=97.8%
If we use exhaustive search, we have 8 features with LOO RR=99.4%

11. Model complexity (# of selected inputs)
Use of Input Selection
Common use of input selection
Increase the model complexity sequentially by adding more inputs
Select the model that has the best test RR
Typical curve of error vs. model complexity
Determine the model structure with the least test error
Test error
Optimal structure
Error rate
Training error
Model complexity (# of selected inputs)