1. Feature Selection and Weighting using Genetic Algorithm for Off-line Character Recognition Systems Faten Hussein Presented by The University of British Columbia Department of Electrical & Computer Engineering

2. Outline Introduction & Problem Definition Motivation & Objectives System Overview Results Conclusions

3. Scanning Pre-Processing Feature Extraction Classification Post-Processing Text document Classified text Introduction Off-line Character Recognition System Address readers Bank Cheques readers Reading data entered in forms (tax forms) Detecting forged signatures

4. Introduction Many variants of character (symbol) shape, size. Different writers have different writing styles. Same person could have different writing style. Thus, unlimited number of variations for a single character exists. For typical handwritten recognition task:

5. Variations in handwritten digits extracted from zip codes To overcome this diversity, a large number of features must be added Introduction An example of features that we used are: moment invariants, number of loops, number of end points, centroid, area, circularity and so on. L=2, E=0 L=1, E=1 L=0, E=3

6. Add more features  Increase problem size Dilemma  Increase run time/memory for classification  To accommodate variations in symbols  Hope to increase classification accuracy Character Recognition System Problem  Add-hoc process, depends on experience and trail and error  Might add redundant/irrelevant features which decrease the accuracy

7. Advantages Feature Selection Solution: Feature Selection Definition: Select a relevant subset of features from a larger set of features while maintaining or enhancing accuracy  Remove irrelevant and redundant features Total of 40 features -> reduced to 16 7 Hu moments -> only first three Area removed -> redundant (Circularity)  Maintain/enhance the classification accuracy 70% recognition rate using 40 features -> 75% after FS & using only 16 features  Faster classification and less memory requirements

8. Feature Selection/Weighting The process of assigning weights (binary or real valued) to features needs a search algorithm to search for the set of weights that results in best classification accuracy (optimization problem) Genetic algorithm is a good search method for optimization problems Feature Selection (FS)Feature Weighting (FW) Special CaseGeneral Case Binary weights (0 for irrelevant/redundant & 1 for relevant) Real-valued weights (variable weights depending on the feature relevance) Number of feature subset combinations

9. Genetic Feature Selection/Weighting Has been proven to be a powerful search method for FS problem Does not require derivative information or any extra knowledge; only the objective function (classifier’s error rate) to evaluate the quality of the feature subset Search a population of solutions in parallel, so they can provide a number of potential solutions not only one GA is resistant to becoming trapped in local minima Why use GA for FS/FW

10. Objectives & Motivations Study the effect of varying weight values on the number of selected features (FS often eliminates more features than FW, how much ??) Compare the performance of genetic feature selection/weighting in the presence of irrelevant & redundant features (not studied before) Compare the performance of genetic feature selection/weighting for regular cases (test the hypothesis that says that FW should have better or at least same results as FS ??) Evaluate the performance of the better method (GFS or GFW) in terms of optimality and time complexity (study the feasibility of genetic search for optimality & time) Build a genetic feature selection/weighting system to be applied to character recognition problem and investigate the following issues:

11. Methodology The recognition problem is to classify isolated handwritten digits Used k-nearest-neighbor as a classifier (k=1) Used genetic algorithm as search method Applied genetic feature selection and weighting in the wrapper approach (i.e. fitness function is the classifier’s error rate) Used two phases during the program run: training/testing phase and validation phase

12. System Overview Pre- Processing Module All Extracted features N Feature selection/weighting Module (GA) Evaluation Module (KNN classifier) Feature subset Assessment of feature subset Evaluation Best feature subset (M <N) Training/Testing Validation Feature Extraction Module Input (isolated handwritten digits images) Clean images

13. Results (Comparison 1) Effect of varying weight values on the number of selected features As the number of weight values increase, the probability of a feature having weight value=0 (POZ) decreases, so the number of eliminated features decreases GFS eliminates more features (thus selects less features) than GFW because of its smaller number of weight values (0/1) and without compromising classification accuracy

14. Results (Comparison 2) Performance of genetic feature selection/weighting in the presence of irrelevant features The performance of 1-NN classifier rapidly degrades by increasing the number of irrelevant features As the number of irrelevant features increases, FS outperform all FW settings in both classification accuracy and elimination of features

15. Results (Comparison 3) Performance of genetic feature selection/weighting in the presence of redundant features The classification accuracy of 1-NN does not suffer so much by adding redundant features, but they increase the problem size As the number of redundant features increases, FS has slightly better classification accuracy than all FW settings, but significantly outperform FW in elimination of features

16. Results (Comparison 4) Performance of genetic feature selection/weighting for regular cases (not necessarily having irrelevant/redundant) FW has better training accuracies than FS, but FS is better in generalization (have better accuracies for unseen validation samples) FW over-fits the training samples

17. Results (Evaluation 1) Convergence of GFS to an Optimal or Near-Optimal Set of Features Number of features Best Exhaustive (class. rate %) Best GA (class. rate %) Average GA (5 runs) 874 1075.2 1277.2 77.04 1479 78.56 1679.27978.28 1879.4 78.92 GFS was able to return optimal or near-optimal values (reached by the exhaustive search) The worst average value obtained by GFS less than 1% away from optimal value

18. Results (Evaluation 2) Number of Features Best Exh. (opt. & near-opt.) Exhaustive Run Time Best GA Average GA (for 5 runs) Number of Generations GA Run Time (single run) 874, 73.82 minutes7473.6852 minutes 1075.2, 7513 minutes75.274.9653 minutes 1277.2, 7747 minutes7776.92105 minutes 1479, 78.83 hours7978.2105.5 minutes 1679.2, 796 hours79.278.48158 minutes 1879.4, 79.21.5 days79.478.922011 minutes Convergence of GFS to an Optimal or Near-Optimal Set of Features within an Acceptable Number of Generations The time needed for GFS is bounded by (lower) linear-fit and (upper) exponential-fit curves The use of GFS for highly dimensional problems need parallel processing

19. Conclusions GFS is superior to GFW in feature reduction and without compromising classification accuracy In the presence of irrelevant features, GFS is better than GFW in both feature reduction and classification accuracy In the presence of redundant features, GFS is also preferred over GFW due its increased ability to feature reduction For regular databases, it is advisable to use 2 or 3 weight values at most to avoid over-fitting GFS is a reliable method to find optimal or near-optimal solution, but need parallel processing for large problem sizes

20. Questions ?