1. Feature Selection Which features work best? One way to rank features: –Make a contingency table for each F –Compute abs ( log ( ad / bc ) ) –Rank the log values ab cd F Madison Hamilton Not F

2. 49 Ranked Features

3. Linear Discriminant Analysis A technique for classifying data Available in the R statistics package Input: –Table of training data –Table of test data Output: –Classification of test data

4. Linear Discriminant Analysis: example Input training data: upon 2-letter 3-letter M 0.000 206.943 194.927 M 0.000 212.915 194.665 M 0.369 202.583 190.775 M 0.000 201.891 213.712 M 0.000 236.943 206.221 H 3.015 235.176 187.940 H 2.458 226.647 201.082 H 4.955 232.432 192.793 H 2.377 232.937 186.078 H 3.788 224.116 196.338 upon 2-letter 3-letter 0.000 226.277 203.163 0.908 205.268 181.653 0.000 225.536 182.627 0.000 217.273 183.053 1.003 232.581 184.962 Input test data: Ouput: m m m m h

5. Some more LDA results 12 to Madison: –upon, 1-letter, 2-letter –upon, enough, there –upon, there 11 to Madison: –upon, 2-letter, 3-letter < 6 to Madison –2-letter, 3-letter –there, 1-letter, 2-letter

6. Some more LDA results ClassOutput of lda Features tested 12 Mm m m m m m upon apt 9 2 12 Mm m m m m m to upon 2 3 11 Mm m m m m m h m m m m mon there 2 13 11 Mh m m m m m m m m m m man by 5 10 10 Mm m m m m m h m m m h mparticularly probability 3 9 8 M m m m m m m h h h m h malso of 1 4 8 M m m m h m m h h m m h malways of 1 3 7 M h m m h m h h m h m m mof work 5 2 6 M m m h m m m h h m h h hthere language 1 8 5 M m h m h h m h h h m m hconsequently direction 5 11

7. Feature Selection Part II Which combinations of features are best for LDA? Are the features independent? We did some random sampling: –Choose features a, b, c, d –Compute x = log a + log b + log x + log d –Compute y = log (a+b+c+d) –Plot x versus y

9. Selecting more features What happens when more than 4 features are used for the lda? Greedy approach –Add features one at a time from two lists –Perform lda on all features chosen so far Is overfitting a problem?

10. First few greedy iterations 6 M 6 H h m h h m h m m h m h m 2-letter words 12 M 0 H m m m m m m upon 12 M 0 H m m m m m m 1-letter words 12 M 0 H m m m m m m 5-letter words 11 M 1 H m m m m m h m m m m m m 4-letter words 12 M 0 H m m m m m m there 12 M 0 H m m m m m m enough 11 M 1 H m m m m m m h m m m m m whilst 12 M 0 H m m m m m m 3-letter words 11 M 1 H m m m m m m h m m m m m 15-letter words