1. A Survey on Text Classification
December 10, 2003
Dongho Kim
KAIST

2. Contents Introduction Statistical Properties of Text Feature Selection
Feature Space Reduction
Classification Methods
Using SVM and TSVM
Hierarchical Text Classification
Summary

3. Introduction Text classification
Assign text to predefined categories based on content
Types of text
Documents (typical)
Paragraphs
Sentences
WWW-Sites
Different types of categories
By topic
By function
By author
By style

4. Text Classification Example

5. Computer-Based Text Classification Technologies
Naive word-matching (Chute, Yang, & Buntrock 1994)
Finding shared words between the text and names of categories
Weakest method
Cannot capture any conceptually relation
Thesaurus-based matching (Lindberg & Humphreys 1990)
Using lexical links
Insensitive to the context
High cost and low adaptivity across domains

6. Computer-Based Text Classification Technologies
Empirical learning of term-category associations
Learning from a training set
Fundamentally different from word-matching
Statistically capturing the semantic association between terms and categories
Context sensitive mapping from terms to categories
For example,
Decision tree methods
Bayesian belief networks
Neural networks
Nearest neighbor classification methods
Least-squares regression techiniques

7. Statistical Properties of Text
There are stable, language-independent patterns in how people use natural language
A few words occur very frequently; most occur rarely
In general
Top 2 words : 10~15% of all word occurrences
Top 6 words : 20% of all word occurrences
Top 50 words : 50% of all word occurrences
Most common words from Tom Sawyer
The
3332
And
2972 A
1775 To
1725 Of
1440 …
Tom
679 1 14

8. Statistical Properties of Text
The most frequent words in one corpus may be rare words in another corpus
Example : ‘computer’ in CACM vs. National Geographic
Each corpus has a different, fairly small “working vocabulary”
InkML is an abbreviation for Ink Markup Language. InkML represents ink by electronic stylus and its format is the extension of XML. It contains several elements such as traces, information about capture device, information about canvas mapping, brushes, semantic labeling and so on. If you wanna know more details, visit this URL.
These properties hold in a wide range of languages

9. Statistical Properties of Text
Summary :
Term usage is highly skewed, but in a predictable pattern
Why is it important to know the characteristics of text?
Optimization of data structures
Statistical retrieval algorithms depend on them

10. Statistical Profiles Can act as a summarization device
Indicate what a document is about
Indicate what a collection is about
1987 WSJ (132MB)
1991 Patent (254MB)
1989 AP (267MB)
stobb (1)
stochast (1)
stock (46704)
stockad (5)
stockard (3)
stockbridg (2)
stockbrok (351)
stockbrokag (1)
stockbrokerag (101)
sto (1)
stochast (21)
stochiometr (1)
stociometr (1)
stock (1910)
stockbarg (30)
stocker (211)
stockholm (1)
stockigt (4)
sto (7)
sto1 (4)
sto3 (1)
stoaker (1)
stoand (1)
stober (6)
stocholm (1)
stock (28505)
stock’ (6)
Where two runs of black pixels appear on a single scan line of the raster image, if there is a run on the line below which spans the distance between these two runs, an upward concavity is formed on the line.
The title of first paper is “Automatic understanding of structure in printed mathematical expressions”.
Recognizing mathematical expressions from document image is a key problem in conversion of scientific documents into electronic form. In this paper, a simple grammar-based approach to recognize complex structures of printed mathematical expressions is presented.
And here is another paper about mathematical formulas. In this article, a system for the recognition of on-line handwritten mathematical formulas which is used in the electronic chalkboard is presented. You can see that mathematical formulas written on e-chalkboard are calculated.
3. This paper propose a method of writer verification based on on-line features extracted from a process of drawing a figure like these.

11. Zipf’s Law Zipf’s Law relates a term’s frequency to its rank
Frequency 1/rank
There is a constant such that
Rank the terms in a vocabulary by frequency, in descending order
Empirical observation :
Hence :
for English
I’d like to introduce some papers presented in this conference. Graphical recognition is the applications of pattern recognition to classification of graphical items. There’re some kinds of graphical items. For example, mathematical formula, drawing, symbols and so on.

12. Precision and Recall Recall Precision
Evaluation Metrics
Recall
Percentage of all relevant documents that are found by a search
Precision
Percentage of retrieved documents that are relevant
retrieved + -

13. Harmonic average of precision and recall
F-measure
Evaluation Metrics
Harmonic average of precision and recall
Rewards results that keep recall and precision close together
R=40, P=60. R/P average=50. F-measure=48
R=45, P=55. R/P average=50. F-measure=49.5
I’ll tell you about research trend of this ICDAR conference. The half of all papers are about character recognition or the classifier. I think that that is still the major part of our field. However, it seems to me that many practical applications also have much interest for the people. And From this year, two new areas, Web documents and graphical on-line recognition were added to existing areas.
And there was the effort for standardization for pen trajectory representatively InkML.
Because there were 33 oral sessions and the oral sessions are organized in 3 parallel tracks, I couldn’t attend all the sessions. And, it’s a little hard for me to understand theoretical and technical papers, so I’ll concentrate upon the some practical papers that I heard the explanation about.

14. Break Even Point The point at which recall equals precision
Evaluation Metrics
The point at which recall equals precision
Evaluation metric :
The value of this point

15. Term Weights: A Brief Introduction
Feature Selection
Term Weights: A Brief Introduction
The words of a text are not equally indicative of its meaning
Important: butterflies, monarchs, scientists, direction, compass
Unimportant : most, think, kind, sky, determine, cues, learn
Term weights reflect the (estimated) importance of each term
“Most scientists think that butterflies use the position of the sun in the sky as a kind of compass that allows them to determine which way is north. Scientists think that butterflies may use other cues, such as the earth’s magnetic field, but we have a lot to learn about monarchs’ sense of direction.”

16. Term Weights Term frequency (TF)
Feature Selection
Term frequency (TF)
The more often a word occurs in a document, the better that term is in describing what the document is about
Often normalized, e.g. by the length of the document
Sometimes biased to range [ ] to represent the fact that even a single occurrence of a term is a significant event
Document Analysis is the applications of pattern recognition to the interpretation of documents. This field contains web documents also.

17. Term Weights Inverse document frequency (IDF)
Feature Selection
Inverse document frequency (IDF)
Terms that occur in many documents in the collection are less useful for discriminating among documents
Document frequency (df) : number of documents containing the term
IDF often calculated as
TF and IDF are used in combination as product

18. Vector Space Similarity
Feature Selection
Similarity is inversely related to the angle between the vectors
Cosine of the angle between the two vectors

19. Feature Space Reduction
Main reasons
Improve accuracy of the algorithm
Decrease the size of data set
Control the computation time
Avoid overfitting
Feature space reduction technique
Stopword removal, stemming
Information gain
Natural language processing

20. Stopword Removal
Feature Space Reduction
Stopwords : words that are discarded from a document representation
Function words : a, an, and, as, for, in, of, the, to, …
About 400 words in English
Other frequent words : ‘Lotus’ in a Lotus Support

21. Stemming Group morphological variants
Feature Space Reduction
Group morphological variants
Plural : ‘streets’  ‘street’
Adverbs : ‘fully’  ‘full’
Other inflected word forms : ‘goes’  ‘go’
Grouping process is called “conflation”
Current stemming algorithms make mistakes
Conflating terms manually is difficult, time-consuming
Automatic conflation using rules
Porter Stemmer
Porter stemming example : ‘police’, ‘policy’  ‘polic’

22. Information Gain
Feature Space Reduction
Measuring information obtained by presence or absence of a term in a document
Feature space reduction by thresholding
Biased to common term  large reduction in size of data set cannot be achieved

23. Natural Language Processing
Feature Space Reduction
Pick out the important words from a document
For example, nouns, proper nouns, or verbs
Ignoring all other parts
Not biased to common terms  reduction in bath feature space and size of data
Named entities
The subset of proper nouns consisting of people, locations, and organization
Effective in cases of news story classification

24. Experimental Results Data set From six news media sources
Robert Cooley, Classification of News Stories Using Support Vector Machines,
Proceedings of the 16th International Joint Conference on Artificial Intelligence Text Mining Workshop, 1999
Data set
From six news media sources
Two print sources (New York Times and Associated Press Wire)
Two television sources (ABC World News Tonight and CNN Headline News)
Two radio sources (Public Radio International and Voice of America)

25. Experimental Results Results
Robert Cooley, Classification of News Stories Using Support Vector Machines,
Proceedings of the 16th International Joint Conference on Artificial Intelligence Text Mining Workshop, 1999
Results
NLP  significant loss in recall and precision
SVM >> kNN (using full text or information gain)
Binary weighting  significant loss in recall

26. kNN Stands for k-nearest neighbor classification Algorithms
Classification Methods
Stands for k-nearest neighbor classification
Algorithms
Given a test document,
Find k nearest neighbors among training documents
Calculate and sort score of candidate categories
Thresholding on these scores
Decision rule

27. LLSF Stands for Linear Least Squares Fit
Classification Methods
Stands for Linear Least Squares Fit
Obtain matrix of word-category regression coefficients by LLSF
FLS : arbitrary document  vector of weighted categories
By thresholding like kNN, assign categories

28. Naïve Bayes Assumption Result
Classification Methods
Assumption
Words are drawn randomly from class dependent lexicons (with replacement)
Word independence
Result
Word independence
Classification rule

29. Estimating the Parameters
Naïve Bayes
Count frequencies in training data
Estimating P(Y)
Fraction of positive / negative examples in training data
Estimating P(W|Y)
Smoothing with Laplace estimate

30. Experiment Results
Yiming Yang and Xin Liu, A re-examination of text categorization methods,
Proceedings of ACM SIGIR Conference on Research and Development in Information Retrieval, 1999.

31. Text Classification using SVM
T. Joachims, A Statistical Learning Model of Text Classification with Support Vector Machines,
Proceedings of the Conference on Research and Development in Information Retrieval (SIGIR), ACM, 2001.
A statistical learning model of text classification with SVMs:
0 if linearly separable

32. Properties 1+2: Sparse Examples in High Dimension
T. Joachims, A Statistical Learning Model of Text Classification with Support Vector Machines,
Proceedings of the Conference on Research and Development in Information Retrieval (SIGIR), ACM, 2001.
Properties 1+2: Sparse Examples in High Dimension
High dimensional feature vectors (30,000 features)
Sparse document vectors : only a few words of the whole language occur in each document
SVMs use overfitting protection which does not depend on the dimension of feature

33. Property 3: Heterogeneous Use of Words
T. Joachims, A Statistical Learning Model of Text Classification with Support Vector Machines,
Proceedings of the Conference on Research and Development in Information Retrieval (SIGIR), ACM, 2001.
Property 3: Heterogeneous Use of Words
No pair of documents shares any words, but ‘it’, ‘the’, ‘and’, ‘of’,
‘for’, ‘an’, ‘a’, ‘not’, ‘that’, ‘in’.

34. Property 4: High Level of Redundancy
T. Joachims, A Statistical Learning Model of Text Classification with Support Vector Machines,
Proceedings of the Conference on Research and Development in Information Retrieval (SIGIR), ACM, 2001.
Property 4: High Level of Redundancy
Few features are irrelevant!
: Feature space reduction causes loss of information

35. Property 5: ‘Zipf’s Law’
T. Joachims, A Statistical Learning Model of Text Classification with Support Vector Machines,
Proceedings of the Conference on Research and Development in Information Retrieval (SIGIR), ACM, 2001.
Most words occur very infrequently!

36. TCat Concepts Modeling real text-classification tasks
T. Joachims, A Statistical Learning Model of Text Classification with Support Vector Machines,
Proceedings of the Conference on Research and Development in Information Retrieval (SIGIR), ACM, 2001.
Modeling real text-classification tasks
Used for previous proof
TCat([20:20:100], # high freq.
[4:1:200],[1:4:200],[5:5:600]. # medium freq.
[9:1:3000],[1:9:3000],[10:10:4000] # low freq. )

37. TCat Concepts Margin of Tcat-Concepts By Zipf’s law, we can bound R2
T. Joachims, A Statistical Learning Model of Text Classification with Support Vector Machines,
Proceedings of the Conference on Research and Development in Information Retrieval (SIGIR), ACM, 2001.
Margin of Tcat-Concepts
By Zipf’s law, we can bound R2
Intuitively, many words with low frequency
 relatively short document vectors
with
Linearly separable

38. TCat Concepts Bound on Expected Error of SVM
T. Joachims, A Statistical Learning Model of Text Classification with Support Vector Machines,
Proceedings of the Conference on Research and Development in Information Retrieval (SIGIR), ACM, 2001.
Bound on Expected Error of SVM

39. Text Classification using TSVM
T. Joachims, Transductive Inference for Text Classification using Support Vector Machines,
Proceedings of the International Conference on Machine Learning (ICML), 1999.
How would you classify the test set?
Training set {D1, D6}
Test set {D2, D3, D4, D5}

40. Why Does Adding Test Examples Reduce Error?
T. Joachims, Transductive Inference for Text Classification using Support Vector Machines,
Proceedings of the International Conference on Machine Learning (ICML), 1999.
Why Does Adding Test Examples Reduce Error?

41. Experiment Results Data set Reuter-21578 dataset-ModApte
T. Joachims, Transductive Inference for Text Classification using Support Vector Machines,
Proceedings of the International Conference on Machine Learning (ICML), 1999.
Data set
Reuter dataset-ModApte
Training : 9,603 test : 3,299
WebKB collection of WWW pages
Only the class ‘course’, ‘faculty’, ‘project’, ‘student’ are used
Stemming and stopword removal are not used
Ohsumed corpus compiled by William Hersh
Training : 10,000 test : 10,000

42. P/R-breakeven point for Reuters categories
Experiment Results
T. Joachims, Transductive Inference for Text Classification using Support Vector Machines,
Proceedings of the International Conference on Machine Learning (ICML), 1999.
Results
P/R-breakeven point for Reuters categories

43. Experiment Results Results Average P/R-breakeven point on WebKB
T. Joachims, Transductive Inference for Text Classification using Support Vector Machines,
Proceedings of the International Conference on Machine Learning (ICML), 1999.
Results
Average P/R-breakeven point on WebKB
Average P/R-breakeven point on Ohsumed

44. Hierarchical Text Classification
Real world classification  complex hierarchical structure
Due to difficulties of training for many classes or features
Class 1-1
Class 1
Class 1-2
documents …
Class 2
Class 1-3
Class 2-1
Class 3 … …
Level 1
Level 2

45. Hierarchical Text Classification
More accurate specialized classifiers
‘computer’ : not discriminating
Hardware
documents
Computers
Software
Chat
Sports
Soccer
Football
‘computer’ : discriminating

46. Experiment Setting Data set : LookSmart’s web directory
S. Dumais and H. Chen, Hierarchical classification of Web content.
Proceedings of SIGIR'00, August 2000, pp
Data set : LookSmart’s web directory
Using short summary from search engine
unique pages
17173 categories
7-level hierarchy
Focus on 13 top-level and 150 second-level categories

47. Experiment Setting Using SVM
S. Dumais and H. Chen, Hierarchical classification of Web content.
Proceedings of SIGIR'00, August 2000, pp
Using SVM
Posterior probabilities by regularized maximum likelihood fitting
Combining probabilities from the first and second level
Boolean scoring function, P(L1) && P(L2) or,
Multiplicative scoring function, P(L1) * P(L2)

48. Experiment Results Non-hierarchical (baseline) : F1 = 0.476
S. Dumais and H. Chen, Hierarchical classification of Web content.
Proceedings of SIGIR'00, August 2000, pp
Non-hierarchical (baseline) : F1 = 0.476
Hierarchical
Top-level
Training set : F1 = 0.649
Test set : F1 = 0.572
Second-level
Multiplicative : F1 = 0.495
Boolean : F1 = 0.497
Assuming top-level classification is correct,
F1 = 0.711

49. Summary Feature space reduction
Performance of SVM and TSVM is better than others
TSVM has merits in text classification
Hierarchical classification is helpful
Other issues
Sampling strategies
Other kinds of feature selection

50. Reference
T. Joachims, Text Categorization with Support Vector Machines: Learning with Many Relevant Features. Proceedings of the European Conference on Machine Learning (ECML), Springer, 1998.
T. Joachims, Transductive Inference for Text Classification using Support Vector Machines. Proceedings of the International Conference on Machine Learning (ICML), 1999.
T. Joachims, A Statistical Learning Model of Text Classification with Support Vector Machines. Proceedings of the Conference on Research and Development in Information Retrieval (SIGIR), ACM, 2001.
Robert Cooley, Classification of News Stories Using Support Vector Machines (1999). Proceedings of the Sixteenth International Joint Conference on Artificial Intelligence Text Mining Workshop, August 1999.
Yiming Yang and Xin Liu, A re-examination of text categorization methods. Proceedings of ACM SIGIR Conference on Research and Development in Information Retrieval, (SIGIR), 1999.
S. Dumais and H. Chen, Hierarchical classification of Web content. Proceedings of SIGIR'00, August 2000, pp