Text Categorization.

Slides:



Advertisements
Similar presentations
3.6 Support Vector Machines
Advertisements

Introduction to Information Retrieval Introduction to Information Retrieval Lecture 7: Scoring and results assembly.
Jeopardy Q 1 Q 6 Q 11 Q 16 Q 21 Q 2 Q 7 Q 12 Q 17 Q 22 Q 3 Q 8 Q 13
Jeopardy Q 1 Q 6 Q 11 Q 16 Q 21 Q 2 Q 7 Q 12 Q 17 Q 22 Q 3 Q 8 Q 13
On Comparing Classifiers : Pitfalls to Avoid and Recommended Approach
Traditional IR models Jian-Yun Nie.
Machine Learning: Intro and Supervised Classification
Improvements and extras Paul Thomas CSIRO. Overview of the lectures 1.Introduction to information retrieval (IR) 2.Ranked retrieval 3.Probabilistic retrieval.
Boolean and Vector Space Retrieval Models
25 seconds left…...
We will resume in: 25 Minutes.
CSE3201/4500 Information Retrieval Systems
Basics of Statistical Estimation
Classification Classification Examples
Mustafa Cayci INFS 795 An Evaluation on Feature Selection for Text Clustering.
Application of Ensemble Models in Web Ranking
Chapter 5: Introduction to Information Retrieval
1 CS 391L: Machine Learning: Instance Based Learning Raymond J. Mooney University of Texas at Austin.
Text Categorization CSC 575 Intelligent Information Retrieval.
Learning for Text Categorization
K nearest neighbor and Rocchio algorithm
Database Management Systems, R. Ramakrishnan1 Computing Relevance, Similarity: The Vector Space Model Chapter 27, Part B Based on Larson and Hearst’s slides.
Ch 4: Information Retrieval and Text Mining
1 CS 430 / INFO 430 Information Retrieval Lecture 3 Vector Methods 1.
Retrieval Models II Vector Space, Probabilistic.  Allan, Ballesteros, Croft, and/or Turtle Properties of Inner Product The inner product is unbounded.
Automatic Indexing (Term Selection) Automatic Text Processing by G. Salton, Chap 9, Addison-Wesley, 1989.
1 Text Categorization. 2 Categorization Given: –A description of an instance, x  X, where X is the instance language or instance space. –A fixed set.
Chapter 5: Information Retrieval and Web Search
1 Text Categorization  Assigning documents to a fixed set of categories  Applications:  Web pages  Recommending pages  Yahoo-like classification hierarchies.
Advanced Multimedia Text Classification Tamara Berg.
Modeling (Chap. 2) Modern Information Retrieval Spring 2000.
Rainbow Tool Kit Matt Perry Global Information Systems Spring 2003.
Processing of large document collections Part 2 (Text categorization) Helena Ahonen-Myka Spring 2006.
1 Text Categorization. 2 Categorization Given: –A description of an instance, x  X, where X is the instance language or instance space. –A fixed set.
Text mining.
Processing of large document collections Part 2 (Text categorization, term selection) Helena Ahonen-Myka Spring 2005.
Text Classification, Active/Interactive learning.
Xiaoying Gao Computer Science Victoria University of Wellington Intelligent Agents COMP 423.
Xiaoying Gao Computer Science Victoria University of Wellington Intelligent Agents COMP 423.
Chapter 6: Information Retrieval and Web Search
Information Retrieval and Web Search IR models: Vectorial Model Instructor: Rada Mihalcea Class web page: [Note: Some.
1 Computing Relevance, Similarity: The Vector Space Model.
CPSC 404 Laks V.S. Lakshmanan1 Computing Relevance, Similarity: The Vector Space Model Chapter 27, Part B Based on Larson and Hearst’s slides at UC-Berkeley.
Text mining. The Standard Data Mining process Text Mining Machine learning on text data Text Data mining Text analysis Part of Web mining Typical tasks.
1 Text Categorization. 2 Categorization Given: –A description of an instance, x  X, where X is the instance language or instance space. –A fixed set.
Classification Techniques: Bayesian Classification
Vector Space Models.
1 CS 391L: Machine Learning Text Categorization Raymond J. Mooney University of Texas at Austin.
CIS 530 Lecture 2 From frequency to meaning: vector space models of semantics.
1 Text Categorization Slides based on R. Mooney (UT Austin)
Xiaoying Gao Computer Science Victoria University of Wellington COMP307 NLP 4 Information Retrieval.
CS Machine Learning Instance Based Learning (Adapted from various sources)
Vector Space Classification 1.Vector space text classification 2.Rochhio Text Classification.
Information Retrieval and Web Search IR models: Vector Space Model Term Weighting Approaches Instructor: Rada Mihalcea.
1 Text Categorization  Assigning documents to a fixed set of categories  Applications:  Web pages  Recommending pages  Yahoo-like classification hierarchies.
IR 6 Scoring, term weighting and the vector space model.
Plan for Today’s Lecture(s)
Instance Based Learning
Intelligent Information Retrieval
Machine Learning. k-Nearest Neighbor Classifiers.
Instance Based Learning (Adapted from various sources)
Information Retrieval and Web Search
Representation of documents and queries
Text Categorization Assigning documents to a fixed set of categories
From frequency to meaning: vector space models of semantics
CS 430: Information Discovery
4. Boolean and Vector Space Retrieval Models
Hankz Hankui Zhuo Text Categorization Hankz Hankui Zhuo
Boolean and Vector Space Retrieval Models
INFORMATION RETRIEVAL TECHNIQUES BY DR. ADNAN ABID
Presentation transcript:

Text Categorization

Text Categorization Assigning documents to a fixed set of categories. Applications: Web pages Recommending Yahoo-like classification Newsgroup Messages spam filtering News articles Personalized newspaper Email messages Routing Prioritizing Folderizing

Learning for Text Categorization Manual development of text categorization functions is difficult. Learning Algorithms: Bayesian (naïve) Neural network Relevance Feedback (Rocchio) Rule based (Ripper) Nearest Neighbor (case based) Support Vector Machines (SVM)

The Vector-Space Model Assume t distinct terms remain after preprocessing; call them index terms or the vocabulary. These “orthogonal” terms form a vector space. Dimension = t = |vocabulary| Each term, i, in a document or query, j, is given a real-valued weight, wij. Both documents and queries are expressed as t-dimensional vectors: dj = (w1j, w2j, …, wtj)

Graphic Representation Example: D1 = 2T1 + 3T2 + 5T3 D2 = 3T1 + 7T2 + T3 Q = 0T1 + 0T2 + 2T3 T3 T1 T2 D1 = 2T1+ 3T2 + 5T3 D2 = 3T1 + 7T2 + T3 Q = 0T1 + 0T2 + 2T3 7 3 2 5 Is D1 or D2 more similar to Q? How to measure the degree of similarity? Distance? Angle? Projection?

Term Weights: Term Frequency More frequent terms in a document are more important, i.e. more indicative of the topic. fij = frequency of term i in document j May want to normalize term frequency (tf) by dividing by the frequency of the most common term in the document: tfij = fij / maxi{fij}

Term Weights: Inverse Document Frequency Terms that appear in many different documents are less indicative of overall topic. df i = document frequency of term i = number of documents containing term i idfi = inverse document frequency of term i, = log2 (N/ df i) (N: total number of documents) An indication of a term’s discrimination power. Log used to dampen the effect relative to tf.

wij = tfij idfi = tfij log2 (N/ dfi) TF-IDF Weighting A typical combined term importance indicator is tf-idf weighting: wij = tfij idfi = tfij log2 (N/ dfi) A term occurring frequently in the document but rarely in the rest of the collection is given high weight. Many other ways of determining term weights have been proposed. Experimentally, tf-idf has been found to work well.

Similarity Measure A similarity measure is a function that computes the degree of similarity between two vectors. Using a similarity measure between the query and each document: It is possible to rank the retrieved documents in the order of presumed relevance. It is possible to enforce a certain threshold so that the size of the retrieved set can be controlled.

Cosine Similarity Measure 2 t3 t1 t2 D1 D2 Q 1 Cosine similarity measures the cosine of the angle between two vectors. Inner product normalized by the vector lengths. CosSim(dj, q) = D1 = 2T1 + 3T2 + 5T3 CosSim(D1 , Q) = 10 / (4+9+25)(0+0+4) = 0.81 D2 = 3T1 + 7T2 + 1T3 CosSim(D2 , Q) = 2 / (9+49+1)(0+0+4) = 0.13 Q = 0T1 + 0T2 + 2T3 D1 is 6 times better than D2 using cosine similarity but only 5 times better using inner product.

Using Relevance Feedback (Rocchio) Relevance feedback methods can be adapted for text categorization. Use standard TF/IDF weighted vectors to represent text documents (normalized by maximum term frequency). For each category, compute a prototype vector by summing the vectors of the training documents in the category. Assign test documents to the category with the closest prototype vector based on cosine similarity.

Illustration of Rocchio Text Categorization

Rocchio Text Categorization Algorithm (Training) Assume the set of categories is {c1, c2,…cn} For i from 1 to n let pi = <0, 0,…,0> (init. prototype vectors) For each training example <x, c(x)>  D Let d be the frequency normalized TF/IDF term vector for doc x Let i = j: (cj = c(x)) (sum all the document vectors in ci to get pi) Let pi = pi + d

Rocchio Text Categorization Algorithm (Test) Given test document x Let d be the TF/IDF weighted term vector for x Let m = –2 (init. maximum cosSim) For i from 1 to n: (compute similarity to prototype vector) Let s = cosSim(d, pi) if s > m let m = s let r = ci (update most similar class prototype) Return class r

Rocchio Properties Does not guarantee a consistent hypothesis. Forms a simple generalization of the examples in each class (a prototype). Prototype vector does not need to be averaged or otherwise normalized for length since cosine similarity is insensitive to vector length. Classification is based on similarity to class prototypes.

Nearest-Neighbor Learning Algorithm Learning is just storing the representations of the training examples in D. Testing instance x: Compute similarity between x and all examples in D. Assign x the category of the most similar example in D. Does not explicitly compute a generalization or category prototypes. Also called: Case-based Memory-based Lazy learning

K Nearest-Neighbor Using only the closest example to determine categorization is subject to errors due to: A single atypical example. Noise (i.e. error) in the category label of a single training example. More robust alternative is to find the k most-similar examples and return the majority category of these k examples. Value of k is typically odd to avoid ties, 3 and 5 are most common.

Illustration of 3 Nearest Neighbor for Text

Rocchio Anomoly Prototype models have problems with polymorphic (disjunctive) categories.

3 Nearest Neighbor Comparison Nearest Neighbor tends to handle polymorphic categories better.

K Nearest Neighbor for Text Training: For each each training example <x, c(x)>  D Compute the corresponding TF-IDF vector, dx, for document x Test instance y: Compute TF-IDF vector d for document y For each <x, c(x)>  D Let sx = cosSim(d, dx) Sort examples, x, in D by decreasing value of sx Let N be the first k examples in D. (get most similar neighbors) Return the majority class of examples in N

Naïve Bayes for Text Modeled as generating a bag of words for a document in a given category by repeatedly sampling with replacement from a vocabulary V = {w1, w2,…wm} based on the probabilities P(wj | ci). Smooth probability estimates with Laplace m-estimates assuming a uniform distribution over all words (p = 1/|V|) and m = |V| Equivalent to a virtual sample of seeing each word in each category exactly once.

Naïve Bayes Generative Model for Text spam legit spam spam legit legit spam spam legit Category science Viagra win PM !! !! hot hot computer ! Friday Nigeria deal deal test homework lottery nude March score ! Viagra Viagra $ May exam spam legit

Naïve Bayes Classification Win lotttery $ ! ?? ?? spam legit spam spam legit legit spam spam legit science Viagra Category win PM !! hot computer ! Friday Nigeria deal test homework lottery nude March score ! Viagra $ May exam spam legit

Text Naïve Bayes Algorithm (Train) Let V be the vocabulary of all words in the documents in D For each category ci  C Let Di be the subset of documents in D in category ci P(ci) = |Di| / |D| Let Ti be the concatenation of all the documents in Di Let ni be the total number of word occurrences in Ti For each word wj  V Let nij be the number of occurrences of wj in Ti Let P(wj | ci) = (nij + 1) / (ni + |V|)

Text Naïve Bayes Algorithm (Test) Given a test document X Let n be the number of word occurrences in X Return the category: where ai is the word occurring the ith position in X

Sample Learning Curve (Yahoo Science Data)