Queensland University of Technology Y. Li Y, A. Algarni, and N. Zhong Mining Positive and Negative Patterns for Relevance Feature Discovery 16th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, Washington, DC (KDD 2010), pp. 753-762. (Presented By) Prof. Yuefeng Li Queensland University of Technology Brisbane, QLD 4001, Australia y2.li@qut.edu.au

Outline Introduction What’s relevance feature discovery, term based Models, Why patterns? Are pattern effective for relevance feature discovery? And, the new solution The Deploying Method The definition, deploying higher level patterns to low-level terms Low-level Features Specificity and Exhaustive. Calculate the specificity score. Classification rules. Revise weights of the low-level features. Evaluation Conclusion

Introduction Relevance is a fundamental concept: Topical relevance, discusses a document's relevance to a given query User relevance, discusses a document's relevance to a user. The objective of relevance feature discovery is to find useful features available in a training set for describing what users want.

Term Based Models Popular term-based IR models The Rocchio algorithm Probabilistic models Okapi BM25 Language models: model-based methods and relevance models The advantages: efficient computational performance; and mature theories for term weighting. Phrases have been used in some IR models, as phrases are more discriminative and carry more “semantics" than words. Currently, many researchers illustrated that phrases are useful and crucial for query expansion in building good ranking functions.

Why Patterns? Challenging issue of using phrases: Finding useful phrases for text mining and classification, since naturally, phrases have inferior statistical properties to words, and there are a large number of redundant and noisy phrases among them. Patterns can be an promising alternative of phrases Like words, patterns enjoy good statistical properties. Data mining has developed some techniques: maximal patterns, closed patterns and master patterns for removing redundant and noisy patterns.

Are Patterns Effective for Relevance Feature Discovery? Pattern Taxonomy Models (PTM) It used closed sequential patterns, and has shown a certain extent improvement on the effectiveness. Two Challenging Issues for using Patterns in Text Mining The low-support problem, given a topic, large patterns are more specific for the topic, but with low supports (or low frequency). If we decrease the minimum support, there are a lot of noisy patterns would be discovered. The misinterpretation problem that means the measures used in pattern mining (e.g., “support” and “confidence”) turn out to be not suitable in using discovered patterns for answering what users want. For example, a highly frequent pattern is usually a general pattern for a topic.

The New Solution Features in two levels: low-level terms and higher level patterns. An innovative approach to evaluate weights of terms according to both their specificity and their distributions in the higher level features where the higher level features include both positive and negative patterns.

The Deploying Method What’s deploying: evaluating weights (supports) of low-level terms based on their distribution (appearances) in higher level patterns where higher level patterns are closed patterns that frequently appeared in paragraphs. A method for interpreting discovered patterns that provide a new method for weighting terms. An efficient and effective way for using patterns in solving problems; especially for using large patterns.

How Deploying Works Let SP1, SP2, ..., SPn be the sets of discovered closed sequential patterns for all documents di  D+ (i = 1, 2, …, n), where n = |D+|.

How Deploying Works cont.

Specificity of Low-Level Features A term’s specificity describes the extent of the term to which the topic focuses on what users want. For example, “JDK” is more specific than term “LIB” for describing “Java Programming Language”. Basically, the specificity of terms based on their positions in a concept hierarchy. For example, terms are more general if they are in the upper part of the LCSH (Library of Congress Subject Headings) hierarchy; otherwise, they are more specific. In many cases, a term’s specificity is measured based on what topics we are talking about. For example, “knowledge discovery” will be a general term in data mining community; however it may be a specific term when we talk about information technology.

Definition of Specificity The concept of relevance is subjective. It is easy for human being to do so. However, it is very difficult to use these concepts for interpreting relevance features in text documents. We define the specificity of a given term t in the training set D = D+  D- as follows:

Classification Rules Based on the specificity score the terms can be categorized into three group used the following classification rules: where G is a general terms, T+ is the specific positive terms and T- is the specific negative terms . 1 T+ | G | T- -0.5

Negative Feedback In general, negative and positive documents could share some background concepts or noises knowledge. There are two main issues of using negative relevance feedback: How to select constructive negative examples to reduce noises and the space of negative examples as well? How to use selected negative examples to refine the discovered knowledge in D+?

The Selection of Constructive Negative Examples Offender documents are negative documents that are most likely to be classified as positive.

How to select constructive negative examples? Re-rank the negative examples used the extracted low-level features from positive feedback. Select the top-K documents as an offenders, Extract higher level patterns and low-level terms from selected negative examples that uses the similar method as for mining in positive documents.

Weight Revision of Low-level Features Generally the specific terms are more important; however, general terms are also necessary for describing what users want. For that reasons, we increased the weights of positive terms and reduce the weights of the specific negative terms as in this paper:

EVALUATION Text Data - Reuters Corpus Volume 1 (RCV1) is used to test the effectiveness of the proposed model, including a total of 806,791 documents. Topics - 50 TREC assessor topics The documents are treated as plain text documents by pre- processing the documents. The tasks of removing stop-words according to a given stop- words list and stemming terms by applying the Porter Stemming algorithm are conducted.

Results Results of the proposed model against the baseline models for all 50 assessing topics

Discussions top-20 Map Fscore p/b IAP RFD PTM Rocchio SVM BM25 Percentage Changes   top-20 Map Fscore p/b IAP RFD 0.557 0.493 0.470 0.472 0.513 PTM 0.496 0.444 0.439 0.430 0.464 Rocchio 0.474 0.420 0.452 SVM 0.453 0.409 0.421 0.408 0.435 BM25 0.445 0.407 0.414 0.428 % Change 12.30% 11.18% 6.92% 9.75% 10.44% Statistical information for RFD with different values of K K Average number of training documents Average number of extracted terms Average weight of extracted terms MAP Positive Negative Offenders T+ G T- w(t+) w(tg) w(t-) K=|D+|/2 12.780 41.300 6.540 23.540 22.360 231.780 4.158 1.400 -0.551 0.493 K=|D-| 39.920 14.200 15.280 539.360 1.858 0.890 -71.202 0.278 K=|D+| 10.180 20.780 20.740 280.180 3.060 1.271 -2.965 0.463 Statistical information for RFD and PTM Average number of Extracted Terms used RFD Average weight(t) before revision Averageweight(t) after revision Terms extracted from D+ used PTM T+ G T- w(t+) w(tg) w(t-) T w(t) 23.54 22.36 231.78 2.842 1.400 0.320 4.158 -0.551 156.9 1.452

Results of using different values of K in RCV1 dataset Offenders selection Results of using different values of K in RCV1 dataset

Results of using different groups of terms in all 50 assessing topics Classification rules Results of using different groups of terms in all 50 assessing topics

Weight revision Weight distributions before and after the revision for the extracted features.

Conclusion Compared with the state-of-the-art models, the proposed model is consistent and very significant on all five measures on all 50 assessing topics. Negative (offender) selection approach is satisfactory. The use of negative relevance feedback is very significant for the relevance feature discovery. It can balance the percentages of the specific terms and the general terms to reduce noises.

Questions?