INFORMATION RETRIEVAL TECHNIQUES BY DR. ADNAN ABID Lecture # 22 Spelling Correction

ACKNOWLEDGEMENTS The presentation of this lecture has been taken from the underline sources “Introduction to information retrieval” by Prabhakar Raghavan, Christopher D. Manning, and Hinrich Schütze “Managing gigabytes” by Ian H. Witten, ‎Alistair Moffat, ‎Timothy C. Bell “Modern information retrieval” by Baeza-Yates Ricardo, ‎  “Web Information Retrieval” by Stefano Ceri, ‎Alessandro Bozzon, ‎Marco Brambilla

Outline Matching trigrams Computing Jaccard coefficient Context-sensitive spell correction General issues in spell correction

Standard postings “merge” will enumerate … Matching trigrams Consider the query lord – we wish to identify words matching 2 of its 3 bigrams (lo, or, rd) lo alone lore sloth or border lore morbid rd 00:01:35  00:02:30 ardent border card Standard postings “merge” will enumerate … Adapt this to using Jaccard (or another) measure.

1. ANSWER LIST Append in the answer list the terms which have 2 or more overlapping bi-grams with the query. This is possible in a single pass. 00:07:30  00:08:05

Standard postings “merge” will enumerate … Matching trigrams Consider the query lord – we wish to identify words matching 2 of its 3 bigrams (lo, or, rd) lo alone lore sloth or border lore morbid rd 00:08:30  00:08:38 00:08:50  00:09:30 00:09:56  00:11:25 00:13:46  00:13:51 00:14:30  00:15:00 00:16:30  00:16:50 ardent border card Standard postings “merge” will enumerate … Adapt this to using Jaccard (or another) measure.

2. ANSWER LIST Improve the walk through to use Jaccard Coefficient so as to identify the candidate terms. HEURISTIC While computing J.C. we may disregard the repeating n-grams in query term as well as in current term. The reason is that we are computing a candidate term in any case, which we shall process later using edit distance. 00:19:17  00:19:50

Computing Jaccard coefficient X = Number of bigram in query tem Y = Number of bigram in current term Threshold: if J.C(Query), (Current term) > 0.8 append current term in answer list |X Y| = Number of pointers at current term |X U Y| = |X| + |Y| - |X Y| 00:20:00  00:20:40 00:22:25  00:22:40 00:25:10  00:25:40

Joining N-grams with Edit Distance N-gram will give Answer list which contains candidate terms. These terms can then be checked for Edit Distance. This helps avoiding the checking of edit distance for all dictionary terms and each term, but restricts it to only candidate terms in the answer list 00:29:43  00:30:05

Context-sensitive spell correction Text: I flew from Lahore to Dubai. Consider the phrase query “flew form Lahore” We’d like to respond Did you mean “flew from Lahore”? because no docs matched the query phrase. 00:36:28  00:36:50 00:38:30  00:38:55

Context-sensitive correction Need surrounding context to catch this. NLP too heavyweight for this. First idea: retrieve dictionary terms close (in weighted edit distance) to each query term Now try all possible resulting phrases with one word “fixed” at a time flew from heathrow fled form heathrow flea form heathrow etc. Suggest the alternative that has lots of hits? Hits in corpus vs. Hits in Query Logs It is more appropriate to look for hits in Qurey Logs 00:41:25  00:41:50

Context-sensitive correction Suppose that for “flew form Heathrow” we have 7 alternatives for flew, 19 for form and 3 for heathrow. 7*19*3  Look for most frequent (in corpus/in query log) replacement of first word, combine it with the second to formulate a bigram, then choose the most frequent and then combine it with the third one. Reduces it to much less than 7*19*3 Alternatively, correct each word separately that will result in 7+19+3 Another alternative is to only correct the misspelled words. 00:42:18  00:43:10 00:44:30  00:44:41 00:48:45  00:49:00

General issues in spell correction We enumerate multiple alternatives for “Did you mean?” Need to figure out which to present to the user The alternative hitting most docs Query log analysis More generally, rank alternatives probabilistically argmaxcorr P(corr | query) From Bayes rule, this is equivalent to argmaxcorr P(query | corr) * P(corr) 00:50:50  00:51:36 00:51:50  00:52:10 Noisy channel Language model

Resources Peter Norvig: How to write a spelling corrector IIR 3, MG 4.2 Efficient spell retrieval: K. Kukich. Techniques for automatically correcting words in text. ACM Computing Surveys 24(4), Dec 1992. J. Zobel and P. Dart.  Finding approximate matches in large lexicons.  Software - practice and experience 25(3), March 1995. http://citeseer.ist.psu.edu/zobel95finding.html Mikael Tillenius: Efficient Generation and Ranking of Spelling Error Corrections. Master’s thesis at Sweden’s Royal Institute of Technology. http://citeseer.ist.psu.edu/179155.html Nice, easy reading on spell correction: Peter Norvig: How to write a spelling corrector http://norvig.com/spell-correct.html