Intelligent Database Systems Lab 國立雲林科技大學 National Yunlin University of Science and Technology 1 Phonetic String Matching:Lessons from Information Retrieval Advisor : Dr. Hsu Graduate : Chih-Ling Wang Authors : Justin Zobel Philip Dart 2003 IEEE.

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 2 Outline Motivation Objective Introduction Phonetic matching versus information retrieval Phonetic matching techniques Performance assessment Combination of evidence Conclusions Personal Opinion

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 3 Motivation We explore the accuracy of the phonetic string matching.

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 4 Objective In this paper we propose a new phonetic matching techniques and describe the results of a new comparative investigation of phonetic matching.

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 5 Introduction Phonetic matching is used to identify strings that may be of similar pronunciation, regardless of their actual spelling.

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 6 Introduction(cont.) There are two pragmatic issues that must be addressed in such a phonetic matching system. One is of speed – answers should be found reasonably quickly. The other pragmatic issue is accuracy. The parallels between information retrieval and phonetic matching mean that They can be measured by the same kinds of techniques. Methods for improving information retrieval performance may also apply to phonetic matching.

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 7 Phonetic matching versus information retrieval In information retrieval, ranking is the process of identifying which of a set of documents are most likely to be similar in content to a given query. Phonetic matching is the process of identifying which of a set of strings are most likely to be similar in sound to a given query string.

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 8 Phonetic matching versus information retrieval(cont.) In both cases the matching process is: fundamentally inexact, since human judgment is required to tell whether the process’s guess is correct Similarity is relative, unable in isolation to determine whether a query and potential answer are matches. It is difficult to give an accurate definition of relevance.

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 9 Phonetic matching versus information retrieval(cont.) We consider phonetic matching to be the process of identifying strings that, after elimination of possible transmission or cognition errors, may sound the same. Transmission errors include, sound-alike mistakes in data entry ;mishearing of a spoken name on a imperfect transmission medium. Cognition errors include, mistaking a pronunciation for an expected word.

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 10 Phonetic matching techniques Soundex  Soundex uses codes based on the sound of each letter to translate a string into a canonical form of at most four characters, preserving the first letter.  Soundex makes the error of transforming dissimilar- sounding strings to the same code, and of transforming similar-sounding strings to different codes.  There is no ranking of matches: strings are either similar or not similar.

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 11 Example: reynold(r005043)=>r543 renauld(r050043) =>r543 Phonetic matching techniques(cont.)

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 12 Phonix Phonix is a Soundex variant. Letters are mapped to a set of codes using the same algorithm, but a slightly different set of codes is used, and prior to mapping about 160 letter-group transformations are used to standardise the string. The sequence tjv is mapped to chv if it occurs at the start of a string, and x is transformed to ecs. These transformations provide context for the phonetic coding and allow c and s to be distinguished. Phonetic matching techniques(cont.)

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 13 Example: reynold(r005043)=>r543 renauld(r050043) =>r543 Phonetic matching techniques(cont.)

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 14 In our experiments we consider a variant of Phonix, here called Phonix+, in which truncation is not applied and a minimal edit distance is used to compare the resulting strings. Phonetic matching techniques(cont.)

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 15 Q-gram methods A q-gram of string s is any substring of s of some fixed length q. Simply counting q-grams does not allow for length differences. Phonetic matching techniques(cont.)

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 16 Example:rhodes;rod We have used this q-gram method with q=2 rhodes= rod= Phonetic matching techniques(cont.)

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 17 Agrep Agrep is a utility that embodies a fast algorithm for identifying strings that contain a substring which is identical to a query but for at most k insertions, deletions, or replacements, where k is a predefined constant. Agrep was not designed for the task of phonetic matching, but rather for fast searching of large files. Phonetic matching techniques(cont.)

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 18 Edit distances A simple edit distance, which counts the minimal number of single-character insertions, deletions, and replacements needed to transform one string into another, could be used for phonetic matching since similar-sounding words are often spelled similarly. Phonetic matching techniques(cont.)

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 19 For two strings s and t of length i and j respectively, this edit distance can computed with the recurrence relation edit(i,j). The function returns 0 if and are identical, and 1otherwise. Phonetic matching techniques(cont.)

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 20 Example: rhodes;rod Phonetic matching techniques(cont.)

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 21 Editex Editex is a phonetic distance measure that combines the properties of edit distances with the letter- grouping strategy used by Soundex and Phonix. Editex also groups letters that can result in similar pronunciations, but doesn't require that the groups be disjoint and can thus reflect the correspondences between letters and possible similar pronunciation more accurately. Phonetic matching techniques(cont.)

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 22 Editex is defined by the edit distance recurrence relation with a redefined function returns 0 if and are identical, 1 if and are both occur in the same group, and 2 otherwise. The function is identical to.If a is h or w and a b then is 1. Phonetic matching techniques(cont.)

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 23 Phonometric methods Our algorithms for phonometric matching consist of two stages: First, the string of letters is converted into a string of phonemes by a string-to-pronunciation conversion algorithm. The second stage is comparison of strings of phonemes. The distance between pronunciations as represented by strings of phonemes can be measured more precisely than the distance between strings of letters. Phonetic matching techniques(cont.)

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 24 Tapering Tapering is a refinement to the edit distance techniques based on a human-factors property: differences at the start of a pronunciation can be more significant than differences at the end. A tapered edit distance of particular interest is one in which the maximum penalty for replacement or deletion at start of string just exceeds twice the minimum penalty for replacement or deletion at end of string. Phonetic matching techniques(cont.)

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 25 We can now compare the various approaches to phonetic matching.Results are shown in Table 1, which is of 11-point recall- precision. For many of the techniques tested, only a few distinct ranks are possible, and some techniques only return two ranks, match and not-match. Performance assessment

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 26 The least effective methods such as Phonix and Soundex only return a small number of answers for most queries. Phonix and Soundex are not only finding many wrong answers but not finding many right ones. The “baseline” results are for a trivial phonetic matching method: find all strings with at most one character – an insertion, deletion, or replacement – different from the query. Performance assessment(cont.)

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 27 A particular problem of best agrep is the tiny number of correct answers returned – less than one per query – but we stress that agrep was not designed for phonetic matching. Performance assessment(cont.)

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 28 An interesting discovery is that even the most successful of the methods fetch rather different sets of answers, sometimes almost without overlap. As for information retrieval, it seems, two methods can perform well without finding the same answers. Performance assessment(cont.)

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 29 Phonetic matching has strong parallels with information retrieval. Matching techniques fetch a ranked list of matches in which each entry has weight attached to it; this weight is the likelihood that the entry is a good match. Combining the ranked lists produced by different retrieval mechanisms can improve performance. Combination of evidence

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 30 The “(none)” lines are the results of running the methods individually. The best performance of all is given by the combination of Phonix+ and the q-gram method, neither of which works particularly well alone. Combination of evidence(cont.)

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 31 More sophisticated techniques for combination could be used: Weighting the ranks from the different techniques. Combining more than two methods. That combination of evidence is successful in this context. Combination of evidence(cont.)

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 32 Conclusions Two of our proposals – the Ipadist and Editex methods – do indeed lead to improved performance, whereas the third – tapering – was not successful. We showed that combination of evidence, which has been successfully applied to information retrieval, consistently improves performance. Our new methods are substantially more effective than existing methods such as edit distances, and that combination of evidence is as valuable in this domain as it is in information retrieval.

Intelligent Database Systems Lab N.Y.U.S.T. I. M. 33 Personal Opinion The concept in this paper may use in our research, but I haven’t have a clear idea to implement it.I need more time to think…think…