Data Structure

Two segments of data structure –Storage –Retrieval

Item normalization Document File Creation Document Manager Document Search Manager Original document file Proc. Token search file

–Stemming –Inverted file system –N-gram –PAT trees and arrays –Signature –Hypertext –Stemming –Inverted file system –N-gram –PAT trees and arrays –Signature –Hypertext

–Inverted file system most common data structure Minimizes secondary storage access –When using multiple search terms Document, inversion list / posting files, dictionary Storing an inversion of documents –Inverted file system most common data structure Minimizes secondary storage access –When using multiple search terms Document, inversion list / posting files, dictionary Storing an inversion of documents

–N-gram Fixed length consecutive series of ‘n’ characters Algorithmically based upon a fixed number of characters Searchable data structure transformed into a overlapping n-grams to create the searchable database (fig. 4.7) Does not involve semantics - concepts –N-gram Fixed length consecutive series of ‘n’ characters Algorithmically based upon a fixed number of characters Searchable data structure transformed into a overlapping n-grams to create the searchable database (fig. 4.7) Does not involve semantics - concepts

–N-gram Symbol # to represent inter-word symbol (fig. 4.7) –Blank, period, semi-colon, colon etc. Word fragments Uses –Spelling error detection and correction (fig. 4.8) –Text compression Ignores words and treat input as a continuous data –N-gram Symbol # to represent inter-word symbol (fig. 4.7) –Blank, period, semi-colon, colon etc. Word fragments Uses –Spelling error detection and correction (fig. 4.8) –Text compression Ignores words and treat input as a continuous data

–N-gram False hits can occur when without # The longer n-gram, the less likely is the error Problems –Increased size of inversion lists –No semantic meaning and concept relationship Can achieve high recall –N-gram False hits can occur when without # The longer n-gram, the less likely is the error Problems –Increased size of inversion lists –No semantic meaning and concept relationship Can achieve high recall

–PAT trees PATRICIA trees –Practical algorithm to retrieve information coded in alphanumerics Each position in the input string is the anchor point for a sub-string that starts at that point and includes all new text up to the end of the input Substrings are termed as sistrings (Figure ) Best for string searching but not widely used commercially –PAT trees PATRICIA trees –Practical algorithm to retrieve information coded in alphanumerics Each position in the input string is the anchor point for a sub-string that starts at that point and includes all new text up to the end of the input Substrings are termed as sistrings (Figure ) Best for string searching but not widely used commercially

Signature To provide a fast test to eliminate the majority of items that are not related to a query A linear scan of the compressed version of items Coding based upon words in the item Words are mapped onto a word signature –A fixed length code with a fixed number of bits set to 1 –Set to 1 determined by the hash function –ORed to create the signature of an item –Fig 4.13 Words in the query are mapped to the word signature Search via template matching Signature To provide a fast test to eliminate the majority of items that are not related to a query A linear scan of the compressed version of items Coding based upon words in the item Words are mapped onto a word signature –A fixed length code with a fixed number of bits set to 1 –Set to 1 determined by the hash function –ORed to create the signature of an item –Fig 4.13 Words in the query are mapped to the word signature Search via template matching

Signature Longer code length reduces probability of collision hashing the same words Fewer bits per code reduce the effect of a code word pattern present in the final signature block while the word is actually not in the item Signature Longer code length reduces probability of collision hashing the same words Fewer bits per code reduce the effect of a code word pattern present in the final signature block while the word is actually not in the item

Hypertext (HTML and XML) Allow one item to reference another item via an embedded pointer A node (separate item) Link (reference pointer) –Similar or different data type than the original Navigates –Managing the loosely structured information Issue –Linkage integrity (no update of the removed or deleted items) Hypertext (HTML and XML) Allow one item to reference another item via an embedded pointer A node (separate item) Link (reference pointer) –Similar or different data type than the original Navigates –Managing the loosely structured information Issue –Linkage integrity (no update of the removed or deleted items)

Hypertext (HTML and XML) Dynamic HTML –Combination of the latest HTML tags and options, style sheets and programming –Creation of animated Web pages and responsive to user interaction Dynamic HTML Object Model –Object-oriented view of Web pages and its elements –Cascading style sheets –Programming addressing the page elements with dynamic fonts Hypertext (HTML and XML) Dynamic HTML –Combination of the latest HTML tags and options, style sheets and programming –Creation of animated Web pages and responsive to user interaction Dynamic HTML Object Model –Object-oriented view of Web pages and its elements –Cascading style sheets –Programming addressing the page elements with dynamic fonts

DOCUMENTSDICTIONARYINVERSION LISTS Doc #1, computer,bit (2)bit - 1, 3 bit, byte DOCUMENTSDICTIONARYINVERSION LISTS Doc #1, computer,bit (2)bit - 1, 3 bit, byte

Inversion list –Weights –Words with special characteristics e.g. date Searching –Locate the inversion lists –Apply appropriate logic between lists –Final hit of the list of items is the result Inversion list –Weights –Words with special characteristics e.g. date Searching –Locate the inversion lists –Apply appropriate logic between lists –Final hit of the list of items is the result

–B trees e.g. of order m A root node with between 2 and 2m keys All other internal nodes have between m and 2m keys All keys are kept in order from smaller to larger All leaves are at the same level or differ by at most one level –B trees e.g. of order m A root node with between 2 and 2m keys All other internal nodes have between m and 2m keys All keys are kept in order from smaller to larger All leaves are at the same level or differ by at most one level

–Inversion list structures Provide optimum performance in searching large databases Minimization of data flow Involve only directly related data Good for storing concepts and their relationship Each list for representing a concept A concordance of all of the items containing the concepts Location of the concepts Do not solely work for natural language processing –Inversion list structures Provide optimum performance in searching large databases Minimization of data flow Involve only directly related data Good for storing concepts and their relationship Each list for representing a concept A concordance of all of the items containing the concepts Location of the concepts Do not solely work for natural language processing

Stemming algorithm Goal: to improve performance and require less system resources by reducing number of unique words that a system has to contain Currently reviewed for potential improvements of recall and associated decline in precision Trade-off: increased overhead for processing token vs. reduced search time overhead for processing query terms with trailing ‘don’t cares’ for the inclusion of all the variants Creates a large index for the stem vs. term masking (ORing) Stemming algorithm Goal: to improve performance and require less system resources by reducing number of unique words that a system has to contain Currently reviewed for potential improvements of recall and associated decline in precision Trade-off: increased overhead for processing token vs. reduced search time overhead for processing query terms with trailing ‘don’t cares’ for the inclusion of all the variants Creates a large index for the stem vs. term masking (ORing)

Stemming algorithm Conflation: refer to mapping multiple morphological variants to a single representation (stem) Stem: carries the meaning of the concept associated with the word Affixes (endings) introduce subtle modifications to the concept or are used for syntactical purposes Languages: grammars defining usage and evolve on human usage Existence of exceptions and non-consistent variants – thus requires exception look-up tables beside normal reduction rules Stemming algorithm Conflation: refer to mapping multiple morphological variants to a single representation (stem) Stem: carries the meaning of the concept associated with the word Affixes (endings) introduce subtle modifications to the concept or are used for syntactical purposes Languages: grammars defining usage and evolve on human usage Existence of exceptions and non-consistent variants – thus requires exception look-up tables beside normal reduction rules

Stemming algorithm Compression – savings in storage and processing? Savings – dictionary, requires weighted positional information in stem inversion list and un-stemmed inversion list Size of inversion list Compression does not significantly reduce storage requirements – small vs. large-sized collection Stemming algorithm Compression – savings in storage and processing? Savings – dictionary, requires weighted positional information in stem inversion list and un-stemmed inversion list Size of inversion list Compression does not significantly reduce storage requirements – small vs. large-sized collection

Stemming algorithm Improve recall? As long as a semantically consistent stem can be identified for a set of words – generalization process of stemming Improve precision? Only if the expansion guarantees every item retrieved by the expansion is relevant Stemming algorithm Improve recall? As long as a semantically consistent stem can be identified for a set of words – generalization process of stemming Improve precision? Only if the expansion guarantees every item retrieved by the expansion is relevant

Stemming algorithm System must recognize the word before stemming Proper names and acronyms – no stemming applied since no common core concept Problems for natural language processing system – loss of information needed for aggregate levels of processing e.g. tenses needed to determine a particular concept Time – important in natural language processing Stemming algorithm System must recognize the word before stemming Proper names and acronyms – no stemming applied since no common core concept Problems for natural language processing system – loss of information needed for aggregate levels of processing e.g. tenses needed to determine a particular concept Time – important in natural language processing

Stemming algorithm Removal of suffixes and prefixes Table look-up – requires a large data structure (e.g. RetrievalWare due to large thesaurus/concept network) Successor stemming – determine prefix overlap as the length of a stem is increased e.g. tenses needed to determine a particular concept Time – important in natural language processing Stemming algorithm Removal of suffixes and prefixes Table look-up – requires a large data structure (e.g. RetrievalWare due to large thesaurus/concept network) Successor stemming – determine prefix overlap as the length of a stem is increased e.g. tenses needed to determine a particular concept Time – important in natural language processing

Stemming algorithm Porter stemming algorithm Dictionary look-up stemmers Successor stemmers Stemming algorithm Porter stemming algorithm Dictionary look-up stemmers Successor stemmers

Porter Stemming algorithm Based upon a set of conditions of the stem, suffix and prefix and associated actions given the condition Measure (m) of a stem is a function of sequences of vowels followed by a consonant. If v is a sequence of vowels and C is a sequence of consonants, then m is: C(VC) m V C and V – optional and m is the number VC repeats *, *v*, *d, *o Porter Stemming algorithm Based upon a set of conditions of the stem, suffix and prefix and associated actions given the condition Measure (m) of a stem is a function of sequences of vowels followed by a consonant. If v is a sequence of vowels and C is a sequence of consonants, then m is: C(VC) m V C and V – optional and m is the number VC repeats *, *v*, *d, *o

Dictionary Look-Up Stemmers Simple stemming rules – fewest exceptions (plural) Original term or stemmed version – looked-up in dictionary and replaced by the stem that best represents it e.g. Kstem – a morphological analyzer conflating word variants to a root form and avoid collapsing words with different meanings into the same root Six major data files: dictionary of words, supplemental list of words, exception list for words that should retain an “e” at the end, direct conflation, country nationality Dictionary Look-Up Stemmers Simple stemming rules – fewest exceptions (plural) Original term or stemmed version – looked-up in dictionary and replaced by the stem that best represents it e.g. Kstem – a morphological analyzer conflating word variants to a root form and avoid collapsing words with different meanings into the same root Six major data files: dictionary of words, supplemental list of words, exception list for words that should retain an “e” at the end, direct conflation, country nationality

Successor Stemmers Based upon length of the prefixes that optimally stem expansions of additional suffixes Based upon the analogy in structural linguistics that investigated word and morpheme boundaries based upon the distribution of phonemes e.g. bag, barn, bring, both, box, bottle (Fig. 4.2) Successor Stemmers Based upon length of the prefixes that optimally stem expansions of additional suffixes Based upon the analogy in structural linguistics that investigated word and morpheme boundaries based upon the distribution of phonemes e.g. bag, barn, bring, both, box, bottle (Fig. 4.2)

Successor Stemmers Methods: cut-off, peak and plateau, complete word method, and entropy method Cut-off method: cut-off value to define stem length, value varies for each possible set of words Peak and plateau: a segment break made after a character whose successor variety exceeds that of the character immediately preceding it and the character immediately following it (not needing cut-off) Complete word method: break on boundaries of complete words (not needing cut-off) Entropy method: uses the distribution of successor variety letters Figure 4.3 Successor Stemmers Methods: cut-off, peak and plateau, complete word method, and entropy method Cut-off method: cut-off value to define stem length, value varies for each possible set of words Peak and plateau: a segment break made after a character whose successor variety exceeds that of the character immediately preceding it and the character immediately following it (not needing cut-off) Complete word method: break on boundaries of complete words (not needing cut-off) Entropy method: uses the distribution of successor variety letters Figure 4.3

Stemming Algorithm Stemming affects recall (positive) in one study, not proven in many studies, but reduce precision – minimized via ranking items, categorization of terms and selective exclusion of some terms from stemming Stemming is dependent upon the nature of the vocabulary Performance measure: Error rate relative to truncation (distance from the origin to the coordinate of the stemmer being evaluated vs. the distance from the origin to the worst case intersection of the line generated by pure truncation), Fig. 4.4 Measure the ability to partition terms semantically and morphologically related to each other into “concept groups” Understemming index – concept groups with multiple stem Overstemming index – same stem is found in multiple groups Stemming Algorithm Stemming affects recall (positive) in one study, not proven in many studies, but reduce precision – minimized via ranking items, categorization of terms and selective exclusion of some terms from stemming Stemming is dependent upon the nature of the vocabulary Performance measure: Error rate relative to truncation (distance from the origin to the coordinate of the stemmer being evaluated vs. the distance from the origin to the worst case intersection of the line generated by pure truncation), Fig. 4.4 Measure the ability to partition terms semantically and morphologically related to each other into “concept groups” Understemming index – concept groups with multiple stem Overstemming index – same stem is found in multiple groups