1. Selecting Relevant Documents Assume: –we already have a corpus of documents defined. –goal is to return a subset of those documents. –Individual documents have been separated into individual files Remaining components must parse, index, find, and rank documents. Traditional approach is based on the words in the documents

2. Extracting Lexical Features Process a string of characters –assemble characters into tokens –choose tokens to index In place (problem for www) Standard lexical analysis problem Lexical Analyser Generator, such as lex,

3. Lexical Analyser Basic idea is a finite state machine Triples of input state, transition token, output state Must be very efficient; gets used a LOT 0 1 2 blank A-Z blank, EOF

4. Design Issues for Lexical Analyser Punctuation –treat as whitespace? –treat as characters? –treat specially? Case –fold? Digits –assemble into numbers? –treat as characters? –treat as punctuation?

5. Lexical Analyser Output of lexical analyser is a string of tokens Remaining operations are all on these tokens We have already thrown away some information; makes more efficient, but limits the power of our search

6. Stemming Additional processing at the token level Turn words into a canonical form: –“cars” into “car” –“children” into “child” –“walked” into “walk” Decreases the total number of different tokens to be processed Decreases the precision of a search, but increases its recall

7. Stemming -- How? Plurals to singulars (eg, children to child) Verbs to infinitives (eg, talked to talk) Clearly non-trivial in English! Typical stemmers use a context-sensitive transformation grammar: –(.*)SSES -> /1SS 50-250 rules are typical

8. Noise Words (Stop Words) Function words that contribute little or nothing to meaning Very frequent words –If a word occurs in every document, it is not useful in choosing among documents –However, need to be careful, because this is corpus-dependent Often implemented as a discrete list (stop.wrd on CD)

9. Example Corpora We are assuming a fixed corpus. Text uses two sample corpora: –AI Abstracts –Email. Anyone’s email. Textual fields, structured attributes Textual: free, unformatted, no meta- information Structured: additional information beyond the content

10. Structured Atributes for AI Theses Thesis # Author Year University Advisor Language

11. Textual Fields for AIT Abstract –Reasonably complete standard academic English capturing the basic meaning of document Title –Short, formalized, captures most critical part of meaning –(proxy for abstract)

12. Indexing We have a tokenized, stemmed sequence of words Next step is to parse document, extracting index terms –Assume that each token is a word and we don’t want to recognize any more complex structures than single words. When all documents are processed, create index

13. Basic Indexing Algorithm For each document in the corpus –get the next token –save the posting in a list doc ID, frequency For each token found in the corpus –calculate #doc, total frequency –sort by frequency (p 53-54)

14. Fine Points Dynamic Corpora Higher-resolution data (eg, char position) Giving extra weight to proxy text (typically by doubling or tripling frequency count) Document-type-specific processing –In HTML, want to ignore tages –In email, maybe want to ignore quoted material

15. Choosing Keyword Don’t necessarily want to index on every word –Takes more space for index –Takes more processing time –May not improve our resolving power How do we choose keywords? –Manually –Statistically Exhaustivity vs specificity

16. Manually Choosing Keywords Unconstrained vocabulary: allow creator of document to choose whatever he/she wants –“best” match –captures new terms easily –easiest for person choosing keywords Constrained vocabulary: hand-crafted ontologies –can include hierarchical and other relations –more consistent –easier for searching; possible “magic bullet” search

17. Examples of Constrained Vocabularies ACM headings Medline Subject Headings

18. Automated Vocabulary Selection Frequency: Zipf’s Law –Within one corpus, words with middle frequencies are typically “best” Document-oriented representation bias: lots of keywords/document Query-Oriented representation bias: only the “most typical” words. Assumes that we are comparing across documents.

19. Choosing Keywords “Best” depends on actual use; if a word only occurs in one document, may be very good for retrieving that document; not, however, very effective overall. Words which have no resolving power within a corpus may be best choices across corpora Not very important for web searching; will be more relevant for text mining.

20. Keyword Choice for WWW We don’t have a fixed corpus of documents New terms appear fairly regularly, and are likely to be common search terms Queries that people want to make are wide- ranging and unpredictable Therefore: can’t limit keywords, except possibly to eliminate stop words. Even stop words are language-dependent. So determine language first.