ISP 433/633 Week 10 Vocabulary Problem & Latent Semantic Indexing Partly based on G.Furnas SI503 slides

Synonymy Same meaning, different words Access, retrieval, look-up

Polysemy Same word, different meaning E.g. bank –River side –Financial institution

Exercise 1 Get out a piece of paper List examplars (members) of category for 30 sec Category is...

Flowers

Extended Free Recall is not Easy Brain is NOT built that way! Consequences: Trouble for generating synonyms for query terms Recall problem or Precision problem?

Exercise 2 On a piece of paper, write the name you would give to a Web site that tells about interesting activities occurring in Albany area –E.g. this site would tell you what is interesting to do on Friday or Saturday night –Make the site name 20 characters or less.

Lexical Variability Very low probability of match in almost every circumstance you examine (p = ) Severe consequences for lexically based access to information or functionality –Performance problems with lexically based IR I.e. querying Precision problem or Recall problem??? Note: The generation problem (a few slides back) contributes to you underestimating the lexical variability problem - you can’t generate many of the alternatives, so you think the variability is lower than it is.

Solutions for Lexical Variability Problem Direct Manipulation –Windows, Icons, Menus, Pointers (WIMP) –Recognition rather than recall –But limited number of items it can work for before navigation gets to be an issue Controlled Vocabulary –Adaptive burden (learning) on users Adaptive Indexing –Adaptive burden on the system Semantic Indexing –E.g., NLP-based retrieval, Latent Semantic Indexing

Adaptive Index An index that Learns from its mistakes Adds links for words it used to miss on Orders results by popularity for the given query Comments: Learns about the most needed words first In the natural context of their use Success requires a sufficient density of usage across the population...

Recall Vector Space Model D 1 = “computer information retrieval” D 2 = “computer retrieval” Q 1 = “information, retrieval” computer information retrieval D1=(1, 1, 1)Q1=(0, 1, 1) D2=(1, 0, 1)

Problem with Term Space Indexing terms contains only a fraction of terms that users may use in the query –Synonymy –Hard for user to come up with alternative terms –Indexing terms may quite different from user’s terms High lexical variability –Precision problem or recall problem?

Problem with Term Space (cont.) Terms are associated with unrelated documents –Polysemy –One solution is Controlled Vocabulary Bank -> financial institution Expensive and restrictive –Another solution is to add more terms in Boolean query Bank AND finance Hard to come up with more terms Terms may not in the index

Problem with Term Space (cont.) Terms are considered independent from each other (Orthogonal term dimensions) –Not the case –Too many dimensions

Semantic Space Space of meanings Both terms and documents are data points Reduced number of dimensions Avoid Synonymy problem Avoid Polysemy problem Meaning 2 Meaning 1 Meaning 3 T1=(1, 1, 1)Q1=(0, 1, 1) D2=(1, 0, 1)

How? Meanings are hidden (latent semantics) Take advantage of dependency among terms –Occurrence of some patterns of words gives clue as to the likely occurrence of others “Bank, mortgage” co-present -> occurrence of “Finance” –These correlated words are close to each other in semantic space –Semantically unrelated words are far away “River” will be far away, meaning of “Bank” will be quite clear

Original Term Document Matrix

Latent Semantic Indexing (LSI) Transform term-document matrix To get some orthogonal factors –Using SVD –Uncover underlying independent meanings Map terms, query and documents to this factor space Do rest as if it is a vector space model –Compute similarity between query and document

Semantic Space

Results

Administrivia Volunteer to pick up, distribute, collect and return teaching evaluation forms?