# Växjö: 23. Jan -04Evaluation of Vector Space...1 Evaluation of Vector Space Models Obtained by Latent Semantic Indexing Leif Grönqvist

## Presentation on theme: "Växjö: 23. Jan -04Evaluation of Vector Space...1 Evaluation of Vector Space Models Obtained by Latent Semantic Indexing Leif Grönqvist"— Presentation transcript:

Växjö: 23. Jan -04Evaluation of Vector Space...1 Evaluation of Vector Space Models Obtained by Latent Semantic Indexing Leif Grönqvist (leifg@ling.gu.se) Växjö University (Mathematics and Systems Engineering) GSLT (Graduate School of Language Technology) Göteborg University (Department of Linguistics)

Växjö: 23. Jan -04Evaluation of Vector Space...2 Outline of the talk  Vector space models in IR (reminder since last seminar) The traditional model Latent semantic indexing (LSI)  Singular value decomposition (SVD)  Evaluation Why How & Data sources

Växjö: 23. Jan -04Evaluation of Vector Space...3 The traditional vector model  One dimension for each index term  A document is a vector in a very high dimensional space  The similarity between a document and a query is:  Gives us a degree of similarity instead of yes/no as for basic keyword search

Växjö: 23. Jan -04Evaluation of Vector Space...4 The traditional vector model, cont.  Assumption used: all terms are unrelated  Could be fixed partially using different weights for each term  Still, we have a lot more dimensions than we want How should we decide the index terms? Similarity between terms are always 0 Very similar documents may have sim0 if they:  use a different vocabulary  don’t use the index terms

Växjö: 23. Jan -04Evaluation of Vector Space...5 Latent semantic indexing (LSI)  Similar to factor analysis  Number of dimensions can be chosen as we like  We make some kind of projection from a vector space with all terms to the smaller dimensionality  Each dimension is a mix of terms  Impossible to know the meaning of the dimension

Växjö: 23. Jan -04Evaluation of Vector Space...6 LSI, cont.  Distance between vectors is cosine just as before  Meaningful to calculate distance between all terms and/or documents  How can we do the projection?  There are some ways: Singular value decomposition (SVD) Random indexing Neural nets, factor analysis, etc.

Växjö: 23. Jan -04Evaluation of Vector Space...7 Why SVD?  I prefer SVD since:  Michael W Berry 1992: “… This important result indicates that A k is the best k-rank approxi- mation (in a least squares sense) to the matrix A.  Leif 2003: What Berry says is that SVD gives the best projection from n to k dimensions, that is the projection that keep distances in the best possible way.

Växjö: 23. Jan -04Evaluation of Vector Space...8 A small example input to SVD

Växjö: 23. Jan -04Evaluation of Vector Space...9 What SVD gives us X=T 0 S 0 D 0 : X, T 0, S 0, D 0 are matrices

Växjö: 23. Jan -04Evaluation of Vector Space...10 Using the SVD  The matrices make it easy to project term and document vectors into a m-dimensional space (m ≤ min (terms, docs)) using ordinary linear algebra  We can select m easily just by using as many rows/columns of T 0, S 0, D 0 as we want  It is possible to calculate a new (approximated) X – it will still be a t x d matrix

Växjö: 23. Jan -04Evaluation of Vector Space...11 Some applications  Automatic generation of a domain specific thesaurus  Keyword extraction from documents  Find sets of similar documents in a collection  Find documents related to a given document or a set of terms

Växjö: 23. Jan -04Evaluation of Vector Space...12 An example based on 50 000 newspaper articles stefan edberg edberg0.918 cincinnatis0.887 edbergs0.883 världsfemman0.883 stefans0.883 tennisspelarna0.863 stefan0.861 turneringsseger0.859 queensturneringen 0.858 växjöspelaren0.852 grästurnering0.847 bengt johansson johansson0.852 johanssons0.704 bengt0.678 centerledare0.674 miljöcentern0.667 landsbygdscentern0.667 implikationer0.645 ickesocialistisk0.643 centerledaren0.627 regeringsalternativet 0.620 vagare0.616

Växjö: 23. Jan -04Evaluation of Vector Space...13 Evaluation  We need evaluation metrics to be able to improve the model!  How can we evaluate millions of vectors? “similar terms have vectors with low cosine” What is similar?  Seems impossible to evaluate the model objectively…  Possible solution: look at specific applications! They may be much easier to evaluate

Växjö: 23. Jan -04Evaluation of Vector Space...14 Applications using the model  Vector models may be evaluated using: A typical IR test suite of queries, documents, and relevance information Texts with lists of manually selected keywords (multiword units included) The Test of English as a Foreign Language (TOEFL), which tests the ability of selecting synonyms from a set of alternatives  Still subjectivity, but the more the vector model improves these applications the better it is!  Let’s look in detail at the first application

Växjö: 23. Jan -04Evaluation of Vector Space...15 An IR testbed  There are such testbeds for English, but Swedish has other problems Very different from English Compounds without spaces “New” letters (åäö) Complex morphology Other stop words …

Växjö: 23. Jan -04Evaluation of Vector Space...16 A new Swedish test collection  A group in Borås is building it Per Ahlgren Johan Eklund Leif Grönqvist  It will contain Documents Topics Relevance judgments

Växjö: 23. Jan -04Evaluation of Vector Space...17 Document collection  Newspaper articles from GP and HD  161 000 articles, 40 MTokens  Good to have more than one newspaper: Same content, different author (not always)  10% of my newspaper article collection  Copyright is a problem

Växjö: 23. Jan -04Evaluation of Vector Space...18 Topics  Borrowed from CLEF  52/90, but not the most difficult  Examples: Filmer av bröderna Kaurismäki.  Description: Sök efter information om filmer som regisserats av någon av de båda bröderna Aki och Mika Kaurismäki.  Narrative: Relevanta dokument namnger en eller flera titlar på filmer som regisserats av Aki eller Mika Kaurismäki. Finlands första EU-kommissionär  Description: Vem utsågs att vara den första EU- kommissionären för Finland i Europeiska unionen?  Narrative: Ange namnet på Finlands första EU- kommissionär. Relevanta dokument kan också nämna sakområdena för den nya kommissionärens uppdrag.

Växjö: 23. Jan -04Evaluation of Vector Space...19 Relevance judgments  Only a subset for each topic Selected by earlier experiments Similar approach to TREC and CLEF  100 documents for 5 strategies: 100  N  500 Important to include relevant and irrelevant documents  A scale of relevance proposed by Sormonen: Irrelevant (0)  Marginally relevant (1)  Fairly relevant (2)  Highly relevant (3)  Manually annotated

Växjö: 23. Jan -04Evaluation of Vector Space...20 Statistics  Some difficult topics got very few relevant documents

Växjö: 23. Jan -04Evaluation of Vector Space...21 Statistics per relevance category

Växjö: 23. Jan -04Evaluation of Vector Space...22 Evaluation metrics  Recall & precision is problematic: Ranked lists – how much better is position 1 than pos 5 and 10? How long should the lists be? Relevance scale – how much better is “highly relevant” than “fairly relevant” What about the unknown documents not judged?  Too many unknown leads to a need of more manual judgments…

Växjö: 23. Jan -04Evaluation of Vector Space...23 The End!  Questions?

Download ppt "Växjö: 23. Jan -04Evaluation of Vector Space...1 Evaluation of Vector Space Models Obtained by Latent Semantic Indexing Leif Grönqvist"

Similar presentations