CpSc 881: Information Retrieval

2 Recall: Term-document matrix This matrix is the basis for computing the similarity between documents and queries. Today: Can we transform this matrix, so that we get a better measure of similarity between documents and queries? Anthony and Cleopatra Julius Caesar The Tempest HamletOthelloMacbeth anthony brutus caesar calpurnia cleopatra mercy worser

3 Latent semantic indexing: Overview  We will decompose the term-document matrix into a product of matrices.  The particular decomposition we’ll use: singular value decomposition (SVD).  SVD: C = U Σ V T (where C = term-document matrix)  We will then use the SVD to compute a new, improved term-document matrix C′.  We’ll get better similarity values out of C′ (compared to C).  Using SVD for this purpose is called latent semantic indexing or LSI.

4 Example of C = U Σ V T : The matrix C This is a standard term-document matrix. Actually, we use a non-weighted matrix here to simplify the example.

5 Example of C = U Σ V T : The matrix U One row perterm, one column per min(M,N) where M is the number of terms and N is the number of documents. This is an orthonormal matrix: (i) Row vectors have unit length. (ii) Any two distinct row vectors are orthogonal to each other. Think of the dimensions as “semantic” dimensions that capture distinct topics like politics, sports, economics. Each number u ij in the matrix indicates how strongly related term i is to the topic represented by semantic dimension j.

6 Example of C = U Σ V T : The matrix Σ This is a square, diagonal matrix of dimensionality min(M,N) × min(M,N). The diagonal consists of the singular values of C. The magnitude of the singular value measures the importance of the corresponding semantic dimension. We’ll make use of this by omitting unimportant dimensions.

7 7 Example of C = U Σ V T : The matrix V T One column per document, one row per min(M,N) where M is the number of terms and N is the number of documents. Again: This is an orthonormal matrix: (i) Column vectors have unit length. (ii) Any two distinct column vectors are orthogonal to each other. These are again the semantic dimensions from the term matrix U that capture distinct topics like politics, sports, economics. Each number v ij in the matrix indicates how strongly related document i is to the topic represented by semantic dimension j. 7

8 8 Example of C = U Σ V T : All four matrices 8

9 LSI: Summary  We’ve decomposed the term-document matrix C into a product of three matrices.  The term matrix U – consists of one (row) vector for each term  The document matrix V T – consists of one (column) vector for each document  The singular value matrix Σ – diagonal matrix with singular values, reflecting importance of each dimension  Next: Why are we doing this?

10 How we use the SVD in LSI  Key property: Each singular value tells us how important its dimension is.  By setting less important dimensions to zero, we keep the important information, but get rid of the “details”.  These details may  be noise – in that case, reduced LSI is a better representation because it is less noisy  make things dissimilar that should be similar – again reduced LSI is a better representation because it represents similarity better.  Analogy for “fewer details is better”  Image of a bright red flower  Image of a black and white flower  Omitting color makes is easier to see similarity

11 Reducing the dimensionality to 2 Actually, we only zero out singular values in Σ. This has the effect of setting the corresponding dimensions in U and V T to zero when computing the product C = U Σ V T.

12 Reducing the dimensionality to 2

13 Recall unreduced decomposition C=U Σ V T

14 Original matrix C vs. reduced C 2 = U Σ 2 V T We can view C 2 as a two-dimensional representation of the matrix. We have performed a dimensionality reduction to two dimensions.

15 Why the reduced matrix is “better” Similarity of d2 and d3 in the original space: 0. Similarity of d2 und d3 in the reduced space: 0.52 * * * * * ≈ 0.52

16 Why the reduced matrix is “better” “boat” and “ship” are semantically similar. The “reduced” similarity measure reflects this. What property of the SVD reduction is responsible for improved similarity?

17 SVD - Interpretation ‘documents’, ‘terms’ and ‘concepts’: U: document-to-concept similarity matrix V: term-to-concept similarity matrix  : its diagonal elements: ‘strength’ of each concept Projection: best axis to project on: (‘best’ = min sum of squares of projection errors)

18 SVD - Example A = U  V T - example: data inf. retrieval brain lung = CS MD xx

19 SVD - Example A = U  V T - example: data inf. retrieval brain lung = CS MD xx CS-concept MD-concept doc-to-concept similarity matrix

20 SVD - Example A = U  V T - example: data inf. retrieval brain lung = CS MD xx ‘strength’ of CS-concept

21 SVD - Example A = U  V T - example: data inf. retrieval brain lung = CS MD xx term-to-concept similarity matrix CS-concept

22 SVD – Dimensionality reduction Q: how exactly is dim. reduction done? A: set the smallest singular values to zero: = xx

23 SVD - Dimensionality reduction ~ xx

24 SVD - Dimensionality reduction ~

25 LSI (latent semantic indexing) Q1: How to do queries with LSI? A: map query vectors into ‘concept space’ – how? data inf. retrieval brain lung = CS MD xx

26 LSI (latent semantic indexing) Q: How to do queries with LSI? A: map query vectors into ‘concept space’ – how? data inf. retrieval brain lung q=q= term1 term2 v1 q v2 A: inner product (cosine similarity) with each ‘concept’ vector v i

27 LSI (latent semantic indexing) compactly, we have: q concept = q V e.g.: data inf. retrieval brain lung q=q= term-to-concept similarities = CS-concept

28 Why we use LSI in information retrieval  LSI takes documents that are semantically similar (= talk about the same topics),... ... but are not similar in the vector space (because they use different words)... ... and re-represents them in a reduced vector space... ... in which they have higher similarity.  Thus, LSI addresses the problems of synonymy and semantic relatedness.  Standard vector space: Synonyms contribute nothing to document similarity.  Desired effect of LSI: Synonyms contribute strongly to document similarity.

29 How LSI addresses synonymy and semantic relatedness  The dimensionality reduction forces us to omit a lot of “detail”.  We have to map differents words (= different dimensions of the full space) to the same dimension in the reduced space.  The “cost” of mapping synonyms to the same dimension is much less than the cost of collapsing unrelated words.  SVD selects the “least costly” mapping (see below).  Thus, it will map synonyms to the same dimension.  But it will avoid doing that for unrelated words.

30 LSI: Comparison to other approaches  Recap: Relevance feedback and query expansion are used to increase recall in information retrieval – if query and documents have (in the extreme case) no terms in common.  LSI increases recall and hurts precision.  Thus, it addresses the same problems as (pseudo) relevance feedback and query expansion... ... and it has the same problems.

31 Implementation  Compute SVD of term-document matrix  Reduce the space and compute reduced document representations  Map the query into the reduced space  This follows from:  Compute similarity of q 2 with all reduced documents in V 2.  Output ranked list of documents as usual  Exercise: What is the fundamental problem with this approach?

32 Optimality  SVD is optimal in the following sense.  Keeping the k largest singular values and setting all others to zero gives you the optimal approximation of the original matrix C. Eckart-Young theorem  Optimal: no other matrix of the same rank (= with the same underlying dimensionality) approximates C better.  Measure of approximation is Frobenius norm:  So LSI uses the “best possible” matrix.  Caveat: There is only a tenuous relationship between the Frobenius norm and cosine similarity between documents.