What is missing? Reasons that ideal effectiveness hard to achieve: 1. Users’ inability to describe queries precisely. 2. Document representation loses information. 3. Same term may have multiple meanings and different terms may have similar meanings. 4. Similarity function used not be good enough. 5. Importance/weight of a term in representing a document and query may be inaccurate.

Some improvements n Query expansion techniques (for 1) u relevance feedback F Vector model F Probabilistic model u co-occurrence analysis (local and global thesauri) n Improving the quality of terms [(2), (3) and (5).] u Latent Semantic Indexing u Phrase-detection

Insight through Principal Components Analysis KL Transform Neural Networks Dimensionality Reduction

Latent Semantic Indexing n Classic IR might lead to poor retrieval due to: u unrelated documents might be included in the answer set u relevant documents that do not contain at least one index term are not retrieved u Reasoning: retrieval based on index terms is vague and noisy n The user information need is more related to concepts and ideas than to index terms n A document that shares concepts with another document known to be relevant might be of interest

Latent Semantic Indexing n Creates modified vector space n Captures transitive co-occurrence information u If docs A & B don’t share any words, with each other, but both share lots of words with doc C, then A & B will be considered similar u Handles polysemy (adam’s apple) & synonymy n Simulates query expansion and document clustering (sort of)

A motivating example n Suppose we have keywords u Car, automobile, driver, elephant n We want queries on car to also get docs about drivers, but not about elephants u Need to realize that driver and car are related while elephant is not n When you scrunch down the dimensions, small differences get glossed over, and you get the desired behavior

Latent Semantic Indexing n Definitions u Let t be the total number of index terms u Let N be the number of documents u Let (Mij) be a term-document matrix with t rows and N columns u To each element of this matrix is assigned a weight wij associated with the pair [ki,dj] u The weight wij can be based on a tf-idf weighting scheme

Everything You Always Wanted to Know About LSI, and More Singular Value Decomposition (SVD): Convert term-document matrix into 3matrices U, D and V  Reduce Dimensionality: Throw out low-order rows and columns Recreate Matrix: Multiply to produce approximate term- document matrix. Use new matrix to process queries

Latent Semantic Indexing n The matrix (Mij) can be decomposed into 3 matrices (singular value decomposition) as follows: u (Mij) = (U) (S) (V) t F (U) is the matrix of eigenvectors derived from (M)(M) t F (V) t is the matrix of eigenvectors derived from (M) t (M) F (S) is an r x r diagonal matrix of singular values r = min(t,N) that is, the rank of (Mij) Singular values are the positive square roots of the eigen values of (M)(M) t (also (M) t (M)) For the special case where M is a square matrix, S is the diagonal eigen value matrix, and K and D are eigen vector matrices K and S are orthogonal matrices

Latent Semantic Indexing n The key idea is to map documents and queries into a lower dimensional space (i.e., composed of higher level concepts which are in fewer number than the index terms) n Retrieval in this reduced concept space might be superior to retrieval in the space of index terms

Latent Semantic Indexing n In the matrix (S), select only the k largest singular values n Keep the corresponding columns in (U) and (V) t n The resultant matrix is called (M) k and is given by u (M) k = (U) k (S) k (D) t k u where k, k < r, is the dimensionality of the concept space n The parameter k should be u large enough to allow fitting the characteristics of the data u small enough to filter out the non-relevant representational details The classic over-fitting issue

Computing an Example n Let (Mij) be given by the matrix u Compute the matrices (K), (S), and (D) t

Example term ch2ch3 ch4 ch5 ch6 ch7 ch8 ch9 controllability observability realization feedback controller observer transfer function polynomial matrices U (9x7) = S (7x7) = V (7x8) = This happens to be a rank-7 matrix -so only 7 dimensions required Singular values = Sqrt of Eigen values of AA T T

U2 (9x2) = S2 (2x2) = V2 (8x2) = U (9x7) = S (7x7) = V (7x8) = U2*S2*V2 will be a 9x8 matrix That approximates original matrix T

K=2 K=6 One component ignored 5 components ignored U6S6V6TU6S6V6T U2S2V2TU2S2V2T USV T U4S4V4TU4S4V4T K=4 =U 7 S 7 V 7 T 3 components ignored What should be the value of k?

Coordinate transformation inherent in LSI M = U S V T Mapping of keywords into LSI space is given by US For k=2, the mapping is: controllability observability realization feedback controller observer Transfer function polynomial matrices LSx LSy controllability controller LSIx LSIy LSIx Mapping of a doc d=[w1….wk] into LSI space is given by dUS -1 The base-keywords of The doc are first mapped To LSI keywords and Then differentially weighted By S -1 ch3

Medline data from Berry’s paper

Querying To query for feedback controller, the query vector would be q = [ ]' (' indicates transpose), since feedback and controller are the 4-th and 5-th terms in the index, and no other terms are selected. Let q be the query vector. Then the document-space vector corresponding to q is given by: q'*U2*inv(S2) = Dq For the feedback controller query vector, the result is: Dq = To find the best document match, we compare the Dq vector against all the document vectors in the 2- dimensional V2 space. The document vector that is nearest in direction to Dq is the best match. The cosine values for the eight document vectors and the query vector are: Centroid of the terms In the query (with scaling)

Within.40 threshold K is the number of singular values used

Latent Ranking (a la text) n The user query can be modelled as a pseudo- document in the original (M) matrix n Assume the query is modelled as the document numbered 0 in the (M) matrix n The matrix (M) t (M) s quantifies the relantionship between any two documents in the reduced concept space n The first row of this matrix provides the rank of all the documents with regard to the user query (represented as the document numbered 0) s Inefficient way

Folding docs -Convert new documents into LSI space using the dUS -1 method Folding terms - find the vectors for new terms as weighted sum of the docs in which they occur Practical Issues: How often do you re-compute SVD when terms or documents are added to the collection? --Folding is a cheaper solution but will worsen quality over time

Summary of LSI n Latent semantic indexing provides an interesting conceptualization of the IR problem n No stemming needed, spelling errors tolerated n Can do true conceptual retrieval u Retrieval of documents that do not share any keywords with the query!

The best fit for the feedback controller query vector is with the second document, which is Chapter 3. The sixth document, or Chapter 7, is also a good match. A query for feedback realization yields the query vector Dq = and cosine values The best matches for feedback realization are Chapters 4 and 6.