1. NLP Workshop Eytan Ruppin http://www.cs.tau.ac.il/~ruppin ruppin@post.tau.ac.il Ben Sandbank http://www.cs.tau.ac.il/~sandban sandban@post.tau.ac.il

2. The problem of language acquisition The argument from the poverty of the stimulus

3. Projects description ADIOS (“Automatic DIstillation Of Structure”) – a grammar induction algorithm developed by Zach Solan. Projects will consist of two phases Implementation of the algorithm Some original work Applying ADIOS to new data Developing & implementing some extensions

4. Latent Semantic Analysis Assumes a high-dimensional “semantic space” Each word/topic is a point in this space Semantic similarity is a function of distance A discourse regarding a certain topic will utilize words from neighboring regions of semantic space

5. The problem of acquisition The learner is presented with sets of words each used in discourse of a single topic For example – the words contained in paragraphs in text The problem – figure out the true distances between the words in semantic space

6. A toy example

7. Bags of words

8. Measuring similarity Question – why not just measure correlations between word vectors? Rationale – words with similar meanings will tend to co-occur in the same contexts Answer – that just doesn’t work

9. Doesn’t it?

10. Oh. Why not, then? That two words rarely co-occur can be because – The words are truly unrelated Like carburetor/semantics The words are related but the sample is too small Many pairwise counts will be zero The words are near-perfect synonyms Like human/user in the toy example

11. The virtues of dimension reduction Proper dimension reduction helps overcome errors in measurement The three houses example Imposes global constraints that give rise to indirect effects Only true if we know the underlying dimensionality

12. Singular Value Decomposition Refers to the fact that every rectangular matrix X can be decomposed thusly – Where m = rank(X) (= min(w, c)) X Contexts Words = W wxcwxcwxmwxm S mxmmxm CTCT mxcmxc

13. Properties of the SVD W, C are orthonormal S is diagonal, with values usually arranged in descending order These are the ‘singular values’ Zeroing all but the r largest values of S will result in a matrix X’ which is the closest rank r matrix to X! X = W wxcwxc wxmwxm S mxmmxm CTCT mxcmxc

14. Toy example SVD

15. (9x9)

16. The original matrix again…

17. And the 2D result

19. Matrix preprocessing The co-occurrence matrix is subjected to a preprocessing transformation that improves LSA’s performance Define tf i,j – term frequency – the number of occurrences of word i in context j. df i – document frequency – the number of documents (context) containing word I

20. The transformation tf i,j  log(tf i,j +1)*log(N/df i ) The ‘log’ compresses the occurrence counts A document containing 3 counts of word i is more important, but not 3 times as important The second term (“Inverse Document Frequency”) is a measure of the word’s informativeness

21. Low dimensional representation Let S r be equal to S with all but the r largest values set to zero. Then WS r gives the r-dimensional representation of the word matrix (each row corresponds to one word) CS r gives the r-dimensional representation of the context matrix (each row corresponds to one column) X = W wxcwxc wxmwxm S mxmmxm CTCT mxcmxc

22. Measuring similarity Actually, similarity is measured by the cosine of the angle between the two words/contexts Equivalent to distance if the vectors are all normalized For a context, its angle is related to its topic; its length indicates how much it has to say about it

23. “Folding” new contexts We would like to be able to determine the low-dimensional representation of new contexts Let q w x1 be the new context, and W w x k the k-dimensional word matrix The k-dimensional embedding of the new context is given by – W w x k T q w x1

24. So how good is this thing? LSA was run on 4.6 million words taken from an online encyclopedia 30,473 articles (= contexts) 60,768 distinct words The LSA representation was tested on the synonym portion of the TOEFL The model got 64.4% of the answers correctly (52.5% corrected for guessing) As compared with an average result of 64.5% (52.7%) by foreign students applying to US colleges

25. Effects of dimensionality Without SVD, 16% of the answers were correct

26. The effects of expertise The performance of LSA was tested with regards to the amount of prior knowledge (i.e corpus size) on two dimensions – The number of occurrences of the tested word The number of contexts NOT containing the tested word A continuous discrimination measured was defined to assess LSA’s knowledge of a word -

27. The effects of expertise

28. Rate of learning The LSA was trained on a corpus the size of a seventh-grader history of reading Adding a single paragraph and retraining the LSA allowed to assess the rate of LSA learning

29. Rate of learning The effect of reading a paragraph on knowledge of words contained in it was a gain of ~0.05 words The effect on words not contained in it was ~0.15 words Given that a seventh-grader reads about 50 paragraphs a day, this means 2.5 words are learned directly 7.5 words are learned indirectly

30. Some more LSA applications

31. An interesting fact about psychologists LSA was trained on the text of introductory psychology textbooks and then used to answer multiple-choice based course exams It passed albeit at a significantly lower grade than class average

32. Judging the quality of essay answers LSA has been used to assign holistic quality scores to answers to essay questions Several methods were used Based on a sample of essays was graded by instructors Based on a pre-existing exemplary text The LSA correlated with expert scores approximately as well as experts correlated with each other

33. Pronominal Anaphora Resolution Anaphora – a word that acquires its meaning in reference to previous words Pronoun – she, he, it, him, himself… For example - The new medicine has been released to the market a week ago. The doctor gave it to the boy.

34. Pronominal Anaphora Resolution ‘The new medicine’, ‘the market’, and ‘a week’ are all possible resolutions for the anaphora Suggestion – compare each candidate to the phrase containing the pronoun (‘the doctor gave to the boy’) ‘The new medicine’ had the highest cosine and was selected as the correct resolution

35. Some LSA shortcomings

36. Some quirks Inexplicable word similarities sometimes arise ‘verbally’ and ‘sadomasochism’ had a cosine of 0.8 in the encyclopedia space LSA sometimes seems more sensitive to contextual associations than to semantic features ‘nurse’ is more similar to ‘physician’ (cos = 0.47) than ‘doctor’ is (cos = 0.41)

37. A limitation? The LSA representation enforces one meaning per word This does not always hold Fly may mean an insect or relate to an airplane The resulting word representation must be a compromise between all its interpretations

38. The effects of context Consider the sentence – The player caught the high fly to left field This sentence has a cosine of 0.37 with ball, 0.31 with baseball, 0.27 with hit In contrast, 0.17 with insect, 0.18 with airplane, 0.13 with bird ‘fly’ by itself has cosines – 0.02, 0.01, -0.02 with ball, baseball and hit 0.69, 0.53, 0.23 with insect, airplane and bird

39. Discussion

40. Questions for the audience What’s missing from LSA? Why does it work so well without it? Attempts to integrate syntax into LSA generally resulted in worse performance Landauer argued at one point that syntax serves no purpose other than a computationally-efficient way to get text into an LSA representation. Is this reasonable?

41. Bibliography and further reading “A Solution to Plato’s Problem: The Latent Semantic Analysis Theory of Acquisition, Induction and Representation of Knowledge”, Landauer & Dumais, Psychological Review, Vol. 104, No. 2, 1997 “An Introduction to Latent Semantic Analysis”, Landauer, Foltz & Laham, Discourse Processes, 25, 1998

42. Bibliography and further reading “Using LSA for Pronominal Anaphora Resolution”, Klebanov & Wiemer- Hastings “Adding Syntactic Information to LSA”, Wiemer-Hastings