1. Conceptual vectors for NLP MMA 2001 Mathieu Lafourcade LIRMM - France www.lirmm.fr/~lafourca

2. Semantic Analysis Word Sense Disambiguation Text Indexing in IR Lexical Transfer in MT Conceptual vector Reminiscent of Vector Models (Salton and all.) & Sowa Applied on preselected concepts (not terms) Concepts are not independent Propagation on morpho-syntactic tree (no surface analysis) Objectives

3. Conceptual vectors An idea = a combination of concepts = a vector The Idea space = vector space A concept = an idea = a vector = combination of itself + neighborhood Sense space = vector space + vector set

4. Conceptual vectors Annotations Helps building vectors Can take the form of vectors Set of k basic concepts example Thesaurus Larousse = 873 concepts A vector = a 873 uple Encoding for each dimension C = 2 15

5. Conceptual vectors Example : cat Kernel c:mammal, c:stroke Augmented c:mammal, c:stroke, c:zoology, c:love … + Iteration for neighborhood augmentation Finer vectors

6. Vector space Basic concepts are not independent Sense space = Generator Space of a real k vector space (unknown) = Dim k <= k Relative position of points

7. Conceptual vector distance Angular Distance D A (x, y) = angle (x, y) 0 <= D A (x, y) <= π if 0 then colinear - same idea if π /2 then nothing in common if π then D A (x, -x) with -x as anti-idea of x x y x

8. Conceptual vector distance Distance = acos(similarity) D A (x, y) = acos( ((x 1 y 1 + … + x n y n )/|x||y|)) D A (x, x) = 0 D A (x, y) = D A (y, x) D A (x, y) + D A (y, z) D A (x, z) D A (0, 0) = 0 and D A (x, 0) = π /2 by definition D A ( x, y) = D A (x, y) with 0 D A ( x, y) = π - D A (x, y) with 0 D A (x+x, x+y) = D A (x, x+y) D A (x, y)

9. Conceptual vector distance Example D A (tit, tit) = 0 D A (tit, passerine) = 0.4 D A (tit, bird) = 0.7 D A (tit, train) = 1.14 D A (tit, insect) = 0.62 tit = kind of insectivorous passerine …

10. Conceptual lexicon Set of (word, vector) = (w, )* Monosemy word 1 meaning 1 vector (w, ) tit

11. Conceptual lexicon Polyseme building Polysemy word n meanings n vectors {(w, ), (w.1, 1 ) … (w.n, n ) } bank

12. Squash isolated meanings against numerous close meanings Conceptual lexicon Polyseme building (w) = (w.i) =.i bank = 1. Mound, 3. river border, … 2. Money institution 3. Organ keyboard 4. … bank

13. Conceptual lexicon Polyseme building (w) = classification(w.i ) bank 1:D A (3,4) & (3+2) 2: (bank 4 ) 7: (bank 2 )6: (bank 1 ) 4: (bank 3 ) 5: D A (6,7)& (6+7) 3: D A (4,5) & (4+5)

14. Lexical scope LS(w) = LS t ( (w)) LS t ( (w)) = 1 if is a leaf LS t ( (w)) = (LS( 1 ) + LS( 2 )) /(2-sin(D( (w))) otherwise (w) = t ( (w)) t ( (w)) = (w) if is a leaf t ( (w)) = LS( 1 ) t ( 1 ) + LS( 2 ) t ( 2 ) otherwise 1:D(3,4), (3+2) 2: 4 7: 2 6: 1 4: 3 5:D(6,7), (6+7) 3:D(4,5), (4+5) Can handle duplicated definitions (w) =

15. Vector Statistics Norm( ) [0, 1] * C (2 15 =32768) Intensity( ) Norm / C Usually = 1 Standard deviation (SD) SD 2 = variance variance = 1/n * (x i - ) 2 with as the arith. mean

16. Vector Statistics Variation coefficient (CV) CV = SD / mean No unity - Norm independent Pseudo Conceptual strength If A Hyperonym B CV(A) > CV(B) (we don t have ) vector « fruit juice » (N) MEAN = 527, SD = 973 CV = 1.88 vector « drink » (N) MEAN = 443, SD = 1014 CV = 2.28

17. Vector operations Sum V = X + Y v i = x i + y i Neutral element : 0 Generalized to n terms : V = V i Normalization of sum : v i /|V|* c Kind of mean

18. Vector operations Term to term product V = X Y v i = x i * y i Neutral element : 1 Generalized to n terms V = V i Kind of intersection

19. Vector operations Amplification V = X ^ n v i = sg(v i ) * |v i |^ n V = V ^ 1/2and n V = V ^ 1/n V V = V ^ 2if v i 0 Normalization of ttm product to n terms V = n V i

20. Vector operations Product + sum V = X Y = ( X Y ) + X + Y Generalized n terms : V = n V i + V i Simplest request vector computation in IR

21. Vector operations Subtraction V = X Y v i = x i y i Dot subtraction V = X Y v i = max (x i y i, 0) Complementary V = C(X) v i = (1 x i c) * c etc. Set operations

22. Intensity Distance Intensity of normalized ttm product 0 ( (X Y)) 1 if |x| = |y| = 1 D I (X, Y) = acos( ( X Y)) D I (X, X) = 0and D I (X, 0) = π/2 D I (tit, tit) = 0(D A = 0) D I (tit, passerine) = 0.25(D A = 0.4) D I (tit, bird) = 0.58(D A = 0.7) D I (tit, train) = 0.89 (D A = 1.14) D I (tit, insect) = 0.50(D A = 0.62)

23. Relative synonymy Syn R (A, B, C) C as reference feature Syn R (A, B, C) = D A (A C, B C) D A (coal,night) = 0.9 Syn R (coal, night, color) = 0.4 Syn R (coal, night, black) = 0.35

24. Relative synonymy Syn R (A, B, C) = Syn R (B, A, C) Syn R (A, A, C) = D (A C, A C) = 0 Syn R (A, B, 0) = D (0, 0) = 0 Syn R (A, 0, C) = π /2 Syn A (A, B) = Syn R (A, B, 1) = D (A 1, B 1) = D (A, B)

25. Subjective synonymy Syn S (A, B, C) C as point of view Syn S (A, B, C) = D(C-A, C-B) 0 Syn S (A, B, C) π normalization: 0 asin(sin(Syn S (A, B, C))) π/2 B A C C C

26. Subjective synonymy When |C| then Syn S (A, B, C) 0 Syn S (A, B, 0) = D(-B, -A) = D(A, B) Syn S (A, A, C) = D(C-A, C-A) = 0 Syn S (A, B, B) = Syn S (A, B, A) = 0 Syn S (tit, swallow, animal) = 0.3 Syn S (tit, swallow, bird) = 0.4 Syn S (tit, swallow, passerine) = 1

27. 68 Semantic analysis Vectors propagate on syntactic tree Lesrapidement P GV GVA GNP termites attaquent lesfermes GN dutoit The white ants strike rapidly the trusses of the roof

28. Semantic analysis Lesrapidement P GV GVA GNP attaquent fermes termites les GN toit GN du farm business farm building truss roof anatomy above to start to attack to Critisize

29. Semantic analysis Initialization - attach vectors to nodes Lesrapidement P GV GVA GNP termites attaquent lesfermes GN dutoit 1 2 3 4 5

30. Semantic analysis Propagation (up) Lesrapidement P GV GVA GNP termites attaquent lesfermes GN dutoit

31. Semantic analysis Back propagation (down) (N i j ) = ( (N i j ) (N i )) + (N i j ) Lesrapidement P GV GVA GNP termites attaquent lesfermes GN dutoit 1 2 3 4 5

32. Semantic analysis Sense selection or sorting Lesrapidement P GV GVA GNP termites attaquent les fermes GN du toit farm business farm building truss roof anatomy above to start to attack to Critisize

33. Sense selection 1:D(3,4) & (3+2) 2: (bank 4 ) 7: (bank 2 )6: (bank 1 ) 4: (bank 3 ) 5:D(6,7)& (6+7) 3:D(4,5) & (4+5) Recursive descent on t(w) as decision tree D A (, i ) Stop on a leaf Stop on an internal node

34. Vector syntactic schemas S: NP(ART,N) (NP) = V(N) S: NP1(NP2,N) (NP1) = (NP1)+ (N)0< <1 (sail boat) = (sail) + 1/2 (boat) (boat sail) = 1/2 (boat) + (sail)

35. Vector syntactic schemas Not necessary linear S: GA(GADV(ADV),ADJ) (GA) = (ADJ)^p(ADV) p(very) = 2 p(mildly) = 1/2 (very happy) = (happy)^2 (mildly happy) = (happy)^1/2

36. Iteration & convergence Iteration with convergence Local D( i, i+1 ) for top Global D( i, i+1 ) for all Good results but costly

37. Lexicon construction Manual kernel Automatic definition analysis Global infinite loop = learning Manual adjustments iterations synonyms Ukn word in definitions kernel

38. Application machine translation Lexical transfer source target Knn search that minimizes D A ( source, target ) Submeaning selection Direct Transformation matrix chat cat

39. Application Information Retrieval on Texts Textual document indexation Language dependant Retrieval Language independent - Multilingual Domain representation horse equitation Granularity Document, paragraphs, etc.

40. Application Information Retrieval on Texts Index = Lexicon = (d i, i )* Knn search that minimizes D A ( (r), (d i )) doc 1 doc n doc 2 request

41. Search engine Distances adjustments Min D A ( (r), (d i )) may pose problems Especially with small documents Correlation between CV & conceptual richness Pathological cases « plane » and « plane plane plane plane … » « inundation » « blood » D = 0.85 (liquid)

42. Search engine Distances adjustments Correction with relative intensity Request vs retrieved doc ( r and d ) D = (D A ( r, d ) * D I ( r, d )) 0 ( r, d ) 1 0 D I ( r, d ) π/2

43. Conclusion Approach statistical (but not probabilistic) thema (and rhema ?) Combination of Symbolic methods (IA) Transformational systems Similarity Neural nets With large Dim (> 50000 ?)

44. Conclusion Self evaluation Vector quality Tests against corpora Unknown words Proper nouns of person, products, etc. Lionel Jospin, Danone, Air France Automatic learning Badly handled phenomena? Negation & Lexical functions (Meltchuk)

45. End 1. extremity 2. death 3. aim …