1. July 17, 2006AAAI-2006 Tutorial1 Language Independent Methods of Clustering Similar Contexts (with applications) Ted Pedersen University of Minnesota, Duluth tpederse@d.umn.edu http://www.d.umn.edu/~tpederse/SCTutorial.html

2. July 17, 2006AAAI-2006 Tutorial2 Language Independent Methods Do not utilize syntactic information –No parsers, part of speech taggers, etc. required Do not utilize dictionaries or other manually created lexical resources Based on lexical features selected from corpora –Assumption: word segmentation can be done by looking for white spaces between strings No manually annotated data of any kind, methods are completely unsupervised in the strictest sense

3. July 17, 2006AAAI-2006 Tutorial3 Clustering Similar Contexts A context is a short unit of text –often a phrase to a paragraph in length, although it can be longer Input: N contexts Output: K clusters –Where each member of a cluster is a context that is more similar to each other than to the contexts found in other clusters

4. July 17, 2006AAAI-2006 Tutorial4 Applications Headed contexts (contain target word) –Name Discrimination –Word Sense Discrimination Headless contexts –Email Organization –Document Clustering –Paraphrase identification Clustering Sets of Related Words

5. July 17, 2006AAAI-2006 Tutorial5 Tutorial Outline Identifying lexical features –Measures of association & tests of significance Context representations –First & second order Dimensionality reduction –Singular Value Decomposition Clustering –Partitional techniques –Cluster stopping –Cluster labeling Evaluation

6. July 17, 2006AAAI-2006 Tutorial6 SenseClusters A package for clustering contexts –http://senseclusters.sourceforge.nethttp://senseclusters.sourceforge.net –SenseClusters Live! (Knoppix CD) Integrates with various other tools –Ngram Statistics Package –CLUTO –SVDPACKC

7. July 17, 2006AAAI-2006 Tutorial7 Many thanks… Amruta Purandare (M.S., 2004) –Founding developer of SenseClusters (2002-2004) –Now PhD student in Intelligent Systems at the University of Pittsburgh http://www.cs.pitt.edu/~amruta/http://www.cs.pitt.edu/~amruta/ Anagha Kulkarni (M.S., 2006, expected) –Enhancing SenseClusters since Fall 2004! –Will start as PhD student at CMU/LTI in Fall 2006 http://www.d.umn.edu/~kulka020/ http://www.d.umn.edu/~kulka020/ NSF for supporting Amruta, Anagha and Ted via CAREER award #0092784

8. July 17, 2006AAAI-2006 Tutorial8 Background and Motivations

9. July 17, 2006AAAI-2006 Tutorial9 Headed and Headless Contexts A headed context includes a target word –Our goal is to cluster the target words based on their surrounding contexts –Target word is center of context and our attention A headless context has no target word –Our goal is to cluster the contexts based on their similarity to each other –The focus is on the context as a whole

10. July 17, 2006AAAI-2006 Tutorial10 Headed Contexts (input) I can hear the ocean in that shell. My operating system shell is bash. The shells on the shore are lovely. The shell command line is flexible. The oyster shell is very hard and black.

11. July 17, 2006AAAI-2006 Tutorial11 Headed Contexts (output) Cluster 1: –My operating system shell is bash. –The shell command line is flexible. Cluster 2: –The shells on the shore are lovely. –The oyster shell is very hard and black. –I can hear the ocean in that shell.

12. July 17, 2006AAAI-2006 Tutorial12 Headless Contexts (input) The new version of Linux is more stable and has better support for cameras. My Chevy Malibu has had some front end troubles. Osborne made one of the first personal computers. The brakes went out, and the car flew into the house. With the price of gasoline, I think I’ll be taking the bus more often!

13. July 17, 2006AAAI-2006 Tutorial13 Headless Contexts (output) Cluster 1: –The new version of Linux is more stable and better support for cameras. –Osborne made one of the first personal computers. Cluster 2: –My Chevy Malibu has had some front end troubles. –The brakes went out, and the car flew into the house. –With the price of gasoline, I think I’ll be taking the bus more often!

14. July 17, 2006AAAI-2006 Tutorial14 Web Search as Application Web search results are headed contexts –Search term is target word (found in snippets) Web search results are often disorganized – two people sharing same name, two organizations sharing same abbreviation, etc. often have their pages “mixed up” If you click on search results or follow links in pages found, you will encounter headless contexts too…

15. July 17, 2006AAAI-2006 Tutorial15 Email Foldering as Application Email (public or private) is made up of headless contexts –Short, usually focused… Cluster similar email messages together –Automatic email foldering –Take all messages from sent-mail file or inbox and organize into categories

16. July 17, 2006AAAI-2006 Tutorial16 Clustering News as Application News articles are headless contexts –Entire article or first paragraph –Short, usually focused Cluster similar articles together

17. July 17, 2006AAAI-2006 Tutorial17 What is it to be “similar”? You shall know a word by the company it keeps –Firth, 1957 (Studies in Linguistic Analysis) Meanings of words are (largely) determined by their distributional patterns (Distributional Hypothesis) –Harris, 1968 (Mathematical Structures of Language) Words that occur in similar contexts will have similar meanings (Strong Contextual Hypothesis) –Miller and Charles, 1991 (Language and Cognitive Processes) Various extensions… –Similar contexts will have similar meanings, etc. –Names that occur in similar contexts will refer to the same underlying person, etc.

18. July 17, 2006AAAI-2006 Tutorial18 General Methodology Represent contexts to be clustered using first or second order feature vectors –Lexical features Reduce dimensionality to make vectors more tractable and/or understandable –Singular value decomposition Cluster the context vectors –Find the number of clusters –Label the clusters Evaluate and/or use the contexts!

19. July 17, 2006AAAI-2006 Tutorial19 Identifying Lexical Features Measures of Association and Tests of Significance

20. July 17, 2006AAAI-2006 Tutorial20 What are features? Features represent the (hopefully) salient characteristics of the contexts to be clustered Eventually we will represent each context as a vector, where the dimensions of the vector are associated with features Vectors/contexts that include many of the same features will be similar to each other

21. July 17, 2006AAAI-2006 Tutorial21 Where do features come from? In unsupervised clustering, it is common for the feature selection data to be the same data that is to be clustered –This is not cheating, since data to be clustered does not have any labeled classes that can be used to assist feature selection –It may also be necessary, since we may need to cluster all available data, and not hold out some for a separate feature identification step Email or news articles

22. July 17, 2006AAAI-2006 Tutorial22 Feature Selection “Test” data – the contexts to be clustered –Assume that the feature selection data is the same as the test data, unless otherwise indicated “Training” data – a separate corpus of held out feature selection data (that will not be clustered) –may need to use if you have a small number of contexts to cluster (e.g., web search results) –This sense of “training” due to Schütze (1998)

23. July 17, 2006AAAI-2006 Tutorial23 Lexical Features Unigram – a single word that occurs more than a given number of times Bigram – an ordered pair of words that occur together more often than expected by chance –Consecutive or may have intervening words Co-occurrence – an unordered bigram Target Co-occurrence – a co-occurrence where one of the words is the target word

24. July 17, 2006AAAI-2006 Tutorial24 Bigrams fine wine (window size of 2) baseball bat house of representatives (window size of 3) president of the republic (window size of 4) apple orchard Selected using a small window size (2-4 words), trying to capture a regular (localized) pattern between two words (collocation?)

25. July 17, 2006AAAI-2006 Tutorial25 Co-occurrences tropics water boat fish law president train travel Usually selected using a larger window (7-10 words) of context, hoping to capture pairs of related words rather than collocations

26. July 17, 2006AAAI-2006 Tutorial26 Bigrams and Co-occurrences Pairs of words tend to be much less ambiguous than unigrams –“bank” versus “river bank” and “bank card” –“dot” versus “dot com” and “dot product” Three grams and beyond occur much less frequently (Ngrams very Zipfian) Unigrams are noisy, but bountiful

27. July 17, 2006AAAI-2006 Tutorial27 “occur together more often than expected by chance…” Observed frequencies for two words occurring together and alone are stored in a 2x2 matrix –Throw out bigrams that include one or two stop words Expected values are calculated, based on the model of independence and observed values –How often would you expect these words to occur together, if they only occurred together by chance? –If two words occur “significantly” more often than the expected value, then the words do not occur together by chance.

28. July 17, 2006AAAI-2006 Tutorial28 2x2 Contingency Table Intelligence!Intelligence Artificial100.0 000.12 300.0 398.8 400 !Artificial200.0 298.8 99,400.0 99,301.2 99,600 30099,700100,000

29. July 17, 2006AAAI-2006 Tutorial29 Measures of Association

30. July 17, 2006AAAI-2006 Tutorial30 Interpreting the Scores… G^2 and X^2 are asymptotically approximated by the chi-squared distribution… This means…if you fix the marginal totals of a table, randomly generate internal cell values in the table, calculate the G^2 or X^2 scores for each resulting table, and plot the distribution of the scores, you *should* get …

31. July 17, 2006AAAI-2006 Tutorial31 Interpreting the Scores… Values above a certain level of significance can be considered grounds for rejecting the null hypothesis –H0: the words in the bigram are independent –3.841 is associated with 95% confidence that the null hypothesis should be rejected

32. July 17, 2006AAAI-2006 Tutorial32 Measures of Association There are numerous measures of association that can be used to identify bigram and co-occurrence features Many of these are supported in the Ngram Statistics Package (NSP) –http://www.d.umn.edu/~tpederse/nsp.htmlhttp://www.d.umn.edu/~tpederse/nsp.html

33. July 17, 2006AAAI-2006 Tutorial33 Summary Identify lexical features based on frequency counts or measures of association – either in the data to be clustered or in a separate set of feature selection data –Language independent Unigrams usually only selected by frequency –Remember, no labeled data from which to learn, so somewhat less effective as features than in supervised case Bigrams and co-occurrences can also be selected by frequency, or better yet measures of association –Bigrams and co-occurrences need not be consecutive –Stop words should be eliminated –Frequency thresholds are helpful (e.g., unigram/bigram that occurs once may be too rare to be useful)

34. July 17, 2006AAAI-2006 Tutorial34 Context Representations First and Second Order Methods

35. July 17, 2006AAAI-2006 Tutorial35 Once features selected… We have a set of unigrams, bigrams, co- occurrences or target co-occurrences –We believe/hope that these are descriptive of the contexts –We also have frequency and measure of association score that have been used in their selection Convert contexts to be clustered into a vector representation based on these features

36. July 17, 2006AAAI-2006 Tutorial36 First Order Representation Each context is represented by a vector with M dimensions, each of which indicates whether or not a particular feature occurred in that context –Value may be binary, a frequency count, or an association score Context by Feature representation

37. July 17, 2006AAAI-2006 Tutorial37 Contexts Cxt1: There was an island curse of black magic cast by that voodoo child. Cxt2: Harold, a known voodoo child, was gifted in the arts of black magic. Cxt3: Despite their military might, it was a serious error to attack. Cxt4: Military might is no defense against a voodoo child or an island curse.

38. July 17, 2006AAAI-2006 Tutorial38 Unigram Feature Set island 1000 black 700 curse 500 magic 400 child 200 (assume these are frequency counts obtained from some corpus…)

39. July 17, 2006AAAI-2006 Tutorial39 First Order Vectors of Unigrams islandblackcursemagicchild Cxt111111 Cxt201011 Cxt300000 Cxt410101

40. July 17, 2006AAAI-2006 Tutorial40 Bigram Feature Set island curse 189.2 black magic 123.5 voodoo child 120.0 military might 100.3 serious error 89.2 island child 73.2 voodoo might 69.4 military error 54.9 black child 43.2 serious curse 21.2 (assume these are log-likelihood scores based on frequency counts from some corpus)

41. July 17, 2006AAAI-2006 Tutorial41 First Order Vectors of Bigrams black magic island curse military might serious error voodoo child Cxt111001 Cxt210001 Cxt300110 Cxt401101

42. July 17, 2006AAAI-2006 Tutorial42 First Order Vectors Can have binary values or weights associated with frequency, etc. Forms a context by feature matrix May optionally be smoothed/reduced with Singular Value Decomposition –More on that later… The contexts are ready for clustering… –More on that later…

43. July 17, 2006AAAI-2006 Tutorial43 Second Order Features First order features encode the occurrence of a feature in a context –Feature occurrence represented by binary value Second order features encode something ‘extra’ about a feature that occurs in a context –Feature occurrence represented by word co-occurrences –Feature occurrence represented by context occurrences

44. July 17, 2006AAAI-2006 Tutorial44 Second Order Representation First, build word by word matrix from features –Based on bigrams or co-occurrences –First word is row, second word is column, cell is score –(optionally) reduce dimensionality w/SVD –Each row forms a vector of first order co-occurrences Second, replace each word in a context with its row/vector as found in the word by word matrix Average all the word vectors in the context to create the second order representation –Due to Schütze (1998), related to LSI/LSA

45. July 17, 2006AAAI-2006 Tutorial45 Word by Word Matrix magiccursemighterrorchild black123.500043.2 island0189.20073.2 military00100.354.90 serious021.2089.20 voodoo0069.40120.0

46. July 17, 2006AAAI-2006 Tutorial46 Word by Word Matrix …can also be used to identify sets of related words In the case of bigrams, rows represent the first word in a bigram and columns represent the second word –Matrix is asymmetric In the case of co-occurrences, rows and columns are equivalent –Matrix is symmetric The vector (row) for each word represent a set of first order features for that word Each word in a context to be clustered for which a vector exists (in the word by word matrix) is replaced by that vector in that context

47. July 17, 2006AAAI-2006 Tutorial47 There was an island curse of black magic cast by that voodoo child. magiccursemighterrorchild black123.500043.2 island0189.20073.2 voodoo0069.40120.0

48. July 17, 2006AAAI-2006 Tutorial48 Second Order Co-Occurrences Word vectors for “black” and “island” show similarity as both occur with “child” “black” and “island” are second order co- occurrence with each other, since both occur with “child” but not with each other (i.e., “black island” is not observed)

49. July 17, 2006AAAI-2006 Tutorial49 Second Order Representation There was an [curse, child] curse of [magic, child] magic cast by that [might, child] child [curse, child] + [magic, child] + [might, child]

50. July 17, 2006AAAI-2006 Tutorial50 There was an island curse of black magic cast by that voodoo child. magiccursemighterrorchild Cxt141.263.124.4078.8

51. July 17, 2006AAAI-2006 Tutorial51 Second Order Representation Results in a Context by Feature (Word) Representation Cell values do not indicate if feature occurred in context. Rather, they show the strength of association of that feature with other words that occur with a word in the context.

52. July 17, 2006AAAI-2006 Tutorial52 Summary First order representations are intuitive, but… –Can suffer from sparsity –Contexts represented based on the features that occur in those contexts Second order representations are harder to visualize, but… –Allow a word to be represented by the words it co- occurs with (i.e., the company it keeps) –Allows a context to be represented by the words that occur with the words in the context –Helps combat sparsity…

53. July 17, 2006AAAI-2006 Tutorial53 Related Work Pedersen and Bruce 1997 (EMNLP) presented first order method of discrimination http://acl.ldc.upenn.edu/W/W97/W97-0322.pdf Schütze 1998 (Computational Linguistics) introduced second order method http://acl.ldc.upenn.edu/J/J98/J98-1004.pdf Purandare and Pedersen 2004 (CoNLL) compared first and second order methods http://acl.ldc.upenn.edu/hlt-naacl2004/conll04/pdf/purandare.pdf –First order better if you have lots of data –Second order better with smaller amounts of data

54. July 17, 2006AAAI-2006 Tutorial54 Dimensionality Reduction Singular Value Decomposition

55. July 17, 2006AAAI-2006 Tutorial55 Effect of SVD SVD reduces a matrix to a given number of dimensions This may convert a word level space into a semantic or conceptual space –If “dog” and “collie” and “wolf” are dimensions/columns in a word co-occurrence matrix, after SVD they may be a single dimension that represents “canines”

56. July 17, 2006AAAI-2006 Tutorial56 Effect of SVD The dimensions of the matrix after SVD are principal components that represent the meaning of concepts –Similar columns are grouped together SVD is a way of smoothing a very sparse matrix, so that there are very few zero valued cells after SVD

57. July 17, 2006AAAI-2006 Tutorial57 How can SVD be used? SVD on first order contexts will reduce a context by feature representation down to a smaller number of features –Latent Semantic Analysis typically performs SVD on a feature by context representation, where the contexts are reduced SVD used in creating second order context representations –Reduce word by word matrix

58. July 17, 2006AAAI-2006 Tutorial58 Word by Word Matrix applebloodcellsibmdataboxtissuegraphicsmemoryorganplasma pc20013100000 body03000020021 disk10020301200 petri02100020101 lab00302020213 sales00023001200 linux20013201100 debt00023402000

59. July 17, 2006AAAI-2006 Tutorial59 Singular Value Decomposition A=UDV’

60. July 17, 2006AAAI-2006 Tutorial60 Word by Word Matrix After SVD applebloodcellsibmdatatissuegraphicsmemoryorganplasma pc.73.00.111.32.0.01.86.77.00.09 body.001.21.3.00.331.6.00.85.841.5 disk.76.00.011.32.1.00.91.72.00 germ.001.11.2.00.491.5.00.86.771.4 lab.211.72.0.351.72.5.181.71.22.3 sales.73.15.391.32.2.35.85.98.17.41 linux.96.00.161.72.7.031.11.0.00.13 debt1.2.00 2.13.2.001.51.1.00

61. July 17, 2006AAAI-2006 Tutorial61 Second Order Representation These two contexts share no words in common, yet they are similar! disk and linux both occur with “Apple”, “IBM”, “data”, “graphics”, and “memory” The two contexts are similar because they share many second order co-occurrences applebloodcellsibmdatatissuegraphicsmemory organplasma disk.76.00.011.32.1.00.91.72.00 linux.96.00.161.72.7.031.11.0.00.13 I got a new disk today! What do you think of linux?

62. July 17, 2006AAAI-2006 Tutorial62 Relationship to LSA Latent Semantic Analysis uses feature by context first order representation –Indicates all the contexts in which a feature occurs –Use SVD to reduce dimensions (contexts) –Cluster features based on similarity of contexts in which they occur –Represent sentences using an average of feature vectors

63. July 17, 2006AAAI-2006 Tutorial63 Feature by Context Representation Cxt1Cxt2Cxt3Cxt4 black magic1101 island curse1001 military might0010 serious error0010 voodoo child1101

64. July 17, 2006AAAI-2006 Tutorial64 References Deerwester, S. and Dumais, S.T. and Furnas, G.W. and Landauer, T.K. and Harshman, R., Indexing by Latent Semantic Analysis, Journal of the American Society for Information Science, vol. 41, 1990 Landauer, T. and Dumais, S., A Solution to Plato's Problem: The Latent Semantic Analysis Theory of Acquisition, Induction and Representation of Knowledge, Psychological Review, vol. 104, 1997 Schütze, H, Automatic Word Sense Discrimination, Computational Linguistics, vol. 24, 1998 Berry, M.W. and Drmac, Z. and Jessup, E.R.,Matrices, Vector Spaces, and Information Retrieval, SIAM Review, vol 41, 1999

65. July 17, 2006AAAI-2006 Tutorial65 Clustering Partitional Methods Cluster Stopping Cluster Labeling

66. July 17, 2006AAAI-2006 Tutorial66 Many many methods… Cluto supports a wide range of different clustering methods –Agglomerative Average, single, complete link… –Partitional K-means (Direct) –Hybrid Repeated bisections SenseClusters integrates with Cluto –http://www-users.cs.umn.edu/~karypis/cluto/http://www-users.cs.umn.edu/~karypis/cluto/

67. July 17, 2006AAAI-2006 Tutorial67 General Methodology Represent contexts to be clustered in first or second order vectors Cluster the context vectors directly –vcluster … or convert to similarity matrix and then cluster –scluster

68. July 17, 2006AAAI-2006 Tutorial68 Partitional Methods Randomly create centroids equal to the number of clusters you wish to find Assign each context to nearest centroid After all contexts assigned, re-compute centroids –“best” location decided by criterion function Repeat until stable clusters found –Centroids don’t shift from iteration to iteration

69. July 17, 2006AAAI-2006 Tutorial69 Partitional Methods Advantages : fast Disadvantages –Results can be dependent on the initial placement of centroids –Must specify number of clusters ahead of time maybe not…

70. July 17, 2006AAAI-2006 Tutorial70 Partitional Criterion Functions Intra-Cluster (Internal) similarity/distance –How close together are members of a cluster? –Closer together is better Inter-Cluster (External) similarity/distance –How far apart are the different clusters? –Further apart is better

71. July 17, 2006AAAI-2006 Tutorial71 Intra Cluster Similarity Ball of String (I1) –How far is each member from each other member Flower (I2) –How far is each member of cluster from centroid

72. July 17, 2006AAAI-2006 Tutorial72 Contexts to be Clustered

73. July 17, 2006AAAI-2006 Tutorial73 Ball of String (I1 Internal Criterion Function)

74. July 17, 2006AAAI-2006 Tutorial74 Flower (I2 Internal Criterion Function)

75. July 17, 2006AAAI-2006 Tutorial75 Inter Cluster Similarity The Fan (E1) –How far is each centroid from the centroid of the entire collection of contexts –Maximize that distance

76. July 17, 2006AAAI-2006 Tutorial76 The Fan (E1 External Criterion Function)

77. July 17, 2006AAAI-2006 Tutorial77 Hybrid Criterion Functions Balance internal and external similarity –H1 = I1/E1 –H2 = I2/E1 Want internal similarity to increase, while external similarity decreases Want internal distances to decrease, while external distances increase

78. July 17, 2006AAAI-2006 Tutorial78 Cluster Stopping

79. July 17, 2006AAAI-2006 Tutorial79 Cluster Stopping Many Clustering Algorithms require that the user specify the number of clusters prior to clustering But, the user often doesn’t know the number of clusters, and in fact finding that out might be the goal of clustering

80. July 17, 2006AAAI-2006 Tutorial80 Criterion Functions Can Help Run partitional algorithm for k=1 to deltaK –DeltaK is a user estimated or automatically determined upper bound for the number of clusters Find the value of k at which the criterion function does not significantly increase at k+1 Clustering can stop at this value, since no further improvement in solution is apparent with additional clusters (increases in k)

81. July 17, 2006AAAI-2006 Tutorial81 H2 versus k T. Blair – V. Putin – S. Hussein

82. July 17, 2006AAAI-2006 Tutorial82 PK2 Based on Hartigan, 1975 When ratio approaches 1, clustering is at a plateau Select value of k which is closest to but outside of standard deviation interval

83. July 17, 2006AAAI-2006 Tutorial83 PK2 predicts 3 senses T. Blair – V. Putin – S. Hussein

84. July 17, 2006AAAI-2006 Tutorial84 PK3 Related to Salvador and Chan, 2004 Inspired by Dice Coefficient Values close to 1 mean clustering is improving … Select value of k which is closest to but outside of standard deviation interval

85. July 17, 2006AAAI-2006 Tutorial85 PK3 predicts 3 senses T. Blair – V. Putin – S. Hussein

86. July 17, 2006AAAI-2006 Tutorial86 References Hartigan, J. Clustering Algorithms, Wiley, 1975 –basis for SenseClusters stopping method PK2 Mojena, R., Hierarchical Grouping Methods and Stopping Rules: An Evaluation, The Computer Journal, vol 20, 1977 –basis for SenseClusters stopping method PK1 Milligan, G. and Cooper, M., An Examination of Procedures for Determining the Number of Clusters in a Data Set, Psychometrika, vol. 50, 1985 –Very extensive comparison of cluster stopping methods Tibshirani, R. and Walther, G. and Hastie, T., Estimating the Number of Clusters in a Dataset via the Gap Statistic,Journal of the Royal Statistics Society (Series B), 2001 Pedersen, T. and Kulkarni, A. Selecting the "Right" Number of Senses Based on Clustering Criterion Functions, Proceedings of the Posters and Demo Program of the Eleventh Conference of the European Chapter of the Association for Computational Linguistics, 2006 –Describes SenseClusters stopping methods

87. July 17, 2006AAAI-2006 Tutorial87 Cluster Labeling

88. July 17, 2006AAAI-2006 Tutorial88 Cluster Labeling Once a cluster is discovered, how can you generate a description of the contexts of that cluster automatically? In the case of contexts, you might be able to identify significant lexical features from the contents of the clusters, and use those as a preliminary label

89. July 17, 2006AAAI-2006 Tutorial89 Results of Clustering Each cluster consists of some number of contexts Each context is a short unit of text Apply measures of association to the contents of each cluster to determine N most significant bigrams Use those bigrams as a label for the cluster

90. July 17, 2006AAAI-2006 Tutorial90 Label Types The N most significant bigrams for each cluster will act as a descriptive label The M most significant bigrams that are unique to each cluster will act as a discriminating label

91. July 17, 2006AAAI-2006 Tutorial91 Evaluation Techniques Comparison to gold standard data

92. July 17, 2006AAAI-2006 Tutorial92 Evaluation If Sense tagged text is available, can be used for evaluation –But don’t use sense tags for clustering or feature selection! Assume that sense tags represent “true” clusters, and compare these to discovered clusters –Find mapping of clusters to senses that attains maximum accuracy

93. July 17, 2006AAAI-2006 Tutorial93 Evaluation Pseudo words are especially useful, since it is hard to find data that is discriminated –Pick two words or names from a corpus, and conflate them into one name. Then see how well you can discriminate. –http://www.d.umn.edu/~tpederse/tools.html Baseline Algorithm– group all instances into one cluster, this will reach “accuracy” equal to majority classifier

94. July 17, 2006AAAI-2006 Tutorial94 Evaluation Pseudo words are especially useful, since it is hard to find data that is discriminated –Pick two or more words or names from a corpus, and conflate them into one name. Then see how well you can discriminate. –http://www.d.umn.edu/~kulka020/kanaghaNa me.html

95. July 17, 2006AAAI-2006 Tutorial95 Baseline Algorithm Baseline Algorithm – group all instances into one cluster, this will reach “accuracy” equal to majority classifier What if the clustering said everything should be in the same cluster?

96. July 17, 2006AAAI-2006 Tutorial96 Baseline Performance S1S2S3Totals C10000 C20000 C3803555170 Totals803555170 S3S2S1Totals C10000 C20000 C3553580170 Totals553580170 (0+0+55)/170 =.32 if C3 is S1 (0+0+80)/170 =.47if C3 is S3

97. July 17, 2006AAAI-2006 Tutorial97 Evaluation Suppose that C1 is labeled S1, C2 as S2, and C3 as S3 Accuracy = (10 + 0 + 10) / 170 = 12% Diagonal shows how many members of the cluster actually belong to the sense given on the column Can the “columns” be rearranged to improve the overall accuracy? –Optimally assign clusters to senses S1S2S3Totals C11030545 C22004060 C35051065 Totals803555170

98. July 17, 2006AAAI-2006 Tutorial98 Evaluation The assignment of C1 to S2, C2 to S3, and C3 to S1 results in 120/170 = 71% Find the ordering of the columns in the matrix that maximizes the sum of the diagonal. This is an instance of the Assignment Problem from Operations Research, or finding the Maximal Matching of a Bipartite Graph from Graph Theory. S2S3S1Totals C13051045 C20402060 C35105065 Totals355580170

99. July 17, 2006AAAI-2006 Tutorial99 Alternatives? Unsupervised methods may not discover clusters equivalent to the classes learned in supervised learning Evaluation based on assuming that sense tags represent the “true” cluster are likely a bit harsh. Alternatives? –Humans could look at the members of each cluster and determine the nature of the relationship or meaning that they all share –Use the contents of the cluster to generate a descriptive label that could be inspected by a human

100. July 17, 2006AAAI-2006 Tutorial100 Thank you! Questions or comments on tutorial or SenseClusters are welcome at any time tpederse@d.umn.edu tpederse@d.umn.edu SenseClusters is freely available via LIVE CD, the Web, and in source code form http://senseclusters.sourceforge.net SenseClusters papers available at: http://www.d.umn.edu/~tpederse/senseclusters-pubs.html