1 Complementarity of Lexical and Simple Syntactic Features: The SyntaLex Approach to S ENSEVAL -3 Saif Mohammad Ted Pedersen University of Toronto, Toronto University of Minnesota, Duluth

2 Supervised Learning: Decision Trees Decision Trees used as classifiers. One classifier for each task (word). Assign sense by asking a series of questions. Questions correspond to features of the instance. Weka’s C4.5 algorithm employed.

3 Decision Tree Feature 4? Feature 2 ? Feature 3 ? Feature 2 ? SENSE 4 SENSE 3SENSE 2 SENSE 1 SENSE Feature 1 ? SENSE 1

4 Why Decision Trees? Context is a rich source of discrete features. The learned model likely meaningful. May provide insight into the interaction of features. Drawbacks – training data fragmentation. Pedersen[2001]*: Choosing the right features is of greater significance than the learning algorithm itself. * “ A Decision Tree of Bigrams is an Accurate Predictor of Word Sense” Pedersen, T., In the Proceedings of the Second Meeting of the North American Chapter of the Association for Computational Linguistics (NAACL-01), June 2-7, 2001, Pittsburgh, PA.

5 Features Represent context with lexical and simple syntactic features. Lexical Features Bigrams of words Syntactic Features Parts of Speech

6 Lexical Features: Bigrams Bigrams Two-word sequences in text. The interest rate is low Bigrams: the interest, interest rate, rate is, is low Ngram Statistics Package*: Identify statistically significant bigrams. * “The Design, Implementation and Use of the Ngram Statistics Package”, Banerjee, S. and Pedersen, T., Proceedings of the Fourth International Conference on Intelligent Text Processing and Computational Linguistics, 2003, February, Mexico City.

7 Part-of-Speech Features A word used in different senses is likely to have different sets of PoS tags around it. Why did Jack turn /VB against /IN his /PRP$ team /NN Why did Jack turn /VB left /NN at /IN the /DT crossing Individual word PoS: P -2, P -1, P 0, P 1, P 2. P 1 = JJ implies that the word to the right of the target word is an adjective.

8 Part-of-Speech Tagging Brill Tagger – widely used tool. Certain target words mis-tagged to completely different parts of speech. Pre-tagging is the act of manually assigning tags to selected words in a text prior to tagging. Brill tagger does not guarantee pre-tagging. BrillPatch* guarantees pre-tagging. * “Guaranteed Pre-Tagging for the Brill Tagger”, Mohammad, S. and Pedersen, T., In Proceedings of Fourth International Conference of Intelligent Systems and Text Processing, February 2003, Mexico.

9 SyntaLex Entries SyntaLex-1 Parts of Speech of a narrow window of words around the target word. P -1 = or P 0 = or P 1 =. SyntaLex-2 Parts of Speech: broader window P -2 = or P -1 = or P 0 = or P 1 = or P 2 =. represents a Part of Speech.

10 SyntaLex Entries (continued) SyntaLex-3 Individual classifiers based on bigrams and parts of speech (narrow context) learned. Given a test instance, both classifiers assign probabilities to every sense. Sense with the highest sum chosen. SyntaLex-4 Single unified decision tree. Either bigrams or part of speech features (narrow context) may exist at a particular node.

11 Fine-Grained Test Results OverallNounsVerbsAdjectives Majority 56.5%55.0%58.0%54.1% SyntaLex-1 PoS (narrow) 62.4%58.7%67.0%48.0% SyntaLex-2 PoS (broad) 61.8%57.7%66.5%55.0% SyntaLex-3 Ens. PoS, Bigram. 64.6%62.5%67.6%51.6% SyntaLex-4 Com. PoS, Bigram. 63.3%62.2%65.3%49.1%

12 Fine-Grained Test Results OverallNounsVerbsAdjectives Majority 56.5%55.0%58.0%54.1% SyntaLex-1 PoS (narrow) 62.4%58.7%67.0%48.0% SyntaLex-2 PoS (broad) 61.8%57.7%66.5%55.0% SyntaLex-3 Ens. PoS, Bigram. 64.6%62.5%67.6%51.6% SyntaLex-4 Com. PoS, Bigram. 63.3%62.2%65.3%49.1%

13 Coarse-Grained Test Results OverallNounsVerbsAdjectives SyntaLex-1 PoS (narrow) 69.1%65.1%73.3%61.7% SyntaLex-2 PoS (broad) 68.4%64.1%73.1%60.1% SyntaLex-3 Ens. PoS, Bigram. 72.0%69.6%74.9%64.2% SyntaLex-4 Com. PoS, Bigram. 71.1%69.5%73.4%62.0%

14 Observations SyntaLex-1 (narrow context) just as good as SyntaLex-2 (broader context). Low training data per task ratio. Weak indicators (P -2 and P 2 ) overwhelmed. Results* on Senseval-1, Senseval-2, line, hard, serve and interest data back the hypothesis. * "Combining Lexical and Syntactic Features for Supervised Word Sense Disambiguation", Mohammad, S. and Pedersen, T., In Proceedings of the Eighth Conference on Natural Language Learning at HLT-NAACL, May 2004, Boston.

15 Observations (continued) Pedersen’s entry (Duluth-ELSS*) Only lexical features Fine grained accuracy: 61.7%. Comparable to SyntaLex-2 (61.8%). Use of both kinds of features (SyntaLex-3 & 4) has provided some improvement. * "The Duluth Lexical Sample Systems in Senseval-3", Pedersen, T., In Proceedings of the Eighth Conference on Natural Language Learning at HLT-NAACL, May 2004, Boston.

16 Thoughts… Lexical and syntactic features perform comparably. Do they get the same instances right ? How much are the individual feature sets redundant. Are there instances correctly disambiguated by one feature set and not by the other ? How much are the individual feature sets complementary. Can we do better than the results obtained by a simple ensemble? Is the effort to combine lexical and syntactic features justified?

17 Measures Baseline Ensemble: accuracy of a hypothetical ensemble which predicts the sense correctly only if both individual feature sets do so. Quantifies redundancy amongst feature sets. SENSEVAL-3, with SyntaLex: 52.9% (fine grained). Optimal Ensemble: accuracy of a hypothetical ensemble which predicts the sense correctly if either of the individual feature sets do so. Difference with individual accuracies quantifies complementarity. SENSEVAL-3, with SyntaLex: 72.1% (fine grained).

18 Conclusions Significant amount of complementarity across the studied lexical and syntactic features. Combination of the two justified. We show that bigram and part of speech features can achieve state-of-the-art results. How best to capitalize on the complementarity still an open issue. Part of speech of words immediately adjacent to target word more suitable than that of larger contexts.

19 Software SyntaLex A system to do WSD using lexical and syntactic features. Weka’s decision tree learning algorithm is utilized. posSenseval Assigns Part-of-Speech to any data in Senseval-2 data format. Outputs tagged data in Senseval-2 data format. Employs the Brill Tagger. BrillPatch Patch to the Brill Tagger to guarantee pre-tagging. Thank You.