1. Class-based nominal semantic role labeling: a preliminary investigation Matt Gerber Michigan State University, Department of Computer Science

2. Introduction: semantic role labeling The semantic role Relation between a constituent and a predication The task Automatically identify semantic roles occurring in natural language Problematic: which roles are the “right” ones? Agent Theme Experiencer “John presented his findings to the committee.”

3. Introduction: PropBank (Kingsbury and Palmer 2003) Annotated corpus of semantic roles Base corpus: TreeBank 2 (Marcus et al., 1993) Evaluation CoNLL Shared Task (Carreras and Marquez, 2005) Implications QA: Kaisser and Webber (2007), Shen and Lapata (2007) Coreference: Ponzetto and Strube (2006) Information extraction: Surdeanu et al. (2003) Arg0 Arg1 Arg2 “John presented his findings to the committee.”

4. Introduction: NomBank (Meyers, 2007) Verbs are not the only lexical category with shallow semantic structure [ Arg0 Judge Curry] [ Predicate ordered] [ Arg1 Edison] [ Arg2 to make average refunds of about $45]. Judge Curry ordered [ Arg0 Edison] to make average [ Predicate refunds] [ Arg1 of about $45]. A more complete semantic interpretation of natural language Nominal Verbal

5. Introduction: NomBank (Meyers, 2007) Corpus information Base corpus: TreeBank 2 Distinct nominalizations: 4704 Total attestations: ~115K NomLex (Macleod et al., 1998) Nominalization classes (22) Nom (deverbals) Example: Sales departments then urged [ Predicate abandonment] [ Arg1 of the Pico Project]. Partitive (part-whole) Example: Hallwood owns about 11 [ Predicate %] [ Arg1 of Integra].

6. Research objectives Investigate the role of NomLex classes in automated NomBank SRL Hypotheses (1) Classes may exhibit consistent realizations of their arguments (2) Modeling each class separately may result in more homogeneous training data and better SRL performance

7. Outline Nominalization interpretation: related work NomBank SRL Class-based NomBank SRL Preliminary results and analysis Conclusions and future work

8. Nominalization interpretation: early work Rule-based methods Associate syntactic configurations with grammatical functions and semantic properties Dahl et al. (1987) Hull and Gomez (1996) Meyers et al. (1998) Statistical models: Lapata (2000) Identify underlying subject/object [subject satellite] observation [object satellite] observation

9. Nominalization interpretation: recent work SemEval (Girju, 2007) Semantic relations between nominals Cause-Effect: laugh wrinkles Instrument-Agency: laser printer Product-Producer: honey bee Origin-Entity: message entity from outer-space origin Theme-Tool: news conference Part-Whole: the door of the car Content-Container: the grocery bag

10. Nominalization interpretation: recent work NomBank SRL: Jiang and Ng (2006), Liu and Ng (2007) Direct application of verbal SRL methods Standard feature set Maximum entropy modeling Best overall f-measure score: 0.7283 NomBank-specific features had little impact

11. Overview of NomBank SRL Judge Curry ordered Edison to make average [Predicate refunds] of about $45. S VP S NP NNSJJ NP PP Full syntactic analysis

12. Overview of NomBank SRL Judge Curry ordered [Edison] to make average [Predicate refunds] [of about $45]. S VP S NP NNSJJ NP PP Argument identification Binary classification problem Argument Non-argument

13. Overview of NomBank SRL Judge Curry ordered [Arg0 Edison] to make average [Predicate refunds] [Arg1 of about $45]. S VP S NP NNSJJ NP PP Argument classification 22-class problem Arg0-Arg9 Temporal, location, etc.

14. NomBank SRL features

15. Class-based NomBank SRL Simple method Cluster nominalizations according to NomLex class membership Train a logistic regression model for each class Single-stage, 23-class strategy Baseline feature set Heuristic post-processing Backoff Trained over all classes

16. Class-based NomBank SRL Model application Hallwood owns about 11 [ Predicate %] of Integra. NomLex abandonment: … abatement: … abduction: … aberration: … ability: … abolition: … abomination: … NomAttributePartitiveRelationalBackoff Hallwood owns about 11 [ Predicate %] [ Arg1 of Integra].

17. Preliminary results and analysis Evaluation configuration Training instances: WSJ 2-21 Testing instances: WSJ 23 Automatically generated parse trees for training and testing Key observations Overall performance Per-class performance Class-based gains over baseline

18. Overall evaluation results Per-class evaluation results

19. General observations Negligible overall gains compared to Liu and Ng (2007), who reported overall f-measure of 0.7283 Some NomLex classes perform very well Classes introduce gains as well as losses

20. Analysis: intra-class regularity Hypothesis 1: classes may exhibit consistent realizations of their arguments Relational class (F1=90.94) Regularity: argument incorporation [ Arg2 Mr. Hunt’s] [ Arg0/Predicate attorney] said his client welcomed the gamble. 100% of Relational nominalizations have an incorporated Arg0 Constitutes 38% of test arguments for the class

21. Analysis: intra-class regularity Hypothesis 1: classes may exhibit consistent realizations of their arguments Partitive class (F1=79.85) Regularity: presence of Arg0 86% of Partitive instances take a single Arg0 Compare: 15% of Nom instances take a single Arg1

22. Analysis: class-based gains Hypothesis 2: modeling each class separately may result in more homogeneous training data and better SRL performance Improvements ClassTest instancesImprovement Nom-like7982.06 Environment1083.97 Group405.87 Job306.29

23. Analysis: class-based gains Hypothesis 2: modeling each class separately may result in more homogeneous training data and better SRL performance Losses ClassTest instancesLossClass ambiguityTraining instances Share4220.8398.5366 of 5211 total Nom-adj-like285.9390.56400 of 5086 total

24. Conclusions and future work NomBank SRL based on classes derived from NomLex Demonstrates negligible gains over Liu and Ng (2007) Intra-class regularity leads to modest gains in some classes NomLex ambiguity causes losses in others

25. Conclusions and future work In-depth class modeling Identification of class-specific regularities not captured by the current feature set Further partitioning of the Nom class? NomLex class disambiguation

26. Thanks! Any questions?

27. References Carreras, X. & Màrquez, L. (2005), 'Introduction to the CoNLL-2005 Shared Task: Semantic Role Labeling'. Dahl, D. A.; Palmer, M. S. & Passonneau, R. J. (1987), Nominalizations in PUNDIT, in 'Proceedings of the 25th annual meeting on Association for Computational Linguistics', Association for Computational Linguistics, Morristown, NJ, USA, pp. 131--139. Girju, R.; Nakov, P.; Nastase, V.; Szpakowicz, S.; Turney, P. & Yuret, D. (2007), SemEval- 2007 Task 04: Classification of Semantic Relations between Nominals, in 'Proceedings of the 4th International Workshop on Semantic Evaluations'. Hull, R. & Gomez, F. (1996), Semantic Interpretation of Nominalizations, in 'Proceedings of AAAI'. Jiang, Z. & Ng, H. (2006), Semantic Role Labeling of NomBank: A Maximum Entropy Approach, in 'Proceedings of the 2006 Conference on Empirical Methods in Natural Language Processing'. Kaisser, M. & Webber, B. (2007), Question Answering based on Semantic Roles, in 'ACL 2007 Workshop on Deep Linguistic Processing', Association for Computational Linguistics, Prague, Czech Republic, pp. 41--48. Kingsbury, P. & Palmer, M. (2003), Propbank: the next level of treebank, in 'Proceedings of Treebanks and Lexical Theories'. Lapata, M. (2000), The Automatic Interpretation of Nominalizations, in 'Proceedings of the Seventeenth National Conference on Artificial Intelligence and Twelfth Conference on Innovative Applications of Artificial Intelligence', AAAI Press / The MIT Press,, pp. 716-- 721.

28. References (cont’d) Liu, C. & Ng, H. (2007), Learning Predictive Structures for Semantic Role Labeling of NomBank, in 'Proceedings of the 45th Annual Meeting of the Association of Computational Linguistics', Association for Computational Linguistics, Prague, Czech Republic, pp. 208-- 215. Macleod, C.; Grishman, R.; Meyers, A.; Barrett, L. & Reeves, R. (1998), Nomlex: A lexicon of nominalizations, in 'Proceedings of the Eighth International Congress of the European Association for Lexicography'. Marcus, M.; Santorini, B. & Marcinkiewicz, M. A. (1993), 'Building a large annotated corpus of English: the Penn TreeBank', Computational Linguistics 19, 313-330. Meyers, A. (2007), 'Annotation Guidelines for NomBank - Noun Argument Structure for PropBank', Technical report, New York University. Meyers, A.; Macleod, C.; Yangarber, R.; Grishman, R.; Barrett, L. & Reeves, R. (1998), Using NOMLEX to produce nominalization patterns for information extraction, in 'Proceedings of the COLING-ACL Workshop on the Computational Treatment of Nominals'. Ponzetto, S. P. & Strube, M. (2006), Exploiting semantic role labeling, WordNet and Wikipedia for coreference resolution, in 'Proceedings of the main conference on Human Language Technology Conference of the North American Chapter of the Association of Computational Linguistics', Association for Computational Linguistics, Morristown, NJ, USA, pp. 192--199. Shen, D. & Lapata, M. (2007), Using Semantic Roles to Improve Question Answering, in 'Proceedings of the Conference on Empirical Methods in Natural Language Processing and on Computational Natural Language Learning', pp. 12-21. Surdeanu, M.; Harabagiu, S.; Williams, J. & Aarseth, P. (2003), Using predicate-argument structures for information extraction, in 'Proceedings of the 41st Annual Meeting on Association for Computational Linguistics'.