1. Semantic Relation Detection in Bioscience Text Marti Hearst SIMS, UC Berkeley http://biotext.berkeley.edu Supported by NSF DBI-0317510 and a gift from Genentech

2. BioText Project Goals Provide flexible, intelligent access to information for use in biosciences applications. Focus on Textual Information from Journal Articles Tightly integrated with other resources Ontologies Record-based databases

3. Project Team Project Leaders: PI: Marti Hearst Co-PI: Adam Arkin Computational Linguistics Barbara Rosario Presley Nakov Database Research Ariel Schwartz Gaurav Bhalotia (graduated) Supported primarily by NSF DBI-0317510 and a gift from Genentech User Interface / IR Adam Newberger Dr. Emilia Stoica Bioscience Dr. TingTing Zhang Janice Hamerja

4. BioText Architecture Sophisticated Text Analysis Annotations in Database Improved Search Interface

5. The Nature of Bioscience Text Claim: Bioscience semantics are simultaneously easier and harder than general text. Fewer subtleties Fewer ambiguities “Systematic” meanings Enormous terminology Complex sentence structure easierharder

6. Sample Sentence “Recent research, in proliferating cells, has demonstrated that interaction of E2F1 with the p53 pathway could involve transcriptional up-regulation of E2F1 target genes such as p14/p19ARF, which affect p53 accumulation [67,68], E2F1-induced phosphorylation of p53 [69], or direct E2F1- p53 complex formation [70].”

7. BioScience Researchers Read A LOT! Cite A LOT! Curate A LOT! Are interested in specific relations, e.g.: What is the role of this protein in that pathway? Show me articles in which a comparison between two values is significant.

8. This Talk Discovering semantic relations Between nouns in noun compounds Between entities in sentences Acquiring labeled data: Idea: use text surrounding citations to documents to identify paraphrases A new direction; preliminary work only

9. Noun Compound Relation Recognition

10. Noun Compounds(NCs) Technical text is rich with NCs Open-labeled long-term study of the subcutaneous sumatriptan efficacy and tolerability in acute migraine treatment. NC is any sequence of nouns that itself functions as a noun asthma hospitalizations health care personnel hand wash

11. NCs: 3 computational tasks Identification Syntactic analysis (attachments) [Baseline [headache frequency]] [[Tension headache] patient] Our Goal: Semantic analysis Headache treatment  treatment for headache Corticosteroid treatment  treatment that uses corticosteroid

12. Descent of Hierarchy Idea: Use the top levels of a lexical hierarchy to identify semantic relations Hypothesis: A particular semantic relation holds between all 2-word NCs that can be categorized by a lexical category pair.

13. Related work ( Semantic analysis of NCs ) Rule-based Finin (1980) Detailed AI analysis, hand-coded Vanderwende (1994) automatically extracts semantic information from an on-line dictionary, manipulates a set of handwritten rules. 13 classes, 52% accuracy Probabilistic Lauer (1995): probabilistic model, 8 classes, 47% accuracy Lapata (2000) classifies nominalizations into subject/object. 2 classes, 80% accuracy

14. Related work ( Semantic analysis of NCs ) Lexical Hierarchy Barrett et al. (2001) WordNet, heuristics to classify a NC given the similarity to a known NC Rosario and Hearst (2001) Relations pre-defined MeSH, Neural Network. 18 classes, 60% accuracy

15. Linguistic Motivation Can cast NC into head-modifier relation, and assume head noun has an argument and qualia structure. (used-in): kitchen knife (made-of): steel knife (instrument-for): carving knife (used-on): putty knife (used-by): butcher’s knife

16. The lexical Hierarchy: MeSH 1. Anatomy [A] 2. Organisms [B] 3. Diseases [C] 4. Chemicals and Drugs [D] 5. Analytical, Diagnostic and Therapeutic Techniques and Equipment [E] 6. Psychiatry and Psychology [F] 7. Biological Sciences [G] 8. Physical Sciences [H] 9. Anthropology, Education, Sociology and Social Phenomena [I] 10. Technology and Food and Beverages [J] 11. Humanities [K] 12. Information Science [L] 13. Persons [M] 14. Health Care [N] 15. Geographic Locations [Z]

17. The lexical Hierarchy: MeSH 1. Anatomy [A] Body Regions [A01] 2. [B] Musculoskeletal System [A02] 3. [C] Digestive System [A03] 4. [D] Respiratory System [A04] 5. [E] Urogenital System [A05] 6. [F] …… 7. [G] 8. Physical Sciences [H] 9. [I] 10. [J] 11. [K] 12. [L] 13. [M]

18. Descending the Hierarchy 1. Anatomy [A] Body Regions [A01] Abdomen [A01.047] 2. [B] Musculoskeletal System [A02] Back [A01.176] 3. [C] Digestive System [A03] Breast [A01.236] 4. [D] Respiratory System [A04] Extremities [A01.378] 5. [E] Urogenital System [A05] Head [A01.456] 6. [F] …… Neck [A01.598] 7. [G] …. 8. Physical Sciences [H] 9. [I] 10. [J] 11. [K] 12. [L] 13. [M]

19. Descending the Hierarchy 1. Anatomy [A] Body Regions [A01] Abdomen [A01.047] 2. [B] Musculoskeletal System [A02] Back [A01.176] 3. [C] Digestive System [A03] Breast [A01.236] 4. [D] Respiratory System [A04] Extremities [A01.378] 5. [E] Urogenital System [A05] Head [A01.456] 6. [F] …… Neck [A01.598] 7. [G] …. 8. Physical Sciences [H] Electronics 9. [I] Astronomy 10. [J] Nature 11. [K] Time 12. [L] Weights and Measures 13. [M] ….

20. Descending the Hierarchy 1. Anatomy [A] Body Regions [A01] Abdomen [A01.047] 2. [B] Musculoskeletal System [A02] Back [A01.176] 3. [C] Digestive System [A03] Breast [A01.236] 4. [D] Respiratory System [A04] Extremities [A01.378] 5. [E] Urogenital System [A05] Head [A01.456] 6. [F] …… Neck [A01.598] 7. [G] …. 8. Physical Sciences [H] Electronics Amplifiers 9. [I] Astronomy Electronics, Medical 10. [J] Nature Transducers 11. [K] Time 12. [L] Weights and Measures 13. [M] ….

21. Descending the Hierarchy 1. Anatomy [A] Body Regions [A01] Abdomen [A01.047] 2. [B] Musculoskeletal System [A02] Back [A01.176] 3. [C] Digestive System [A03] Breast [A01.236] 4. [D] Respiratory System [A04] Extremities [A01.378] 5. [E] Urogenital System [A05] Head [A01.456] 6. [F] …… Neck [A01.598] 7. [G] …. 8. Physical Sciences [H] Electronics Amplifiers 9. [I] Astronomy Electronics, Medical 10. [J] Nature Transducers 11. [K] Time 12. [L] Weights and Measures Calibration 13. [M] …. Metric System Reference Standard

22. Descending the Hierarchy 1. Anatomy [A] Body Regions [A01] Abdomen [A01.047] 2. [B] Musculoskeletal System [A02] Back [A01.176] 3. [C] Digestive System [A03] Breast [A01.236] 4. [D] Respiratory System [A04] Extremities [A01.378] 5. [E] Urogenital System [A05] Head [A01.456] 6. [F] …… Neck [A01.598] 7. [G] …. 8. Physical Sciences [H] Electronics Amplifiers 9. [I] Astronomy Electronics, Medical 10. [J] Nature Transducers 11. [K] Time 12. [L] Weights and Measures Calibration 13. [M] …. Metric System Reference Standard Homogeneous Heterogeneous

23. Mapping Nouns to MeSH Concepts headache recurrence C23.888.592.612.441 C23.550.291.937 headache pain C23.888.592.612.441 G11.561.796.444

24. Levels of Description headache pain Level 0: C.23 G.11 Level 1: C23.888 G11.561 Level 1: C23.888.592 G11.561.796 … Original: C23.888.592.612.441 G11.561.796.444

25. Descent of Hierarchy Idea: Words falling in homogeneous MeSH subhierarchies behave “similarly” with respect to relation assignment Hypothesis: A particular semantic relation holds between all 2-word NCs that can be categorized by a MeSH category pairs

26. Grouping the NCs CP: A02 C04 (Musculoskeletal System, Neoplasms) skull tumors, bone cysts, bone metastases, skull osteosarcoma… CP: C04 M01 (Neoplasms, Person) leukemia survivor, lymphoma patients, cancer physician, cancer nurses…

27. Distribution of Category Pairs

28. Collection ~70,000 NCs extracted from titles and abstracts of Medline 2,627 CPs at level 0 (with at least 10 unique NCs) We analyzed 250 CPs with Anatomy (A) 21 CPs with Natural Science (H01) 3 CPs with Neoplasm (C04) This represents 10% of total CPs and 20% of total NCs

29. For each CP Divide its NCs into “training-testing” sets “Training”: inspect NCs by hand Start from level 0 0 While NCs are not all similar descend one level of the hierarchy Repeat until all NCs for that CP are similar Classification Method

30. Classification Decisions A02 C04 B06 B06 C04 M01 C04 M01.643 C04 M01.526 A01 H01 A01 H01.770 A01 H01.671 A01 H01.671.538 A01 H01.671.868 A01 M01 A01 M01.643 A01 M01.526 A01 M01.898

31. Classification Decisions + Relations A02 C04  Location of Disease B06 B06  Kind of Plants C04 M01 C04 M01.643  Person afflicted by Disease C04 M01.526  Person who treats Disease A01 H01 A01 H01.770 A01 H01.671 A01 H01.671.538 A01 H01.671.868 A01 M01 A01 M01.643 A01 M01.526 A01 M01.898

32. Classification Decisions + Relations A02 C04  Location of Disease B06 B06  Kind of Plants C04 M01 C04 M01.643  Person afflicted by Disease C04 M01.526  Person who treats Disease A01 H01 A01 H01.770 A01 H01.671 A01 H01.671.538 A01 H01.671.868 A01 M01 A01 M01.643  Person afflicted by Disease A01 M01.526 A01 M01.898

33. Classification Decision Levels Anatomy: 250 CPs 187 (75%) remain first level 56 (22%) descend one level 7 (3%) descend two levels Natural Science (H01): 21 CPs 1 ( 4%) remain first level 8 (39%) descend one level 12 (57%) descend two levels Neoplasms (C04) 3 CPs: 3 (100%) descend one level

34. Evaluation Test the decisions on “testing” set Count how many NCs that fall in the groups defined in the classification decisions are similar to each other Accuracy (for 2 nd noun): Anatomy: 91% Natural Science: 79% Neoplasm: 100% Total Accuracy : 90.8% Generalization: our 415 classification decisions cover ~ 46,000 possible CP pairs

35. Ambiguity – Two Types Lexical ambiguity: mortality state of being mortal death rate Relationship ambiguity: bacteria mortality death of bacteria death caused by bacteria

36. Four Cases Single MeSH sensesMultiple MeSH senses Only one possible relationship: abdomen radiography, aciclovir treatment Multiple relationships: hospital databases, education efforts, kidney metabolism Only one possible relationship: alcoholism treatment Ambiguity of relationship Multiple relationships bacteria mortality

37. Four Cases Single MeSH sensesMultiple MeSH senses Only one possible relationship: abdomen radiography, aciclovir treatment Multiple relationships: hospital databases, education efforts, kidney metabolism Only one possible relationship: alcoholism treatment Ambiguity of relationship Multiple relationships bacteria mortality Most problematic cases … but rare!

38. Conclusions on NN Relation Classification Very simple method for assigning semantic relations to two-word technical NCs 90.8% accuracy Lexical resource (MeSH) useful for this task Probably works because of the relative lack of ambiguity in this kind of technical text.

39. Entity-Entity Relation Recognition

40. Problem: Which relations hold between 2 entities? TreatmentDisease Cure? Prevent? Side Effect?

41. Hepatitis Examples Cure These results suggest that con A-induced hepatitis was ameliorated by pretreatment with TJ-135. Prevent A two-dose combined hepatitis A and B vaccine would facilitate immunization programs Vague Effect of interferon on hepatitis B

42. Two tasks Relationship Extraction: Identify the several semantic relations that can occur between the entities disease and treatment in bioscience text Entity extraction: Related problem: identify such entities

43. The Approach Data: MEDLINE abstracts and titles Graphical models Combine in one framework both relation and entity extraction Both static and dynamic models Simple discriminative approach: Neural network Lexical, syntactic and semantic features

44. Related Work We allow several DIFFERENT relations between the same entities Thus differs from the problem statement of other work on relations Many find one relation which holds between two entities (many based on ACE) Agichtein and Gravano (2000), lexical patterns for location of Zelenko et al. (2002) SVM for person affiliation and organization-location Hasegawa et al. (ACL 2004) Person-Organization -> President “relation” Craven (1999, 2001) HMM for subcellular-location and disorder-association Doesn’t identify the actual relation

45. Related work: Bioscience Many hand-built rules Feldman et al. (2002), Friedman et al. (2001) Pustejovsky et al. (2002) Saric et al.; this conference

46. Data and Relations MEDLINE, abstracts and titles 3662 sentences labeled Relevant: 1724 Irrelevant: 1771 e.g., “Patients were followed up for 6 months” 2 types of Entities, many instances treatment and disease 7 Relationships between these entities

47. Semantic Relationships 810: Cure Intravenous immune globulin for recurrent spontaneous abortion 616: Only Disease Social ties and susceptibility to the common cold 166: Only Treatment Flucticasone propionate is safe in recommended doses 63: Prevent Statins for prevention of stroke

48. Semantic Relationships 36: Vague Phenylbutazone and leukemia 29: Side Effect Malignant mesodermal mixed tumor of the uterus following irradiation 4: Does NOT cure Evidence for double resistance to permethrin and malathion in head lice

49. Features Word Part of speech Phrase constituent Orthographic features ‘is number’, ‘all letters are capitalized’, ‘first letter is capitalized’ … MeSH (semantic features) Replace words, or sequences of words, with generalizations via MeSH categories Peritoneum -> Abdomen

50. Models 2 static generative models 3 dynamic generative models 1 discriminative model (neural network)

51. Static Graphical Models S1: observations dependent on Role but independent from Relation given roles S2: observations dependent on both Relation and Role S1S2

52. Dynamic Graphical Models D1, D2 as in S1, S2 D3: only one observation per state is dependent on both the relation and the role D1 D2 D3

53. Graphical Models Relation node: Semantic relation (cure, prevent, none..) expressed in the sentence

54. Graphical Models Role nodes: 3 choices: treatment, disease, or none

55. Graphical Models Feature nodes (observed): word, POS, MeSH…

56. Graphical Models Different dependencies between the features and the relation nodes D3 D1 S1 D2 S2

57. Graphical Models For Dynamic Model D1: Joint probability distribution over relation, roles and features nodes Parameters estimated with maximum likelihood and absolute discounting smoothing

58. Neural Network Feed-forward network (MATLAB) Training with conjugate gradient descent One hidden layer (hyperbolic tangent function) Logistic sigmoid function for the output layer representing the relationships Same features Discriminative approach

59. Role extraction Results in terms of F-measure Graphical models Junction tree algorithm (BNT) Relation hidden and marginalized over Neural Net Couldn’t run it (features vectors too large) (Graphical models can do role extraction and relationship classification simultaneously)

60. Role Extraction: Results F-measures D1 best when no smoothing

61. Role Extraction: Results F-measures D2 best with smoothing, but doesn’t boost scores as much as in relation classification

62. Role Extraction: Results Static models better than Dynamic for Note: No Neural Networks

63. Relation classification: Results With Smoothing and Roles, D1 best GM

64. Features impact: Role Extraction Most important features: 1)Word, 2)MeSH Models D1 D2 All features 0.67 0.71 No word 0.58 0.61 -13.4% -14.1% No MeSH 0.63 0.65 -5.9% -8.4% (rel. + irrel.)

65. Most important features: Roles Accuracy: D1 D2 NN All feat. + roles 91.6 82.0 96.9 All feat. – roles 68.9 74.9 79.6 -24.7% -8.7% -17.8% All feat. + roles – Word 91.6 79.8 96.4 0% -2.8% -0.5% All feat. + roles – MeSH 91.6 84.6 97.3 0% 3.1% 0.4% Features impact: Relation classification (rel. + irrel.)

66. Relation extraction Results in terms of classification accuracy (with and without irrelevant sentences) 2 cases: Roles hidden Roles given Graphical models NN: simple classification problem

67. Relation classification: Results Neural Net always best

68. Relation classification: Results With Smoothing and No Roles, D2 best GM

69. Relation classification: Results Dynamic models always outperform Static

70. Relation classification: Results With no smoothing, D1 best Graphical Model

71. Relation classification: Confusion Matrix Computed for the model D2, “rel + irrel.”, “only features”

72. Features impact: Relation classification Most realistic case: Roles not known Most important features: 1) Mesh 2) Word for D1 and NN (but vice versa for D2) Accuracy: D1 D2 NN All feat. – roles 68.9 74.9 79.6 All feat. - roles – Word 66.7 66.1 76.2 -3.3% -11.8% -4.3% All feat. - roles – MeSH 62.7 72.5 74.1 -9.1% -3.2% -6.9% (rel. + irrel.)

73. Relation Recognition: Conclusions Classification of subtle semantic relations in bioscience text Discriminative model (neural network) achieves high classification accuracy Graphical models for the simultaneous extraction of entities and relationships Importance of lexical hierarchy Next Step: Different entities/relations Semi-supervised learning to discover relation types

74. Acquiring Labeled Data using Citances

75. A discovery is made … A paper is written …

76. That paper is cited … and cited … … as the evidence for some fact(s) F.

77. Each of these in turn are cited for some fact(s) … … until it is the case that all important facts in the field can be found in citation sentences alone!

78. Citances Nearly every statement in a bioscience journal article is backed up with a cite. It is quite common for papers to be cited 30-100 times. The text around the citation tends to state biological facts. (Call these citances.) Different citances will state the same facts in different ways … … so can we use these for creating models of language expressing semantic relations?

79. Using Citances Potential uses of citation sentences (citances) creation of training and testing data for semantic analysis, synonym set creation, database curation, document summarization, and information retrieval generally. Some preliminary results: Citances to a document align well with a hand-built curation. Citances are good candidates for paraphrase creation.

80. Citances for Acquiring Examples of Semantic Relations A relationship type R between entities of type A and B can be expressed in many ways. Use citances to build a model the different ways to express the relationship: Seed learning algorithms with examples that mention A and B, for which relation R holds. Train a model to recognize R when the relation is not known. Results may extend to sentences that are not citances as well.

81. Issues for Processing Citances Text span Identification of the appropriate phrase, clause, or sentence that constructs a citance. Correct mapping of citations when shown as lists or groups (e.g., “[22-25]”). Grouping citances by topic Citances that cite the same document should be grouped by the facts they state. Normalizing or paraphrasing citances For IR, summarization, learning synonyms, relation extraction, question answering, and machine translation.

82. Related Work Traditional citation analysis dates back to the 1960’s (Garfield). Includes: Citation categorization, Context analysis, Citer motivation. Citation indexing systems, such as ISI’s SCI, and CiteSeer. Mercer and Di Marco (2004) propose to improve citation indexing using citation types. Bradshaw (2003) introduces Reference Directed Indexing (RDI), which indexes documents using the terms in the citances citing them.

83. Related Work (cont.) Teufel and Moens (2002) identify citances to improve summarization of the citing paper.. Nanba et. al. (2000) use citances as features for classifying papers into topics. Related field to citation indexing is the use of link structure and anchor text of Web pages. Applications include: IR, classification, Web crawlers, and summarization.

84. Example: protein-protein

85. Early results: Paraphrase Creation from Citances

86. Sample Sentences NGF withdrawal from sympathetic neurons induces Bim, which then contributes to death. Nerve growth factor withdrawal induces the expression of Bim and mediates Bax dependent cytochrome c release and apoptosis. The proapoptotic Bcl-2 family member Bim is strongly induced in sympathetic neurons in response to NGF withdrawal. In neurons, the BH3 only Bcl2 member, Bim, and JNK are both implicated in apoptosis caused by nerve growth factor deprivation.

87. Their Paraphrases NGF withdrawal induces Bim. Nerve growth factor withdrawal induces the expression of Bim. Bim has been shown to be upregulated following nerve growth factor withdrawal. Bim implicated in apoptosis caused by nerve growth factor deprivation. They all paraphrase: Bim is induced after NGF withdrawal.

88. Paraphrase Creation Algorithm 1. Extract the sentences that cite the target. 2. Mark the NEs of interest (genes/proteins, MeSH terms) and normalize. 3. Dependency parse (MiniPar). 4. For each parse For each pair of NEs of interest i. Extract the path between them. ii. Create a paraphrase from the path. 5. Rank the candidates for a given pair of NEs. 6. Select only the ones above a threshold. 7. Generalize.

89. Creating a Paraphrase Given the path from the dependency parse: Restore the original word order. Add words to improve grammaticality. Bim … shown … be … following nerve growth factor withdrawal. Bim [has] [been] shown [to] be [upregulated] following nerve growth factor withdrawal.

90. 2-word Heuristic Demonstration NGF withdrawal induces Bim. Nerve growth factor withdrawal induces [the] expression of Bim. Bim [has] [been] shown [to] be [upregulated] following nerve growth factor withdrawal. Bim [is] induced in [sympathetic] neurons in response to NGF withdrawal. member Bim implicated in apoptosis caused by nerve growth factor deprivation.

91. Evaluation (1) An influential journal paper from Neuron: J. Whitfield, S. Neame, L. Paquet, O. Bernard, and J. Ham. Dominantnegative c-jun promotes neuronal survival by reducing bim expression and inhibiting mitochondrial cytochrome c release. Neuron, 29:629–643, 2001. 99 journal papers citing it 203 citances in total 36 different types of important biological factoids But we concentrated on one model sentence: “Bim is induced after NGF withdrawal.”

92. Evaluation (2) Set 1: 67 citances pointing to the target paper and manually found to contain a good or acceptable paraphrase (do not necessarily contain Bim or NGF); (Ideal conditions) Set 2: 65 citances pointing to the target paper and containing both Bim and NGF; Set 3: 102 sentences from the 99 texts, containing both Bim and NGF (Do citances do better than arbitrarily chosen sentences?)

93. Correctness (Judgments) Bad (0.0), if: different relation (often phosphorylation aspect); opposite meaning; vagueness (wording not clear enough). Acceptable (0.5), If it was not Bad and: contains additional terms (e.g., DP5 protein) or topics (e.g., PPs like in sympathetic neurons); the relation was suggested but not definitely. Else Good (1.0)

94. Results Obtained 55, 65 and 102 paraphrases for sets 1, 2 and 3 Only one paraphrase from each sentence comparison of the dependency path to that of the model sentence % - good (1.0) or acceptable (0.5)

95. Correctness (Recall) Calculated on Set 1 60 paraphrases (out of 67 citances) 5 citances produced 2 paraphrases system recall: 55/67, i.e. 82.09% 10 of the 67 relevant in Set 1 initially missed by the human annotator 8 good, 2 acceptable. human recall is 57/67, i.e. 85.07%

96. Misses Sample system miss (no NGF): Growth factor withdrawal was shown to cause increased Bim expression in various populations of neuronal cell types. Sample human miss: The precise targets of c-Jun necessary for the induction of apoptosis have been the subject of intense interest and recently, Bim and Dp5, both “BH3-domain only” family members, have been identified as pro-apoptotic genes induced in a c- Jun-dependent manner in both sympathetic neurons subjected to NGF withdrawal and in cerebellar granule cells deprived of KCl.

97. Grammaticality Missing coordinating “and”: “Hrk/DP5 Bim [have] [been] found [to] be upregulated after NGF withdrawal” Verb subcategorization “caused by NGF role for Bim” Extra subject words member Bim implicated in apoptosis caused by NGF deprivation sentence: “In neurons, the BH3-only Bcl2 member, Bim, and JNK are both implicated in apoptosis caused by NGF deprivation.”

98. Related Work Word-level paraphrases. Grefenstette uses a semantic parser to compare the distributional similarity of local contexts for synonyms extraction. Phrase-level paraphrases. Barzilay&McKeown use POS information from the local context and co- training. Template paraphrases. Lin&Pantel apply the idea of Grefenstette to dependency tree paths. Later refined by Shinyama&al. Sentence-level paraphrases. Barzilay&Lee use multiple sequence alignment. Pang&al. merge parse trees into a transducer.

99. Relevant Papers Citances: Citation Sentences for Semantic Analysis of Bioscience Text, Preslav Nakov, Ariel Schwartz, and Marti Hearst, in the SIGIR'04 workshop on Search and Discovery in Bioinformatics. Classifying Semantic Relations in Bioscience Text, Barbara Rosario and Marti Hearst, in ACL 2004. The Descent of Hierarchy, and Selection in Relational Semantics, Barbara Rosario, Marti Hearst, and Charles Fillmore, in ACL 2002.

100. Thank you! Marti Hearst SIMS, UC Berkeley http://biotext.berkeley.edu

101. Additional slides

102. Our D1 Thompson et al. 2003 Frame classification and role labeling for FrameNet sentences Target word must be observed More relations and roles

103. Smoothing: absolute discounting Lower the probability of seen events by subtracting a constant from their count (ML estimate: ) The remaining probability is evenly divided by the unseen events

104. F-measures for role extraction in function of smoothing factors

105. Relation accuracies in function of smoothing factors