1. Automatically Acquiring a Linguistically Motivated Genic Interaction Extraction System Mark A. Greenwood Mark Stevenson Yikun Guo Henk Harkema Angus Roberts Natural Language Processing Group Department of Computer Science University of Sheffield, UK

2. August 7th 2005LLL05: Automatically Acquiring a Linguistically Motivated Genic Interaction Extraction System Outline of Talk The Challenge Extraction Patterns Acquiring And Using Extracting Patterns Challenge Evaluation Analysis Conclusions and Future Work

3. August 7th 2005LLL05: Automatically Acquiring a Linguistically Motivated Genic Interaction Extraction System The Challenge The challenge is to extract Genic Interactions from biomedical texts, such as MedLine abstracts.  A genic interaction involve genes and proteins  The interactions are directional but no guarantee that genes and proteins always fill the same slot. GerE stimulates cotD transcription and inhibits cotA transcription in vitro by sigma K RNA polymerase, as expected from in vivo studies, and, unexpectedly, profoundly inhibits in vitro transcription of the gene (sigK) that encode sigma K.  6 genes and proteins mentioned  Five pairs interact: GerE  cotD, GerE  cotA, sigma K  cotA, GerE  SigK and sigK  sigma K

4. August 7th 2005LLL05: Automatically Acquiring a Linguistically Motivated Genic Interaction Extraction System Outline of Talk The Challenge Extraction Patterns Acquiring And Using Extracting Patterns Challenge Evaluation Analysis Conclusions and Future Work

5. August 7th 2005LLL05: Automatically Acquiring a Linguistically Motivated Genic Interaction Extraction System We represent extraction patterns as paths in a dependency tree  Dependency trees represent text by linking each sentence word with those words which directly modify it.  For example the noun phrase “the brown dog” is represented by two dependency relations:  In these experiments we used MINIPAR (Lin, 1999) to generate the dependency trees from which the extraction patterns were taken. Extraction Patterns the browndog det adj

6. August 7th 2005LLL05: Automatically Acquiring a Linguistically Motivated Genic Interaction Extraction System Extraction Patterns Given the dependency tree representing the phrase “…AGENT represses the transcription of TARGET…” we extract chain shaped paths as extraction patterns. verb[v/repress](subj[n/AGENT]) verb[v/repress](obj[n/transcription](of[n/TARGET])) verb[v/repress](obj[n/transcription]+subj[n/AGENT]) verb[v/repress](obj[n/transcription](of[n/TARGET])+subj[n/AGENT])

7. August 7th 2005LLL05: Automatically Acquiring a Linguistically Motivated Genic Interaction Extraction System Extraction Patterns The nodes in the dependency trees can be either:  Lexical items (i.e. words)  Semantic categories such as gene, protein, agent, target, etc. Lexical items are represented in lower case Semantic categories are capitalised For example in the pattern verb[v/transcribe](subj[n/GENE]+obj[n/PROTEIN]) transcribe is a lexical item and GENE and PROTEIN are semantic categories.

8. August 7th 2005LLL05: Automatically Acquiring a Linguistically Motivated Genic Interaction Extraction System Outline of Talk The Challenge Extraction Patterns Acquiring And Using Extracting Patterns Challenge Evaluation Analysis Conclusions and Future Work

9. August 7th 2005LLL05: Automatically Acquiring a Linguistically Motivated Genic Interaction Extraction System Learning Extraction Patterns Iterative Learning Algorithm 1.Begin with set of seed patterns which are known to be good extraction patterns 2.Compare every other pattern with the ones known to be good 3.Choose the highest scoring of these and add them to the set of good patterns 4.Stop if enough patterns have been learned, else repeat from step 2. Seeds Candidates Rank Patterns

10. August 7th 2005LLL05: Automatically Acquiring a Linguistically Motivated Genic Interaction Extraction System Pattern Similarity We determine the similarity between two patterns using a vector space model inspired by that commonly used in IR.  Each pattern can be represented by a set of pattern element-filler pairs  The set of pattern element-filler pairs in a corpus forms the basis for a vector space where the value is 1 if a pattern contains the pair, 0 otherwise. The similarity of two patterns can then be computed as: This is the cosine measure augmented with a matrix W which lists the similarity between each pattern element-filler pair.  The similarity between pattern element-filler pairs is computed using a WordNet similarity measure proposed by Banerjee and Pederson (2002) referred to as Adapted Lesk.

11. August 7th 2005LLL05: Automatically Acquiring a Linguistically Motivated Genic Interaction Extraction System Pattern Similarity Extraction Patterns a. verb[v/block](subj[n/protein]) b. verb[v/repress](subj[n/enzyme]) c. verb[v/promote](subj[n/protein]) Matrix Labels 1. subj_protein, 2. subj_enzyme, 3. verb_block, 4. verb_repress, 5. verb_promote 10.95000 1000 0010.90.1 000.910.1 00 1 Similarity Values sim(a, b) = 0.925 sim(a, c) = 0.55 sim(b, c) = 0.525 Similarity Matrix

12. August 7th 2005LLL05: Automatically Acquiring a Linguistically Motivated Genic Interaction Extraction System Acquiring Patterns We use this approach to learn patterns containing a known agent or target from the training data. The texts are pre-processed to include AGENT and TARGET as semantic class labels. We restricted certain terms (e.g. repress) so that only certain domain specific senses in WordNet were used for similarity calculations. At each iteration of the algorithm we accepted up to 4 new patterns which were within 0.95 of the best pattern being accepted. The algorithm was allowed to run until no more patterns could be acquired.

13. August 7th 2005LLL05: Automatically Acquiring a Linguistically Motivated Genic Interaction Extraction System Seed Patterns We used the following seed patterns in all experiments: verb[v/transcribe](by[n/AGENT]+obj[n/TARGET]) verb[v/be](of[n/AGENT]+s[n/expression](of[n/TARGET])) verb[v/inhibit](obj[n/activity](nn[n/TARGET])+subj[n/AGENT]) verb[v/bind](mod[r/specifically](to[n/TARGET])+subj[n/AGENT]) verb[v/block](obj[n/capacity](of[n/TARGET])+subj[n/AGENT]) verb[v/regulate](obj[n/expression](nn[n/TARGET])+subj[n/AGENT]) verb[v/require](obj[n/AGENT]+subj[n/gene](nn[n/TARGET])) verb[v/repress](obj[n/transcription](of[n/TARGET])+subj[n/AGENT])

14. August 7th 2005LLL05: Automatically Acquiring a Linguistically Motivated Genic Interaction Extraction System Extracting Relations Text from which we wish to extract relations is processed to produce extraction patterns in the same way as before. Any pattern which matches an acquired pattern is used to extract information.  The acquired patterns match with AGENT and TARGET matching anything  Not all patterns contain both an AGENT and TARGET so post- processing links part relations together. So for example  The pattern verb[v/stimulates](subj[n/AGENT]+obj[n/TARGET])  Matches against verb[v/stimulates](subj[n/GerE]+obj[n/cotD])  Resulting in the interaction GerE  cotD

15. August 7th 2005LLL05: Automatically Acquiring a Linguistically Motivated Genic Interaction Extraction System Outline of Talk The Challenge Extraction Patterns Acquiring And Using Extracting Patterns Challenge Evaluation Analysis Conclusions and Future Work

16. August 7th 2005LLL05: Automatically Acquiring a Linguistically Motivated Genic Interaction Extraction System Challenge Evaluation We submitted three runs for evaluation  Baseline: A simple baseline system which pairs all dictionary elements in a sentence with each other in both orders.  Basic: A system trained on the basic data set without coreference as provided for the LLL-05 challenge.  Expanded: A system trained on the basic data set augmented with 78 automatically acquired weakly labelled (Craven & Kumlien, 1999) MedLine sentences. The basic and expanded systems differ only in the training data used to acquire the extraction patterns.

17. August 7th 2005LLL05: Automatically Acquiring a Linguistically Motivated Genic Interaction Extraction System Challenge Evaluation SystemPrecisionRecallF-measure Baseline10.6% (53/500)98.1% (53/54)19.1% Basic22.2% (6/27)11.1% (6/54)14.8% Expanded21.6% (8/37)14.8% (8/54)17.5% The baseline system did not achieve 100% recall as some constructs, such as “… A activates or represses B…” requires two interactions between A and B to be recognised. Both approaches have low recall but a precision twice that of the baseline system. While the performance is low it seems that supplying extra training data improves the performance of our approach.

18. August 7th 2005LLL05: Automatically Acquiring a Linguistically Motivated Genic Interaction Extraction System Outline of Talk The Challenge Extraction Patterns Acquiring And Using Extracting Patterns Challenge Evaluation Analysis Conclusions and Future Work

19. August 7th 2005LLL05: Automatically Acquiring a Linguistically Motivated Genic Interaction Extraction System Analysis If we examine the algorithm at each iteration instead of just the final result we can see that:  The seed patterns are unable to extract a single interaction, i.e. the initial F-measure is zero.  As the seeds do not extract relations the performance of the system is solely due to the acquired patterns.  The algorithm is fairly resilient to the acquisition of bad patterns, i.e. with few exceptions, the F- measure steadily increases.

20. August 7th 2005LLL05: Automatically Acquiring a Linguistically Motivated Genic Interaction Extraction System Analysis Previously our implementation had been used only to perform sentence filtering (Stevenson & Greenwood, 2005), i.e. determining if a given sentence contains an interaction or not. Using the acquired patterns to perform sentence filtering results in an F-measure of 47.5%. Given the small amount of training data (181 sentences) this looks promising.  Nédellec et al. (2001) reported F- measure of 80% over similar data but using 900 training sentences.

21. August 7th 2005LLL05: Automatically Acquiring a Linguistically Motivated Genic Interaction Extraction System Outline of Talk The Challenge Extraction Patterns Acquiring And Using Extracting Patterns Challenge Evaluation Analysis Conclusions and Future Work

22. August 7th 2005LLL05: Automatically Acquiring a Linguistically Motivated Genic Interaction Extraction System Conclusions & Future Work We used a pattern representation based on dependency trees and an iterative algorithm to learn representative patterns.  The seed patterns were not well suited to the task and future work will include experimenting with different seed sets.  The small amount of training data seems to hinder our approach (adding 78 extra sentences saw a 2.7% increase in F-measure) The similarity measure we adopted seems well suited to this task where similar meaning can be conveyed in different ways. Other issues for future work:  We used MINIPAR to produce the dependency trees. We intend to try other dependency parser to see if they are more suited to biomedical texts.  We intend to continue our work on sentence filtering as this would provide a useful first step in any extraction system.

23. Any Questions? Copies of these slides can be found at: http://www.dcs.shef.ac.uk/~mark/nlp/pubs/

24. August 7th 2005LLL05: Automatically Acquiring a Linguistically Motivated Genic Interaction Extraction System Bibliography Satanjeev Banerjee and Ted Pedersen. An Adapted Lesk Algorithm for Word Sense Disambiguation Using WordNet. In Proceedings of the Fourth International Conference on Computational Linguistics and Intelligent Text Processing (CICLING-02), 2002. Mark Craven and Johan Kumlien. Constructing Biological Knowledge Bases by Extracting Information from Text Sources. In Proceedings of the Seventh International Conference on Intelligent Systems for Molecular Biology, 1999. Dekan Lin. MINIPAR: a minimalist parser. Maryland Linguistics Colloquium. University of Maryland, College Park. 1999. Claire Nédellec and Mohamed Ould Abdel Vetah and Philippe Bessières. Sentence Filtering for Information Extraction in Genomics, a Classification Problem. In Proceedings of the Conference on Practical Knowledge Discovery in Databases (PKDD'2001), 2001. Mark Stevenson and Mark A. Greenwood. A Semantic Approach to IE Pattern Induction. In Proceedings of the 43rd Annual Meeting of the Association for Computational Linguistics (ACL-05), 2005.