1. Natural Language Tools and Resources for Biomedical Information Extraction Yoshimasa Tsuruoka Tsujii laboratory University of Tokyo

2. Outline NLP resources for bioNLP –GENIA corpus NLP tools –Machine learning Maximum entropy modeling for feature forest Maximum entropy modeling with inequality constraints –Part-of-speech tagger –Chunker (shallow parser) –HPSG Parser Applications of NLP –Extracting disease-gene relationships from MEDLINE abstracts

3. Application of NLP to the Biomedical domain Plenty of text –MEDLINE database: 12 million abstracts –Needs of effective IE and IR Domain knowledge –Gene ontology, KEGG, UMLS, ICD, … Other Information sources –Molecular databases DNA sequences, motifs, diseases, molecular interactions, etc…

4. Developing NLP resources Resources for NLP research –Domain knowledge –Training data for ML-based techniques –Test data for evaluating the transferability of a system GENIA resources –Ontology –Corpus

5. GENIA corpus 4,000 MEDLINE abstracts –Selected by MeSH Terms (Human, Blood cells, Transcription factors) XML format Contents –Named-entity (Kim et al 2003) –Part-of-speech (Tateisi et al 2004) –Parse tree –Co-reference (Institute of Infocomm Research, Singapore)

6. GENIA part-of-speech corpus Each token is annotated with its part-of-speech tag. Size –2,000 abstracts –20,544 sentences –50,1054 words (about half the size of Penn Treebank) The peri-kappa B site mediates human immunodeficiency virus type 2 enhancer activation in monocytes … DT NN NN NN VBZ JJ NN NN NN CD NN NN IN NNS

7. The peri-kappa B site mediates human immunodeficiency virus type 2 enhancer activation in monocytes … GENIA named-entity corpus Terms are annotated based on the semantic classes in the GENIA ontology Size –2,000 abstracts –Number of the terms: 92,723 –Vocabulary size: 36,568 DNA virus cell_type

8. GENIA treebank Based on the standard of the Penn TreeBank Size –500 abstracts –(1500 abstracts by the end of this summer) CD3-episilon expression is controlled by a downstream T lymphocyte-specific enhancer element NP ADJP NP PP VP S

9. Few known genes (IL-2, members of the IL-8 family, interferon-gamma) are induced in T cells only through the combined effect of phorbol myristic acetatete (PMA) and a Ca(2+)-ionophore, and expression of only these genes can be fully suppressed by Cyclosporin A (CyA). T cell IL-2 Interferon-gamma IL-8 family IL-2 IL-8 IFN-γ Ca(2+)-i PMA Ca(2+)-i PMA Ca(2+)-i PMA CyA × × × Target Interaction Agent Location Event Annotation Few known genes (IL-2, members of the IL-8 family, interferon-gamma) are induced in T cells only through the combined effect of phorbol myristic acetatete (PMA) and a Ca(2+)-ionophore, and expression of only these genes can be fully suppressed by Cyclosporin A (CyA).

10. T cell IL-2 Interferon-gamma IL-8 family IL-2 IL-8 IFN-γ Ca(2+)-i PMA Ca(2+)-i PMA Ca(2+)-i PMA Target Interaction Agent Location Event annotation

11. Few known genes (IL-2, members of the IL-8 family, interferon-gamma) are induced in T cells only through the combined effect of phorbol myristic acetatete (PMA) and a Ca(2+)-ionophore, and expression of only these genes can be fully suppressed by Cyclosporin A (CyA). T cell IL-2 Interferon-gamma IL-8 family IL-2 IL-8 IFN-γ CyA × × × Target Interaction Agent Location Event annotation

12. GENIA corpus Used in more than 240 institutions –Japan (28), Asia (54), North America (63), Europe (62), etc… De facto standard for evaluating biomedical named- entity recognition systems –BioNLP workshop at Coling 2004 Named-entity recognition shared task –Institute for Infocomm Research (Singapore), –Stanford University (USA), –University of Edinburgh (UK), –University of Wisconsin-Madison (USA), –Pohang University of Science and Technology (Korea), –University of Alberta (Canada), –University Duisburg-Essen (Germany), –Korea University (Korea), –National Taiwan University (Taiwan),

13. NLP tools Biomedical text mining –Huge amount of text. Machine learning –Training set can be very large. –Efficient training algorithms. Taggers (and parsers) –Decoding should be fast.

14. Machine learning Supervised learning –learns the rules for classifying samples into predefined classes by seeing a large number of training samples with class labels. Algorithms –Naïve Bayes, Decision Tree, SVMs, AdaBoost, Perceptron, Random forests, Maximum Entropy, etc...

15. Maximum entropy learning Log-linear modeling Maximum likelihood estimation –determines the parameters so that they maximizes the likelihood of the training data Feature function Feature weight

16. Maximum entropy modeling with inequality constraints (Kazama and Tsujii 2003) Advantages over the standard ME modeling. –Good regularization effects (as good as Gaussian prior) –Sparse solution C++ implementation –offers fast training. –can be used as a library. –can incorporate the model into your source code. The C++ library is used in many NLP programs (e.g. POS tagger, chunkers, IE modules)

17. Part-of-speech tagging A PoS tagger annotates each token with its part-of- speech tag. The peri-kappa B site mediates human immunodeficiency DT NN NN NN VBZ JJ NN virus type 2 enhancer activation in monocytes … NN NN CD NN NN IN NNS

18. Chunking (shallow parsing) A chunker (shallow parser) segments a sentence into non-recursive phrases. He reckons the current account deficit will narrow to NP VP NP VP PP only # 1.8 billion in September. NP PP NP

19. Chunking (shallow parsing) Chunking tasks can be converted into a standard tagging task. He reckons the current account deficit will narrow to B NP B VP B NP I NP I NP I NP B VP I VP B PP only # 1.8 billion in September. B NP I NP I NP I NP B PP B NP

20. Sequential Classification Approaches Sequence tagging tasks –Find the tag sequence that maximizes the following probability given the observation (e.g. words): Left to right decomposition (with the first-order markov assumption) Right to left decomposition (with the first-order markov assumption) classification problem

21. Bidirectional Inference Possible decomposition structures Bidirectional inference algorithm (Tsuruoka et al.) –We can find the best structure and tag sequences in polynomial time t1t1 t2t2 t3t3 (a) t1t1 t2t2 t3t3 (b) t1t1 t2t2 t3t3 (c) t1t1 t2t2 t3t3 (d)

22. State-of-the-art PoS taggers Tagging speed and accuracy on Penn Treebank Tagging SpeedAccuracy Dependency Net (2003)Very slow97.24 Perceptron (2002)?97.11 SVM (2003)Fast97.05 HMM (2000)Extremely fast96.48 Bidirectional MEMMVery fast97.10

23. State-of-the-art Chunkers Chunking speed an accuracy on Penn Treebank Tagging SpeedAccuracy Perceptron (2003)?93.74 SVM + voting (2003)Slow?93.91 SVM (2000)Fast93.48 Bidirectional MEMMVery fast93.70

24. The peri-kappa B site mediates human immunodeficiency virus type 2 enhancer activation in monocytes … Named-entity recognition Recognizing named-entities in text Similar to chunking –IOB tagging Named entities in the biomedical domain are long. –Sliding window DNA virus cell_type

25. A sliding window approach to biomedical NE recogition We want to use rich features on a term. Enumerate all sub-word sequences in a sentence. Classify them into semantic classes. W1W2W3W4W1W2W3W4

26. Accuracy of biomedical NE recognition RecallPrecisionF-score SVM+HMM (Zho 2004)76.069.472.6 Sliding window71.570.270.8 MEMM (Fin 2004)71.668.670.1 CRF (Set 2004)70.369.369.8 Shared task at Coling 2004 BioNLP workshop

27. HPSG parsing HPSG –A few schema –Many lexical entries –Deep syntactic analysis Grammar –Corpus-based grammar construction (Miyao et al 2004) Parser –Beam search (Tsuruoka et al.) Lexical entry HEAD: verb SUBJ: <> COMPS: <> Mary walked slowly HEAD: noun SUBJ: <> COMPS: <> HEAD: verb SUBJ: COMPS: <> HEAD: adv MOD: verb HEAD: verb SUBJ: COMPS: <> Subject-head schema Head-modifier schema

28. Phrase structure The company is run by him DT NN VBZ VBN IN PRP dt np vp vp pp np np pp vp s

29. Predicate-argument structure The company is run by him DT NN VBZ VBN IN PRP dt np vp vp pp np np pp vp s arg1 arg2 mod

30. IR search engine using predicate- argument structures

31. A maximum entropy model is defined for the entire tree structure –e.g. HPSG parse trees Exponentially-many trees are represented with a packed forest of polynomial size A probability of each tree is estimated without unpacking the feature forest Feature forest model (Miyao and Tsujii 2002) S NP 1 NP 2 VP 1 VP 2 number of trees: size: feature forest

32. Automatic Generation of Spelling Variants Variant Generator NF-Kappa B(1.0) NF Kappa B (0.9) NF kappa B(0.6) NF kappaB(0.5) NFkappaB(0.3) : Generator NF-Kappa B Each generated variant is associated with its generation probability

33. Generation Algorithm T cell (1.0) T-cell (0.5)T cells (0.2) T-cells (0.1) 0.5 0.2 Recursive generation P = P x P op

34. Learning Operation Rules Operations for generating variants –Substitution –Deletion –Insertion Context –Character-level context: preceding (following) two characters Operation Probability

35. Example of variant generation (1) Generation Probability Generated VariantsFrequency 1.0 (input)antiinflammatory effect7 0.462anti-inflammatory effect33 0.393antiinflammatory effects6 0.356Antiinflammatory effect0 0.286antiinflammatory-effect0 0.181anti-inflammatory effects23 :::

36. Example of variant generation (2) Generation Probabilitiy Generated VariantsFrequency 1.0 (Input)tumour necrosis factor alpha15 0.492tumor necrosis factor alpha126 0.356tumour necrosis factor-alpha30 0.235Tumour necrosis factor alpha2 0.175tumor necrosis factor alpha182 0.115Tumor necrosis factor alpha8 :::

37. Domain Adaptation Newspaper articles are widely used as training data for machine learning-based NLP systems. Domain Adaptability –Part-of-speech tagging –HPSG parsing

38. Tagging errors by TnT tagger (Brants 2000) … and membrane potential after mitogen binding. CC NN NN IN NN JJ … two factors, which bind to the same kappa B enhancers… CD NNS WDT NN TO DT JJ NN NN NNS … by analysing the Ag amino acid sequence. IN VBG DT VBG JJ NN NN … to contain more T-cell determinants than … TO VB RBR JJ NNS IN Stimulation of interferon beta gene transcription in vitro by NN IN JJ JJ NN NN IN NN IN

39. Accuracy of TnT tagger on the GENIA corpus Ignoring unessential errors Accuracy TnT (original)84.4% NNP = NN, NNPS = NNS90.0% LS = NN91.3% JJ = NN94.9% About 94% in practice

40. GENIA tagger training WSJGENIA WSJ97.084.3 GENIA75.298.1 WSJ+GENIA96.998.1 An MEMM tagger trained on WSJ and GENIA corpus The tagger works well on both types of texts.

41. Parsing MEDLINE with the HPSG parser Parsing accuracy on the GENIA Treebank #sentencesLP / LRUP / UR All sentences1,55682.8 / 81.586.4 / 85.1 Covered sentences1,10486.8 / 86.588.7 / 88.4

42. Extracting Disease-Gene Associations from MEDLINE abstracts These results suggested that targeted disruption of Cyp19 caused anovulation and precocious depletion of ovarian follicles Furthermore, AML cells with methylated p15(INAK4B) tended to express higher levels of DNMT1 and 3B.

43. Text 1.5 million MEDLINE abstracts –Selected by MeSH Terms Disease Category AND (Amino Acids, Peptides, and Proteins OR Genetic Structures) Parsing –All the sentences were parsed by the HPSG parser –Using a PC cluster (100 processors with GXP) –Time: 10 days

44. Training data All foals with OLWS were homozygous for the Ile118Lys EDNRB mutation, and adults that were homozygous were not found. Dominant radial drusen and Arg345Trp EFEMP1 mutation. The 5 year overall survival (OS) and event-free survival (EFS) were 94 and 90 +/- 8%, respectively, with a median follow-up of 48 months. These data may indicate that formation of parathyroid adenoma in young patients is related to a mechanism involving EGFR. All co-occurrences are classified into relevant or irrelevant by a domain expert.

45. Predicate-argument features (1) Dedifferentiation of adenoid cystic carcinoma: report of a case implicating p53 gene mutation. X gene/disease ARG2

46. Predicate-argument features (2) These results suggested that targeted disruption of Cyp19 caused anovulation and precocious depletion of ovarian follicles. Furthermore, AML cells with methylated p15(INAK4B) tended to express higher levels of DNMT1 and 3B. X disease/gene ARG2ARG1 gene/disease

47. Extraction accuracy Training/test data: 2,253 sentences 10-fold cross validation featuresrecallprecisionf-score N/A1.00.3510.520 + bag of words0.7330.6820.706 + local context0.7330.6950.714 + predicate-argument structures 0.7590.7100.733

48. DGA explorer

49. Summary The GENIA corpus –Part-of-speech: 2000 abstracts –Named-entities: 2000 abstracts –Parse tree: 500 abstracts Machine learning –Maximum entropy modeling Inequality constraints Feature forests –Bidirectional inference for sequence tagging NLP tools –Part-of-speech tagger: 97.11% –Chunker: 93.7% –HPSG parser: 87.5% –Term variant generation Extracting disease-gene associations from MEDLINE

50. Software and resources Machine learning packages –Maximum entropy with inequality constraints –Maximum entropy for feature forests Taggers and Parsers –PoS tagger –Chunker –Named-entity tagger –HPSG parser GENIA resource –Named-entity corpus –Part-of-speech corpus –Tree corpus –Co-reference corpus (Singapore Univ.) –HPSG parsed results (100,000 MEDLINE abstracts)