Presentation on theme: "Natural Language Tools and Resources for Biomedical Information Extraction Yoshimasa Tsuruoka Tsujii laboratory University of Tokyo."— Presentation transcript:
Natural Language Tools and Resources for Biomedical Information Extraction Yoshimasa Tsuruoka Tsujii laboratory University of Tokyo
Outline NLP resources for bioNLP –GENIA corpus NLP tools –Machine learning Maximum entropy modeling for feature forest Maximum entropy modeling with inequality constraints –Part-of-speech tagger –Chunker (shallow parser) –HPSG Parser Applications of NLP –Extracting disease-gene relationships from MEDLINE abstracts
Application of NLP to the Biomedical domain Plenty of text –MEDLINE database: 12 million abstracts –Needs of effective IE and IR Domain knowledge –Gene ontology, KEGG, UMLS, ICD, … Other Information sources –Molecular databases DNA sequences, motifs, diseases, molecular interactions, etc…
Developing NLP resources Resources for NLP research –Domain knowledge –Training data for ML-based techniques –Test data for evaluating the transferability of a system GENIA resources –Ontology –Corpus
GENIA corpus 4,000 MEDLINE abstracts –Selected by MeSH Terms (Human, Blood cells, Transcription factors) XML format Contents –Named-entity (Kim et al 2003) –Part-of-speech (Tateisi et al 2004) –Parse tree –Co-reference (Institute of Infocomm Research, Singapore)
GENIA part-of-speech corpus Each token is annotated with its part-of-speech tag. Size –2,000 abstracts –20,544 sentences –50,1054 words (about half the size of Penn Treebank) The peri-kappa B site mediates human immunodeficiency virus type 2 enhancer activation in monocytes … DT NN NN NN VBZ JJ NN NN NN CD NN NN IN NNS
The peri-kappa B site mediates human immunodeficiency virus type 2 enhancer activation in monocytes … GENIA named-entity corpus Terms are annotated based on the semantic classes in the GENIA ontology Size –2,000 abstracts –Number of the terms: 92,723 –Vocabulary size: 36,568 DNA virus cell_type
GENIA treebank Based on the standard of the Penn TreeBank Size –500 abstracts –(1500 abstracts by the end of this summer) CD3-episilon expression is controlled by a downstream T lymphocyte-specific enhancer element NP ADJP NP PP VP S
Few known genes (IL-2, members of the IL-8 family, interferon-gamma) are induced in T cells only through the combined effect of phorbol myristic acetatete (PMA) and a Ca(2+)-ionophore, and expression of only these genes can be fully suppressed by Cyclosporin A (CyA). T cell IL-2 Interferon-gamma IL-8 family IL-2 IL-8 IFN-γ Ca(2+)-i PMA Ca(2+)-i PMA Ca(2+)-i PMA CyA × × × Target Interaction Agent Location Event Annotation Few known genes (IL-2, members of the IL-8 family, interferon-gamma) are induced in T cells only through the combined effect of phorbol myristic acetatete (PMA) and a Ca(2+)-ionophore, and expression of only these genes can be fully suppressed by Cyclosporin A (CyA).
Few known genes (IL-2, members of the IL-8 family, interferon-gamma) are induced in T cells only through the combined effect of phorbol myristic acetatete (PMA) and a Ca(2+)-ionophore, and expression of only these genes can be fully suppressed by Cyclosporin A (CyA). T cell IL-2 Interferon-gamma IL-8 family IL-2 IL-8 IFN-γ CyA × × × Target Interaction Agent Location Event annotation
GENIA corpus Used in more than 240 institutions –Japan (28), Asia (54), North America (63), Europe (62), etc… De facto standard for evaluating biomedical named- entity recognition systems –BioNLP workshop at Coling 2004 Named-entity recognition shared task –Institute for Infocomm Research (Singapore), –Stanford University (USA), –University of Edinburgh (UK), –University of Wisconsin-Madison (USA), –Pohang University of Science and Technology (Korea), –University of Alberta (Canada), –University Duisburg-Essen (Germany), –Korea University (Korea), –National Taiwan University (Taiwan),
NLP tools Biomedical text mining –Huge amount of text. Machine learning –Training set can be very large. –Efficient training algorithms. Taggers (and parsers) –Decoding should be fast.
Machine learning Supervised learning –learns the rules for classifying samples into predefined classes by seeing a large number of training samples with class labels. Algorithms –Naïve Bayes, Decision Tree, SVMs, AdaBoost, Perceptron, Random forests, Maximum Entropy, etc...
Maximum entropy learning Log-linear modeling Maximum likelihood estimation –determines the parameters so that they maximizes the likelihood of the training data Feature function Feature weight
Maximum entropy modeling with inequality constraints (Kazama and Tsujii 2003) Advantages over the standard ME modeling. –Good regularization effects (as good as Gaussian prior) –Sparse solution C++ implementation –offers fast training. –can be used as a library. –can incorporate the model into your source code. The C++ library is used in many NLP programs (e.g. POS tagger, chunkers, IE modules)
Part-of-speech tagging A PoS tagger annotates each token with its part-of- speech tag. The peri-kappa B site mediates human immunodeficiency DT NN NN NN VBZ JJ NN virus type 2 enhancer activation in monocytes … NN NN CD NN NN IN NNS
Chunking (shallow parsing) A chunker (shallow parser) segments a sentence into non-recursive phrases. He reckons the current account deficit will narrow to NP VP NP VP PP only # 1.8 billion in September. NP PP NP
Chunking (shallow parsing) Chunking tasks can be converted into a standard tagging task. He reckons the current account deficit will narrow to B NP B VP B NP I NP I NP I NP B VP I VP B PP only # 1.8 billion in September. B NP I NP I NP I NP B PP B NP
Sequential Classification Approaches Sequence tagging tasks –Find the tag sequence that maximizes the following probability given the observation (e.g. words): Left to right decomposition (with the first-order markov assumption) Right to left decomposition (with the first-order markov assumption) classification problem
Bidirectional Inference Possible decomposition structures Bidirectional inference algorithm (Tsuruoka et al.) –We can find the best structure and tag sequences in polynomial time t1t1 t2t2 t3t3 (a) t1t1 t2t2 t3t3 (b) t1t1 t2t2 t3t3 (c) t1t1 t2t2 t3t3 (d)
State-of-the-art PoS taggers Tagging speed and accuracy on Penn Treebank Tagging SpeedAccuracy Dependency Net (2003)Very slow97.24 Perceptron (2002)?97.11 SVM (2003)Fast97.05 HMM (2000)Extremely fast96.48 Bidirectional MEMMVery fast97.10
State-of-the-art Chunkers Chunking speed an accuracy on Penn Treebank Tagging SpeedAccuracy Perceptron (2003)?93.74 SVM + voting (2003)Slow?93.91 SVM (2000)Fast93.48 Bidirectional MEMMVery fast93.70
The peri-kappa B site mediates human immunodeficiency virus type 2 enhancer activation in monocytes … Named-entity recognition Recognizing named-entities in text Similar to chunking –IOB tagging Named entities in the biomedical domain are long. –Sliding window DNA virus cell_type
A sliding window approach to biomedical NE recogition We want to use rich features on a term. Enumerate all sub-word sequences in a sentence. Classify them into semantic classes. W1W2W3W4W1W2W3W4
Accuracy of biomedical NE recognition RecallPrecisionF-score SVM+HMM (Zho 2004)76.069.472.6 Sliding window71.570.270.8 MEMM (Fin 2004)71.668.670.1 CRF (Set 2004)70.369.369.8 Shared task at Coling 2004 BioNLP workshop
Phrase structure The company is run by him DT NN VBZ VBN IN PRP dt np vp vp pp np np pp vp s
Predicate-argument structure The company is run by him DT NN VBZ VBN IN PRP dt np vp vp pp np np pp vp s arg1 arg2 mod
IR search engine using predicate- argument structures
A maximum entropy model is defined for the entire tree structure –e.g. HPSG parse trees Exponentially-many trees are represented with a packed forest of polynomial size A probability of each tree is estimated without unpacking the feature forest Feature forest model (Miyao and Tsujii 2002) S NP 1 NP 2 VP 1 VP 2 number of trees: size: feature forest
Automatic Generation of Spelling Variants Variant Generator NF-Kappa B(1.0) NF Kappa B (0.9) NF kappa B(0.6) NF kappaB(0.5) NFkappaB(0.3) : Generator NF-Kappa B Each generated variant is associated with its generation probability
Generation Algorithm T cell (1.0) T-cell (0.5)T cells (0.2) T-cells (0.1) 0.5 0.2 Recursive generation P = P x P op
Learning Operation Rules Operations for generating variants –Substitution –Deletion –Insertion Context –Character-level context: preceding (following) two characters Operation Probability
Example of variant generation (1) Generation Probability Generated VariantsFrequency 1.0 (input)antiinflammatory effect7 0.462anti-inflammatory effect33 0.393antiinflammatory effects6 0.356Antiinflammatory effect0 0.286antiinflammatory-effect0 0.181anti-inflammatory effects23 :::
Domain Adaptation Newspaper articles are widely used as training data for machine learning-based NLP systems. Domain Adaptability –Part-of-speech tagging –HPSG parsing
Tagging errors by TnT tagger (Brants 2000) … and membrane potential after mitogen binding. CC NN NN IN NN JJ … two factors, which bind to the same kappa B enhancers… CD NNS WDT NN TO DT JJ NN NN NNS … by analysing the Ag amino acid sequence. IN VBG DT VBG JJ NN NN … to contain more T-cell determinants than … TO VB RBR JJ NNS IN Stimulation of interferon beta gene transcription in vitro by NN IN JJ JJ NN NN IN NN IN
Accuracy of TnT tagger on the GENIA corpus Ignoring unessential errors Accuracy TnT (original)84.4% NNP = NN, NNPS = NNS90.0% LS = NN91.3% JJ = NN94.9% About 94% in practice
GENIA tagger training WSJGENIA WSJ97.084.3 GENIA75.298.1 WSJ+GENIA96.998.1 An MEMM tagger trained on WSJ and GENIA corpus The tagger works well on both types of texts.
Parsing MEDLINE with the HPSG parser Parsing accuracy on the GENIA Treebank #sentencesLP / LRUP / UR All sentences1,55682.8 / 81.586.4 / 85.1 Covered sentences1,10486.8 / 86.588.7 / 88.4
Extracting Disease-Gene Associations from MEDLINE abstracts These results suggested that targeted disruption of Cyp19 caused anovulation and precocious depletion of ovarian follicles Furthermore, AML cells with methylated p15(INAK4B) tended to express higher levels of DNMT1 and 3B.
Text 1.5 million MEDLINE abstracts –Selected by MeSH Terms Disease Category AND (Amino Acids, Peptides, and Proteins OR Genetic Structures) Parsing –All the sentences were parsed by the HPSG parser –Using a PC cluster (100 processors with GXP) –Time: 10 days
Training data All foals with OLWS were homozygous for the Ile118Lys EDNRB mutation, and adults that were homozygous were not found. Dominant radial drusen and Arg345Trp EFEMP1 mutation. The 5 year overall survival (OS) and event-free survival (EFS) were 94 and 90 +/- 8%, respectively, with a median follow-up of 48 months. These data may indicate that formation of parathyroid adenoma in young patients is related to a mechanism involving EGFR. All co-occurrences are classified into relevant or irrelevant by a domain expert.
Predicate-argument features (1) Dedifferentiation of adenoid cystic carcinoma: report of a case implicating p53 gene mutation. X gene/disease ARG2
Predicate-argument features (2) These results suggested that targeted disruption of Cyp19 caused anovulation and precocious depletion of ovarian follicles. Furthermore, AML cells with methylated p15(INAK4B) tended to express higher levels of DNMT1 and 3B. X disease/gene ARG2ARG1 gene/disease
Extraction accuracy Training/test data: 2,253 sentences 10-fold cross validation featuresrecallprecisionf-score N/A1.00.3510.520 + bag of words0.7330.6820.706 + local context0.7330.6950.714 + predicate-argument structures 0.7590.7100.733