Presentation is loading. Please wait.

Presentation is loading. Please wait.

Lexical Tools Briefing The Lexical Systems Group NLMNLM. LHNCBC. CGSBLHNCBCCGSB June, 2006.

Published byAlexia Norris Modified over 8 years ago

Similar presentations

Presentation on theme: "Lexical Tools Briefing The Lexical Systems Group NLMNLM. LHNCBC. CGSBLHNCBCCGSB June, 2006."— Presentation transcript:

1 Lexical Tools Briefing The Lexical Systems Group NLMNLM. LHNCBC. CGSBLHNCBCCGSB June, 2006

2 Introduction Lexical Tools Lvg Norm Text Categorization Questions Table of Contents

3 Introduction

4 Introduction - LB

5 Introduction - Lexicon

6 Introduction - LC

7 Introduction - LA

8 Introduction - Numbers

9 Introduction - SCRT

10 Introduction – Lexical Tools

11 Introduction - GSpell

12 Introduction – Text Tools

13 Introduction - TC

14 Lexical Tools Lexical Tools A suite of text utilities

15 Lexical Tools Input Lexical Tools A suite of text utilities take the given input

16 Lexical Tools Input Output… Output.3 Output.2 Output.1 Lexical Tools A suite of text utilities that generate, mutate, and filter out lexical variants from the given input

17 Four Tools Input Output… Output.3 Output.2 Output.1 Lvg Norm LuiNorm WordIndex

18 Tool Types Command line tools –lvg (Lexical Variants Generation)lvg –normnorm –luiNormluiNorm –wordIndwordInd Lexical Gui Tool (lgt)Lexical Gui Tool Web Tools Java API’s

19 Functions Used in nature language processing for –aggressive text pattern matching –creating normalized and expanded terms –making word, term, phrase indexes –matching queries with indexed entries –increasing recall and/or precision

20 Facts Release annually 100% Java (since 2002) Free distributed with open source code Run on different platforms One complete package Documents & support

21 Lexical Variants Generation

22 LVG, 2006 58 flow componentsflow components 37 options –input filter options (3)input filter options –global behavior options (13)global behavior options –flow specific options (2)flow specific options –output filter options (19)output filter options

23 Flow Components leave leaves leaving left inflect

24 Command Line Tool > lvg –f:i leave leave|leave|128|1|i|1|1281 leave|leave|128|512|i|1| leave|leaves|128|8|i|1| leave|left|1024|64|i|1| leave|left|1024|32|i|1| leave|leave|1024|1|i|1| leave|leave|1024|262144|i|1| leave|leave|1024|1024|i|1| leave|leaves|1024|128|i|1| leave|leaving|1024|16|i|1|

25 Fielded Output Input Term Output Term Categories Inflections Flow history Flow Number leave 128 1 1 i | || | | > lvg –f:i leave

26 A Serial Flow Input term Remove possessive lowercase Strip punctuation Remove stop words Strip diacritics Word order sort Output term Flow components can be arranged so that the output of one is the input to another.

27 A Serial Flow - Example > lvg –f:l:q:g:t:p:w The Gougerot-Sjögren's Syndrome The Gougerot-Sjögren's Syndrome| gougerotsjogren syndrome|2047| 16777215|l+q+g+t+p+w|1|

28 Parallel Flows Input term Output term Multiple flows can be defined noOperation Uninflect synonyms Output terms

29 Parallel Flows - Example > lvg –f:n –f:B:y ear ear|ear|2047|1048575|n|1| ear|aural|1|1|B+y|2| ear|auricularis|1|1|B+y|2| ear|otic|1|1|B+y|2| ear|otor|1|1|B+y|2|

30 Input Filter Options Output terms Input term > lvg -f:u -t:7 -F:8:6 C0035440|ENG|S|L0035434|VW|S0003894| Rheumatic carditis, acute acute Rheumatic carditis|S0003894 Take field 7 from the input

31 Global Behavior Options Output terms Input term Output terms > lvg -f:L –f:E –s:”\” otitis otitis\otitis\128\513\L\1 otitis\E0044452\128\513\E\2 Change separator to “\”

32 Output Filter Options > lvg -f:L -SC -SI hot hot|hot| |<base+positive+infin itive+pres1p23p>|L|1| Show the category and inflection names Output terms Input term

33 Composed of 11 Lvg flow components to abstract away from: –case –punctuation –possessive forms –inflections –spelling variants –stop words –diacritics & ligatures –word order Norm

34 g: remove genitives t: strip stop words o: replace punctuation with spaces l: lowercase B: uninflect each words in a term w: sort words by order rs: remove parenthetic plural forms q: strip diacritics q2: split ligature Ct: retrieve citations q4: get symbol names synonymy

35 g: remove genitives t: strip stop words o: replace punctuation with spaces l: lowercase B: uninflect each words in a term w: sort words by order rs: remove parenthetic plural forms q: strip diacritics q2: split ligature Ct: retrieve citations q4: get symbol names synonymy Hodgkin's Diseases, NOS Norm

36 g: remove genitives t: strip stop words o: replace punctuation with spaces l: lowercase B: uninflect each words in a term w: sort words by order rs: remove parenthetic plural forms q: strip diacritics q2: split ligature Ct: retrieve citations q4: get symbol names synonymy Hodgkin Diseases, NOS Hodgkin's Diseases, NOS Norm

37 g: remove genitives t: strip stop words o: replace punctuation with spaces l: lowercase B: uninflect each words in a term w: sort words by order rs: remove parenthetic plural forms q: strip diacritics q2: split ligature Ct: retrieve citations q4: get symbol names synonymy Hodgkin Diseases, NOS Hodgkin's Diseases, NOS Hodgkin Diseases, NOS Norm

38 g: remove genitives t: strip stop words o: replace punctuation with spaces l: lowercase B: uninflect each words in a term w: sort words by order rs: remove parenthetic plural forms q: strip diacritics q2: split ligature Ct: retrieve citations q4: get symbol names synonymy Hodgkin Diseases, NOS Hodgkin's Diseases, NOS Hodgkin Diseases, NOS Hodgkin Diseases NOS Norm

39 g: remove genitives t: strip stop words o: replace punctuation with spaces l: lowercase B: uninflect each words in a term w: sort words by order rs: remove parenthetic plural forms q: strip diacritics q2: split ligature Ct: retrieve citations q4: get symbol names synonymy Hodgkin Diseases, NOS Hodgkin's Diseases, NOS Hodgkin Diseases, NOS Hodgkin Diseases NOS Hodgkin Diseases Norm

40 g: remove genitives t: strip stop words o: replace punctuation with spaces l: lowercase B: uninflect each words in a term w: sort words by order rs: remove parenthetic plural forms q: strip diacritics q2: split ligature Ct: retrieve citations q4: get symbol names synonymy Hodgkin Diseases, NOS Hodgkin's Diseases, NOS Hodgkin Diseases, NOS Hodgkin Diseases NOS Hodgkin Diseases Norm

41 g: remove genitives t: strip stop words o: replace punctuation with spaces l: lowercase B: uninflect each words in a term w: sort words by order rs: remove parenthetic plural forms q: strip diacritics q2: split ligature Ct: retrieve citations q4: get symbol names synonymy Hodgkin Diseases, NOS Hodgkin's Diseases, NOS Hodgkin Diseases, NOS Hodgkin Diseases NOS Hodgkin Diseases Norm

42 g: remove genitives t: strip stop words o: replace punctuation with spaces l: lowercase B: uninflect each words in a term w: sort words by order rs: remove parenthetic plural forms q: strip diacritics q2: split ligature Ct: retrieve citations q4: get symbol names synonymy Hodgkin Diseases, NOS Hodgkin's Diseases, NOS Hodgkin Diseases, NOS Hodgkin Diseases NOS Hodgkin Diseases hodgkin diseases Norm

43 g: remove genitives t: strip stop words o: replace punctuation with spaces l: lowercase B: uninflect each words in a term w: sort words by order rs: remove parenthetic plural forms q: strip diacritics q2: split ligature Ct: retrieve citations q4: get symbol names synonymy Hodgkin Diseases, NOS Hodgkin's Diseases, NOS Hodgkin Diseases, NOS Hodgkin Diseases NOS Hodgkin Diseases hodgkin diseases hodgkin disease Norm

44 g: remove genitives t: strip stop words o: replace punctuation with spaces l: lowercase B: uninflect each words in a term w: sort words by order rs: remove parenthetic plural forms q: strip diacritics q2: split ligature Ct: retrieve citations q4: get symbol names synonymy Hodgkin Diseases, NOS Hodgkin's Diseases, NOS Hodgkin Diseases, NOS Hodgkin Diseases NOS Hodgkin Diseases hodgkin diseases hodgkin disease Norm

45 g: remove genitives t: strip stop words o: replace punctuation with spaces l: lowercase B: uninflect each words in a term w: sort words by order rs: remove parenthetic plural forms q: strip diacritics q2: split ligature Ct: retrieve citations q4: get symbol names synonymy Hodgkin Diseases, NOS Hodgkin's Diseases, NOS Hodgkin Diseases, NOS Hodgkin Diseases NOS Hodgkin Diseases hodgkin diseases hodgkin disease disease hodgkin Norm

46 g: remove genitives t: strip stop words o: replace punctuation with spaces l: lowercase B: uninflect each words in a term w: sort words by order rs: remove parenthetic plural forms q: strip diacritics q2: split ligature Ct: retrieve citations q4: get symbol names synonymy Hodgkin Diseases, NOS Hodgkin's Diseases, NOS Hodgkin Diseases, NOS Hodgkin Diseases NOS Hodgkin Diseases hodgkin diseases hodgkin disease disease hodgkin Norm

47 Norm: Example disease hodgkin Hodgkin Disease HODGKINS DISEASE Hodgkin's Disease Disease, Hodgkin's HODGKIN'S DISEASE Hodgkin's disease Hodgkins Disease Hodgkin's disease NOS Hodgkin's disease, NOS Disease, Hodgkins Diseases, Hodgkins Hodgkins Diseases Hodgkins disease hodgkin's disease Disease;Hodgkins Disease, Hodgkin

48 Text Categorization Based on Journal Descriptor Indexing (JDI) methodology Uses a small set of high level descriptors, such as Journal Descriptors (JDs), Semantic Types (STs), Mesh subcategories, etc.. Used for categorize text, index contents, retrieve records, and word sense disambiguation

49 Text Categorization Free distributed with open source code 100 % in Java Run on different platforms One complete package Documents & support Provides Java APIs, command line tools, GUI tools, and Web tools Planned first release, TC 2007

50 Text Categorization Words Senses disambiguation (WSD) Free Text Metathesaurus Concept MetaMap (MMTX)

51 Text Categorization Words Senses disambiguation (WSD) Free Text Concept n Concept 2 Concept 1 MetaMap (MMTX)

52 Text Categorization Words Senses disambiguation (WSD) Free Text Concept n Concept 2 Concept 1 MetaMap (MMTX) TC Best Concept

53 Text Categorization Words Senses disambiguation (WSD) ….. transport... Patient Transport (ST: Health Care Activity) Biological Transport (ST: Cell Function) MetaMap (MMTX) TC Best Concept

54 Questions Lexical Systems Group: http://umlslex.nlm.nih.govhttp://umlslex.nlm.nih.gov Lexical Tools: http://umlslex.nlm.nih.gov/lvghttp://umlslex.nlm.nih.gov/lvg

55 Application Metathesaurus English Strings norm Normalized string index Normalized word index WordInd MRXNS.ENG MRXNW.ENG

56 Application norm Normalized string index Normalized word index Metathesaurus Concepts Query Normed term SUIS Metathesaurus concepts that match the normalized query

57 Example norm Query Normed term dry eye syndrome Dry Eyes Syndrome

58 ENG|dry eye syndrome|C0013238|L0013238|S0004019| ENG|dry eye syndrome|C0013238|L0013238|S0035652| ENG|dry eye syndrome|C0013238|L0013238|S0090228| ENG|dry eye syndrome|C0013238|L0013238|S0090454| ENG|dry eye syndrome|C0013238|L0013238|S0220550| ENG|dry eye syndrome|C0013238|L0013238|S0368350| ENG|dry eye syndrome|C0013238|L0013238|S1459074| Normed term SUIS Example (Cont.)

59 C0013238|ENG|P|L0013238|VS |S0004019|Dry eye syndrome C0013238|ENG|P|L0013238|VS |S0368350|Dry Eye Syndrome C0013238|ENG|P|L0013238|VS |S1459074|dry eye syndrome C0013238|ENG|P|L0013238|VWS|S0090228|Syndrome, Dry Eye C0013238|ENG|P|L0013238|VWS|S0220550|Dry, eye syndrome C0013238|ENG|P|L0013238|VW |S0090454|Syndromes, Dry Eye SUIS MRCON C0013238|ENG|P|L0013238|PF |S0035652| Dry Eye Syndromes Example (Cont.)

60 Questions Lexical Systems Group: http://umlslex.nlm.nih.govhttp://umlslex.nlm.nih.gov Lexical Tools: http://umlslex.nlm.nih.gov/lvghttp://umlslex.nlm.nih.gov/lvg

Download ppt "Lexical Tools Briefing The Lexical Systems Group NLMNLM. LHNCBC. CGSBLHNCBCCGSB June, 2006."

Similar presentations

LIS618 lecture 6 Thomas Krichel 2003-03-05. structure DIALOG –basic vs additional index –initial database file selection (files) Lexis/Nexis.

LIS618 lecture 6 Thomas Krichel structure DIALOG –basic vs additional index –initial database file selection (files) Lexis/Nexis.
Effective Searching Strategies and Techniques

Effective Searching Strategies and Techniques
Search Techniques Boolean Logic and Keyword Searching.

Search Techniques Boolean Logic and Keyword Searching.
Properties of Text CS336 Lecture 3:. 2 Generating Document Representations Want to automatically generate with little human intervention Use significant.

Properties of Text CS336 Lecture 3:. 2 Generating Document Representations Want to automatically generate with little human intervention Use significant.
PubMed and its search options Jan Emmerich, Sonja Jacobi, Kerstin Müller (5th Semester Library Management)

PubMed and its search options Jan Emmerich, Sonja Jacobi, Kerstin Müller (5th Semester Library Management)
Intelligent Information Retrieval CS 336 –Lecture 3: Text Operations Xiaoyan Li Spring 2006.

Intelligent Information Retrieval CS 336 –Lecture 3: Text Operations Xiaoyan Li Spring 2006.
U. S. National Library of Medicine NLM Indexing Initiative Tools for NLP: MetaMap and the Medical Text Indexer Natural Language Processing: State of the.

U. S. National Library of Medicine NLM Indexing Initiative Tools for NLP: MetaMap and the Medical Text Indexer Natural Language Processing: State of the.
Codifying Semantic Information in Medical Questions Using Lexical Sources Paul E. Pancoast Arthur B. Smith Chi-Ren Shyu.

Codifying Semantic Information in Medical Questions Using Lexical Sources Paul E. Pancoast Arthur B. Smith Chi-Ren Shyu.
The Role of the UMLS in Vocabulary Control CENDI Conference “Controlled Vocabulary and the Internet” Stuart J. Nelson, MD.

The Role of the UMLS in Vocabulary Control CENDI Conference “Controlled Vocabulary and the Internet” Stuart J. Nelson, MD.
Text Operations: Preprocessing. Introduction Document preprocessing –to improve the precision of documents retrieved –lexical analysis, stopwords elimination,

Text Operations: Preprocessing. Introduction Document preprocessing –to improve the precision of documents retrieved –lexical analysis, stopwords elimination,
Www.ei.org Engineering Village ™ ® Basic Searching On Compendex ®

Engineering Village ™ ® Basic Searching On Compendex ®
U. S. National Library of Medicine Welcome to the first MMTx User’s Group Meeting AMIA 2003 November 11, 2003.

U. S. National Library of Medicine Welcome to the first MMTx User’s Group Meeting AMIA 2003 November 11, 2003.
April 22, 20041 Text Mining: Finding Nuggets in Mountains of Textual Data Jochen Doerre, Peter Gerstl, Roland Seiffert IBM Germany, August 1999 Presenter:

April 22, Text Mining: Finding Nuggets in Mountains of Textual Data Jochen Doerre, Peter Gerstl, Roland Seiffert IBM Germany, August 1999 Presenter:
June 12, 2015 ©2005 Ovid Technologies Jörn Hope Ovid.

June 12, 2015 ©2005 Ovid Technologies Jörn Hope Ovid.
Brian A. Carlsen Apelon, Inc. Tools For Classification Integration Networked Knowledge Organization Systems/Services Workshop June 28, 2001.

Brian A. Carlsen Apelon, Inc. Tools For Classification Integration Networked Knowledge Organization Systems/Services Workshop June 28, 2001.
QuASI: Question Answering using Statistics, Semantics, and Inference Marti Hearst, Jerry Feldman, Chris Manning, Srini Narayanan Univ. of California-Berkeley.

QuASI: Question Answering using Statistics, Semantics, and Inference Marti Hearst, Jerry Feldman, Chris Manning, Srini Narayanan Univ. of California-Berkeley.
UCB BioText TREC 2003 Participation Participants: Marti Hearst Gaurav Bhalotia, Presley Nakov, Ariel Schwartz Track: Genomics, tasks 1 and 2.

UCB BioText TREC 2003 Participation Participants: Marti Hearst Gaurav Bhalotia, Presley Nakov, Ariel Schwartz Track: Genomics, tasks 1 and 2.
CSE 730 Information Retrieval of Biomedical Data The use of medical lexicon in biomedical IR.

CSE 730 Information Retrieval of Biomedical Data The use of medical lexicon in biomedical IR.
© 2013 The McGraw-Hill Companies, Inc. All rights reserved. Introduction to CPT Chapter Six.

© 2013 The McGraw-Hill Companies, Inc. All rights reserved. Introduction to CPT Chapter Six.
HIKM’2006AMTEx Automatic Document Indexing in Large Medical Collections Angelos Hliaoutakis, Kalliopi Zervanou, Euripides G.M. Petrakis Technical University.

HIKM’2006AMTEx Automatic Document Indexing in Large Medical Collections Angelos Hliaoutakis, Kalliopi Zervanou, Euripides G.M. Petrakis Technical University.

Similar presentations

Ads by Google