Presentation is loading. Please wait.

Presentation is loading. Please wait.

Extracting bilingual terminologies from comparable corpora By: Ahmet Aker, Monica Paramita, Robert Gaizauskasl CS671: Natural Language Processing Prof.

Published byMervyn Nash Modified over 8 years ago

Similar presentations

Presentation on theme: "Extracting bilingual terminologies from comparable corpora By: Ahmet Aker, Monica Paramita, Robert Gaizauskasl CS671: Natural Language Processing Prof."— Presentation transcript:

1 Extracting bilingual terminologies from comparable corpora By: Ahmet Aker, Monica Paramita, Robert Gaizauskasl CS671: Natural Language Processing Prof. Amitabha Mukerjee Presented By: Ankit Modi (10104)

2 Introduction » Bilingual terminologies are important for various applications of human language technologies » Earlier studies may be distinguished by whether they work on parallel or comparable corpora » Focus on Comparable corpora is crucial as Parallel corpora is tough to find for all language pairs

3 Task To extract bilingual terminologies from comparable Corpora

4 Task To extract bilingual terminologies from comparable Corpora Comparable corpora: Collection of source-target language document pairs that are not direct translations but topically related.

5 Method » Pair each term extracted from S with each term extracted from T Term: Contiguous sequence of words (No particular syntactic restriction)

6 Method » Pair each term extracted from S with each term extracted from T » Treat term alignment as a binary classification task

7 Method » Pair each term extracted from S with each term extracted from T » Treat term alignment as a binary classification task » Extract features for each S-T potential term pair Decide whether to classify it as term equivalent or not ( SVM binary classifier with linear kernel)

8 Feature Extraction » Dictionary Based Features 1. isFirstWordTranslated ( Binary Feature ) 2. isLastWordTranslated 3. percentageOfTranslatedWord 4. percentageOfNotTranslatedWords

9 Feature Extraction » Dictionary Based Features 5. longestTranslatedUnitInPercentage 6. longestNotTranslatedUnitInPercentage 7. averagePercentageOfTranslatedWords » First 6 features are computed in both directions (S - > T and T -> S). In total, we have 13 Dictionary Based Features

10 Feature Extraction » Cognate Based Features 1. Longest Common Subsequence Ratio: Ex: LCSR (‘dollar’, ‘dolari’) = 5/6 2. Longest Common Substring Ratio: Ex: LCSTR (‘dollar’, ‘dolari’) = 3/6 3 Dice Similarity: Dice = 2*LCST / (len(X) + len(Y))

11 Feature Extraction » Cognate Based Features 4. Needlemann Wunsch Distance (NWD): NWD = LCST /min[ len(X) + len(Y)] 5. Levenshtein Distance: LDn = 1 - ( LD / max[len(X), len(Y)] ) » We have 5 Cognate Based Features

12 Feature Extraction » Cognate based features with term matching Applicable to those pair of languages whose alphabets belong to a common character set A mapping is performed from a source term to a target writing system or vice versa. Same cognate features as previous are calculated in both directions » We have 10 such features

13 Feature Extraction » Combined Features 1. isFirstWordCovered: Translation + Transliteration 2. isLastWordCovered: 3. percentageOfCoverage: 4. percentageOfNonCoverage 5. difBetweenCoverageAndNonCoverage » Calculated in both directions - 10 Combined Features

14 Feature Extraction » We have 38 features Dictionary based features : 13 Cognate based features : 5 Cognate based features with term matching : 10 Combined features :10

15 Evaluation 1 » Some positive and negative examples are created » Precision, recall and f-score are calculated » The precision score ranges from 100 to 67 percent

16 Evaluation 2 » Manual Evaluation » Human assessors are asked to categorize each term pair into one of the following categories: Equivalence, Inclusion, Overlap and Unrelated » Over 80 percent of the term pairs were assessed to be of the first category i.e. Equivalence.

17 Dataset » Training data taken from EUROVOC thesarus » English-German term-tagged comparable corpora for manual evaluation

18 Thank You

19 Manual Evaluation » Equivalence: Exact translation/ transliteration of each other » Inclusion: An exact translation/ transliteration of one term contained within the other » Overlap: Terms share at least one translated/ transliterated word » Unrelated: No word in either term is a translation/ transliteration of a word in other

20 Error » Error percentage was generally low » Reason for errors: Existence of words with very similar spellings but completely different meanings

21 SVM Binary Classifier » Pair each term extracted from S with each term extracted from T » Treat term alignment as a binary classification task » Linear Kernel » Trade-off between training error and margin parameter, c = 10.

22 Future Work » Looking into the usefulness of the term pairs in various application scenarios such as machine translation etc

Download ppt "Extracting bilingual terminologies from comparable corpora By: Ahmet Aker, Monica Paramita, Robert Gaizauskasl CS671: Natural Language Processing Prof."

Similar presentations

Pseudo-Relevance Feedback For Multimedia Retrieval By Rong Yan, Alexander G. and Rong Jin Mwangi S. Kariuki 2008-11629.

Pseudo-Relevance Feedback For Multimedia Retrieval By Rong Yan, Alexander G. and Rong Jin Mwangi S. Kariuki
Proceedings of the Conference on Intelligent Text Processing and Computational Linguistics (CICLing-2007) Learning for Semantic Parsing Advisor: Hsin-His.

Proceedings of the Conference on Intelligent Text Processing and Computational Linguistics (CICLing-2007) Learning for Semantic Parsing Advisor: Hsin-His.
Automatic Identification of Cognates, False Friends, and Partial Cognates University of Ottawa, Canada University of Ottawa, Canada.

Automatic Identification of Cognates, False Friends, and Partial Cognates University of Ottawa, Canada University of Ottawa, Canada.
Automatic Identification of Cognates and False Friends in French and English Diana Inkpen and Oana Frunza University of Ottawa and Greg Kondrak University.

Automatic Identification of Cognates and False Friends in French and English Diana Inkpen and Oana Frunza University of Ottawa and Greg Kondrak University.
Made with OpenOffice.org 1 Sentiment Classification using Word Sub-Sequences and Dependency Sub-Trees Pacific-Asia Knowledge Discovery and Data Mining.

Made with OpenOffice.org 1 Sentiment Classification using Word Sub-Sequences and Dependency Sub-Trees Pacific-Asia Knowledge Discovery and Data Mining.
Explorations in Tag Suggestion and Query Expansion Jian Wang and Brian D. Davison Lehigh University, USA SSM 2008 (Workshop on Search in Social Media)

Explorations in Tag Suggestion and Query Expansion Jian Wang and Brian D. Davison Lehigh University, USA SSM 2008 (Workshop on Search in Social Media)
Evaluating Search Engine

Evaluating Search Engine
SUPPORT VECTOR MACHINES PRESENTED BY MUTHAPPA. Introduction Support Vector Machines(SVMs) are supervised learning models with associated learning algorithms.

SUPPORT VECTOR MACHINES PRESENTED BY MUTHAPPA. Introduction Support Vector Machines(SVMs) are supervised learning models with associated learning algorithms.
Identifying Translations Philip Resnik, Noah Smith University of Maryland.

Identifying Translations Philip Resnik, Noah Smith University of Maryland.
Confidence Estimation for Machine Translation J. Blatz et.al, Coling 04 SSLI MTRG 11/17/2004 Takahiro Shinozaki.

Confidence Estimation for Machine Translation J. Blatz et.al, Coling 04 SSLI MTRG 11/17/2004 Takahiro Shinozaki.
1 Learning to Detect Objects in Images via a Sparse, Part-Based Representation S. Agarwal, A. Awan and D. Roth IEEE Transactions on Pattern Analysis and.

1 Learning to Detect Objects in Images via a Sparse, Part-Based Representation S. Agarwal, A. Awan and D. Roth IEEE Transactions on Pattern Analysis and.
Automatic Name Transliteration via OCR and NLP Yu Cao Tao Wang.

Automatic Name Transliteration via OCR and NLP Yu Cao Tao Wang.
Image Categorization by Learning and Reasoning with Regions Yixin Chen, University of New Orleans James Z. Wang, The Pennsylvania State University Published.

Image Categorization by Learning and Reasoning with Regions Yixin Chen, University of New Orleans James Z. Wang, The Pennsylvania State University Published.
Designing clustering methods for ontology building: The Mo’K workbench Authors: Gilles Bisson, Claire Nédellec and Dolores Cañamero Presenter: Ovidiu Fortu.

Designing clustering methods for ontology building: The Mo’K workbench Authors: Gilles Bisson, Claire Nédellec and Dolores Cañamero Presenter: Ovidiu Fortu.
Bioinformatics Challenge  Learning in very high dimensions with very few samples  Acute leukemia dataset: 7129 # of gene vs. 72 samples  Colon cancer.

Bioinformatics Challenge  Learning in very high dimensions with very few samples  Acute leukemia dataset: 7129 # of gene vs. 72 samples  Colon cancer.
Symmetric Probabilistic Alignment Jae Dong Kim Committee: Jaime G. Carbonell Ralf D. Brown Peter J. Jansen.

Symmetric Probabilistic Alignment Jae Dong Kim Committee: Jaime G. Carbonell Ralf D. Brown Peter J. Jansen.
1 The Web as a Parallel Corpus  Parallel corpora are useful  Training data for statistical MT  Lexical correspondences for cross-lingual IR  Early.

1 The Web as a Parallel Corpus  Parallel corpora are useful  Training data for statistical MT  Lexical correspondences for cross-lingual IR  Early.
Introduction to Machine Learning Approach Lecture 5.

Introduction to Machine Learning Approach Lecture 5.
Multiclass object recognition

Multiclass object recognition
Slide Image Retrieval: A Preliminary Study Guo Min Liew and Min-Yen Kan National University of Singapore Web IR / NLP Group (WING)

Slide Image Retrieval: A Preliminary Study Guo Min Liew and Min-Yen Kan National University of Singapore Web IR / NLP Group (WING)

Similar presentations

Ads by Google