English-Persian SMT Reza Saeedi 1 WTLAB Wednesday, May 25, 2011

Outline 2  MT Introduction  SMT Introduction  Requirements for SMT  Evaluation metrics  English-Persian MT challenges  English-Persian SMT  System1  System2  Problems in English-Persian SMT

MT Introduction 3  Automatic translation of text written in a natural language into another one by the use of computers is referred to as Machine Translation.  There are several way to do this work:  Dictionary-based  Rule-based  Example-based  Statistical approach

SMT Introduction 4  First ideas of Statistical machine translation was proposed by Warren Weaver in  Statistical machine translation tries to learn the translation by examining the translations made by humans.

SMT Introduction(Cont.) 5  Statistical MT models take the view that every sentence in the target language is a translation of the source language sentence with some probability.  The best translation, of course, is the sentence that has the highest probability.  The key problems in statistical MT are:  estimating the probability of a translation  and efficiently finding the sentence with the highest probability.

SMT Introduction(Cont.) 6  Given a Source sentence f, we seek the target sentence e that maximizes P(e | f). e‘ = argmax e P(e | f)  Intuitively, P(e|f) should depend on two factors:  P(e|f) = P(e) * P(f | e) / P(f)  argmax e P(e | f) = argmax e P(e) * P(f | e) fluency faithfulness

SMT Introduction(Cont.) 7  Philipp koehn 

Why SMT? 8  Better use of resources  Not need linguistic knowledge  It can use for any pair of language  But  We need a big training corpus

Steps of SMT 9

Requirements for SMT 10  Bilingual and Monolingual Corpus:  For bilingual need tow file aligned sentence by sentence (one file for source language and other for target language)  Microsoft Bi-Lingual sentence Aligner  Language Model:  We need a tool to compute P(e)  For this step we need to monolingual corpus  SRILM: a tool for create N-grams

LM output 11

Requirements for SMT 12  Translation Model:  We need a tool for compute P(f|e)  For this step we need to bilingual corpus  GIZA++  The output of this tool is a phrase table  Decode:  For search and find best translation  Moses

Phrase table 13

Moses tool 14

The training steps 15  Prepare data  Run GIZA++  Align words  Get lexical translation table  Extract phrases  Score phrases  Build reordering model  Build generation models  Create configuration file

Evaluation metrics 16  BLEU(BiLingual Evaluation Understudy)  Developed at IBM’s  The closer a MT is to a professional human translation, the better it is  NIST

English-Persian MT challenges 17  The Persian language structure is very different in comparison to English  The structure of Persian language is very complex  There has been little previous work done for this language pair  Effective SMT systems rely on very large bilingual corpora but there are not readily available for the English/Persian language pair

English-Persian SMT 18  There have been few English-Persian MT systems developed  Most of them are purely rule-based  There are two work on English-Persian SMT  Mohaghegh and Sarrafzadeh (Massey University)  Pilevar and Faili (Tehran University)

System1 19  Corpus: BBC news

System1(Cont.) 20  Tools: SRILM, GIZA++, Moses

System1: Improved Language Modeling 21

System2 22  Corpus:  Bidirectional(TEP): Subtitle of films, 3 books, KDE4

System2(Cont.) 23  Corpus:  Monolingual: Hamshahri, subtitle of films

System2(Cont.) 24  Tools: SRILM, GIZA++, Moses PersianSMT with 4-gram Sub-LM

Comparison PersianSMT with Google Translator 25

Problems in English-Persian SMT 26  compound verbs (aligning problem)  Use a phrase-based SMT system  But problem is inflectional morphology  Large number of inflected verb forms does not let the system learn to translate all the individual forms of a compound verb  Persian takes personal pronouns as an optional element in the sentence (aligning problem)

Problems(Cont.) 27  failure of the system to place the elements of the sentence in the right order  Use a phrase-based SMT system  Re-rank the n-best output list and/or reorder the output sentences  Prior to translation, the input sentence is reordered using morpho-syntactic information, so that the word order resembles better that of the target language.

28

References 29  [1] A. Ramanathan, "Statistical Machine Translation", Ph.D. Seminar Report, Department of Computer Science and Engineering Indian Institute of Technology,  [2] A. LOPEZ, "Statistical Machine Translation", ACM Computing Surveys,  [3] M. Mohaghegh, & A. Sarrafzadeh, “The first english-persian statistical machine translation”, New Zealand Postgraduate Conference,  [4] M. Mohaghegh, & A. Sarrafzadeh, " An analysis of the effect of training data variation in English-Persian Statistical Machine Translation”, 2009 International Conference on Innovations in Information Technology (IIT 2009)  [5] M. Mohaghegh, & A. Sarrafzadeh, " Performance evaluation of various training data in English-Persian statistical machine translation “, Appear in Proceedings of the 10th International Conference on the Statistical Analysis of Textual Data (JADT 2010), Rome, Italy, June 9-11,  [6] M. Mohaghegh, & A. Sarrafzadeh, " Improved Language Modeling for English-Persian Statistical Machine Translation”, COLING 2010 / SIGMT Workshop 23rd International Conference on Computational Linguistics Beijing, China 28 August 2010

References(Cont.) 30  [7] M.T. Pilevar and H. Faili, "PersianSMT: A First Attempt to English-Persian Statistical Machine Translation", to appear in Proc. of 10th International Conference on statistical analysis of textual data (JADT 2010)