Statistical Machine Translation SMT – Basic Ideas Stephan Vogel MT Class Spring Semester 2011 Stephan Vogel - Machine Translation

Overview Deciphering foreign text – an example Principles of SMT Data processing Stephan Vogel - Machine Translation

Deciphering Example Apinaye – English Apinaye belongs to the Ge family of Brazil Spoken by 800 (according to SIL, 1994) http://www.ethnologue.com/show_family.asp?subid=90784 http://www.language-museum.com/a/apinaye.php Example from Linguistic Olympics 2008, see http://www.naclo.cs.cmu.edu Parallel Corpus (some characters adapted) Kukre kokoi The monkey eats Ape kre The child works Ape kokoi rats The big monkey works Ape mi mets The good man works Ape mets kra The child works well Ape punui mi pinjets The old man works badly Can we translate new sentence? Stephan Vogel - Machine Translation

Deciphering Example Parallel Corpus (some characters adapted) Can we build a lexicon from these sentence pairs? Observations: Apinaye: Kukre (1) Ape (5), English: The (6), works (5) Aha! -> first guess: Ape – works monkey in 1, 3; child in 2, 4; man in 4, 6 different distribution over corpus: do we find words with similar distribution on the Apinaye side? Kukre kokoi The monkey eats Ape kra The child works Ape kokoi rats The big monkey works Ape mi mets The good man works Ape mets kra The child works well Ape punui mi pinjets The old man works badly Stephan Vogel - Machine Translation

… Vocabularies Corpus Vocabularies Observations: Expectations: Kukre kokoi The monkey eats Ape kra The child works Ape kokoi rats The big monkey works Ape mi mets The good man works Ape mets kra The child works well Ape punui mi pinjets The old man works badly Apinaye English kukre The kokoi monkey ape eats kra child rats works mi big mets good punui man pinjets well old badly Observations: 9 Apinaye words, 11 English words Expectations: English words without translation? Apinaye words corresponding to more then 1 English word? Stephan Vogel - Machine Translation

… Word Frequencies Corpus Vocabularies, with frequencies Suggestions: Kukre kokoi The monkey eats Ape kra The child works Ape kokoi rats The big monkey works Ape mi mets The good man works Ape mets kra The child works well Ape punui mi pinjets The old man works badly Apinaye English kukre 1 The 6 kokoi 2 monkey ape 5 eats kra child rats works mi big mets good punui man pinjets well old badly Suggestions: ‘ape’ (5) could align to ‘The’ (6) or ‘works’ (5) More likely that content word ‘works’ has match, i.e. ‘ape’ = ‘works’ Other word pairs difficult to predict – too many similar frequencies Stephan Vogel - Machine Translation

… Location in Corpus Corpus Vocabularies, with occurrences Kukre kokoi The monkey eats Ape kra The child works Ape kokoi rats The big monkey works Ape mi mets The good man works Ape mets kra The child works well Ape punui mi pinjets The old man works badly Apinaye Sentences English kukre 1 The 1 2 3 4 5 6 kokoi 1 3 monkey ape 2 3 4 5 6 eats kra 2 5 child rats 3 works mi 4 6 big mets 4 5 good 4 punui 6 man pinjets well 5 old badly Observations: Same sentences: ‘kukre’ – ‘eats’, ‘kokoi’ – ‘monkey’, ‘ape’ – ‘works’, ‘kra’ – ‘child’, ‘rats’ – ‘big’, ‘mi’ – ‘man’ ‘mets’ (4 and 5) =? ‘good’ (4) and ‘well’ (5); makes sense ‘punui’ and ‘pinjets’ match ‘old’ and ‘badly’ – which is which? Stephan Vogel - Machine Translation

… Location in Sentence Corpus Observations: Hypothesis: Apinaye English Alignment EN - AP Kukre kokoi The monkey eats 1-0 2-2 3-1 Ape kra The child works Ape kokoi rats The big monkey works 1-0 2-3 3-2 4-1 Ape mi mets The good man works Ape mets kra The child works well 1-0 2-3 3-1 4-2 Ape punui mi pinjets The old man works badly 1-0 2-??? 3-3 4-1 5-??? Observations: First English word (‘The’) does not align; we say it aligns to the NULL word Apinaye verb in first position English last word aligns to 1st or 2nd position English -> Apinaye: reverse word order (not strictly in sentence pair 5) Hypothesis: alignment for last sentence pair is 1-0 2-4 3-3 4-1 5-2 I.e: ‘pinjets’ – ‘old’ and ‘punui’ – ‘badly’ Stephan Vogel - Machine Translation

… POS Information Corpus Observations: Hypothesis: Kukre kokoi V N The monkey eats DET N V Ape kra The child works Det N V Ape kokoi rats V N Adj The big monkey works Det Adj N V Ape mi mets The good man works Ape mets kra V Adv N The child works well Det N V Adv Ape punui mi pinjets V ??? N ??? The old man works badly Det Adj N V Adv Observations: English determiner (‘The’) does not align; perhaps no determiners in Apinaye English Verb Adverb -> Apinaye: Verb Adverb -> no reordering English Adjective Noun -> Apinaye: Noun Adjective -> reordering Hypothesis: ‘pinjets’ is Adj to make it N Adj, ‘punui’ is Adv (consistent with alignment hypothesis) Stephan Vogel - Machine Translation

Translate New Sentences: Ap - En Source Sentence: Ape rats mi mets Lexical information: works big man good/well Reordering information: The good man works big Better lexical choice: The good man works hard Compare: Ape mi mets -> The good man works Source Sentence: Kukre rats kokoi punui Lexical information: eats big monkey badly Reordering information: The bad monkey eats big Better lexical choice: The bad monkey eats a lot Stephan Vogel - Machine Translation

Translate New Sentences: En - Ap Source Sentence: The old monkey eats a lot Lexical information: NULL pinjets kokio kukre rats Reordering information: kukre rats kokio pinjets Or Deleting words: old monkey eats a lot Rephrase: old monkey eats big Reorder: eats big monkey old Lexical information: kukre rats kokio pinjets Source Sentence: The big child works a long time Delete plus rephrase: big child works big Reorder: works big child big Lexical information: Ape rats kra rats Stephan Vogel - Machine Translation

Overview Deciphering foreign text – an example Principles of SMT Data processing Stephan Vogel - Machine Translation

Principles of SMT We will use the same approach – learning from data Build translation models using frequency, co-occurrence, word position, etc. information Use the models to translate new sentences Not manually, but fully automatically The training will be automatically The is still lots of manual work left: designing models, preparing data, running experiments, etc. Stephan Vogel - Machine Translation

Machine Translation Approaches Grammar-based Interlingua-based Transfer-based Direct Example-based Statistical Stephan Vogel - Machine Translation

Statistical Approach Using statistical models Advantages Disadvantages Create many alternatives; we call them hypotheses Give a score to each hypothesis; based on statistical models Select the best -> search problem Advantages Avoid hard decisions Sometimes, optimality can be guaranteed Speed can be traded with quality, not all-or-nothing It works better ! Disadvantages Difficulties in handling structurally rich models, mathematically and computationally (but that’s also true for non-statistical systems) Need data to train the model parameters Stephan Vogel - Machine Translation

Statistical versus Grammar-Based Often statistical and grammar-based MT are seen as alternatives, even opposing approaches – wrong !!! Dichotomies are: Use probabilities || everything is equally likely, yes/no decision Rich (deep) structure || no or only flat structure Both dimensions are continuous Examples EBMT: no/little structure and heuristics SMT: (initially only) flat structure and probabilities XFER: deep(er) structure and heuristics Goal: structurally rich probabilistic models statXFER: deep structure and probabilities Syntax-augmented SMT: deep structure and probabilities No Probs Probs Flat Structure EBMT SMT Deep Structure XFER, Interlingua Holy Grail Stephan Vogel - Machine Translation

Statistical Machine Translation Translator translates source text Use machine learning techniques to extract useful knowledge Translation model: word and phrase translations Language model: how likely words follow in a particular sequence Translation system (decoder) uses these models to translates new sentences Advantages: Can quickly train for new languages Can adopt to new domains Problems: Need parallel data All words, even punctuation, are equal Difficult to pin-point the causes of errors Source Target Translation Model Language Model Source Sentence Translation Stephan Vogel - Machine Translation

Tasks in SMT Modelling build statistical models which capture characteristic features of translation equivalences and of the target language Training train translation model on bilingual corpus, train language model on monolingual corpus Decoding find best translation for new sentences according to models Evaluation Subjective evaluation: fluency, adequacy Automatic evaluation: WER, Bleu, etc And all the nitty-gritty stuff Text preprocessing, data cleaning Parameter tuning (minimum error rate training) Stephan Vogel - Machine Translation

Noisy Channel View “French is actually English, which has been garbled during transmission; recover the correct, original English” Noisy channel distorts into French Speaker speaks English You hear French, but need to recover the English Stephan Vogel - Machine Translation

Bayesian Approach Select translations which has highest probability ê = argmax{ p(e | f) } = argmax{ p(e) p(f | e) } Model Channel Search Process Model Source Stephan Vogel - Machine Translation

SMT Architecture p(e) – language model p(f | e) – translation model Stephan Vogel - Machine Translation

Log-Linear Model In practice: ê = argmax{ log(p(e)) + log( p(f | e)) } Translaiton model (TM) and language model (LM) may be of different quality: - simplifying assumptions - trained on different abounts of data Give different weights to both models ê = argmax{ w1 * log(p(e)) + w2 * log( p(f | e)) } Why not add more features? ê = argmax{ w1 * h1(e,f) + ... wn * hn(e, f) } Note: We don‘t need the normalization constant for the argmax Stephan Vogel - Machine Translation

Overview Deciphering foreign text – an example Principles of SMT Data processing Stephan Vogel - Machine Translation

Corpus Statistics We want to know how much data Corpus size: not file size, not documents, but words and sentences Why is file size not important? Vocabulary: number of word types We want to know some distributions How many words are seen only once? Why is this interesting? Does it help to increase the corpus? … How long are the sentence Does it matter if we have many short of fewer, but longer sentences? Stephan Vogel - Machine Translation

All Simple, Basic, Important Important: When you publish, these numbers are important To be able to interpret the results E.g. what works on small corpora may not work on large corpora To make them comparable to other papers Basic: no deep thinking, no fancy Simple: a few unix commands, a few simple scripts wc, grep, sed, sort, uniq perl, awk (my favorite), perhaps python, … Let’s look at some data! Stephan Vogel - Machine Translation

BTEC Spa-Eng Corpus Statistics Sentence length balance Corpus and vocabulary size Percentage of singletons Number of unknown words, out-of-vocabulary (OOV) rate Sentence length balance Text normalization Spoken language forms: I’ll, we’ar, but also I will, we are Note: this was shown online Stephan Vogel - Machine Translation

Tokenization Punctuation attached to words Tokenization can be tricky Example: ‘you’ ‘you,’ ‘you.’ ‘you?’ All different strings, i.e. all are different words Tokenization can be tricky What about punctuation in numbers What about appreviations(A5-0104/1999) Numbers are not just numbers Percentages: 1.2% Ordinals: 1st, 2. Ranges: 2000-2006, 3:1 And more: (A5-0104/1999) Stephan Vogel - Machine Translation

GigaWord Corpus Distributed by LDC Collection of new papers: NYT, Xinhua News, … > 3 billion words How large is vocabulary? Some observations in vocabulary Number of entries with digits Number of entries with special characters Number of strange ‘words’ Some observations in corpus Sentences with lots of numbers Sentences with lots of punctuation Sentences with very long words Note: this was shown online Stephan Vogel - Machine Translation

And then the more interesting Stuff POS tagging Parsing For syntax-based MT systems How parallel are the parse trees? Word segmentation Morphological processing In all these tasks the central problem is: How to make the corpus more parallel? Stephan Vogel - Machine Translation