Statistical Machine Translation Raghav Bashyal

Statistical Machine Translation Uses pre-translated text (copora) Compare translated text to original Notice patterns, associate words

SMT Process Knight – A Statistical Translation Workbook Basic probabilities – P(word) Conditional probabilities – P(word | word) … Pick the most probable translation

SMT process

Project Translate basic text from Spanish to English Test effectiveness with/without hard-coded components (syntax) Specific procedures/algorithms that add speed

Literature Guides on Statistical Machine Translation Most research project follow the same procedure as outlined by Knight “state of the art” implementation – Google

Literature NLTK – Christina Wallin UC Berkeley – Modifications – Larger corpora more useful Syntax based – hard-code – Higher translation quality when used with SMT

Procedure NLTK – Natural Language ToolKit Python Made from Natural Language processing projects Current procedure – read the SMT worksheet Code along with worksheet

Development Create corpora Tokenization – Clean string Probability – P(word) in corpora

Smoothing Coefficients used to modify probability – Large coefficients for trigrams – Small for bigrams and single words Normalizes the weight of all the words/phrases – Trigrams are more valuable

Algorithm For translation, IMB Model 3 is used: 1. For each English word ei indexed by i = 1, 2,..., 1, choose fertility phi-i with probability n(phi-i | ei) 2. Choose the number phi-0 of "spurious" French words to be generated from e0 = NULL, using probability p1 and the sum of fertilities from step 1 3. Let m be the sum of fertilities for all words, including NULL 4. For each i = 0, 1, 2,...., 1, and each k = 1, 2,..., phi-i, choose a French word tau-ik with probability t(tau-ik | ei) 5. For each i = 1, 2,..., 1, and each k = 1, 2,..., phi-i, choose target French position pi-ik with probability d(pi-ik | i, l, m) 6. For each k = 1, 2,..., phi-0, choose a position pi-0k from the phi-0 - k + 1 remaining vacant positions in 1, 2,...m, for a total probability of 1/phi-0! 7. Output the French sentence with words tau-ik in positions pi-ik (0<=i<=1, 1<=k<phi-i)

Expected Results Probably will be very basic translation Usually perform better with “sample” text than “real” text Highlighted errors Program should use reference data to find some errors Error frequency plots for certain words Test the effectiveness of adjustments Hard coding, other algorithms

GUI