THE MATHEMATICS OF STATISTICAL MACHINE TRANSLATION Sriraman M Tallam

2 Sriraman Tallam August 18, 2015 The Problem  The problem of machine translation is discussed.  Five Statistical Models are proposed for the translation process. Algorithms for estimating their parameters are described.  For the learning process, pairs of sentences that are translations of one another are used.  Previous work shows statistical methods to be useful in achieving linguistically interesting goals. natural extension - matching up words within pairs of aligned sentences.  Results show the power of statistical methods in extracting linguistically interesting correlations.

3 Sriraman Tallam August 18, 2015 Statistical Translation  Warren Weaver first suggested the use of statisitical techniques for machine translation. [Weaver 1955]  Fundamental Equation for Machine Translation Pr(e|f) = Pr(e) Pr(f|e) Pr(f) ê = argmax Pr(e) Pr(f|e)

4 Sriraman Tallam August 18, 2015 Statistical Translation  A translator when writing a French sentence, even a native speaker, conceives an English sentence and then mentally translates it. Machine translation’s goal is to find that English sentence.  Equation summarizes the 3 computational challenges presented by statistical translation. Language Model Probability Estimation - Pr(e) Translational Model Probability Estimation - Pr(f|e) Search Problem - maximizing their product  Why not reverse the translation models ? Class Discussion !!

5 Sriraman Tallam August 18, 2015 Alignments  What is a translation ? Pair of strings that are translations of one another (Qu’ aurions-nous pu faire ? | What could we have done ?)  What is an alignment ?

6 Sriraman Tallam August 18, 2015 Alignments  The mapping in an alignment could be from one-one to many-many.  The alignment in the figure is expressed as (Le programme a ete mis en application | And the(1) program(2) has(3) been(4) implemented(5,6,7)).  This alignment though acceptable has a lower probability. (Le programme a ete mis en application | And(1,2,3,4,5,6,7) the program has been implemented).  A(e,f) is the set of alignments of (f|e) If e has length ‘l’ and f has length ‘m’, there are 2 lm alignments in all.

7 Sriraman Tallam August 18, 2015 Cepts  What is a cept ? To express the fact that each word is related to a concept, in a figurative sense, a sentence is a web of concepts woven together The cepts in the example are The, poor and don’t have any money There is the notion of an empty cept.

8 Sriraman Tallam August 18, 2015 Translation Models  Five Translation models have been developed.  Each model is a recipe for computing Pr(f|e), which is called the likelihood of the translation (f,e).  The likelihood is a function of many parameters (  !).  The idea is to guess values for these parameters and to apply the EM algorithm iteratively.

9 Sriraman Tallam August 18, 2015 Translation Models  Models 1 and 2. all possible lengths are equally possible In Model 1, all connections for each french position are equally likely. In Model 2, connection probabilities are more realistic These models lead to unsatisfactory alignments very often  Models 3,4 and 5. No assumptions on the length of the French string Models 3 and 4 make more realistic assumptions on the connection probabilities Models are a stepping stone for the training of Model 5 Start with Model 1 for initial estimates and pipe thru the models,

10 Sriraman Tallam August 18, 2015 Translation Models  The likelihood of f | e is, over all elements of A(e,f)  Then, choose the length of the French string given the English for each french word position, choose the alignment, given previous alignments and words choose the identity of the word at this position given our knowledge of the previous alignments and words.

11 Sriraman Tallam August 18, 2015 Model 1 Assumptions  We assume Pr(m|e) is independent of e and m All reasonable lengths of the French string are equally likely.  Also, depends only on l. All connections are equally likely, and for a word there are (l + 1) connections, so this quantity is equal to (l + 1) -1  is called the translation probability of f j given e a j

12 Sriraman Tallam August 18, 2015 Model 1  The joint likelihood function for Model 1 is, and for j = 1 … m, and a j from 1 … l  Therefore,  subject to,

13 Sriraman Tallam August 18, 2015 Model 1  Technique of Lagrange Multipliers,  EM algorithm is applied repeatedly.  = X = Y =  The expected number of times e connects to f is t (f | e) f, e, l set of a j

14 Sriraman Tallam August 18, 2015 Model 1

15 Sriraman Tallam August 18, 2015 Model 1 -> Model 2  Model 1 does not take into account where words appear in either string All connections are equally probable  In Model 2, alignment probabilities are introduced and, which satisfy the constraints,

16 Sriraman Tallam August 18, 2015 Model 2  The likelihood function now is, and the cost function is,

17 Sriraman Tallam August 18, 2015 Fertitlity and Tablet  Fertility of a english word is the number of French words it is connected to -  i  Each english word translates to a set of French words called the Tablet - T i  The collection of Tablets is the Tableau - T.  The final French string is a permutation of the words in the Tableau - 

18 Sriraman Tallam August 18, 2015 Joint Likelihood of a Tableau and Permutation  The joint likelihood of a Tableau and Permutation is,  and,

19 Sriraman Tallam August 18, 2015 Model 3 Assumptions  The fertility probability of an english word only depends on the word.  The translation probability is,  The distortion probability is,

20 Sriraman Tallam August 18, 2015 Model 3  The likelihood function for Model 3 is now,

21 Sriraman Tallam August 18, 2015 Deficiency of Model 3  The fertility of word i does not depend on the fertility of previous words. Does not always concentrate its probability on events of interest.  This deficiency is no serious problem.  It might decrease the probability of all well-formed strings by a constant factor.

22 Sriraman Tallam August 18, 2015 Model 4  Allowing Phrases in the English String to move and be translated as units in the French String Model 3 doesn’t account for this well, because of the word by word movement. where, A and B are functions of the French and English words.  Using this they account for facts that an adjective appears before a noun in English and reverse in Frernch. - THIS IS GOOD !

23 Sriraman Tallam August 18, 2015 Model 4  For example, implemented produces mis en application, all occuring together, whereas not produces ne pas which occurs with a word in between.  So, d >1 (2 | B(pas)) is relatively large when compared to d >1 (2 | B(en))  Models 3 and 4 are both deficient. Words can be placed before the first position or beyond the last position in the French string. Model 5 removes this deficiency.

24 Sriraman Tallam August 18, 2015 Model 5  They defineto be the number of vacancies up to and including position j just before forming the words of the i th cept.  And, this gives rise to the following distortion probability equation,  Model 5 is powerful but must be used in tandem with the other 4 models.

25 Sriraman Tallam August 18, 2015 Results

26 Sriraman Tallam August 18, 2015 Changing Viterbi Alignments with Iterations

27 Sriraman Tallam August 18, 2015 Key Points from Results  Words like nodding have a large fertility because they don’t slip gracefully into French.  Words like should do not have a fertility greater than one but they translate into many different possible words, their translation probability is spread more.  Words like the have zero fertlility some times since English prefers an article in some places where French does not.