1. LIN6932 Topics in Computational Linguistics
Lecture 9: Machine Translation
Hana Filip
3/22/07
LIN 6932

2. Outline for MT Week Intro and a little history
Language Similarities and Divergences
Three classic MT Approaches
Transfer
Interlingua
Direct
Modern Statistical MT
Evaluation
3/22/07
LIN 6932

3. What is MT?
Translating a text from one language to another automatically.
Fully automatic translation lies at one end of the scale and the work of the human translator armed with pencil and paper at the other.
Between them are a number of possibilities for collaboration between man and computer which include word processing, terminology databases, voice recognition and translation memory systems.
3/22/07
LIN 6932

4. Machine Translation
dai yu zi zai chuang shang gan nian bao chai you ting jian chuang wai zhu shao xiang ye zhe shang, yu sheng xi li, qing han tou mu, bu jue you di xia lei lai.
Dai-yu alone on bed top think-of-with-gratitude Bao-chai again listen to window outside bamboo tip plantain leaf of on-top rain sound sigh drop clear cold penetrate curtain not feeling again fall down tears come
As she lay there alone, Dai-yu’s thoughts turned to Bao-chai… Then she listened to the insistent rustle of the rain on the bamboos and plantains outside her window. The coldness penetrated the curtains of her bed. Almost without noticing it she had begun to cry.
3/22/07
LIN 6932

5. Machine Translation
3/22/07
LIN 6932

6. Machine Translation The Story of the Stone also called
The Dream of the Red Chamber (Cao Xueqin 1792)
Issues:
Word segmentation, word order
Sentence segmentation: 4 English sentences to 1 Chinese
Grammatical differences
Chinese does not grammatically mark tense on the verb:
additional words and tense marking used in E: as, turned to, had begun
tou -> penetrated
Zero anaphora: a gap, in a phrase or clause, that has an anaphoric function similar to a pro-form
No articles
Stylistic and cultural differences
Bamboo tip plaintain leaf -> bamboos and plantains
Ma ‘curtain’ -> curtains of her bed
Rain sound sigh drop -> insistent rustle of the rain
3/22/07
LIN 6932

7. Not just literature Hansard: Canadian parliamentary proceeedings
Hansard is the traditional name for the printed transcripts of parliamentary debates in the Westminster system of government.
3/22/07
LIN 6932

8. Canadian Hansard and MT
The bilingual nature of the Canadian federal government requires that two equivalent Canadian Hansards be maintained: one in French and one in English.
This makes it a natural parallel text, and it is often used to train French-English machine translation programs.
In addition to being already translated and aligned, the size of the Hansards and the fact new material is always being added makes it an attractive corpus.
Problem: translations are accurate in meaning, but they are not always literally exact
3/22/07
LIN 6932

9. What is MT not good for? Really hard stuff Really important stuff
Literature
Natural spoken speech (meetings, court reporting)
Really important stuff
Medical translation in hospitals
3/22/07
LIN 6932

10. What is MT good for? Tasks for which a rough translation is fine
Web pages,
Tasks for which MT can be post-edited
MT as first pass
Computer-aided human translation
Tasks in sublanguage domains where high-quality MT is possible
FAHQT [faah-quit]
(Fully Automatic High-Quality Translation)
3/22/07
LIN 6932

11. Sublanguage domain Weather forecasting
“Cloudy with a chance of showers today and Thursday”
“Low tonight 4”
Can be modeled completely enough to use raw MT output
Word classes and semantic features like
MONTH, PLACE, DIRECTION, TIME POINT
3/22/07
LIN 6932

12. Some MT History
1946 Warren Weaver (mathematician, and science administrator, director of the Division of Natural Sciences at the Rockefeller Foundation, ) and Andrew D. Booth (British crystallographer) first discuss the possibility of MT in New York
idea of dictionary-based (word-for-word) direct translation
1949 Weaver’s “Translation” memorandum popularized the MT idea
3/22/07
LIN 6932

13. Some MT History
1949 Weaver’s “Translation” memorandum popularized the MT idea
limitations of any simplistic word-for-word approach
four proposals:
Approach the problem of multiple meanings by the examination of immediate context
logical elements in language
cryptographic methods were possibly applicable
linguistic universals:
“Think, by analogy, of individuals living in a series of tall closed towers, all erected over a common foundation. When they try to communicate with one another, they shout back and forth, each from his own closed tower. It is difficult to make the sound penetrate even the nearest towers, and communication proceeds very poorly indeed. But, when an individual goes down his tower, he finds himself in a great open basement, common to all the towers. Here he establishes easy and useful communication with the persons who have also descended from their towers.”
3/22/07
LIN 6932

14. Some MT History
1952: the first MT (“mechanical translation”) conference held at MIT, 18 MT researchers
1954 First public demo of computer translation at Georgetown University: 49 Russian sentences are translated into English using a 250-word vocabulary and 6 grammar rules.
a number of labs take up MT
3/22/07
LIN 6932

15. History of MT: Pessimism
1959/1960: Bar-Hillel “Report on the state of MT in US and GB”
Argued FAHQT too hard (semantic ambiguity, etc)
Should work on semi-automatic instead of automatic translations
His argument Little John was looking for his toy box. Finally, he found it. The box was in the pen. John was very happy.
Only human knowledge let’s us know that ‘playpens’ are bigger than boxes, but ‘writing pens’ are smaller
His claim: in order for MT to succeed, we would have to encode all of human knowledge
3/22/07
LIN 6932

16. Bar-Hillel’s report 1959/1960
MT research – now a “multimillion dollar affair”, as he pointed out – was, with few exceptions, set on a mistaken and unattainable goal, namely, fully automatic translation of a quality equal to that of a good human translator. This he held to be utterly unrealistic, and in his view resources were being wasted which could be more fruitfully be devoted to the development of less ambitious and more practical computer aids for translators.
3/22/07
LIN 6932

17. History of MT: Pessimism
1966
The ALPAC (Automatic Language Processing Advisory Committee) report
Headed by John R. Pierce of Bell Labs
Main Conclusion: years of research produced no useful results,
all current MT work had to be post-edited
Supply of human translators exceeds demand
All the Soviet literature is already being translated
Sponsored evaluations which showed that intelligibility and informativeness was worse than human translations
Results:
MT research suffered
halt in federal funding for machine translation in the US
Number of research labs declined
Association for Machine Translation and Computational Linguistics dropped MT from its name
3/22/07
LIN 6932

18. History of MT
1977 METEO System, developed at the Université de Montréal was installed in Canada to translate weather forecasts from English to French
1968 Systran founded by Peter Toma, one of the oldest machine translation companies, extensive work for the United States Department of Defense and the European Commission, provides the technology for Yahoo!, AltaVista's (Babel Fish) and Google's online translation services, among others
1970’s:
European focus in MT; mainly ignored in US
1980’s
ideas of using AI techniques in MT (Knowledge-based MT at Carnegie Mellon University)
1990’s
Commercial MT systems
Statistical MT
Speech-to-speech translation
3/22/07
LIN 6932

19. Language Similarities and Divergences
Some aspects of human language are universal or near-universal, others diverge greatly.
Typology: the study of systematic cross-linguistic similarities and differences
What are the dimensions along with human languages vary?
3/22/07
LIN 6932

20. Morphological Variation
Number of morphemes per word:
Isolating languages
Cantonese, Vietnamese: each word generally has one morpheme
Polysynthetic languages
Siberian Yupik (`Eskimo’): single word may have very many morphemes
Degree to which morphemes are segmentable:
Agglutinative languages
Turkish: morphemes have clean boundaries
Vs. Fusion languages
Russian: single affix conflate more than one grammatical categories (e.g., a case suffix fuses number, gender and case)
3/22/07
LIN 6932

21. Syntactic Variation SVO (Subject-Verb-Object) languages SOV Languages
English, German, French, Mandarin
SOV Languages
Japanese, Hindi
VSO languages
Irish, Classical Arabic
SVO lgs generally prepositions: to Yuriko
VSO lgs generally postpositions: Yuriko ni
3/22/07
LIN 6932

22. Segmentation Variation
Not every writing system has word boundaries marked by visual cues for
Chinese, Japanese, Thai, Vietnamese
Some languages tend to have sentences that are quite long, closer to English paragraphs than sentences:
Modern Standard Arabic, Chinese
3/22/07
LIN 6932

23. Lexical Divergences Word to phrases: POS divergences
English “computer science” = French “informatique”
POS divergences
English ‘she likes/VERB to sing’
German ‘Sie singt gerne’/ADV
English ‘I’m hungry’/ADJ
Italian ‘Ho fame’/NOUN
3/22/07
LIN 6932

24. Lexical Divergences: Specificity
Grammatical constraints
English has gender on pronouns, Mandarin not.
So translating “3rd person” from Chinese to English, need to figure out gender of the person!
Similarly from English “they” to French “ils” (masc. plural) or “elles” (feminine plual)
Semantic constraints
English ‘brother’
Mandarin ‘gege’ (older) versus ‘didi’ (younger)
English ‘wall’
German ‘Wand’ (inside) vs. ‘Mauer’ (outside) cp. die Berliner Mauer
German ‘Berg’
English ‘hill’ or ‘mountain’
3/22/07
LIN 6932

25. Lexical Divergence: many-to-many
3/22/07
LIN 6932

26. Lexical Divergence: lexical gaps
Japanese: no word for ‘privacy’
English (and other languages): no single word for German ‘Schadenfreude’ (the enjoyment of another’s misfortune)
English: no word for Japanese ‘oyakoko’ (something like ‘filial piety’)
English ‘blue’ versus Russian ‘sinevoy’ (dark blue) and ‘goluboy’ (light blue)
3/22/07
LIN 6932

27. Lexicalization Patterns divergences
Leonard Talmy (1985)
“Lexicalization patterns: Semantic structure in lexical forms”
English
The bottle floated out.
Manner of motion lexicalized in the verb
Direction of motion lexicalized in the ‘satellite’ (here V particle)
Spanish
La botella salió flotando.
Lit: the bottle exited floating
Manner of motion lexicalized in the gerund
Direction of motion lexicalized in the verb
3/22/07
LIN 6932

28. Lexicalization Patterns divergences
Verb-framed lg: mark direction of motion on verb
Romance, Arabic, Hebrew, Japanese, Tamil, Polynesian, Mayan, Bantu familiies
Satellite-framed lg: mark direction of motion on satellite
Crawl out, float off, jump down, walk over to, run after
Rest of Indo-European (e.g., Germanic, Slavic), Hungarian, Finnish, Chinese
3/22/07
LIN 6932

29. Structural divergences
German: Wir treffen uns am Mittwoch
English: We’ll meet on Wednesday
3/22/07
LIN 6932

30. Thematic divergence German: Mir fällt der Termin ein
English: I remember the date
3/22/07
LIN 6932

31. MT on the web Babelfish: Google: http://babelfish.altavista.com/
3/22/07
LIN 6932

32. 3 methods for MT
Direct
Transfer
Interlingua
3/22/07
LIN 6932

33. Three MT Approaches: Direct, Transfer, Interlingual
3/22/07
LIN 6932

34. Direct Translation Proceed word-by-word through text
Translating each word
No intermediate structures except morphology
Knowledge is in the form of
Huge bilingual dictionary
word-to-word translation information
After word translation, can do simple reordering
Adjective ordering English -> French/Spanish
3/22/07
LIN 6932

35. Direct MT
3/22/07
LIN 6932

36. Problems with direct MT
German
Complex reordering of words and phrases are necessary
3/22/07
LIN 6932

37. The Transfer Model
Idea: Starting from a structural analysis, use rules about differences between languages to translate directly from one surface structure to another: syntactic transformations (adjusting word order) and lexical transfer (selecting equivalents).
Steps:
Analysis: Syntactically parse Source language
Transfer: Rules to turn this parse into parse for Target language
Generation: Generate Target sentence from parse tree and lexical transfer via lookup in the bilingual dictionary
3/22/07
LIN 6932

38. English to French Generally English: Adjective Noun
French: Noun Adjective
Note: not always true
Route mauvaise ‘bad road, badly-paved road’
Mauvaise route ‘wrong road’
But is a reasonable first approximation
Rule:
3/22/07
LIN 6932

39. Transfer rules
From English SVO to Japanese SOV
3/22/07
LIN 6932

40. Transfer rules
3/22/07
LIN 6932

41. Lexical transfer
Transfer-based systems also need lexical transfer rules
Bilingual dictionary (like for direct MT)
English home (lexical ambiguity)
German
nach Hause (going home)
Heimat (homeland, home country)
zu Hause (at home)
Can list “at home <-> zu Hause”
Or do Word Sense Disambiguation
3/22/07
LIN 6932

42. Systran: combining direct and transfer
Shallow syntactic parsing
Morphological analysis, POS tagging
Chunking of NPs, PPs, phrases
Shallow dependency parsing (subjects, passives, head-modifiers)
Transfer
Translation of idioms
Word sense disambiguation
Assigning prepositions based on governing verbs
Synthesis
Apply rich bilingual dictionary
Deal with reordering
Morphological generation
3/22/07
LIN 6932

43. Transfer: some problems
A distinct set of transfer rules for each pair of languages
Grammar and lexicon full of language-idiosyncratic generalizations
Hard to build, hard to maintain
3/22/07
LIN 6932

44. Interlingua
Intuition: Instead of lg-lg knowledge rules, use the meaning of the sentence to help
Steps:
translate source sentence into meaning representation
generate target sentence from meaning
3/22/07
LIN 6932

45. Interlingua for Mary did not slap the green witch
3/22/07
LIN 6932

46. Direct MT: pros and cons (Bonnie Dorr)
Fast
Simple
Cheap
No translation rules hidden in lexicon
Cons
Unreliable
Not powerful
Rule proliferation
Requires lots of context
Major restructuring after lexical substitution
3/22/07
LIN 6932

47. Interlingual MT: pros and cons (B. Dorr)
Avoids the proliferation of specific rules
Easier to write rules
Cons:
Semantics is HARD
Useful information lost (paraphrase)
3/22/07
LIN 6932

48. What makes a good translation
Translators often talk about two factors we want to maximize:
Faithfulness or fidelity
How close is the meaning of the translation to the meaning of the original
(Even better: does the translation cause the reader to draw the same inferences as the original would have)
Fluency or naturalness
How natural the translation is, just considering its fluency in the target language
3/22/07
LIN 6932

49. The impossibility of translation
Hebrew “adonai roi” (= The Lord is my Shepherd)
How do you translate it into a language whose culture has no sheep or shepherds
Something fluent and understandable, but not faithful:
“The Lord will look after me”
Something faithful, but not fluent and natural
“The Lord is for me like somebody who looks after animals with cotton-like hair”
3/22/07
LIN 6932

50. Statistical MT: Faithfulness and Fluency formalized
Best-translation of a source sentence S into the target sentence T:
Idea: build probabilistic models of faithfulness and fluency, and then combine these models to choose the most probable (= best) translation
3/22/07
LIN 6932

51. The IBM model those two factors might look familiar…
Yup, it’s Bayes rule:
3/22/07
LIN 6932

52. Noisy channel model for statistical MT
Idea:
Statistical machine translation (MT) typically takes as its basis a noisy channel model in which the target language sentence, by tradition labelled E, is seen as distorted by the channel into the foreign language F.
3/22/07
LIN 6932

53. Noisy channel model for MT
Background:
The Shannon-Weaver Model of Communication
3/22/07
LIN 6932

54. Noisy channel model for MT
Idea:
Assume that the foreign (source language) input F we must translate into English is a corrupted version of some English (target language) sentence E, and that our task is to discover the hidden (target language) sentence E that generated our observation sentence F.
Hidden Markov Model
3/22/07
LIN 6932

55. Noisy channel model for MT
Given a Spanish sentence to translate (source L sentence), we treat it as the output of an English sentence (target L sentence) having gone through the noisy channel, and search for the best possible ‘source’ English sentence: I.e., the probability of the foreign sentence F given the existence of E: P(F|E)
3/22/07
LIN 6932

56. More formally
Assume we are translating from a foreign language sentence F to an English sentence E:
F = f1, f2, f3,…, fm
We want a decoder which is given F and produces the most probable (= best) English sentence
E-hat = e1, e2, e3,…, en
E-hat = argmaxE P(E|F)1
= argmaxE P(F|E)P(E)/P(F)2 Bayes rule
= argmaxE P(F|E)P(E)
Translation Model
Language Model
1 The conditional probability of an English sentence E, given a foreign sentence F
2 We can ignore the denominator P(F) inside the argmax since we are choosing the best English sentence for a fixed foreign sentence F, and hence P(F) is a constant.
3/22/07
LIN 6932

57. More formally argmaxE P(E|F) = argmaxE P(F|E)P(E)
This equation leaves much unresolved concerning how the actual translation is to be performed, systems that presuppose it are derived from the early IBM Models originally designed for speech recognition at IBM and work at the word level.
Called the IBM model of MT
the translation process involves translating words and then rearranging them to recover the target language sentence.
3/22/07
LIN 6932

58. Fluency: P(T) How to measure that this sentence
That car was almost crash onto me
is less fluent than this one:
That car almost hit me.
Answer: language models (N-grams!)
For example P(hit|almost) > P(almost|was)
But we can use any other more sophisticated model of grammar
Advantage: this is monolingual knowledge!
3/22/07
LIN 6932

59. Faithfulness: P(S|T) French: ça me plait [that me pleases] English:
probability that each word in target sentence would generate each word in source sentence.
French: ça me plait [that me pleases]
English:
that pleases me - most faithful
I like it - most fluent
How to quantify faithfulness?
Intuition: degree to which words in one sentence are plausible translations of words in other sentence
3/22/07
LIN 6932

60. Faithfulness P(S|T)
Need to know, for every target language word, probability of it mapping to every source language word.
How do we learn these probabilities?
Parallel texts!
two texts that are translations of each other
3/22/07
LIN 6932

61. Word Alignment
All statistical translation models are based on the idea of a word alignment
French - English word alignment
3/22/07
LIN 6932

62. Word Alignment
The IBM models require that each French word comes from exactly one English word: one-to-one and one-to-many alignments sanctioned
Many-to-many and many-to-one alignments disallowed by basic MT models
We can represent the above alignment by giving the index number of the English word that the French word comes from: A = 2,3,4,5,6,6,6.
3/22/07
LIN 6932

63. Word Alignment TRAINING ALIGNMENT MODELS
All statistical translation models are trained using a large parallel corpus.
A parallel corpus, parallel text, or bitext is a text that is available in two languages.
For example, the proceedings of the Canadian parliament are kept in both French and English. Each sentence spoken in parliament is translated, producing a volume with running text in both languages.
3/22/07
LIN 6932

64. Word Alignment First step: Sentence alignment
Figuring out which source language sentence maps to which target language sentence
Second step: Word alignment
Figuring out which source language word maps to which target language word for each sentence pair (F, E).
3/22/07
LIN 6932

65. Back to Faithfulness and Fluency
Job of the faithfulness model P(S|T) is to model “bag of words”; e.g., which words align from English to Spanish, when translating from Spanish to English.
P(S|T) does not have to worry about lg particular facts about Spanish word order: that’s the job of P(T) (language model)
P(T) can do Bag generation: rearrange the words so that they recover the correct word order of the target sentence (from Kevin Knight, USC/Information Sciences Institute) -
3/22/07
LIN 6932

66. P(T) and bag generation: problem
Problem: the ‘bag of words’ statistical MT does not model relations among words
How about:
loves Mary John
3/22/07
LIN 6932

67. Phrase-Based MT
Recently there has been considerable interest in MT systems based not upon words, but rather syntactic phrases
Such MT systems perform the translation by assuming that during the training phase the target language (but not the source language) specifies not just the words, but rather the complete parse of the sentence.
Eugene Charniak, Kevin Knight and Kenji Yamada (2003)
“Syntax-based Language Models for Statistical Machine Translation”
3/22/07
LIN 6932