1 Machine Translation Dai Xinyu

2 Outline  Introduction  Architecture of MT  Rule-Based MT vs. Data-Driven MT  Evaluation of MT  Development of MT  MT problems in general  Some Thinking about MT from recognition

3 Introduction machine translation - the use of computers to translate from one language to another The classic acid test for natural language processing. Requires capabilities in both interpretation and generation. About $10 billion spent annually on human translation. "I have a text in front of me which is written in Russian but I am going to pretend that it is really written in English and that it has been coded in some strange symbols. All I need do is strip off the code in order to retrieve the information contained in the text"

4 Introdution - MT past and present  mid-1950's : Great expectations  The dark ages for MT: Academic research projects  1980's 's: Successful specialized applications  1990's: Human-machine cooperative translation  1990's - now: Statistical-based MT Hybrid-strategies MT  Future prospects: ???

5 Interest in MT  Commercial interest: U.S. has invested in MT for intelligence purposes MT is popular on the web — it is the most used of Google ’ s special features EU spends more than $1 billion on translation costs each year. (Semi-)automated translation could lead to huge savings

6 Interest in MT  Academic interest: One of the most challenging problems in NLP research Requires knowledge from many NLP sub-areas, e.g., lexical semantics, parsing, morphological analysis, statistical modeling, … Being able to establish links between two languages allows for transferring resources from one language to another

7 Related Area to MT  Linguistics  Computer Science AI Compile Formal Semantics …  Mathematics Probability Statistics …  Informatics  Recognition

8 Architecture of MT -- (Levers of Transfer)

9 Rule-Based MT vs. Data-Driven MT  Rule-Based MT  Data-Driven MT Example-Based MT Statistics-Based MT

10 Rule-Based MT 翻译系统 规则 语言学 语义学 认知科学 人工智能 写规则 自然语言输入 翻译结果

11 Rule-Based MT

12 Hmm, every time he sees “banco”, he either types “bank” or “bench” … but if he sees “banco de…”, he always types “bank”, never “bench”… Man, this is so boring. Translated documents

13 Example-Based MT  origins: Nagao (1981)  first motivation: collocations, bilingual differences of syntactic structures  basic idea: human translators search for analogies (similar phrases) in previous translations MT should seek matching fragment in bilingual database, extract translations  aim to have less complex dictionaries, grammars, and procedures  improved generation (using actual examples of TL sentences)

14 EBMT still going  Bi-lingual corpus Collection  Store  Searching and matching  …

15 Statistical MT Basics  Based on assumption that translations observed statistical regularities origins: Warren Weaver (1949) Shannon ’ s information theory  core process is the probabilistic ‘ translation model ’ taking SL words or phrases as input, and producing TL words or phrases as output  succeeding stage involves a probabilistic ‘ language model ’ which synthesizes TL words as ‘ meaningful ’ TL sentences

16 Statistical MT 学习系统 预测系统 概率模型 统计学习 建立模型 自然语言输入 预测

17 Statistical MT schema

18 Statistical MT processes  Bilingual corpora: original and translation  little or no linguistic ‘ knowledge ’, based on word co- occurrences in SL and TL texts (of a corpus), relative positions of words within sentences, length of sentences  Alignment: sentences aligned statistically (according to sentence length and position)  Decoding: compute probability that a TL string is the translation of a SL string ( ‘ translation model ’ ), based on: frequency of co-occurrence in aligned texts of corpus position of SL words in SL string  Adjustment: compute probability that a TL string is a valid TL sentence (based on a ‘ language model ’ of allowable bigrams and trigrams)  search for TL string that maximizes these probabilities argmax e P(e/f) = argmax e P (f/e) P (e)

19 Language Modeling  Determines the probability of some English sequence of length l  P(e) is normally approximated as: where m is size of the context, i.e. number of previous words that are considered, m=1, bi-gram language model m=2, tri-gram language model

20 Translation Modeling  Determines the probability that the foreign word f is a translation of the English word e  How to compute P(f | e) from a parallel corpus?  Statistical approaches rely on the co- occurrence of e and f in the parallel data: If e and f tend to co-occur in parallel sentence pairs, they are likely to be translations of one another

21 SMT issues  ignores previous MT research (new start, new ‘ paradigm ’ ) basically ‘ direct ’ approach:  replaces SL word by most probable TL word,  reorders TL words decoding is effectively kind of ‘ back translation ’  originally wholly word-based (IBM ‘ Candide ’ 1988) ; now predominantly phrase-based (i.e. alignment of word groups); some research on syntax-based  mathematically simple, but huge amount of training (large databases)  problems for SMT: translation is not just selecting the most frequent ‘ equivalent ’ (wider context) no quality control of corpora lack of monolingual data for some languages insufficient bilingual data (Internet as resource) lack of structure information of language  merit of SMT: evaluation as integral process of system development

22 Rule-Based MT & SMT  SMT black box: no way of finding how it works in particular cases, why it succeeds sometimes and not others  RBMT: rules and procedures can be examined  RBMT and SMT are apparent polar opposites, but gradually ‘ rules ’ incorporated in SMT models first, morphology (even in versions of first IBM model) then, ‘ phrases ’ (with some similarity to linguistic phrases) now also, syntactic parsing

23 Rule-Based MT & SMT  Comparison from following perspectives: Theory background Knowledge expression Knowledge discovery Robust Extension Development Cycle

24 Evaluation of MT  Manual: Precise / fluency / integrality 信 达 雅  Automatically evaluation: BLEU: percentage of word sequences (n-grams) occurring in reference texts NIST

25 Development of MT - MT System

26 Knowledge Acquisition Strategy Knowledge Representation Strategy All manual Deep/ Complex Shallow/ Simple Fully automated Learn from un- annotated data Phrase tables Word-based only Learn from annotated data Example-based MT Original statistical MT Typical transfer system Classic interlingual system Original direct approach Syntactic Constituent Structure Interlingua New Research Goes Here! Semantic analysis Hand-built by non-experts Hand-built by experts Electronic dictionaries MT Development - Research

27 MT problems in general  Characters of language Ambiguous Dynamic Flexible  Knowledge How to express How to discovery How to use

28 Some Thinking about MT from recognition  Human Cerebra Memory Progress - Learning Model Pattern  Translation by human …  Translation by machine …

29 Further Reading  Arturo Trujillo, Translation Engines: Techniques for Machine Translation, Springer-Verlag London Limited 1999  P.F. Brown, et al., A Statistical Approach to MT, Computational Linguistics, 1990,16(2)  P.F. Brown, et al., The Mathematics of Statistical Machine Translation: Parameter Estimation, Computational Linguistics, 1993, 19(2)  Bonnie J. Dorr, et al, Survey of Current Paradigms in Machine Translation  Makoto Nagao, A Framework of a Mechanical Translation between Japanese and English by Analog Principle, In A. Elithorn and R. Banerji(Eds.), Artificial and Human Intelligence. NATO Publications, 1984  Hutchins WJ, Machine Translation: Past, Present, Future. Chichester: Ellis Horwood, 1986  Daniel Jurafsky & James H. Martin, Speech and Language Processing, Prentice-Hall, 2000  Christopher D. Manning & Hinrich Schutze, Foundations of Statistical Natural Langugae Processing, Massachusetts Institute of Technology, 1999  James Allen, Natural Language Understanding, The Benjamin/Cummings Publishing Company, Inc. 1987