MT Rule-Based MT vs. Data-Driven MT Evaluation of MT Development of MT MT problems in general Some Thinking about MT from recognition 3 Introduction machine translation - the use of computers to translate from one language to another The classic acid test for natural language processing. Requires capabilities in both interpretation and generation. About $10 billion spent annually on human translation. http://www.google.com/
the MT output with the reference? Improved levels of correlation with human judgments of MT Quality Contact Faculty: Alon Lavie May 21, 2009ITEC Dinner28 MT at the LTI LTI originated as the Center for Machine Translation (CMT) in 1985 MT continues to be a prominent sub-/ Subj Verb Obj I saw the man Modern Arabic is VSO: Verb Subj Obj –Different verb syntax: Verb complexes in English vs. in German I can eat the apple Ich kann den apfel essen –Case marking and free constituent order German and other languages /
Malay-English EMNLP-CoNLL 2012, July 12, 2012, Jeju, Korea 7 Source Language Adaptation for Resource-Poor Machine Translation (Wang, Nakov, & Ng) 7 Malay vs. Indonesian Malay Semua manusia dilahirkan bebas dan samarata dari segi kemuliaan dan hak-hak. / EMNLP-CoNLL 2012, July 12, 2012, Jeju, Korea 30 Source Language Adaptation for Resource-Poor Machine Translation (Wang, Nakov, & Ng) Human Judgments Morphology yields worse top-3 adaptations but better phrase tables, due to coverage. Is the adapted/
and Generators Input/Output compatability –Translation Lexicons Word-based vs. Transfer/Interlingua –Parallel Corpora Domain of interest Bigger is better Time Availability –Statistical training, resource building Road Map Multilingual Challenges for MT MT Approaches MT Evaluation More art than science Wide range of Metrics/Techniques –interface, …, scalability, …, faithfulness,... space/time complexity, … etc. Automatic vs. Human-based –Dumb Machines vs. Slow Humans Human-based Evaluation Example Accuracy/
Report: MT recognized as an extremely difficult, “AI- complete” problem. Funding disappears 1968: SYSTRAN founded 1985: CMU “Center for Machine Translation” (CMT) founded Late 1980s and early 1990s: Field dominated by rule-based approaches – KBMT, KANT, Eurotra, etc. 1992: /for the translation that statistically looks best August 22, 2011LTI IC 20118 Rule-based vs. Data-driven Approaches to MT How does the MT system pick the correct (or best) translation among many options? –Rule-based: Human experts encode/
counts go down) Mobile user, hand held devices Real time/Just in time demand Cross-lingual translation challenges Balance of cost, timeliness and quality Uncertain Open (collaborative) vs Closed (competitive)? Fee vs free? Human vs Machine? (incremental step or technology breakthrough) From TAUS Copenhagen Forum (May 2010) Machines Open (Collaborative) Closed (Competitive) Human & Machine ? ? Industry in 5 Years Content disruption SWOT Data assessment Innovation dilemma Embedding technology SWOT for/
The Possibility of Language: A discussion of the Nature of Language, with implications for Human and Machine Translation. Amsterdam: John Benjamins Pub. Co. Machine Translation (MT) – a few dates 1947 Warren Weaver - his ideas led to heavy/a neutral language (real, artificial, logical, mathematical..) > L2 Major Methods, Techniques and Approaches today Statistical vs. Linguistic MT assimilation tasks: lower quality, broad domains – statistical techniques predominate dissemination tasks: higher quality, /
-2/f04/lectures/mt-lecture.ppt Statistical MT Automatic Word Alignment GIZA++ A statistical machine translation toolkit used to train word alignments. Uses Expectation-Maximization with various constraints to bootstrap / interface, …, scalability, …, faithfulness, ... space/time complexity, … etc. Automatic vs. Human-based Dumb Machines vs. Slow Humans Human-based Evaluation Example Accuracy Criteria Human-based Evaluation Example Fluency Criteria Automatic Evaluation Example Bleu Metric (Papineni et al 2001/
of machine translation We can evaluate machine translation at corpus, document, sentence or word level –Remember that in MT the unit of translation is the sentence Human evaluation of machine translation quality is/translations (done by humans!) BLEU correlates well with average of fluency and adequacy at a corpus level –But not at a sentence level! 19 Alex Fraser IMS Stuttgart BLEU discussion BLEU works well for comparing two similar MT systems –Particularly: SMT system built on fixed training data vs/
translators the big picture How does MT affect the translation process and the translator? Integrating MT in the translation process Phase 1: Translation Memories Source text 0% translated Translation Memory (TM) Hybrid x% Phase 2: Machine Translation (MT) Hybrid text 100% translated but with MT errors Un-translated segments Phase 3: Post-editing Target text 100% translated Post-editor The role of translators/ effort Does PEMT save time vs. human translation? Does PEMT save time vs. TM fuzzy matches? Depends/
Translation Technology Continuum automation human involvement Automatic Translation Unaided Translation Computer-aided Translation (CAT) Translation process automated by use of Machine Translation Translation process aided by electronic tools such as Translation Memory Translation process not aided by any electronic tools Adapted from Hutchins & Somers (1992) 5 Machine Translation (MT) ………..Translation/ in MT in 80s and 90s 11 Machine Translation (MT) MT Design Rule-based vs Data-driven Systems (SBMT & EBMT)/
Translation Memory ICT Development TT Development 5 Translation Technology Continuum automation human involvement Automatic Translation Unaided Translation Computer-aided Translation (CAT) Translation process automated by use of Machine Translation Translation process aided by electronic tools such as Translation Memory Translation/ to multilingual information Renewed interest in MT in 80s and 90s 13 Machine Translation (MT) MT Design Rule-based vs Data-driven Systems (SMT & EBMT) –Rule-based systems by far/
Obj I saw the man Modern Arabic is VSO: Verb Subj Obj –Different verb syntax: Verb complexes in English vs. in German I can eat the apple Ich kann den apfel essen –Case marking and free constituent order German and/BLEU metric developed by IBM and used extensively in recent years Main ideas: –Assess the similarity between a machine-produced translation and (several) human reference translations –Similarity is based on word-to-word matching that matches: Identical words Morphological variants of same word (/
with the assistance of a computer program. The machine supports a human translator Internet Web-based human translation is generally favored by companies and individuals that seek more accurate translators. In view of the frequent inaccuracy of machine translators, human translation remains the most reliable, most accurate form of translation available Classwork Write one paragraph discussing either two advantages or disadvantages of human vs. machine translation. Project Language related topics: (should be/
For Samsung, February 20, 2008 Carnegie Mellon Slide 2 Languge Technologies Insitute Outline of Presentation Machine Translation (MT): –Three major paradigms for MT (transfer, example-based, statistical) –New method/ Better (Requiring None is Best ) Challenge –There is just not enough to approach human-quality MT for most major language pairs (we need ~100X more) –Much parallel/alignments, seed rules form s-boundary of VS; generalize with validation Seeded Version Space Learning Group seed rules into version spaces: /
reform plan Washing machine manual Google translate web UI 12/18 SMT Offers Croatian as source and target languageOffers Croatian as source and target language Statistically basedStatistically based Monolingual target language texts Aligned texts (human translations) Fluency and adequacy highly depend on available corporaFluency and adequacy highly depend on available corpora Reference translations vs. candidate translationsReference translations vs. candidate translations Levels of analysis:Levels/
Nakov & Min-Yen Kan Human Evaluation 4 native Finnish speakers 50 random test sentences follow WMT’09 evaluation: provided judges with the source sentence, its reference translation & outputs of (m-system, w-system, ourSystem) shown in random order asked for three pairwise judgments 25 A Hybrid Morpheme-Word Representation for Machine Translation of Morphologically Rich Languages our vs. mour vs. ww vs. m Judge 1251819122119 Judge 2241619152514/
Obj I saw the man Modern Arabic is VSO: Verb Subj Obj –Different verb syntax: Verb complexes in English vs. in German I can eat the apple Ich kann den apfel essen –Case marking and free constituent order German and/School49 English-Hindi Example June 27, 2006JHU Summer School50 Why Machine Translation for Minority and Indigenous Languages? Commercial MT economically feasible for only a handful of major languages with large resources (corpora, human developers) Is there hope for MT for languages with limited/
Extract bilingual terms in Patent Bilingual Titles Automatic Terminology Recognition and Human Translation Estimating the Number of Terms (1/4) Coverage of Single Terms and/ the patent attorneys and the patent examiners with the on-line English-Korean machine translation service (http://www.ipac.or.kr) In 2007, KIPO (Korean /documents) For average precision for all 1,000 queries Full text (baseline) vs. segmented text Full text Full text Query DocumentDocument set 55 Comparative experimentation /
Text quality evaluation (TQE) – issues 1/2 Quality evaluation vs. error identification / analysis Black box vs. glass box evaluation Error correction on the user side dictionary updating do-not-translate lists, etc. 2. Text quality evaluation (TQE) – / #PADJ – #ADVADJ) 4.2 Reference proximity methods Assumption of Reference Proximity (ARP): …the closer the machine translation is to a professional human translation, the better it is (Papineni et al., 2002: 311) Finding a distance between 2 texts Minimal edit /
, pp. 11-1 19 Advantages of direct systems Saving resources Translation is much faster & requires less memory Machine-learning techniques could be applied straightforwardly to create a direct MT system/translation X s defeat == X s loss X s defeat of Y == X s victory ORI: Swedish playmaker scored a hat-trick in the 4- 2 defeat of Heusden-Zolder Vs –… its defeat of last night –… their FA Cup defeat of last season –… their defeat of last seasons Cup winners –… last seasons defeat of Durham 54 … MT and human/
, pp. 11-1 18 Advantages of the direct systems Saving resources Translation is much faster & requires less memory Machine-learning techniques could be applied straightforwardly to create a direct MT system Direct/translation Xs defeat== Xs loss Xs defeat of Y == Xs victory ORI: Swedish playmaker scored a hat-trick in the 4-2 defeat of Heusden-Zolder Vs … its defeat of last night … their FA Cup defeat of last season … their defeat of last seasons Cup winners … last seasons defeat of Durham 51 … MT and human/
of Statistical Machine Translation Machine Translation Introduction to MT Machine Translation Language Divergences Dan Jurafsky Language Similarities and Divergences Typology: the study of systematic cross-linguistic similarities and differences What are the dimensions along which human languages vary?/ be necessary for MT between these languages and languages like English Dan Jurafsky Inferential Load: cold vs. hot languages Hot languages: Who did what to whom is marked explicitly English Cold languages: /
Areas: A Whirlwind Tour Machine Translation Focus on low-resource languages Elicit: translations, alignments, morphology, … Computational Biology Mapping the interactome (protein-protein) Host-pathogen interactome (e.g. HIV-human) Wind Energy Optimization of / Cannot optimize w/o knowing wind-speed map Different locations, altitudes, seasons, … Cost vs reliability (ground vs. tower sensors) Sensor type, placement, duration, reliability Analytic models reduce sensor net density Prediction /
(e.g. clauses/phrases) Force same linguistic phenomena in S an T? Vs translated as Ns How to model different linguistic phenomena? S = linguistic unit in source; T /Human scores (1-4): to which degree does the translation convey the meaning of the original text? 1: requires complete retranslation2: a lot of post-editing needed, but quicker than retranslation 3: a little post-editing needed4: fit for purpose 1: completely inadequate2: poorly adequate 3: Fairly Adequate 4: Highly Adequate 16 Machine/
and applications of MT 4.2 Reference proximity methods Assumption of Reference Proximity (ARP): “…the closer the machine translation is to a professional human translation, the better it is” (Papineni et al., 2002: 311) Finding a distance between 2 texts Minimal/ scores have to be validated Are they meaningful, whether of not predict any human evaluation measures, e.g., Fluency, Adequacy, Informativeness Agreement human vs. automated scores measured by Pearson’s correlation coefficient r a number in the /
Prentice Hall 13.3 Text Considerations Translation: either Use human translators exclusively Use semiautomatic translation with human oversight In either case: Avoid jargon and slang, which translate poorly Avoid sports metaphors Fully automatic machine translation is not a viable option—/ not complete Chapter 13: GlobalizationCopyright © 2004 by Prentice Hall Giving user a choice: opt-in vs. opt-out Chapter 13: GlobalizationCopyright © 2004 by Prentice Hall Giving information about expected delivery date /
modification (Hindi – Tamil, Telugu etc) (i) relative clause vs relative participles Telugu nenu tinnina camcaa Hindi : *meraa khaayaa cammaca /Human aided MT) (NCST,now C-DAC, Mumbai) General Purpose (not yet in use) Angla Bharati approach (IIT Kanpur ) UNL based MT (IIT Bombay) Shiva: EBMT (IIIT Hyderabad/IISc Bangalore) Shakti: English-Hindi MT system (IIIT Hyderabad) MT Efforts in India (3/4) Major Government funded MT projects in consortium mode Indian Language to Indian Language Machine Translation/
(search engines) Routing, personalization Anticipatory analysis Info extraction, speech Machine translation Summarization, expansion 3 Exploiting Context for LT’s Machine Translation –Why context is so necessary –Context-Based MT (new/is Best ) Challenge –There is just not enough to approach human-quality MT for major language pairs (we need ~1000X) –/, or for selecting optimal passages in summary generation. 39 MMR Ranking vs Standard IR query documents MMR IR λ controls spiral curl 40 Personalized Context/
systems (post-editing, controlled languages, domain-specific systems) research (new approaches, new methods) computational linguistics born in the aftermath Machine Translation (Pass 0 – From Intro Lectures) CIS 530 - Intro to NLP 6 Why use computers in translation? Too much translation for humans Technical materials too boring for humans Greater consistency required Need results more quickly Not everything needs to be top quality Reduce costs Any one/
grammar for machines talking to each other in biomedicine e.g., HL7 –semantic interoperation: reliable exchange of common meaning among humans and machines requires standard vocabularies and standard data models March 17, 2009: I. Sim Translational eScience Epi / decision support systems Fundamental tradeoff of coding effort vs. “smartness” of system limits both EHR and CDSS return on investment March 17, 2009: I. Sim Translational eScience Epi – 206 Medical Informatics Standardization Standardization of/
machine translation – a usability test for professional translation settings EyeTrackBehavior 2012 | October 9-10 | Leuven Post-editing? “term used for the correction of machine translation output by human linguists/editors” (Veale & Way 1997) “taking raw machine translated output and then editing it to produce a translation/TT processing metrics EyeTrackBehavior 2012 | October 9-10 | Leuven Processing of ST Translation vs. Post-editing => post-editing more efficientWHY? EyeTrackBehavior 2012 | October 9-10 /
Machine Translation in Academia and in the Commercial World: a Contrastive Perspective Alon Lavie Research Professor – LTI, Carnegie Mellon University Co-founder, President and CTO – Safaba Translation/for commercially- relevant content-types and domains WMT MT Systems vs Safaba MT Systems 10 WMT: MT for Assimilation (/ [Denkowski, Lavie, Lacruz and Dyer, 2014] EACL 2014 Workshop on Humans and Computer-assisted Translation Real-time Online Adaptation 68 Static MT System: Grammar: precompiled corpus/
the data March 24, 2006LTI Open House34 English-Hindi Example March 24, 2006LTI Open House35 GEBMT vs. Statistical MT Generalized-EBMT (GEBMT) uses examples at run time, rather than training a parameterized model/) liderespoliticosrusosfirmanpactodepazcivil March 24, 2006LTI Open House37 Why Machine Translation for Minority and Indigenous Languages? Commercial MT economically feasible for only a handful of major languages with large resources (corpora, human developers) Is there hope for MT for languages/
Page 57 The Analysis of the Term Project Polytechnic Playing Machine, Ppm The term project is human vs. machine There are two other Ppm versions which are not term projects Machine vs. machine Human vs. human Experiment 1 Lab 4CS 2204 Spring 2014 Page 58 The Term/ Xilinx IMPLEMENTATION consists of 3 major steps Synthesis to translate the schematic to a netlist file after converting the schematic to a VHDL file Implement Design which consists of Translate, Map, Place & Route Generate Programming File to/
checking, optimization, code generation etc.) Representations become more machine specific and less language specific as the translation proceeds 7 The first few steps The first few steps can be understood by analogies to how humans comprehend a natural language The first step is recognizing/a big impact on the complexity of the compiler M*N vs. M+N problem –Compilers are required for all the languages and all the machines –For M languages and N machines we need to developed M*N compilers –However, there is /
Important to explore the limits of each metric 29 May 2008LREC 2008 Sensitivity of BLEU vs task-based evaluation 3 Classification of MT evaluation models Reference proximity methods (BLEU, Edit Distance) –Measuring distance between MT and a “gold standard” translation “…the closer the machine translation is to a professional human translation, the better it is” (Papineni et al., 2002) Task-based methods (X-score, IE from/
go down) Mobile user, hand held devices Real time/Just in time demand Cross-lingual translation challenges Balance of cost, timeliness and quality Uncertain Open (collaborative) vs Closed (competitive)? Fee vs free? Human vs Machine? (incremental step or technology breakthrough) From TAUS Copenhagen Forum (May 2010) Machines Open (Collaborative) Closed (Competitive) Human & Machine ? ? Industry in 5 Years Content disruption SWOT Data assessment Innovation dilemma Embedding technology Copyright/
/without phenomenon under study Automatic vs. human evaluation http://www.meta-net.eu7 Hybrid MT Systems http://www.meta-net.eu7 8 Machine Translation Paradigms RB-MT – Rule-Based Machine translation EB-MT – Example-Based Machine Translation SMT – Statistical Machine Translation PB-SMT – Phrase-Based Statistical Machine Translation HPB-SMT – Hierachical Phrase-Based Statistical Machine Translation SB-SMT – Syntax-Based Statistical Machine Translation ... Observation: Different systems have/
– better interface design mistake – better understanding of system Human error – slips and mistakes Human errors are often classified into slips and mistakes. We /the machine is through the Input, and so the task must be articulated within the input language. The input language is translated into/the most commonly used controls being the most easily accessible. Industrial interfaces Office interface vs. industrial interface? Context matters! office industrial type of datatextualnumeric rate of changeslowfast /
of MT 10 4.2 Reference proximity methods Assumption of Reference Proximity (ARP): –“…the closer the machine translation is to a professional human translation, the better it is” (Papineni et al., 2002: 311) Finding a distance between 2 / scores Automatic scores have to be validated –Are they meaningful, whether or not predict any human evaluation measures, e.g., Fluency, Adequacy, Informativeness Agreement human vs. automated scores –measured by Pearson’s correlation coefficient r a number in the range of/
a challenging task 23 Translational Medicine Enable physicians to more effectively translate relevant findings and hypotheses into therapies for human health Support the blending /relations Data can be merged into complete graphs Multiple ontologies supported 90 RDF vs. XML example Wang et al., Nature Biotechnology, Sept 2005 AGML HUPML/first stage is underway: Using NLP and other automated technologies, extract machine-readable representations of neuroscience-related knowledge as contained in free text and /
introductory guide to MT by D.J.Arnold (1994) http://www.essex.ac.uk/linguistics/clmt/MTbook/ Free-to-use machine translation on the web: Free-to-use machine translation on the web: http://www.translatorsbase.com/ (Free human translation service) http://www.translatorsbase.com/ (Free human translation service) http://www.translatorsbase.com/ http://www.google.com/language_tools (uses Systran software) http://www.google.com/language_tools (uses Systran/
“percentage” (e.g., * 100) 85 BLEU results circa 2002 [from Papineni et al., ACL 2002][from G. Doddington, NIST] Distinguishes humans from machines……correlates well with human judgments 86 However nowadays we’re starting to see problems: - Some systems score better than human translations - In competitions, some “gaming of BLEU” - Rule based systems are at a disadvantage after tuning Next Time MT & Word Alignment Application/
Kalmár’s activity in machine translation (conference in 1962): „Representation of Languages with the Help of Mathematical Structures” Kalmár Workshop 2003 Gábor Prószéky: An Intuitive Representation of Human Languages for Translation Linguistic representation problems of the/a specific situation? Kalmár Workshop 2003 Gábor Prószéky: An Intuitive Representation of Human Languages for Translation Open classes t Open vs. closed classes: that is, features can or cannot be overridden t Proper names, jabbers, folk/
score: -45. Score: 9/10 Comments: Long convoluted sentences. Diagrammatical explanations. Minor grammatical and typo errors. 23 Human vs machine: unfair competition? One kind to combs the type generator using a phase lock agility frequency modulation output signal to / by subtracting the input signal and negating the phase error. Systran Human translation Original text Is such an MT useful? 24 EPO Research The case of Machine Translation Our Vision & Mission MT versus Patents The Chinese language case Our/
memory as binary, but displayed as human- readable ASCII characters Outline Binary vs. Text Files Compiler vs. Assembler Mnemonic Assembly Process Development Process Debugging Example 8 this analogy will make sense… Compiler vs. Assembler 9 high-level language assembly language compile assemble machine language (object code) Compiler 10 high-level language assembly language compile assemble machine language (object code) Compiler Translates high-level language into object code/
A Multi-Path Architecture for Machine Translation of English Text into American Sign Language Animation Matt Huenerfauth Student Workshop of the Human Language Technologies conference / North American chapter of the Association for/? How close are they? Where does path start/stop? How show path is bumpy vs. windy vs. hilly? Some English sentences require a classifier predicate to be translated fluently. Spatial prepositions, adverbs, other phrases… Signers use classifier predicates frequently. Depending on /
does not disambiguate It makes you wonder why you did such a beautiful Translation The machine Translation Often follow the Law of Diminishing Returns – Asymptotic – Require Huge Human, material and computer resources. Assume that the user is unaware of either/ are Word Order Free Languages- Hence good lexical Corpora would help in making nearly good enough Translation Generalization for Indian Languages- Phrase level vs. Sentence level Should we generalize –at the phrase level? –at the sentence level? Input/
Semantic Analysis Question-Answering Precision/Recall Watson Machine Translation Change to Schedule Wednesday Oct. 26: Support Vector Machines instead of Bayesian Networks Readings for this week (all on class webpage) B. Christian, Mind vs. MachineMind vs. Machine D. Ferrucci et al. Building /By analyzing the contexts in which the words appear The word user has co-occurred with words that human has co-occurred with (e.g., system and interface) It downgrades associations when such contextual similarities are/
