Download presentation
Presentation is loading. Please wait.
Published byKerry Hunt Modified over 8 years ago
2
Rapid Prototyping of Machine Translation Systems A Tale of Two Case Studies Srinivas Bangalore Giuseppe Riccardi AT&T Labs-Research Joint work with German Bordel and Vanessa Gaudin
3
Many thanks…to Alicia Abella Tirso Alonso Iker Arizmendi Barbara Hollister Mike Peñagarikano
4
Outline Machine Translation (MT) Past and Present Data Bottleneck and MT bootstrapping Consensus-based MT MT Evaluation Subjective and Objective Measures The Two Case Studies Demo
5
Machine Translation: Past and Present 1947- 1954 1954- 1966 1966- 1980s 1980- 1990 1990- present MT as code breaking, IBM-Georgetown Univ. demonstration Large bilingual dictionaries, linguistic and formal grammar motivated syntactic reordering, lots of funding, little progress ALPAC report: “there is no immediate or predictable prospect of useful machine translation”. 1966 Translation continued in Canada, France and Germany. Beyond English- Russian translation. Meteo for translating weather reports. Systran in 1970 Emphasis on ‘indirect’ translation: semantic and knowledge-based. Advent of microcomputers. Translation companies: Systran, Logos, GlobalLink. Domain specific machine-aided translation systems. Corpus-based methods: IBM’s Candide, Japanese ‘example-based’ translation. Speech-to-Speech translation: Verbmobil, Janus. ‘Pure’ to practical MT for embedded applications: Cross-lingual IR
6
Corpus-based Translation Direct-translation methods relying on large parallel corpora. –Statistical Translation (IBM in early 90’s) stochastic generative model; parameters estimated for lexical choice, lexical reordering reordering based on string positions robust when encountered with new data –Example-based Translation (Japanese research) corpus of example translations match previous instances, retrieve closest match performs well for minor variants of previously encountered examples; typical in limited domains
7
Tree-based Alignment English: I’d like to charge this to my home phone Japanese: 私は これを 私の 家の 電話にチャージ したいのです Automatic algorithm (Alshawi, Bangalore and Douglas, 1998) 私は (I) これを (this) したいのです (like) チャージ (charge) 家の (home) 私の (my) 電話に (phone) I’d this like charge home my phone to
8
Statistical Translation Models Head Transducer Model (Alshawi, Bangalore and Douglas, 1998) –Context-free grammar based transduction model –Parsing complexity: O(n^6) Stochastic Finite-State Transducer Model (Bangalore and Riccardi 2000) –Approximation of context-free grammar based transduction model –Parsing complexity: O(n) –Tightly integrated with ASR
9
Translation in HMIHY
10
Bilingual Parallel Corpus Statistical translation techniques crucially depend on bilingual parallel corpus Typically, monolingual corpus is available How to create bilingual parallel corpus? Solution: Create bilingual parallel data with the help of translation houses + high quality translations -expensive and longer turn around time
11
Alleviating the Bilingual Data Bottleneck Creating Parallel Corpora: –Use of off-the-shelf translation engines (via the web) + Per sentence translation – No translation engine may be perfect; combine multiple translations Inducing Parallel Corpora: –Use of documents in multiple languages + Highly accurate translations + Unlimited data source – Document translations not sentence translations
12
Acquiring Bilingual Data Use of translation systems over the Web MT1 MT2MTn Web MT Interface Consensus Translation Monolingual Data Bilingual Data
13
Consensus Translation Translations differ in –Lexical choice –Word order Create consensus among different translations: –Multi-string alignment English: give me driving directions please MT1: deme direccionnes impulsoras por favor MT2: deme direccionnes por favor MT3: deme direccionnes conductores por favor MT4: deme las direccionnes que conducen satisfacen MT5: deme que las direccionnes tendencia a gradan
14
String Alignment Alignment of tokens between two strings –Insertion, deletion and substitution operations Two string alignment complexity: O(n^2) Multi-string alignment complexity: O(n^m) –Exponential in the number of strings (m) MT1: deme direccionnes impulsoras por favor MT2: deme direccionnes por favor Profile: * * d * *
15
Multi-String Alignment Progressive multi-sequence alignment (Feng and Doolittle 87) –Compute the edit distance and profiles for m*(m-1)/2 pairs –Repeat the following until one profile remains Construct profile strings for least edit distance string-string, string-profile or profile-profile pairs. Compute the edit distance between selected profile and the remaining strings and profiles
16
Multi-String Alignment S2 S1S3 P13 S4S5 P45 P1345 P12345 Strings to Align: S1, S2, S3, S4, S5
17
Consensus Translation (1) Result of multi-string alignment can be viewed as a “sausage” –Arcs represent words or phrases (possibly ) –Arcs between two states represent different translations of a word or phrase –Fan out at a states indicates disagreement in translation –Weights can be associated with each arc
18
Consensus Translation (2) Retrieving the consensus translation –Concatenate substrings from each segment of sausage –Majority vote: Substring with most number of votes from each segment of the lattice CMV = BestCostPath(Sausage) Some segments do not have a clear majority Use a posterior n-gram language model (λ) with weighting factor (α) CMV+LM = BestCostPath(Sausage o α*λ)
19
Consensus Translation (3) Retrieving a consensus translation English: give me driving directions please MT1: deme direccionnes impulsoras por favor MT2: deme direccionnes por favor MT3: deme direccionnes conductores por favor MT4: deme las direccionnes que conducen satisfacen MT5: deme que las direccionnes tendencia a gradan CT: deme direccionnes por favor
20
Outline Machine Translation (MT) Past and Present Data Bottleneck and MT bootstrapping Consensus-based MT MT Evaluation Subjective and Objective Measures The two Case Studies Demo
21
Spoken Dialog Corpus –Conference Registration System (“Innovation Forum”) –Average sentence length ~7 words/utt –Utterance from all dialog contexts Evaluation data –Small (~0.5K) (labeler agreement) –Large (~4K) (MT performance) Spoken Language Database
22
Criteria –Objective (string accuracy, parse accuracy) –Subjective (Labeler Annotation) Translator agreement (disagreement) –Not as straightforward as speech utterance transcriptions (ASR) –One-to-Many mapping (Language Generation) –Local phenomena ENGLISH Would you like to go out tomorrow night? ITALIAN Vuoi uscire domani sera? Vorresti uscire domani sera? Vuoi uscire fuori con me domani sera? Vuoi uscire con me domani sera? MT Evaluation
23
MT Evaluation (1) objective String alignment – no direct relation with semantics/syntax + objective + system incremental evaluation Test set of manual translation (300 sentences) String edit distance between reference string and result string (length in words: R) Translation String Accuracy = 1 – (M + I + D + S) / R
24
Evaluation Results (1) objective Translation accuracy CMV+LM51.0% CMV47.7% MT 129.8% MT 223.7% MT 335.2% MT 446.9% MT 549.7%
25
Semantic/Syntactic scale (1-3) 1 = The translation is semantically and syntactically correct 2 = The translation is semantically correct and the syntax has some flaws. 3 = The translation is neither semantically nor syntactically correct. Two Labelers The source language text was presented together with all hypotheses for the target language MT Evaluation (2) subjective
26
Labeler Score Distribution (Small test set) Labeler A Labeler B
27
Labeler Distributional Agreement CMV+LM0.02 MT system 10.01 MT system 20.05 MT system 30.02 MT system 40.13 MT system 50.08 Binary random variable p_A(x=1) = 0.8 p_B(x=1) = 0.2 KL(p_A || p_B) ~ 1 Kulback-Leibler Distance
28
Evaluation (Large test set) Improve any MT system Decrease # Bad translation (score =3) Increase # Good translation ( score =1 or 2)
29
Hubbub…me (Instant Messaging)
30
Characteristics of Hubbub Data Human-human text-based interactions Open domain, conversations can be on any topic, may not be even task oriented Spontaneous chatty style of language (average 8 words per turn) Ungrammatical utterances and spelling errors Visual conversation context plays a crucial role in disambiguation Translation errors may be compensated based on the context of the conversation
31
Translation Accuracy Test set: 300 sentences
32
Summary Data Bottleneck solved by bootstrapping off existing MT systems Refine and Improve MT accuracy with Consensus-based MT Subjective and Objective Evaluation supports the improvement
33
Hubbub…me
Similar presentations
© 2024 SlidePlayer.com Inc.
All rights reserved.