Stat-XFER: A General Framework for Search-based Syntax-driven MT Alon Lavie Language Technologies Institute Carnegie Mellon University Joint work with: Erik Peterson, Alok Parlikar, Vamshi Ambati, Christian Monson, Ari Font Llitjos, Lori Levin, Jaime Carbonell – Carnegie Mellon University Shuly Wintner, Danny Shacham, Nurit Melnik - University of Haifa Roberto Aranovitch – University of Pittsburgh

February 18, 2008CICLing Outline Context and Rationale CMU Statistical Transfer MT Framework Broad Resource Scenario: Chinese-to-English Low Resource Scenario: Hebrew-to-English Open Research Challenges Conclusions

February 18, 2008CICLing Current State-of-the-Art in Machine Translation MT underwent a major paradigm shift over the past 15 years: –From manually crafted rule-based systems with manually designed knowledge resources –To search-based approaches founded on automatic extraction of translation models/units from large sentence- parallel corpora Current Dominant Approach: Phrase-based Statistical MT: –Extract and statistically model large volumes of phrase-to- phrase correspondences from automatically word-aligned parallel corpora –“Decode” new input by searching for the most likely sequence of phrase matches, using a combination of features, including a statistical Language Model for the target language

February 18, 2008CICLing Current State-of-the-art in Machine Translation Phrase-based MT State-of-the-art: –Requires minimally several million words of parallel text for adequate training –Mostly limited to language-pairs for which such data exists: major European languages, Arabic, Chinese, Japanese, a few others… –Linguistically shallow and highly lexicalized models result in weak generalization –Best performance levels (BLEU=~0.6) on Arabic-to- English provide understandable but often still ungrammatical or somewhat disfluent translations –Ill suited for Hebrew and most of the world’s minor and resource-poor languages

February 18, 2008CICLing Rule-based vs. Statistical MT Traditional Rule-based MT: –Expressive and linguistically-rich formalisms capable of describing complex mappings between the two languages –Accurate “clean” resources –Everything constructed manually by experts –Main challenge: obtaining broad coverage Phrase-based Statistical MT: –Learn word and phrase correspondences automatically from large volumes of parallel data –Search-based “decoding” framework: Models propose many alternative translations Effective search algorithms find the “best” translation –Main challenge: obtaining high translation accuracy

Research Goals Long-term research agenda (since 2000) focused on developing a unified framework for MT that addresses the core fundamental weaknesses of previous approaches: –Representation – explore richer formalisms that can capture complex divergences between languages –Ability to handle morphologically complex languages –Methods for automatically acquiring MT resources from available data and combining them with manual resources –Ability to address both rich and poor resource scenarios Main research funding sources: NSF (AVENUE and LETRAS projects) and DARPA (GALE) February 18, 20086CICLing-2008

February 18, 2008CICLing CMU Statistical Transfer (Stat-XFER) MT Approach Integrate the major strengths of rule-based and statistical MT within a common framework: –Linguistically rich formalism that can express complex and abstract compositional transfer rules –Rules can be written by human experts and also acquired automatically from data –Easy integration of morphological analyzers and generators –Word and syntactic-phrase correspondences can be automatically acquired from parallel text –Search-based decoding from statistical MT adapted to find the best translation within the search space: multi-feature scoring, beam-search, parameter optimization, etc. –Framework suitable for both resource-rich and resource- poor language scenarios

February 18, 2008CICLing Stat-XFER Main Principles Framework: Statistical search-based approach with syntactic translation transfer rules that can be acquired from data but also developed and extended by experts Automatic Word and Phrase translation lexicon acquisition from parallel data Transfer-rule Learning: apply ML-based methods to automatically acquire syntactic transfer rules for translation between the two languages Elicitation: use bilingual native informants to produce a small high-quality word-aligned bilingual corpus of translated phrases and sentences Rule Refinement: refine the acquired rules via a process of interaction with bilingual informants XFER + Decoder: –XFER engine produces a lattice of possible transferred structures at all levels –Decoder searches and selects the best scoring combination

February 18, 2008CICLing Stat-XFER MT Approach Interlingua Syntactic Parsing Semantic Analysis Sentence Planning Text Generation Source (e.g. Quechua) Target (e.g. English) Transfer Rules Direct: SMT, EBMT Statistical-XFER

Transfer Engine Language Model + Additional Features Transfer Rules {NP1,3} NP1::NP1 [NP1 "H" ADJ] -> [ADJ NP1] ((X3::Y1) (X1::Y2) ((X1 def) = +) ((X1 status) =c absolute) ((X1 num) = (X3 num)) ((X1 gen) = (X3 gen)) (X0 = X1)) Translation Lexicon N::N |: ["$WR"] -> ["BULL"] ((X1::Y1) ((X0 NUM) = s) ((Y0 lex) = "BULL")) N::N |: ["$WRH"] -> ["LINE"] ((X1::Y1) ((X0 NUM) = s) ((Y0 lex) = "LINE")) Source Input בשורה הבאה Decoder English Output in the next line Translation Output Lattice (0 1 (1 1 (2 2 (1 2 "THE (0 2 "IN (0 4 "IN THE NEXT Preprocessing Morphology

February 18, 2008CICLing Transfer Rule Formalism Type information Part-of-speech/constituent information Alignments x-side constraints y-side constraints xy-constraints, e.g. ((Y1 AGR) = (X1 AGR)) ; SL: the old man, TL: ha-ish ha-zaqen NP::NP [DET ADJ N] -> [DET N DET ADJ] ( (X1::Y1) (X1::Y3) (X2::Y4) (X3::Y2) ((X1 AGR) = *3-SING) ((X1 DEF = *DEF) ((X3 AGR) = *3-SING) ((X3 COUNT) = +) ((Y1 DEF) = *DEF) ((Y3 DEF) = *DEF) ((Y2 AGR) = *3-SING) ((Y2 GENDER) = (Y4 GENDER)) )

February 18, 2008CICLing Transfer Rule Formalism Value constraints Agreement constraints ;SL: the old man, TL: ha-ish ha-zaqen NP::NP [DET ADJ N] -> [DET N DET ADJ] ( (X1::Y1) (X1::Y3) (X2::Y4) (X3::Y2) ((X1 AGR) = *3-SING) ((X1 DEF = *DEF) ((X3 AGR) = *3-SING) ((X3 COUNT) = +) ((Y1 DEF) = *DEF) ((Y3 DEF) = *DEF) ((Y2 AGR) = *3-SING) ((Y2 GENDER) = (Y4 GENDER)) )

February 18, 2008CICLing Translation Lexicon: Examples PRO::PRO |: ["ANI"] -> ["I"] ( (X1::Y1) ((X0 per) = 1) ((X0 num) = s) ((X0 case) = nom) ) PRO::PRO |: ["ATH"] -> ["you"] ( (X1::Y1) ((X0 per) = 2) ((X0 num) = s) ((X0 gen) = m) ((X0 case) = nom) ) N::N |: ["$&H"] -> ["HOUR"] ( (X1::Y1) ((X0 NUM) = s) ((Y0 NUM) = s) ((Y0 lex) = "HOUR") ) N::N |: ["$&H"] -> ["hours"] ( (X1::Y1) ((Y0 NUM) = p) ((X0 NUM) = p) ((Y0 lex) = "HOUR") )

February 18, 2008CICLing Hebrew Transfer Grammar Example Rules {NP1,2} ;;SL: $MLH ADWMH ;;TL: A RED DRESS NP1::NP1 [NP1 ADJ] -> [ADJ NP1] ( (X2::Y1) (X1::Y2) ((X1 def) = -) ((X1 status) =c absolute) ((X1 num) = (X2 num)) ((X1 gen) = (X2 gen)) (X0 = X1) ) {NP1,3} ;;SL: H $MLWT H ADWMWT ;;TL: THE RED DRESSES NP1::NP1 [NP1 "H" ADJ] -> [ADJ NP1] ( (X3::Y1) (X1::Y2) ((X1 def) = +) ((X1 status) =c absolute) ((X1 num) = (X3 num)) ((X1 gen) = (X3 gen)) (X0 = X1) )

February 18, 2008CICLing The Transfer Engine Input: source-language input sentence, or source- language confusion network Output: lattice representing collection of translation fragments at all levels supported by transfer rules Basic Algorithm: “bottom-up” integrated “parsing- transfer-generation” guided by the transfer rules –Start with translations of individual words and phrases from translation lexicon –Create translations of larger constituents by applying applicable transfer rules to previously created lattice entries –Beam-search controls the exponential combinatorics of the search-space, using multiple scoring features

February 18, 2008CICLing The Transfer Engine Some Unique Features: –Works with either learned or manually-developed transfer grammars –Handles rules with or without unification constraints –Supports interfacing with servers for morphological analysis and generation –Can handle ambiguous source-word analyses and/or SL segmentations represented in the form of lattice structures

February 18, 2008CICLing XFER Output Lattice (28 28 "AND" "W" "(CONJ,0 'AND')") (29 29 "SINCE" "MAZ " "(ADVP,0 (ADV,5 'SINCE')) ") (29 29 "SINCE THEN" "MAZ " "(ADVP,0 (ADV,6 'SINCE THEN')) ") (29 29 "EVER SINCE" "MAZ " "(ADVP,0 (ADV,4 'EVER SINCE')) ") (30 30 "WORKED" "&BD " "(VERB,0 (V,11 'WORKED')) ") (30 30 "FUNCTIONED" "&BD " "(VERB,0 (V,10 'FUNCTIONED')) ") (30 30 "WORSHIPPED" "&BD " "(VERB,0 (V,12 'WORSHIPPED')) ") (30 30 "SERVED" "&BD " "(VERB,0 (V,14 'SERVED')) ") (30 30 "SLAVE" "&BD " "(NP0,0 (N,34 'SLAVE')) ") (30 30 "BONDSMAN" "&BD " "(NP0,0 (N,36 'BONDSMAN')) ") (30 30 "A SLAVE" "&BD " "(NP,1 (LITERAL 'A') (NP2,0 (NP1,0 (NP0,0 (N,34 'SLAVE')) ) ) ) ") (30 30 "A BONDSMAN" "&BD " "(NP,1 (LITERAL 'A') (NP2,0 (NP1,0 (NP0,0 (N,36 'BONDSMAN')) ) ) ) ")

February 18, 2008CICLing The Lattice Decoder Simple Stack Decoder, similar in principle to simple Statistical MT decoders Searches for best-scoring path of non-overlapping lattice arcs No reordering during decoding Scoring based on log-linear combination of scoring features, with weights trained using Minimum Error Rate Training (MERT) Scoring components: –Statistical Language Model –Rule Scores –Lexical Probabilities –Fragmentation: how many arcs to cover the entire translation? –Length Penalty: how far from expected target length?

February 18, 2008CICLing XFER Lattice Decoder 0 0 ON THE FOURTH DAY THE LION ATE THE RABBIT TO A MORNING MEAL Overall: , Prob: , Rules: 0, Frag: , Length: 0, Words: 13, < : B H IWM RBI&I (PP,0 (PREP,3 'ON')(NP,2 (LITERAL 'THE') (NP2,0 (NP1,1 (ADJ,2 (QUANT,0 'FOURTH'))(NP1,0 (NP0,1 (N,6 'DAY')))))))> 918 < : H ARIH AKL AT H $PN (S,2 (NP,2 (LITERAL 'THE') (NP2,0 (NP1,0 (NP0,1 (N,17 'LION')))))(VERB,0 (V,0 'ATE'))(NP,100 (NP,2 (LITERAL 'THE') (NP2,0 (NP1,0 (NP0,1 (N,24 'RABBIT')))))))> 584 < : L ARWXH BWQR (PP,0 (PREP,6 'TO')(NP,1 (LITERAL 'A') (NP2,0 (NP1,0 (NNP,3 (NP0,0 (N,32 'MORNING'))(NP0,0 (N,27 'MEAL')))))))>

February 18, 2008CICLing Stat-XFER MT Systems General Stat-XFER framework under development for past seven years Systems so far: –Chinese-to-English –Hebrew-to-English –Urdu-to-English –Hindi-to-English –Dutch-to-English –Mapudungun-to-Spanish In progress or planned: –Brazilian Portuguese-to-English –Native-Brazilian languages to Brazilian Portuguese –Hebrew-to-Arabic –Quechua-to-Spanish –Turkish-to-English

MT Resource Acquisition in Resource-rich Scenarios Scenario: Significant amounts of parallel-text at sentence-level are available –Parallel sentences can be word-aligned and parsed (at least on one side, ideally on both sides) Goal: Acquire both broad-coverage translation lexicons and transfer rule grammars automatically from the data Syntax-based translation lexicons: –Broad-coverage constituent-level translation equivalents at all levels of granularity –Can serve as the elementary building blocks for transfer trees constructed at runtime using the transfer rules February 18, CICLing-2008

Acquisition Process Automatic Process for Extracting Syntax-driven Rules and Lexicons from sentence-parallel data: 1.Word-align the parallel corpus (GIZA++) 2.Parse the sentences independently for both languages 3.Run our new PFA Constituent Aligner over the parsed sentence pairs 4.Extract all aligned constituents from the parallel trees 5.Extract all derived synchronous transfer rules from the constituent-aligned parallel trees 6.Construct a “data-base” of all extracted parallel constituents and synchronous rules with their frequencies and model them statistically (assign them relative-likelihood probabilities) February 18, CICLing-2008

PFA Constituent Node Aligner Input: a bilingual pair of parsed and word-aligned sentences Goal: find all sub-sentential constituent alignments between the two trees which are translation equivalents of each other Equivalence Constraint: a pair of constituents are considered translation equivalents if: –All words in yield of are aligned only to words in yield of (and vice-versa) –If has a sub-constituent that is aligned to, then must be a sub-constituent of (and vice-versa) Algorithm is a bottom-up process starting from word- level, marking nodes that satisfy the constraints February 18, CICLing-2008

PFA Node Alignment Algorithm Example Words don’t have to align one-to-one Constituent labels can be different in each language Tree Structures can be highly divergent

PFA Node Alignment Algorithm Example Aligner uses a clever arithmetic manipulation to enforce equivalence constraints Resulting aligned nodes are highlighted in figure

PFA Node Alignment Algorithm Example Extraction of Phrases: Get the Yields of the aligned nodes and add them to a phrase table tagged with syntactic categories on both source and target sides Example: NP # NP :: 澳洲 # Australia

PFA Node Alignment Algorithm Example All Phrases from this tree pair: 1.IP # S :: 澳洲 是 与 北韩 有 邦交 的 少数 国家 之一 。 # Australia is one of the few countries that have diplomatic relations with North Korea. 2.VP # VP :: 是 与 北韩 有 邦交 的 少数 国家 之一 # is one of the few countries that have diplomatic relations with North Korea 3.NP # NP :: 与 北韩 有 邦交 的 少数 国家 之一 # one of the few countries that have diplomatic relations with North Korea 4.VP # VP :: 与 北韩 有 邦交 # have diplomatic relations with North Korea 5.NP # NP :: 邦交 # diplomatic relations 6.NP # NP :: 北韩 # North Korea 7.NP # NP :: 澳洲 # Australia

PFA Constituent Node Alignment Performance Evaluation Data: Chinese-English Treebank – Parallel Chinese-English Treebank with manual word- alignments – 3342 Sentence Pairs Created a “Gold Standard” constituent alignments using the manual word-alignments and treebank trees –Node Alignments: (About 12/tree pair) –NP to NP Alignments: 5427 Manual inspection confirmed that the constituent alignments are extremely accurate (>95%) Evaluation: Run PFA Aligner with automatic word alignments on same data and compare with the “gold Standard” alignments February 18, CICLing-2008

PFA Constituent Node Alignment Performance Viterbi Combination PrecisionRecallF-Measure Intersection Union Sym-1 (Thot Toolkit) Sym-2 (Thot Toolkit) Grow-Diag-Final Viterbi word alignments from Chinese-English and reverse directions were merged using different algorithms Tested the performance of Node-Alignment with each resulting alignment

Transfer Rule Learning Input: Constituent-aligned parallel trees Idea: Aligned nodes act as possible decomposition points of the parallel trees –The sub-trees of any aligned pair of nodes can be broken apart at any lower-level aligned nodes, creating an inventory of “treelet” correspondences –Synchronous “treelets” can be converted into synchronous rules Algorithm: –Find all possible treelet decompositions from the node aligned trees –“Flatten” the treelets into synchronous CFG rules February 18, CICLing-2008

Rule Extraction Algorithm Sub-Treelet extraction: Extract Sub-tree segments including synchronous alignment information in the target tree. All the sub-trees and the super-tree are extracted.

Rule Extraction Algorithm Flat Rule Creation: Each of the treelets pairs is flattened to create a Rule in the ‘Avenue Formalism’ – Four major parts to the rule: 1. Type of the rule: Source and Target side type information 2. Constituent sequence of the synchronous flat rule 3. Alignment information of the constituents 4. Constraints in the rule (Currently not extracted)

Rule Extraction Algorithm Flat Rule Creation: Sample rule: IP::S [ NP VP.] -> [NP VP.] ( ;; Alignments (X1::Y1) (X2::Y2) ;;Constraints )

Rule Extraction Algorithm Flat Rule Creation: Sample rule: NP::NP [VP 北 CD 有 邦交 ] -> [one of the CD countries that VP] ( ;; Alignments (X1::Y7) (X3::Y4) ) Note: 1.Any one-to-one aligned words are elevated to Part-Of-Speech in flat rule. 2.Any non-aligned words on either source or target side remain lexicalized

Rule Extraction Algorithm All rules extracted: VP::VP [VC NP] -> [VBZ NP] ( (*score* 0.5) ;; Alignments (X1::Y1) (X2::Y2) ) VP::VP [VC NP] -> [VBZ NP] ( (*score* 0.5) ;; Alignments (X1::Y1) (X2::Y2) ) NP::NP [NR] -> [NNP] ( (*score* 0.5) ;; Alignments (X1::Y1) (X2::Y2) ) VP::VP [ 北 NP VE NP] -> [ VBP NP with NP] ( (*score* 0.5) ;; Alignments (X2::Y4) (X3::Y1) (X4::Y2) ) All rules extracted: NP::NP [VP 北 CD 有 邦交 ] -> [one of the CD countries that VP] ( (*score* 0.5) ;; Alignments (X1::Y7) (X3::Y4) ) IP::S [ NP VP ] -> [NP VP ] ( (*score* 0.5) ;; Alignments (X1::Y1) (X2::Y2) ) NP::NP [ “ 北韩 ”] -> [“North” “Korea”] ( ;Many to one alignment is a phrase )

Chinese-English System Developed over past year under DARPA/GALE funding (within IBM-led “Rosetta” team) Participated in recent NIST MT-08 Evaluation Large-scale broad-coverage system Integrates large manual resources with automatically extracted resources Current performance-level is still inferior to state-of-the-art phrase-based systems February 18, CICLing-2008

Chinese-English System Lexical Resources: –Manual Lexicons (base forms): LDC, ADSO, Wiki Total number of entries: 1.07 million –Automatically acquired from parallel data: Approx 5 million sentences LDC/GALE data Filtered down to phrases < 10 words in length Full formed Total number of entries: 2.67 million February 18, CICLing-2008

Chinese-English System Transfer Rules: –61 manually developed transfer rules –High-accuracy rules extracted from manually word- aligned parallel data CorpusSize (sens)Rules with Structure Rules (count>=2) Complete Lexical rules Parallel Treebank (3K)3,34345,2661,96211, sentences99312, ,199 Parallel Treebank (7K)6,54141,9981,75616,081 Merged Corpus set10K94, ,340 February 18, CICLing-2008

February 5, 2008CMU MT Update for Joe Olive 39 Translation Example SrcSent 3 澳洲是与北韩有邦交的少数国家之一。 Gloss: Australia is with north korea have diplomatic relations DE few country world Reference: Australia is one of the few countries that have diplomatic relations with North Korea. Translation:Australia is one of the few countries that has diplomatic relations with north korea. Overall: , Prob: , Rules: , TransSGT: , TransTGS: , Frag: , Length: , Words: 11,15 ( 0 10 "Australia is one of the few countries that has diplomatic relations with north korea" " 澳洲 是 与 北韩 有 邦交 的 少数 国家 之一 " "(S1, (S, (NP,2 (NB,1 (LDC_N,1267 'Australia') ) ) (VP, (MISC_V,1 'is') (NP, (LITERAL 'one') (LITERAL 'of') (NP, (NP, (NP,1 (LITERAL 'the') (NUMNB,2 (LDC_NUM,420 'few') (NB,1 (WIKI_N,62230 'countries') ) ) ) (LITERAL 'that') (VP, (LITERAL 'has') (FBIS_NP,11916 'diplomatic relations') ) ) (FBIS_PP,84791 'with north korea') ) ) ) ) ) ") ( "." " 。 " "(MISC_PUNC,20 '.')")

February 5, 2008CMU MT Update for Joe Olive 40 Example: Syntactic Lexical Phrases (LDC_N,1267 'Australia') (WIKI_N,62230 'countries') (FBIS_NP,11916 'diplomatic relations') (FBIS_PP,84791 'with north korea')

February 5, 2008CMU MT Update for Joe Olive 41 Example: XFER Rules ;;SL::(2,4) 对 台 贸易 ;;TL::(3,5) trade to taiwan ;;Score::22 {NP, } NP::NP [PP NP ] -> [NP PP ] ((*score* ) (X2::Y1) (X1::Y2)) ;;SL::(2,7) 直接 提到 伟 哥 的 广告 ;;TL::(1,7) commercials that directly mention the name viagra ;;Score::5 {NP, } NP::NP [VP " 的 " NP ] -> [NP "that" VP ] ((*score* ) (X3::Y1) (X1::Y3)) ;;SL::(4,14) 有 一 至 多 个 高 新 技术 项目 或 产品 ;;TL::(3,14) has one or more new, high level technology projects or products ;;Score::4 {VP, } VP::VP [" 有 " NP ] -> ["has" NP ] ((*score* 0.1) (X2::Y2))

MT Resource Acquisition in Resource-poor Scenarios Scenario: Very limited amounts of parallel-text at sentence-level are available –Significant amounts of monolingual text available for one of the two languages (i.e. English, Spanish) Approach: –Manually acquire and/or construct translation lexicons –Transfer rule grammars can be manually developed and/or automatically acquired from an elicitation corpus Strategy: –Learn transfer rules by syntax projection from major language to minor language –Build MT system to translate from minor language to major language February 18, CICLing-2008

February 18, 2008CICLing Learning Transfer-Rules for Languages with Limited Resources Rationale: –Large bilingual corpora not available –Bilingual native informant(s) can translate and align a small pre-designed elicitation corpus, using elicitation tool –Elicitation corpus designed to be typologically comprehensive and compositional –Transfer-rule engine and new learning approach support acquisition of generalized transfer-rules from the data

February 18, 2008CICLing Elicitation Tool: English-Hindi Example

February 18, 2008CICLing Elicitation Tool: English-Arabic Example

February 18, 2008CICLing Elicitation Tool: Spanish-Mapudungun Example

February 18, 2008CICLing Hebrew-to-English MT Prototype Initial prototype developed within a two month intensive effort Accomplished: –Adapted available morphological analyzer –Constructed a preliminary translation lexicon –Translated and aligned Elicitation Corpus –Learned XFER rules –Developed (small) manual XFER grammar –System debugging and development –Evaluated performance on unseen test data using automatic evaluation metrics

February 18, 2008CICLing Challenges for Hebrew MT Puacity in existing language resources for Hebrew –No publicly available broad coverage morphological analyzer –No publicly available bilingual lexicons or dictionaries –No POS-tagged corpus or parse tree-bank corpus for Hebrew –No large Hebrew/English parallel corpus Scenario well suited for Stat-XFER framework for languages with limited resources

February 18, 2008CICLing Modern Hebrew Spelling Two main spelling variants –“KTIV XASER” (difficient): spelling with the vowel diacritics, and consonant words when the diacritics are removed –“KTIV MALEH” (full): words with I/O/U vowels are written with long vowels which include a letter KTIV MALEH is predominant, but not strictly adhered to even in newspapers and official publications  inconsistent spelling Example: –niqud (spelling): NIQWD, NQWD, NQD –When written as NQD, could also be niqed, naqed, nuqad

February 18, 2008CICLing Morphological Analyzer We use a publicly available morphological analyzer distributed by the Technion’s Knowledge Center, adapted for our system Coverage is reasonable (for nouns, verbs and adjectives) Produces all analyses or a disambiguated analysis for each word Output format includes lexeme (base form), POS, morphological features Output was adapted to our representation needs (POS and feature mappings)

February 18, 2008CICLing Morphology Example Input word: B$WRH | B$WRH | |-----B-----|$WR|--H--| |--B--|-H--|--$WRH---|

February 18, 2008CICLing Morphology Example Y0: ((SPANSTART 0) Y1: ((SPANSTART 0) Y2: ((SPANSTART 1) (SPANEND 4) (SPANEND 2) (SPANEND 3) (LEX B$WRH) (LEX B) (LEX $WR) (POS N) (POS PREP)) (POS N) (GEN F) (GEN M) (NUM S) (NUM S) (STATUS ABSOLUTE)) (STATUS ABSOLUTE)) Y3: ((SPANSTART 3) Y4: ((SPANSTART 0) Y5: ((SPANSTART 1) (SPANEND 4) (SPANEND 1) (SPANEND 2) (LEX $LH) (LEX B) (LEX H) (POS POSS)) (POS PREP)) (POS DET)) Y6: ((SPANSTART 2) Y7: ((SPANSTART 0) (SPANEND 4) (SPANEND 4) (LEX $WRH) (LEX B$WRH) (POS N) (POS LEX)) (GEN F) (NUM S) (STATUS ABSOLUTE))

February 18, 2008CICLing Translation Lexicon Constructed our own Hebrew-to-English lexicon, based primarily on existing “Dahan” H-to-E and E-to-H dictionary made available to us, augmented by other public sources Coverage is not great but not bad as a start –Dahan H-to-E is about 15K translation pairs –Dahan E-to-H is about 7K translation pairs Base forms, POS information on both sides Converted Dahan into our representation, added entries for missing closed-class entries (pronouns, prepositions, etc.) Had to deal with spelling conventions Recently augmented with ~50K translation pairs extracted from Wikipedia (mostly proper names and named entities)

February 18, 2008CICLing Manual Transfer Grammar (human-developed) Initially developed by Alon in a couple of days, extended and revised by Nurit over time Current grammar has 36 rules: –21 NP rules –one PP rule –6 verb complexes and VP rules –8 higher-phrase and sentence-level rules Captures the most common (mostly local) structural differences between Hebrew and English

February 18, 2008CICLing Transfer Grammar Example Rules {NP1,2} ;;SL: $MLH ADWMH ;;TL: A RED DRESS NP1::NP1 [NP1 ADJ] -> [ADJ NP1] ( (X2::Y1) (X1::Y2) ((X1 def) = -) ((X1 status) =c absolute) ((X1 num) = (X2 num)) ((X1 gen) = (X2 gen)) (X0 = X1) ) {NP1,3} ;;SL: H $MLWT H ADWMWT ;;TL: THE RED DRESSES NP1::NP1 [NP1 "H" ADJ] -> [ADJ NP1] ( (X3::Y1) (X1::Y2) ((X1 def) = +) ((X1 status) =c absolute) ((X1 num) = (X3 num)) ((X1 gen) = (X3 gen)) (X0 = X1) )

February 18, 2008CICLing Example Translation Input: – לאחר דיונים רבים החליטה הממשלה לערוך משאל עם בנושא הנסיגה –Gloss: After debates many decided the government to hold referendum in issue the withdrawal Output: –AFTER MANY DEBATES THE GOVERNMENT DECIDED TO HOLD A REFERENDUM ON THE ISSUE OF THE WITHDRAWAL

February 18, 2008CICLing Noun Phrases – Construct State decision.3SF-CSthe-president.3SMthe-first.3SM החלטת הנשיא הראשון החלטת הנשיא הראשונה decision.3SF-CSthe-president.3SMthe-first.3SF THE DECISION OF THE FIRST PRESIDENT THE FIRST DECISION OF THE PRESIDENT

February 18, 2008CICLing Noun Phrases - Possessives HNSIAHKRIZ$HM$IMHHRA$WNH$LWTHIH the-presidentannouncedthat-the-task.3SFthe-first.3SFof-himwill.3SF LMCWA PTRWNLSKSWKBAZWRNW to-findsolutionto-the-conflictin-region-POSS.1P הנשיא הכריז שהמשימה הראשונה שלו תהיה למצוא פתרון לסכסוך באזורנו Without transfer grammar: THE PRESIDENT ANNOUNCED THAT THE TASK THE BEST OF HIM WILL BE TO FIND SOLUTION TO THE CONFLICT IN REGION OUR With transfer grammar: THE PRESIDENT ANNOUNCED THAT HIS FIRST TASK WILL BE TO FIND A SOLUTION TO THE CONFLICT IN OUR REGION

February 18, 2008CICLing Subject-Verb Inversion ATMWLHWDI&HHMM$LH yesterdayannounced.3SFthe-government.3SF אתמול הודיעה הממשלה שתערכנה בחירות בחודש הבא $T&RKNHBXIRWTBXWD$HBA that-will-be-held.3PFelections.3PFin-the-monththe-next Without transfer grammar: YESTERDAY ANNOUNCED THE GOVERNMENT THAT WILL RESPECT OF THE FREEDOM OF THE MONTH THE NEXT With transfer grammar: YESTERDAY THE GOVERNMENT ANNOUNCED THAT ELECTIONS WILL ASSUME IN THE NEXT MONTH

February 18, 2008CICLing Subject-Verb Inversion LPNIKMH$BW&WTHWDI&HHNHLTHMLWN beforeseveralweeksannounced.3SFmanagement.3SF.CSthe-hotel לפני כמה שבועות הודיעה הנהלת המלון שהמלון יסגר בסוף השנה $HMLWNISGRBSWFH$NH that-the-hotel.3SMwill-be-closed.3SMat-end.3SM.CSthe-year Without transfer grammar: IN FRONT OF A FEW WEEKS ANNOUNCED ADMINISTRATION THE HOTEL THAT THE HOTEL WILL CLOSE AT THE END THIS YEAR With transfer grammar: SEVERAL WEEKS AGO THE MANAGEMENT OF THE HOTEL ANNOUNCED THAT THE HOTEL WILL CLOSE AT THE END OF THE YEAR

February 18, 2008CICLing Evaluation Results Test set of 62 sentences from Haaretz newspaper, 2 reference translations SystemBLEUNISTPRMETEOR No Gram Learned Manual

Open Research Questions Our large-scale Chinese-English system is still significantly behind phrase-based SMT. Why? –Weaker decoder? –Feature set is not sufficiently discriminant? –Problems with the parsers for the two sides? –Syntactic constituents don’t provide sufficient coverage? –Bugs and deficiencies in the underlying algorithms? The ISI experience indicates that it may take a couple of years to catch up with and surpass the phrase-based systems Significant engineering issues to improve speed and efficient runtime processing and improved search February 18, CICLing-2008

Open Research Questions Immediate Research Issues: –Rule Learning: Study effects of learning rules from manually vs automatically word aligned data Study effects of parser accuracy on learned rules Effective discriminant methods for modeling rule scores Rule filtering strategies –Syntax-based LMs: Our translations come out with a syntax-tree attached to them Add a syntax-based LM feature that can discriminate between good and bad trees February 18, CICLing-2008

Conclusions Stat-XFER is a promising general MT framework, suitable to a variety of MT scenarios and languages Provides a complete solution for building end-to-end MT systems from parallel data, akin to phrase-based SMT systems (training, tuning, runtime system) No open-source publically available toolkits (yet), but we welcome further collaboration activities Complex but highly interesting set of open research issues Prediction: this is the future direction of MT! February 18, CICLing-2008

February 18, 2008CICLing Questions?

February 18, 2008CICLing Current and Future Work Issues specific to the Hebrew-to-English system: –Coverage: further improvements in the translation lexicon and morphological analyzer –Manual Grammar development –Acquiring/training of word-to-word translation probabilities –Acquiring/training of a Hebrew language model at a post- morphology level that can help with disambiguation General Issues related to XFER framework: –Discriminative Language Modeling for MT –Effective models for assigning scores to transfer rules –Improved grammar learning –Merging/integration of manual and acquired grammars

February 18, 2008CICLing Conclusions Test case for the CMU XFER framework for rapid MT prototyping Preliminary system was a two-month, three person effort – we were quite happy with the outcome Core concept of XFER + Decoding is very powerful and promising for MT We experienced the main bottlenecks of knowledge acquisition for MT: morphology, translation lexicons, grammar...

February 18, 2008CICLing Sample Output (dev-data) maxwell anurpung comes from ghana for israel four years ago and since worked in cleaning in hotels in eilat a few weeks ago announced if management club hotel that for him to leave israel according to the government instructions and immigration police in a letter in broken english which spread among the foreign workers thanks to them hotel for their hard work and announced that will purchase for hm flight tickets for their countries from their money

February 18, 2008CICLing Some Syntactic Challenges for Hebrew-English MT Possessor Dative Construction Anaphor resolution hitkalkelala-nuha-mexonit broke-downto-usthe-car Our car broke down. ha-memSalaarxaetyeSivata ha-riSona the-government held ACC her-meeting the-first The government held its first meeting.

February 18, 2008CICLing {NP0,2} NP0::NP0 [N PRO] -> [N] ( (X1::Y1) ((X2 case) = possessive) ((X0 possessor) = X2) ((X0 def) = +) ((Y1 num) = (X1 num)) (X0 = X1) (Y0 = X0) ) פגישתם PGI$TM pgiSat-am meeting.3SF-POSS.3PM ( ( SPANSTART 0 ) ( SPANEND 1 ) ( SCORE 1 ) ( LEX PGI$H ) ( POS N ) ( GEN feminine ) ( NUM singular ) ( STATUS absolute ) ) ( ( SPANSTART 1 ) ( SPANEND 2 ) ( SCORE 1 ) ( LEX *PRO* ) ( POS PRO ) ( TRANS *PRO* ) ( GEN masculine ) ( NUM plural ) ( PER 3 ) ( CASE possessive ) ) Transfer RulesMorph. AnalysisInput {NP,3} NP::NP [NP2] -> [PRO NP2] ( (X1::Y2) ((X1 possessor) =c *DEFINED*) ((Y1 case) = (X1 possessor case)) ((Y1 per) = (X1 possessor person)) ((Y1 num) = (X1 possessor num)) ((Y1 gen) = (X1 possessor gen)) (X0 = X1) (Y0 = Y2) ) THEIR MEETING Output

February 18, 2008CICLing Morphological Processing Split attached prefixes and suffixes into separate words for translation Produce feature-structures as output Convert feature-value codes to our conventions “All analyses mode”: all possible analyses for each input word returned, represented in the form of a input lattice Analyzer installed as a server integrated with input pre-processer

February 18, 2008CICLing Challenges and Future Directions Our approach for learning transfer rules is applicable to the large parallel data scenario, subject to solutions for several big challenges: –No elicitation corpus  break-down parallel sentences into reasonable learning examples –Working with less reliable automatic word alignments rather than manual alignments –Effective use of reliable parse structures for ONE language (i.e. English) and automatic word alignments in order to decompose the translation of a sentence into several compositional rules. –Effective scoring of resulting very large transfer grammars, and scaled up transfer + decoding

February 18, 2008CICLing Challenges and Future Directions Automatic Transfer Rule Learning: –Learning mappings for non-compositional structures –Effective models for rule scoring for Decoding: using scores at runtime Pruning the large collections of learned rules –Learning Unification Constraints –In the absence of morphology or POS annotated lexica Integrated Xfer Engine and Decoder –Improved models for scoring tree-to-tree mappings, integration with LM and other knowledge sources in the course of the search

February 18, 2008CICLing Hebrew Text Encoding Issues Input texts are (most commonly) in standard Windows encoding for Hebrew, but also unicode (UTF-8) and others… Morphology analyzer and other resources already set to work in a romanized “ascii-like” representation  Converter script converts the input into the romanized representation – 1-to-1 mapping! All further processing is done in the romanized representation Lexicon and grammar rules are also converted into romanized representation

February 18, 2008CICLing XFER + Decoder XFER engine produces a lattice of all possible transferred fragments Decoder searches for and selects the best scoring sequence of fragments as a final translation output Main advantages: –Very high robustness always some translation output no transfer grammar  word-to-word translation –Scoring can take into account word-to-word translation probabilities, transfer rule scores, target statistical language model –Effective framework for late-stage disambiguation Main Difficulty: lattice size too big  pruning

February 18, 2008CICLing Modern Hebrew Native language of about 3-4 Million in Israel Semitic language, closely related to Arabic and with similar linguistic properties –Root+Pattern word formation system –Rich verb and noun morphology –Particles attach as prefixed to the following word: definite article (H), prepositions (B,K,L,M), coordinating conjuction (W), relativizers ($,K$)… Unique alphabet and Writing System –22 letters represent (mostly) consonants –Vowels represented (mostly) by diacritics –Modern texts omit the diacritic vowels, thus additional level of ambiguity: “bare” word  word –Example: MHGR  mehager, m+hagar, m+h+ger

February 18, 2008CICLing The Transfer Engine Analysis Source text is parsed into its grammatical structure. Determines transfer application ordering. Example: 他 看 书。 (he read book) S NP VP N V NP 他 看 书 Transfer A target language tree is created by reordering, insertion, and deletion. S NP VP N V NP he read DET N a book Article “a” is inserted into object NP. Source words translated with transfer lexicon. Generation Target language constraints are checked and final translation produced. E.g. “reads” is chosen over “read” to agree with “he”. Final translation: “He reads a book”

February 18, 2008CICLing Elicitation Tool: English-Chinese Example

February 18, 2008CICLing Elicitation Tool: English-Chinese Example

February 18, 2008CICLing English-Hindi Example

February 18, 2008CICLing Rule Learning - Overview Goal: Acquire Syntactic Transfer Rules Use available knowledge from the source side (grammatical structure) Three steps: 1.Flat Seed Generation: first guesses at transfer rules; flat syntactic structure 2.Compositionality: use previously learned rules to add hierarchical structure 3.Seeded Version Space Learning: refine rules by learning appropriate feature constraints

February 18, 2008CICLing Flat Seed Rule Generation Learning Example: NP Eng: the big apple Heb: ha-tapuax ha-gadol Generated Seed Rule: NP::NP [ART ADJ N]  [ART N ART ADJ] ((X1::Y1) (X1::Y3) (X2::Y4) (X3::Y2))

February 18, 2008CICLing Compositionality Initial Flat Rules: S::S [ART ADJ N V ART N]  [ART N ART ADJ V P ART N] ((X1::Y1) (X1::Y3) (X2::Y4) (X3::Y2) (X4::Y5) (X5::Y7) (X6::Y8)) NP::NP [ART ADJ N]  [ART N ART ADJ] ((X1::Y1) (X1::Y3) (X2::Y4) (X3::Y2)) NP::NP [ART N]  [ART N] ((X1::Y1) (X2::Y2)) Generated Compositional Rule: S::S [NP V NP]  [NP V P NP] ((X1::Y1) (X2::Y2) (X3::Y4))

February 18, 2008CICLing Seeded Version Space Learning Input: Rules and their Example Sets S::S [NP V NP]  [NP V P NP] {ex1,ex12,ex17,ex26} ((X1::Y1) (X2::Y2) (X3::Y4)) NP::NP [ART ADJ N]  [ART N ART ADJ] {ex2,ex3,ex13} ((X1::Y1) (X1::Y3) (X2::Y4) (X3::Y2)) NP::NP [ART N]  [ART N] {ex4,ex5,ex6,ex8,ex10,ex11} ((X1::Y1) (X2::Y2)) Output: Rules with Feature Constraints: S::S [NP V NP]  [NP V P NP] ((X1::Y1) (X2::Y2) (X3::Y4) (X1 NUM = X2 NUM) (Y1 NUM = Y2 NUM) (X1 NUM = Y1 NUM))

Statistical XFER: Hybrid Statistical Rule-based Machine Translation Alon Lavie Language Technologies Institute Carnegie Mellon University Joint work with: Jaime Carbonell, Lori Levin, Bob Frederking, Erik Peterson, Christian Monson, Vamshi Ambati, Greg Hanneman, Kathrin Probst, Ariadna Font-Llitjos, Alison Alvarez, Roberto Aranovich

February 18, 2008CICLing Outline Background and Rationale Stat-XFER Framework Overview Elicitation Learning Transfer Rules Automatic Rule Refinement Example Prototypes Major Research Challenges

February 18, 2008CICLing Progression of MT Started with rule-based systems –Very large expert human effort to construct language- specific resources (grammars, lexicons) –High-quality MT extremely expensive  only for handful of language pairs Along came EBMT and then Statistical MT… –Replaced human effort with extremely large volumes of parallel text data –Less expensive, but still only feasible for a small number of language pairs –We “traded” human labor with data Where does this take us in 5-10 years? –Large parallel corpora for maybe language pairs What about all the other languages? Is all this data (with very shallow representation of language structure) really necessary? Can we build MT approaches that learn deeper levels of language structure and how they map from one language to another?

Transfer Engine Scoring Features Transfer Rules {NP1,3} NP1::NP1 [NP1 "H" ADJ] -> [ADJ NP1] ((X3::Y1) (X1::Y2) ((X1 def) = +) ((X1 status) =c absolute) ((X1 num) = (X3 num)) ((X1 gen) = (X3 gen)) (X0 = X1)) Translation Lexicon N::N |: ["$WR"] -> ["BULL"] ((X1::Y1) ((X0 NUM) = s) ((Y0 lex) = "BULL")) N::N |: ["$WRH"] -> ["LINE"] ((X1::Y1) ((X0 NUM) = s) ((Y0 lex) = "LINE")) Hebrew Input בשורה הבאה Decoder English Output in the next line Translation Output Lattice (0 1 (1 1 (2 2 (1 2 "THE (0 2 "IN (0 4 "IN THE NEXT Preprocessing Morphology February 18, CICLing-2008

February 18, 2008CICLing Transfer Rule Formalism Type information Part-of-speech/constituent information Alignments x-side constraints y-side constraints xy-constraints, e.g. ((Y1 AGR) = (X1 AGR)) ; SL: the old man, TL: ha-ish ha-zaqen NP::NP [DET ADJ N] -> [DET N DET ADJ] ( (X1::Y1) (X1::Y3) (X2::Y4) (X3::Y2) ((X1 AGR) = *3-SING) ((X1 DEF = *DEF) ((X3 AGR) = *3-SING) ((X3 COUNT) = +) ((Y1 DEF) = *DEF) ((Y3 DEF) = *DEF) ((Y2 AGR) = *3-SING) ((Y2 GENDER) = (Y4 GENDER)) )

February 18, 2008CICLing Transfer Rule Formalism (II) Value constraints Agreement constraints ;SL: the old man, TL: ha-ish ha-zaqen NP::NP [DET ADJ N] -> [DET N DET ADJ] ( (X1::Y1) (X1::Y3) (X2::Y4) (X3::Y2) ((X1 AGR) = *3-SING) ((X1 DEF = *DEF) ((X3 AGR) = *3-SING) ((X3 COUNT) = +) ((Y1 DEF) = *DEF) ((Y3 DEF) = *DEF) ((Y2 AGR) = *3-SING) ((Y2 GENDER) = (Y4 GENDER)) )

February 18, 2008CICLing Hebrew Manual Transfer Grammar (human-developed) Initially developed in a couple of days, with some later revisions by a CL post-doc Current grammar has 36 rules: –21 NP rules –one PP rule –6 verb complexes and VP rules –8 higher-phrase and sentence-level rules Captures the most common (mostly local) structural differences between Hebrew and English

February 18, 2008CICLing Source-language Confusion Network Hebrew Example Input word: B$WRH | B$WRH | |-----B-----|$WR|--H--| |--B--|-H--|--$WRH---|

February 18, 2008CICLing XFER Output Lattice (28 28 "AND" "W" "(CONJ,0 'AND')") (29 29 "SINCE" "MAZ " "(ADVP,0 (ADV,5 'SINCE')) ") (29 29 "SINCE THEN" "MAZ " "(ADVP,0 (ADV,6 'SINCE THEN')) ") (29 29 "EVER SINCE" "MAZ " "(ADVP,0 (ADV,4 'EVER SINCE')) ") (30 30 "WORKED" "&BD " "(VERB,0 (V,11 'WORKED')) ") (30 30 "FUNCTIONED" "&BD " "(VERB,0 (V,10 'FUNCTIONED')) ") (30 30 "WORSHIPPED" "&BD " "(VERB,0 (V,12 'WORSHIPPED')) ") (30 30 "SERVED" "&BD " "(VERB,0 (V,14 'SERVED')) ") (30 30 "SLAVE" "&BD " "(NP0,0 (N,34 'SLAVE')) ") (30 30 "BONDSMAN" "&BD " "(NP0,0 (N,36 'BONDSMAN')) ") (30 30 "A SLAVE" "&BD " "(NP,1 (LITERAL 'A') (NP2,0 (NP1,0 (NP0,0 (N,34 'SLAVE')) ) ) ) ") (30 30 "A BONDSMAN" "&BD " "(NP,1 (LITERAL 'A') (NP2,0 (NP1,0 (NP0,0 (N,36 'BONDSMAN')) ) ) ) ")

February 18, 2008CICLing The Lattice Decoder Simple Stack Decoder, similar in principle to simple Statistical MT decoders Searches for best-scoring path of non-overlapping lattice arcs No reordering during decoding Scoring based on log-linear combination of scoring components, with weights trained using MERT Scoring components: –Statistical Language Model –Fragmentation: how many arcs to cover the entire translation? –Length Penalty –Rule Scores –Lexical Probabilities

February 18, 2008CICLing XFER Lattice Decoder 0 0 ON THE FOURTH DAY THE LION ATE THE RABBIT TO A MORNING MEAL Overall: , Prob: , Rules: 0, Frag: , Length: 0, Words: 13, < : B H IWM RBI&I (PP,0 (PREP,3 'ON')(NP,2 (LITERAL 'THE') (NP2,0 (NP1,1 (ADJ,2 (QUANT,0 'FOURTH'))(NP1,0 (NP0,1 (N,6 'DAY')))))))> 918 < : H ARIH AKL AT H $PN (S,2 (NP,2 (LITERAL 'THE') (NP2,0 (NP1,0 (NP0,1 (N,17 'LION')))))(VERB,0 (V,0 'ATE'))(NP,100 (NP,2 (LITERAL 'THE') (NP2,0 (NP1,0 (NP0,1 (N,24 'RABBIT')))))))> 584 < : L ARWXH BWQR (PP,0 (PREP,6 'TO')(NP,1 (LITERAL 'A') (NP2,0 (NP1,0 (NNP,3 (NP0,0 (N,32 'MORNING'))(NP0,0 (N,27 'MEAL')))))))>

February 18, 2008CICLing Data Elicitation for Languages with Limited Resources Rationale: –Large volumes of parallel text not available  create a small maximally-diverse parallel corpus that directly supports the learning task –Bilingual native informant(s) can translate and align a small pre-designed elicitation corpus, using elicitation tool –Elicitation corpus designed to be typologically and structurally comprehensive and compositional –Transfer-rule engine and new learning approach support acquisition of generalized transfer-rules from the data

February 18, 2008CICLing Designing Elicitation Corpora Goal: Create a small representative parallel corpus that contains examples of the most important translation correspondences and divergences between the two languages Method: –Elicit translations and word alignments for a broad diversity of linguistic phenomena and constructions Current Elicitation Corpus: ~3100 sentences and phrases, constructed based on a broad feature-based specification Open Research Issues: –Feature Detection: discover what features exist in the language and where/how they are marked Example: does the language mark gender of nouns? How and where are these marked? –Dynamic corpus navigation based on feature detection: no need to elicit for combinations involving non-existent features

February 18, 2008CICLing Rule Learning - Overview Goal: Acquire Syntactic Transfer Rules Use available knowledge from the source side (grammatical structure) Three steps: 1.Flat Seed Generation: first guesses at transfer rules; flat syntactic structure 2.Compositionality Learning: use previously learned rules to learn hierarchical structure 3.Constraint Learning: refine rules by learning appropriate feature constraints

February 18, 2008CICLing Flat Seed Rule Generation Learning Example: NP Eng: the big apple Heb: ha-tapuax ha-gadol Generated Seed Rule: NP::NP [ART ADJ N]  [ART N ART ADJ] ((X1::Y1) (X1::Y3) (X2::Y4) (X3::Y2))

February 18, 2008CICLing Compositionality Learning Initial Flat Rules: S::S [ART ADJ N V ART N]  [ART N ART ADJ V P ART N] ((X1::Y1) (X1::Y3) (X2::Y4) (X3::Y2) (X4::Y5) (X5::Y7) (X6::Y8)) NP::NP [ART ADJ N]  [ART N ART ADJ] ((X1::Y1) (X1::Y3) (X2::Y4) (X3::Y2)) NP::NP [ART N]  [ART N] ((X1::Y1) (X2::Y2)) Generated Compositional Rule: S::S [NP V NP]  [NP V P NP] ((X1::Y1) (X2::Y2) (X3::Y4))

February 18, 2008CICLing Constraint Learning Input: Rules and their Example Sets S::S [NP V NP]  [NP V P NP] {ex1,ex12,ex17,ex26} ((X1::Y1) (X2::Y2) (X3::Y4)) NP::NP [ART ADJ N]  [ART N ART ADJ] {ex2,ex3,ex13} ((X1::Y1) (X1::Y3) (X2::Y4) (X3::Y2)) NP::NP [ART N]  [ART N] {ex4,ex5,ex6,ex8,ex10,ex11} ((X1::Y1) (X2::Y2)) Output: Rules with Feature Constraints: S::S [NP V NP]  [NP V P NP] ((X1::Y1) (X2::Y2) (X3::Y4) (X1 NUM = X2 NUM) (Y1 NUM = Y2 NUM) (X1 NUM = Y1 NUM))

February 18, 2008CICLing Automated Rule Refinement Bilingual informants can identify translation errors and pinpoint the errors A sophisticated trace of the translation path can identify likely sources for the error and do “Blame Assignment” Rule Refinement operators can be developed to modify the underlying translation grammar (and lexicon) based on characteristics of the error source: –Add or delete feature constraints from a rule –Bifurcate a rule into two rules (general and specific) –Add or correct lexical entries See [Font-Llitjos, Carbonell & Lavie, 2005]

February 18, 2008CICLing Stat-XFER MT Prototypes General Statistical XFER framework under development for past five years (funded by NSF and DARPA) Prototype systems so far: –Chinese-to-English –Dutch-to-English –French-to-English –Hindi-to-English –Hebrew-to-English –Mapudungun-to-Spanish In progress or planned: –Brazilian Portuguese-to-English –Native-Brazilian languages to Brazilian Portuguese –Hebrew-to-Arabic –Iñupiaq-to-English –Urdu-to-English –Turkish-to-English

February 18, 2008CICLing Chinese-English Stat-XFER System Bilingual lexicon: over 1.1 million entries (multiple resources, incl. ADSO, Wikipedia, extracted base NPs) Manual syntactic XFER grammar: 76 rules! (mostly NPs, a few PPs, and reordering of NPs/PPs within VPs) Multiple overlapping Chinese word segmentations English morphology generation Uses CMU SMT-group’s Suffix-Array LM toolkit for LM Current Performance (GALE dev-test): –NW: XFER: 10.89(B)/0.4509(M) Best (UMD): 15.58(B)/0.4769(M) –NG XFER: 8.92(B)/0.4229(M) Best (UMD): 12.96(B)/0.4455(M) In Progress: –Automatic extraction of “clean” base NPs from parallel data –Automatic learning and extraction of high-quality transfer- rules from parallel data

February 18, 2008CICLing Translation Example REFERENCE: When responding to whether it is possible to extend Russian fleet's stationing deadline at the Crimean peninsula, Yanukovych replied, "Without a doubt. Stat-XFER (0.3989): In reply to whether the possibility to extend the Russian fleet stationed in Crimea Pen. left the deadline of the problem, Yanukovich replied : " of course. IBM-ylee (0.2203): In response to the possibility to extend the deadline for the presence in Crimea peninsula, the Queen Vic said : " of course. CMU-SMT (0.2067): In response to a possible extension of the fleet in the Crimean Peninsula stay on the issue, Yanukovych vetch replied : " of course. maryland-hiero (0.1878): In response to the possibility of extending the mandate of the Crimean peninsula in, replied: "of course. IBM-smt (0.1862):The answer is likely to be extended the Crimean peninsula of the presence of the problem, Yanukovych said: " Of course. CMU-syntax (0.1639): In response to the possibility of extension of the presence in the Crimean Peninsula, replied : " of course.

February 18, 2008CICLing Major Research Directions Automatic Transfer Rule Learning: –From manually word-aligned elicitation corpus –From large volumes of automatically word-aligned “wild” parallel data –In the absence of morphology or POS annotated lexica –Compositionality and generalization –Identifying “good” rules from “bad” rules –Effective models for rule scoring for Decoding: using scores at runtime Pruning the large collections of learned rules –Learning Unification Constraints

February 18, 2008CICLing Major Research Directions Extraction of Base-NP translations from parallel data: –Base-NPs are extremely important “building blocks” for transfer-based MT systems Frequent, often align 1-to-1, improve coverage Correctly identifying them greatly helps automatic word- alignment of parallel sentences –Parsers (or NP-chunkers) available for both languages: Extract base-NPs independently on both sides and find their correspondences –Parsers (or NP-chunkers) available for only one language (i.e. English): Extract base-NPs on one side, and find reliable correspondences for them using word-alignment, frequency distributions, other features… Promising preliminary results

February 18, 2008CICLing Major Research Directions Algorithms for XFER and Decoding –Integration and optimization of multiple features into search-based XFER parser –Complexity and efficiency improvements (i.e. “Cube Pruning”) –Non-monotonicity issues (LM scores, unification constraints) and their consequences on search

February 18, 2008CICLing Major Research Directions Building Elicitation Corpora: –Feature Detection –Corpus Navigation Automatic Rule Refinement Translation for highly polysynthetic languages such as Mapudungun and Iñupiaq

February 18, 2008CICLing Questions?

February 18, 2008CICLing Recent Performance Analysis What fraction of the time does each MT system produce the best translation (sentence-by-sentence)? Evaluated on Chinese GALE dev-test (text) data BLEU METEOR CMU-PhraseSyntaxCombination 60 of 284 (21.1%) 41 of 284 (14.4%) IBM-smt 50 of 284 (17.6%) 49 of 284 (17.2%) IBM-ylee 64 of 284 (22.5%) 50 of 284 (17.6%) maryland-jhu-combination 71 of 284 (25.0%) 77 of 284 (27.1%) Stat-XFER 32 of 284 (11.2%) 56 of 284 (19.7%)

February 18, 2008CICLing Outline Rationale for learning-based MT Roadmap for learning-based MT Framework overview Elicitation Learning transfer Rules Automatic rule refinement Example prototypes Major Research Challenges

February 18, 2008CICLing Outline Rationale for learning-based MT Roadmap for learning-based MT Framework overview Elicitation Learning transfer Rules Automatic rule refinement Example prototypes Implications for MT with vast parallel data Conclusions and future directions

February 18, 2008CICLing Outline Rationale for learning-based MT Roadmap for learning-based MT Framework overview Elicitation Learning transfer Rules Automatic rule refinement Example prototypes Implications for MT with vast parallel data Conclusions and future directions

February 18, 2008CICLing Stat-XFER Prototypes General XFER framework under development for past five years Prototype systems so far: –German-to-English, Dutch-to-English –Chinese-to-English –Hindi-to-English –Hebrew-to-English –Portuguese-to-English In progress or planned: –Mapudungun-to-Spanish –Quechua-to-Spanish –Arabic-to-English –Native-Brazilian languages to Brazilian Portuguese

February 18, 2008CICLing CMU’s Statistical-Transfer (XFER) Approach Framework: Statistical search-based approach with syntactic translation transfer rules that can be acquired from data but also developed and extended by experts Elicitation: use bilingual native informants to produce a small high-quality word-aligned bilingual corpus of translated phrases and sentences Transfer-rule Learning: apply ML-based methods to automatically acquire syntactic transfer rules for translation between the two languages XFER + Decoder: –XFER engine produces a lattice of possible transferred structures at all levels –Decoder searches and selects the best scoring combination Rule Refinement: refine the acquired rules via a process of interaction with bilingual informants Word and Phrase bilingual lexicon acquisition

February 18, 2008CICLing The Transfer Engine Main algorithm: chart-style bottom-up integrated parsing+transfer with beam pruning –Seeded by word-to-word translations –Driven by transfer rules –Generates a lattice of transferred translation segments at all levels Some Unique Features: –Works with either learned or manually-developed transfer grammars –Handles rules with or without unification constraints –Supports interfacing with servers for morphological analysis and generation –Can handle ambiguous source-word analyses and/or SL segmentations represented in the form of lattice structures

February 18, 2008CICLing Why Machine Translation for Languages with Limited Resources? We are in the age of information explosion –The internet+web+Google  anyone can get the information they want anytime… But what about the text in all those other languages? –How do they read all this English stuff? –How do we read all the stuff that they put online? MT for these languages would Enable: –Better government access to native indigenous and minority communities –Better minority and native community participation in information-rich activities (health care, education, government) without giving up their languages. –Civilian and military applications (disaster relief) –Language preservation

February 18, 2008CICLing The Roadmap to Learning-based MT Automatic acquisition of necessary language resources and knowledge using machine learning methodologies: –Learning morphology (analysis/generation) –Rapid acquisition of broad coverage word-to-word and phrase-to-phrase translation lexicons –Learning of syntactic structural mappings Tree-to-tree structure transformations [Knight et al], [Eisner], [Melamed] require parse trees for both languages Learning syntactic transfer rules with resources (grammar, parses) for just one of the two languages –Automatic rule refinement and/or post-editing A framework for integrating the acquired MT resources into effective MT prototype systems Effective integration of acquired knowledge with statistical/distributional information

February 18, 2008CICLing Why Machine Translation for Languages with Limited Resources? We are in the age of information explosion –The internet+web+Google  anyone can get the information they want anytime… But what about the text in all those other languages? –How do they read all this English stuff? –How do we read all the stuff that they put online? MT for these languages would Enable: –Better government access to native indigenous and minority communities –Better minority and native community participation in information-rich activities (health care, education, government) without giving up their languages. –Civilian and military applications (disaster relief) –Language preservation

February 18, 2008CICLing CMU’s AVENUE Approach Elicitation: use bilingual native informants to create a small high-quality word-aligned bilingual corpus of translated phrases and sentences –Building Elicitation corpora from feature structures –Feature Detection and Navigation Transfer-rule Learning: apply ML-based methods to automatically acquire syntactic transfer rules for translation between the two languages –Learn from major language to minor language –Translate from minor language to major language XFER + Decoder: –XFER engine produces a lattice of possible transferred structures at all levels –Decoder searches and selects the best scoring combination Rule Refinement: refine the acquired rules via a process of interaction with bilingual informants Morphology Learning Word and Phrase bilingual lexicon acquisition

February 18, 2008CICLing AVENUE Architecture Learning Module Transfer Rules {PP,4894} ;;Score: PP::PP [NP POSTP] -> [PREP NP] ((X2::Y1) (X1::Y2)) Translation Lexicon Run Time Transfer System Lattice Decoder English Language Model Word-to-Word Translation Probabilities Word-aligned elicited data

February 18, 2008CICLing The Transfer Engine Analysis Source text is parsed into its grammatical structure. Determines transfer application ordering. Example: 他 看 书。 (he read book) S NP VP N V NP 他 看 书 Transfer A target language tree is created by reordering, insertion, and deletion. S NP VP N V NP he read DET N a book Article “a” is inserted into object NP. Source words translated with transfer lexicon. Generation Target language constraints are checked and final translation produced. E.g. “reads” is chosen over “read” to agree with “he”. Final translation: “He reads a book”

February 18, 2008CICLing The Transfer Engine Some Unique Features: –Works with either learned or manually-developed transfer grammars –Handles rules with or without unification constraints –Supports interfacing with servers for morphological analysis and generation –Can handle ambiguous source-word analyses and/or SL segmentations represented in the form of lattice structures

February 18, 2008CICLing The Lattice Decoder Simple Stack Decoder, similar in principle to SMT/EBMT decoders Searches for best-scoring path of non- overlapping lattice arcs Scoring based on log-linear combination of scoring components (no MER training yet) Scoring components: –Standard trigram LM –Fragmentation: how many arcs to cover the entire translation? –Length Penalty –Rule Scores (not fully integrated yet)

February 18, 2008CICLing Typological Elicitation Corpus Feature Detection –Discover what features exist in the language and where/how they are marked Example: does the language mark gender of nouns? How and where are these marked? –Method: compare translations of minimal pairs – sentences that differ in only ONE feature Elicit translations/alignments for detected features and their combinations Dynamic corpus navigation based on feature detection: no need to elicit for combinations involving non-existent features

February 18, 2008CICLing Typological Elicitation Corpus Initial typological corpus of about 1000 sentences was manually constructed New construction methodology for building an elicitation corpus using: –A feature specification: lists inventory of available features and their values –A definition of the set of desired feature structures Schemas define sets of desired combinations of features and values Multiplier algorithm generates the comprehensive set of feature structures –A generation grammar and lexicon: NLG generator generates NL sentences from the feature structures

February 18, 2008CICLing Structural Elicitation Corpus Goal: create a compact diverse sample corpus of syntactic phrase structures in English in order to elicit how these map into the elicited language Methodology: –Extracted all CFG “rules” from Brown section of Penn TreeBank (122K sentences) –Simplified POS tag set –Constructed frequency histogram of extracted rules –Pulled out simplest phrases for most frequent rules for NPs, PPs, ADJPs, ADVPs, SBARs and Sentences –Some manual inspection and refinement Resulting corpus of about 120 phrases/sentences representing common structures See [Probst and Lavie, 2004]

February 18, 2008CICLing Flat Seed Rule Generation Create a “flat” transfer rule specific to the sentence pair, partially abstracted to POS –Words that are aligned word-to-word and have the same POS in both languages are generalized to their POS –Words that have complex alignments (or not the same POS) remain lexicalized One seed rule for each translation example No feature constraints associated with seed rules (but mark the example(s) from which it was learned)

February 18, 2008CICLing Compositionality Learning Detection: traverse the c-structure of the English sentence, add compositional structure for translatable chunks Generalization: adjust constituent sequences and alignments Two implemented variants: –Safe Compositionality: there exists a transfer rule that correctly translates the sub-constituent –Maximal Compositionality: Generalize the rule if supported by the alignments, even in the absence of an existing transfer rule for the sub-constituent

February 18, 2008CICLing Constraint Learning Goal: add appropriate feature constraints to the acquired rules Methodology: –Preserve general structural transfer –Learn specific feature constraints from example set Seed rules are grouped into clusters of similar transfer structure (type, constituent sequences, alignments) Each cluster forms a version space: a partially ordered hypothesis space with a specific and a general boundary The seed rules in a group form the specific boundary of a version space The general boundary is the (implicit) transfer rule with the same type, constituent sequences, and alignments, but no feature constraints

February 18, 2008CICLing Constraint Learning: Generalization The partial order of the version space: Definition: A transfer rule tr 1 is strictly more general than another transfer rule tr 2 if all f- structures that are satisfied by tr 2 are also satisfied by tr 1. Generalize rules by merging them: –Deletion of constraint –Raising two value constraints to an agreement constraint, e.g. ((x1 num) = *pl), ((x3 num) = *pl)  ((x1 num) = (x3 num))

February 18, 2008CICLing Challenges for Hebrew MT Paucity in existing language resources for Hebrew –No publicly available broad coverage morphological analyzer –No publicly available bilingual lexicons or dictionaries –No POS-tagged corpus or parse tree-bank corpus for Hebrew –No large Hebrew/English parallel corpus Scenario well suited for CMU transfer-based MT framework for languages with limited resources

February 18, 2008CICLing Hebrew-to-English MT Prototype Initial prototype developed within a two month intensive effort Accomplished: –Adapted available morphological analyzer –Constructed a preliminary translation lexicon –Translated and aligned Elicitation Corpus –Learned XFER rules –Developed (small) manual XFER grammar as a point of comparison –System debugging and development –Evaluated performance on unseen test data using automatic evaluation metrics

February 18, 2008CICLing Morphology Example Y0: ((SPANSTART 0) Y1: ((SPANSTART 0) Y2: ((SPANSTART 1) (SPANEND 4) (SPANEND 2) (SPANEND 3) (LEX B$WRH) (LEX B) (LEX $WR) (POS N) (POS PREP)) (POS N) (GEN F) (GEN M) (NUM S) (NUM S) (STATUS ABSOLUTE)) (STATUS ABSOLUTE)) Y3: ((SPANSTART 3) Y4: ((SPANSTART 0) Y5: ((SPANSTART 1) (SPANEND 4) (SPANEND 1) (SPANEND 2) (LEX $LH) (LEX B) (LEX H) (POS POSS)) (POS PREP)) (POS DET)) Y6: ((SPANSTART 2) Y7: ((SPANSTART 0) (SPANEND 4) (SPANEND 4) (LEX $WRH) (LEX B$WRH) (POS N) (POS LEX)) (GEN F) (NUM S) (STATUS ABSOLUTE))

February 18, 2008CICLing Sample Output (dev-data) maxwell anurpung comes from ghana for israel four years ago and since worked in cleaning in hotels in eilat a few weeks ago announced if management club hotel that for him to leave israel according to the government instructions and immigration police in a letter in broken english which spread among the foreign workers thanks to them hotel for their hard work and announced that will purchase for hm flight tickets for their countries from their money

February 18, 2008CICLing Evaluation Results Test set of 62 sentences from Haaretz newspaper, 2 reference translations SystemBLEUNISTPRMETEOR No Gram Learned Manual

February 18, 2008CICLing Hebrew-English: Test Suite Evaluation GrammarBLEUMETEOR Baseline (NoGram) Learned Grammar Manual Grammar

February 18, 2008CICLing Outline Rationale for learning-based MT Roadmap for learning-based MT Framework overview Elicitation Learning transfer Rules Automatic rule refinement Learning Morphology Example prototypes Implications for MT with vast parallel data Conclusions and future directions

February 18, 2008CICLing Implications for MT with Vast Amounts of Parallel Data Phrase-to-phrase MT ill suited for long-range reorderings  ungrammatical output Recent work on hierarchical Stat-MT [Chiang, 2005] and parsing-based MT [Melamed et al, 2005] [Knight et al] Learning general tree-to-tree syntactic mappings is equally problematic: –Meaning is a hybrid of complex, non-compositional phrases embedded within a syntactic structure –Some constituents can be translated in isolation, others require contextual mappings

February 18, 2008CICLing Implications for MT with Vast Amounts of Parallel Data Our approach for learning transfer rules is applicable to the large data scenario, subject to solutions for several large challenges: –No elicitation corpus  break-down parallel sentences into reasonable learning examples –Working with less reliable automatic word alignments rather than manual alignments –Effective use of reliable parse structures for ONE language (i.e. English) and automatic word alignments in order to decompose the translation of a sentence into several compositional rules. –Effective scoring of resulting very large transfer grammars, and scaled up transfer + decoding

February 18, 2008CICLing Implications for MT with Vast Amounts of Parallel Data Example: 他 经常 与 江泽民 总统 通 电话 He freq with J Zemin Pres via phone He freq talked with President J Zemin over the phone

February 18, 2008CICLing Implications for MT with Vast Amounts of Parallel Data Example: 他 经常 与 江泽民 总统 通 电话 He freq with J Zemin Pres via phone He freq talked with President J Zemin over the phone NP1 NP2 NP3

February 18, 2008CICLing Conclusions There is hope yet for wide-spread MT between many of the worlds language pairs MT offers a fertile yet extremely challenging ground for learning-based approaches that leverage from diverse sources of information: –Syntactic structure of one or both languages –Word-to-word correspondences –Decomposable units of translation –Statistical Language Models AVENUE’s XFER approach provides a feasible solution to MT for languages with limited resources Promising approach for addressing the fundamental weaknesses in current corpus-based MT for languages with vast resources

February 18, 2008CICLing

February 18, 2008CICLing Mapudungun-to-Spanish Example Mapudungun pelafiñ Maria Spanish No vi a María English I didn’t see Maria

February 18, 2008CICLing Mapudungun-to-Spanish Example Mapudungun pelafiñ Maria pe-la-fi-ñMaria see-neg-3.obj-1.subj.indicativeMaria Spanish No vi a María negsee.1.subj.past.indicativeaccMaria English I didn’t see Maria

February 18, 2008CICLing V pe pe-la-fi-ñ Maria

February 18, 2008CICLing V pe pe-la-fi-ñ Maria VSuff la Negation = +

February 18, 2008CICLing V pe pe-la-fi-ñ Maria VSuff la VSuffG Pass all features up

February 18, 2008CICLing V pe pe-la-fi-ñ Maria VSuff la VSuffG VSuff fi object person = 3

February 18, 2008CICLing V pe pe-la-fi-ñ Maria VSuff la VSuffGVSuff fi VSuffG Pass all features up from both children

February 18, 2008CICLing V pe pe-la-fi-ñ Maria VSuff la VSuffGVSuff fi VSuffGVSuff ñ person = 1 number = sg mood = ind

February 18, 2008CICLing V pe pe-la-fi-ñ Maria VSuff la VSuffGVSuff fi VSuffGVSuff ñ Pass all features up from both children VSuffG

February 18, 2008CICLing V V pe pe-la-fi-ñ Maria VSuff la VSuffGVSuff fi VSuffGVSuff ñ Pass all features up from both children VSuffG Check that: 1) negation = + 2) tense is undefined

February 18, 2008CICLing V pe pe-la-fi-ñ Maria VSuff la VSuffGVSuff fi VSuffGVSuff ñ VSuffG V NP N Maria N person = 3 number = sg human = +

February 18, 2008CICLing Pass features up from V pe pe-la-fi-ñ Maria VSuff la VSuffGVSuff fi VSuffGVSuff ñ VSuffG NP N Maria N S V Check that NP is human = + V VP

February 18, 2008CICLing V pe Transfer to Spanish: Top-Down VSuff la VSuffGVSuff fi VSuffGVSuff ñ VSuffG NP N Maria N S V VP S

February 18, 2008CICLing V pe Transfer to Spanish: Top-Down VSuff la VSuffGVSuff fi VSuffGVSuff ñ VSuffG NP N Maria N S V VP S NP“a” V Pass all features to Spanish side

February 18, 2008CICLing V pe Transfer to Spanish: Top-Down VSuff la VSuffGVSuff fi VSuffGVSuff ñ VSuffG NP N Maria N S V VP S NP“a” V Pass all features down

February 18, 2008CICLing V pe Transfer to Spanish: Top-Down VSuff la VSuffGVSuff fi VSuffGVSuff ñ VSuffG NP N Maria N S V VP S NP“a” V Pass object features down

February 18, 2008CICLing V pe Transfer to Spanish: Top-Down VSuff la VSuffGVSuff fi VSuffGVSuff ñ VSuffG NP N Maria N S V VP S NP“a” V Accusative marker on objects is introduced because human = +

February 18, 2008CICLing V pe Transfer to Spanish: Top-Down VSuff la VSuffGVSuff fi VSuffGVSuff ñ VSuffG NP N Maria N S V VP S NP“a” V VP::VP [VBar NP] -> [VBar "a" NP] ((X1::Y1) (X2::Y3) ((X2 type) = (*NOT* personal)) ((X2 human) =c +) (X0 = X1) ((X0 object) = X2) (Y0 = X0) ((Y0 object) = (X0 object)) (Y1 = Y0) (Y3 = (Y0 object)) ((Y1 objmarker person) = (Y3 person)) ((Y1 objmarker number) = (Y3 number)) ((Y1 objmarker gender) = (Y3 ender)))

February 18, 2008CICLing V pe Transfer to Spanish: Top-Down VSuff la VSuffGVSuff fi VSuffGVSuff ñ VSuffG NP N Maria N S V VP S NP“a” V V“no” Pass person, number, and mood features to Spanish Verb Assign tense = past

February 18, 2008CICLing V pe Transfer to Spanish: Top-Down VSuff la VSuffGVSuff fi VSuffGVSuff ñ VSuffG NP N Maria N S V VP S NP“a” V V“no” Introduced because negation = +

February 18, 2008CICLing V pe Transfer to Spanish: Top-Down VSuff la VSuffGVSuff fi VSuffGVSuff ñ VSuffG NP N Maria N S V VP S NP“a” V V“no” ver

February 18, 2008CICLing V pe Transfer to Spanish: Top-Down VSuff la VSuffGVSuff fi VSuffGVSuff ñ VSuffG NP N Maria N S V VP S NP“a” V V“no” ver vi person = 1 number = sg mood = indicative tense = past

February 18, 2008CICLing V pe Transfer to Spanish: Top-Down VSuff la VSuffGVSuff fi VSuffGVSuff ñ VSuffG NP N Maria N S V VP S NP“a” V V“no” vi N María N Pass features over to Spanish side

February 18, 2008CICLing V pe I Didn’t see Maria VSuff la VSuffGVSuff fi VSuffGVSuff ñ VSuffG NP N Maria N S V VP S NP“a” V V“no” vi N María N

February 18, 2008CICLing