July 24, 2007GALE Update: Alon Lavie1 Statistical Transfer and MEMT Activities Chinese-to-English Statistical Transfer MT system (Stat-XFER) –Developed over past year –Included in Rosetta Phase-II GNG evaluation –Team: Erik Peterson and Alon Lavie Multi-Engine Machine Translation –MEMT service within the cross-GALE IOD –MEMT system combination for GNG evaluation within Rosetta consortium –Team: Greg Hanneman, Shilpa Aurora, Dave Svoboda, Alon Lavie and Eric Nyberg

July 24, 2007GALE Update: Alon Lavie2 CMU Statistical-XFER System Truly hybrid “rule-based”/statistical system Scaled up version of our XFER approach developed for low-resource languages (under NSF funding) Large-coverage “clean” bilingual lexicon + syntactic transfer rules (human written + extracted from data) XFER formalism is a Synchronous CFG + feature unification constraints Supports morphological analysis and generation as “plug in” components Two-stage translation process: –Build lattice of translation constituents at all grammar levels “bottom-up” –Monotonic decoder selects best combination of lattice edges –Beam-search with multiple features at both stages –Features include: LM, fragmentation, lexical probabilities, length, etc. –Optimized Log-linear combination of feature scores

July 24, 2007GALE Update: Alon Lavie3 Chinese-English S-XFER System Bilingual lexicon: over 1.1 million entries (multiple resources, incl. ADSO, Wikipedia, extracted base NPs) Manual syntactic XFER grammar: 76 rules! (mostly NPs, a few PPs, and reordering of NPs/PPs within VPs) Multiple overlapping Chinese word segmentations English morphology generation Uses CMU SMT-group’s Suffix-Array LM toolkit for LM Current Performance (GALE dev-test): –NW XFER: 10.89(B)/0.4509(M) Best (UMD): 15.58(B)/0.4769(M) –NG XFER: 8.92(B)/0.4229(M) Best (UMD): 12.96(B)/0.4455(M) In Progress: –Automatic extraction of “clean” base NPs from parallel data –Automatic learning and extraction of high-quality transfer-rules from parallel data

July 24, 2007GALE Update: Alon Lavie4 Recent Performance Analysis What fraction of the time does each MT system produce the best translation (sentence-by-sentence)? Evaluated on Chinese GALE dev-test (text) data BLEU METEOR CMU-PhraseSyntaxCombination 60 of 284 (21.1%) 41 of 284 (14.4%) IBM-smt 50 of 284 (17.6%) 49 of 284 (17.2%) IBM-ylee 64 of 284 (22.5%) 50 of 284 (17.6%) maryland-jhu-combination 71 of 284 (25.0%) 77 of 284 (27.1%) Stat-XFER 32 of 284 (11.2%) 56 of 284 (19.7%)

July 24, 2007GALE Update: Alon Lavie5 Translation Example REFERENCE: When responding to whether it is possible to extend Russian fleet's stationing deadline at the Crimean peninsula, Yanukovych replied, "Without a doubt. Stat-XFER (0.3989): In reply to whether the possibility to extend the Russian fleet stationed in Crimea Pen. left the deadline of the problem, Yanukovich replied : " of course. IBM-ylee (0.2203): In response to the possibility to extend the deadline for the presence in Crimea peninsula, the Queen Vic said : " of course. CMU-SMT (0.2067): In response to a possible extension of the fleet in the Crimean Peninsula stay on the issue, Yanukovych vetch replied : " of course. maryland-hiero (0.1878): In response to the possibility of extending the mandate of the Crimean peninsula in, replied: "of course. IBM-smt (0.1862):The answer is likely to be extended the Crimean peninsula of the presence of the problem, Yanukovych said: " Of course. CMU-syntax (0.1639): In response to the possibility of extension of the presence in the Crimean Peninsula, replied : " of course.

July 24, 2007GALE Update: Alon Lavie6 Synthetic Combination MEMT Two Stage Approach: 1.Identify common words and phrases across the translations provided by the engines 2.Decode: search the space of synthetic combinations of words/phrases and select the highest scoring combined translation Example: 1.announced afghan authorities on saturday reconstituted four intergovernmental committees 2.The Afghan authorities on Saturday the formation of the four committees of government

July 24, 2007GALE Update: Alon Lavie7 Synthetic Combination MEMT Two Stage Approach: 1.Identify common words and phrases across the translations provided by the engines 2.Decode: search the space of synthetic combinations of words/phrases and select the highest scoring combined translation Example: 1.announced afghan authorities on saturday reconstituted four intergovernmental committees 2.The Afghan authorities on Saturday the formation of the four committees of government MEMT: the afghan authorities announced on Saturday the formation of four intergovernmental committees

July 24, 2007GALE Update: Alon Lavie8 Scoring MEMT Hypotheses Scoring Features: –Word confidence score [0,1] based on engine confidence and reinforcement from alignments –LM score based on suffix-array 5-gram LM –Exponential “occurrence-based” long n-gram model –Select best scoring hypothesis based on optimized log- linear combination of feature scores

July 24, 2007GALE Update: Alon Lavie9

July 24, 2007GALE Update: Alon Lavie10 MEMT – Main Activities Preserving Source Alignments: target phrases that originate from same source word can be marked as unbreakable units (performance effects are currently under testing…) LM experiments using CMU’s Suffix-Array LM toolkit and new LM features (work in progress) Case Restoration: scheme for selecting the case of words in final MEMT output Handling of varying number of MEMT input engines Parameter Optimization for MEMT decoder Upgrades to MEMT software infrastructure to support IOD-3 requirements: UIMA v2.0 and ActiveMQ MEMT server is up 24/7 for ongoing IOD runs Active participation in phase-II GNG evaluation

July 24, 2007GALE Update: Alon Lavie11 Recent Evaluation Results

July 24, 2007GALE Update: Alon Lavie12 MEMT Fresh Example Extracted from IOD overnight run of Sunday, July 22 IBM_MT: “defence minister announced that the Afghan security forces and Afghan joint NATO began this morning the process of wide security” RWTH_MT: "afghan defence minister that the security forces, a joint us-afghan started today in the morning, the implementation of a wide security operation.” CMU_MEMT: "afghan defence minister announced that the security forces and Afghan joint NATO began this morning in the process of the a wide security.”

July 24, 2007GALE Update: Alon Lavie13 Future Plans and New Directions Classifiers for Hypothesis Selection –Simpler than MEMT, but perhaps more effective (given recent analysis results) Constrained Search-spaces for MEMT –Other groups use more constrained combination spaces with good results – can we also do better? Discriminative feature-rich LMs for MT (and MEMT) –Standard statistical LMs are not sufficiently discriminative for MT –New NSF grant (with Rebecca Hwa) to explore novel feature-rich “occurrence-based” models M-TER –Create a fully automatic metric that approximates H-TER using the stemming and synonymy capabilities of METEOR to create “targeted” references

July 24, 2007GALE Update: Alon Lavie14 Translation Example REFERENCE: I believe there is a detail that worth mentioning. Stat-XFER (0.5799):There is a detail, I believe worth together. maryland-hiero(0.3221):There is a one details, I think they deserve. IBM-ylee (0.2525):There is one of the details, I think it is worth. IBM-smt (0.2062):There is a details, I think is a way. CMU-SMT (0.2000):One of the details, I think it is worth to join in. CMU-syntax (0.1546):a detail, I think it is a cold front. maryland-jhu (0.1031):One to details, I think they deserve.

July 24, 2007GALE Update: Alon Lavie15 Translation Example REFERENCE: Xinhua News Agency, Zhuhai, Nov. 2 (reporters Wang Hongshan, Li Xuanliang) - Yang Liwei, Deputy Director of China Astronaut Research and Training Center and 'Space Hero', said in Zhuhai on the 1st that China has no plan to select female astronaut yet. Stat-XFER (0.6349): The Xinhua News Agency and the Zhuhai November seconds ( reporters Wang Hongshan, li announced good ) Chinese astronaut research and training center deputy director, " aerospace hero " Yang Liwei and the first said in zhuhai, china now not yet to select woman astronaut plan. IBM-ylee (0.5325): Zhuhai November 2 ( Xinhua News Agency, Chinese astronauts scientific training center, deputy director of the " space hero Yang Liwei said on July 1 in Zhuhai, China is still no plan and the flight, astronauts. IBM-smt (0.5194): Xinhua Zhuhai, November 2nd (Xinhua) (Reporter,) Chinese astronauts scientific training, deputy director of the Center, " Space hero ' yang Liwei, 1st in Zhuhai, said that China has no plan of selecting astronaut. maryland-hiero (0.5180): ZHUHAI, November 2 (Xinhua, ) deputy director of the Chinese astronauts scientific research and training center, "1" space hero Yang Liwei said in Zhuhai, China does not plan to make astronaut selection. CMU-combination (0.4701): zhuhai, november 2nd ( xinhua ) deputy director of china's astronaut training centre, " space hero yang liwei, 1st in zhuhai, china currently has no plans of the selected astronaut.

July 24, 2007GALE Update: Alon Lavie16 Chinese-English Example - Before THE SCIENTISTS IN ORDER TO Øü TO CLOSE IN THE EARLY PERIOD TO GO THE THE KNOWLEDGE THE THE DISEASE IN THE CHROMOSOME HAS BEEN COMPLETED IS SCHEDULED TO ORDER Overall: , Prob: , Rules: , Frag: 0.4, Length: , Words: 13,

July 24, 2007GALE Update: Alon Lavie17 Chinese-English Example - After THE SCIENTISTS COMPLETED SEQUENCING FOR THE CHROMOSOMES WHICH RELATED TO THE INITIAL STAGE DEMENTIA Overall: , Prob: , Rules: , Frag: 0, Length: , Words: 8, < : ¿Æѧ¼Ò Ϊ Øü¹Ø ³õÆÚ Ê§ÖÇÖ¢ µÄ ȾɫÌå Íê³É ¶¨Ðò (S,1 (NP,1 (LITERAL 'THE') (NB,1 (N,21601 'SCIENTISTS'))) (VP,4 (VP,1 (V,7513 'COMPLETED')(NP,2 (NB,1 (N, 'SEQUENCING')))) (PP,1 (PREP,5 'FOR')(NPRC,1 (NP,1 (LITERAL 'THE') (NB,1 (N, 'CHROMOSOMES'))) (LITERAL 'WHICH') (VP,1 (V,18 'RELATED TO') (NPASSOC,5 (NP,1 (LITERAL 'THE') (NB,1 (N,7637 'INITIAL STAGE'))) (NP,2 (NB,1 (N,445 'DEMENTIA')))))))))>

July 24, 2007GALE Update: Alon Lavie18 C/E Stat-XFER Lexicon Lexical sources and their sizes: –Named Entities from LDC lexical entries –Filtered base NPs from Parallel Corpus lexical entries –ADSO bilingual lexicon lexical entries –LDC word bilingual glossary lexical entries –Wikipedia extracted bilingual lexicon lexical entries –Phrases from Parallel Corpus lexical entries –Manual bilingual lexicon (high freq) 1149 lexical entries