Presentation is loading. Please wait.

Presentation is loading. Please wait.

100 years of progress and innovation © 2011 IBM Corporation The value of Post Editing - IBM Case Study Frank X. Rojas, Jian Ming Xu, Santi Pont Nesta,

Similar presentations


Presentation on theme: "100 years of progress and innovation © 2011 IBM Corporation The value of Post Editing - IBM Case Study Frank X. Rojas, Jian Ming Xu, Santi Pont Nesta,"— Presentation transcript:

1 100 years of progress and innovation © 2011 IBM Corporation The value of Post Editing - IBM Case Study Frank X. Rojas, Jian Ming Xu, Santi Pont Nesta, Álex Martínez Corrià, Salim Roukos, Helena Chapman, Saroj K. Vohra June 2011

2 © 2011 IBM Corporation2 IBM Case Study – MT Post Editing  Introduction  MT Innovation  Process Overview  Findings  Conclusion / Recommendations

3 © 2011 IBM Corporation3 IBM World Wide Translation Operations  24 Centers World Wide  ~115 Translation Suppliers  Process ~2.8 B Words  Translate ~0.4 B Words  ~ 60 language pairs One Stop Shop for all Translation Services Marketing Material Web Product Integrated Information Publications Legal/Safety/ Contracts Machine Translation Multimedia Francization Cultural Consultancy Centralized DTP Overall End to End Process Management

4 © 2011 IBM Corporation4 IBM Professional Translation Services Professional Memory 72%  85% Re-Use Unit Cost >50% Reduction Traditional Technology Process Mgmt Human Skill Consistent Quality Standards Global Brand Identity Professional Quality Standards  Future: –Ability to reduce cost using conventional methods reaching limits –Business pressure for additional cost elimination –Looking to MT Technology as next wave to reach business goals

5 © 2011 IBM Corporation5 - MT portal - Generic crowdsourcing - Text translation services June 2008 Historical Perspective RTTS introduced in 2006 as platform for speech and text translation, developed by IBM Research 2010 MT piloting Pilot: SPA, ITA, FRE, GER New E2E process Partnership: WWTO/n.Fluent 8.6 M words 2011 MT Training Pilot: GER, BPR, JPN, CHS MT payment profiles ready 16.0 M words target eSupport (www) “Translate This Page” JPN pilot / rule engine n.Fluent customized with WWTO translation memories eSupport “Translate This Page” switch to n.Fluent RTTS licensed to IBM partners Initial n.Fluent/WWTO Spanish MT pilot Improve efficiency of professional translators

6 © 2011 IBM Corporation6 MT Critical Success Metrics  Necessary and sufficient condition to measure success –5.0 M words sampled –Minimum of 3 languages –Net Contribution to ROI by MT Engine: 10% of payable words should be MT –No more than 5% adverse impact to Overall Quality Index –No more than 5% impact to Customer Satisfaction  Lack of industry metrics and guidance. –Active research on MT technology... no guidance on operational impacts –A business vacuum existed on how to integrate MT services –No operational process had been defined for MT services

7 © 2011 IBM Corporation7  IBM’s Watson Q&A computer  Google’s autonomous car  Technologies to understand and produce natural human speech  Instantaneous, high-quality machine translation  Smartphones / App phones in the developing world * Andrew McAfee is a principal research scientist in the MIT Sloan School of Business Recent Digital Innovations with Biggest Impact in the Business World*

8 © 2011 IBM Corporation8 Real-Time Translation Server (RTTS) & n.Fluent  Real Time Translation Server (RTTS)  IBMs MT Engine  RTTS provides machine translation for n.Fluent & other applications  APIs allow other applications to access these translation services.  Customization tools – Domains, chat-specific models, …  Commercially licensed to IBM partners  Language Pairs to/from English:  n.Fluent  IBMs MT translation application  Providing machine translation services for:  Text, web pages, and documents (Word, Excel, …)  Instant Messaging chats (via IM plug-in)  Mobile translation application (BlackBerry and others)  Enabled with LEARNING via crowdsourcing (internal 450K IBMers)  Deployed for eSupport self serving tech support (external) العربية 中文 Deutsch English Français Italiano 日本語 Português Русский Español 한국어

9 © 2011 IBM Corporation9 - MT portal - Generic crowdsourcing - Text translation services June 2008 Historical Perspective RTTS introduced in 2006 as platform for speech and text translation, developed by IBM Research 2010 MT piloting Pilot: SPA, ITA, FRE, GER New E2E process Partnership: WWTO/n.Fluent 8.6 M words 2011 MT Training Pilot: GER, BPR, JPN, CHS MT payment profiles ready 16.0 M words target eSupport (www) “Translate This Page” JPN pilot / rule engine n.Fluent customized with WWTO translation memories eSupport “Translate This Page” switch to n.Fluent RTTS licensed to IBM partners Initial n.Fluent/WWTO Spanish MT pilot Improve efficiency of professional translators

10 © 2011 IBM Corporation10 TM MT New / Changed 100% Exact Match MT Pre-Process Editing Session MT Post Editing End to End Workflow  Upfront & on-going MT tuning via IBM TM professional translations –Professional translation = Best context  Matching methods –Traditional TM – breaks down segment level –Machine TM – breaks down block level using MT models – reconstructs segments preserving formats/mark-up tags  MT service level integration TM Pre-Process Shipment English TM Match Analysis CAT Translation 1.Show best choice vs vs 2.Select best choice (Post Edit rules) 3. Commit language TESTING QUALITY MT Model & Trans. = Localization Kit (NLV Folder)

11 © 2011 IBM Corporation1118-sept.-08 MT Pre-processing TM New / Changed 100% Exact Match Build dynamic, domain specific MT model MT MT initial corpus General parallel training corpus Domain specific parallel training corpus ALL segment “no match segments ” Translation of no match segments  Initial MT corpus –done before start of project Localization kit TM MT New / Changed 100% Exact Match

12 © 2011 IBM Corporation1218-sept.-08 Xxx xxx xx xxx xxx xxx. La aplicación desprotege los archivos antes de exportarlos. Yy yyy yyy TM Editing Environment TM Environment Xxx xxx xx xxx xxx xxx. The application unprotects files before exporting them. Yy yyy yyy Translation Memory 0 - The application unprotects files before exporting them. 1[m] – La aplicación desprotege archivos antes de exportarlos. 2[f 85%] - La aplicación protege los archivos antes de exportarlos TM Environment [Ctrl + 1] Typed Translator options  Ignore fuzzy and MT  Post edit MT  Post edit fuzzy Two Seconds Rule: Translators are trained on several strategies to make a quick choice TM MT

13 © 2011 IBM Corporation13 Productivity Measurements  Start segment –Choose action  End segment  MT productivity evaluation log (MTeval Log) –N events –Words | Time | Existing Proposal | Used Proposal |...  Examine productivity per payment category –SUM(Words) / SUM(Time) –Use of IBM Business Analytic Tool (SPSS) –Trim events that fall into 5% (slowest) and 95% (fastest) percentile 1.accept match [~0 time] 2.edit match [X time] 3.reject match [manual translation] Each event EM : Exact RM : Replace FM : Fuzzy MT : Machine NP : No Proposal A) = “best” Existing Proposal B) = “alternative” Existing Proposal C) = reject all Existing Proposal, 100% human labor

14 © 2011 IBM Corporation14  Total # events : 2,309 (377+1,932)  Total words: 24,150 Total time: 27,362 –3,911 w/ MT match 11,377 w/ MT match –20,239 w/o MT match 15,985 w/o MT match  MT impact to productivity –MT : 0.44 words/sec [1777 words / 4071 sec] –NP 0.21 w/ MT match 0.32 w/o MT match  Baseline (placebo)  MT Leverage : 71.8% [1777 / ( )] Single Shipment EXAMPLE rate(MT) / rate(NP) : 1.37 i.e. Translator can complete 37% more words in the same time. Key metrics

15 © 2011 IBM Corporation15 MT Impact on Fuzzy Match : 4Q10 Findings  When FM & MT matches exist simultaneously  Productivity: rate(MT) / rate(NP): a.Case : Translator edits FM b.FM-MT Combined case c.Case: Translator edits MT ** Findings subject to change with additional sampling.  Overall –Machine matches not as good as professional (fuzzy) matches –No statistical impact to fuzzy productivity to include MT matches. SPA highest sample case 28.6% 4.4% 57.6% 46.9% FM-MT Pick Rate: FREGERITASPA Productivity ratio FM FM-MT MT

16 © 2011 IBM Corporation16 MT Key Metrics: 4Q10 Findings  8.6 M words sampled in real time translation service.  SPA : Qualified MT engine 4Q10  ITA : Qualified MT engine 4Q10  FRA : Qualified MT engine 1Q11 While rate(MT) / rate(NP) is high, the findings were not statistically significant in 4Q.  GER : Insufficient productivity from MT engine ** Findings subject to change with additional sampling.

17 © 2011 IBM Corporation17 Overall Savings Assessment  Overall savings % –Word savings due to MT efficiency Convert time savings  MT payment factor % –MT payment factor X [MT % words + NP % words] Results in less payable words.  MT productivity savings drives a overall savings –These are not the same due to MT % distribution.  Supply chain has to consider cost of MT services ** Findings subject to change with additional sampling.

18 © 2011 IBM Corporation18 Pay for MT Words Translated not MT Matches  We pay for final results (MT payable words) not MT matches –MT matches considered “opinion” until chosen by a human –Too many opinions & opinions by immature MT models are less efficient.  Actual MT payable words have value beyond the specific project –Post Edited words are reused in future and unknown MT context  Engine has to deliver consistent MT payable words –Minimum needed to quality an MT engine for compensation High MT productivity[rate(MT) / rate(NP)] High MT leverage[% of MT matches used] –Compensation to be based on MT payment factor

19 © 2011 IBM Corporation19 Variance across Languages  There is no single maturity path when modeling MT engines across many languages.  IBM Pilot: each trained MT engine is a unique asset. –Some languages require more modeling/tuning than others. –Language pairs that service “Loose -> Structured” languages are struggling German requires more effort than Spanish  Are there limitations to statistical MT engines? –New thinking may need to be explored?  Each MT engine will have separate MT payment factors.

20 © 2011 IBM Corporation20 Perspective of MT Post Edit Pilots Translation Service Hierarchy Professional Translation Services (Professional LSP) Community Translation Services (Controlled Social Crowd) Volunteer Translation Services (General Crowds) Free Services (Individual) Quality / Reliability LOWER HIGHER General Domain Specific internal IBM All IBM external/internal Pubs / UI external (2011 Pilots) internal IBM n.Fluent “machine” WWTO “human” New Memory Assets MT Post Editing has impacts across entire Translation Service Hierarchy

21 © 2011 IBM Corporation21 1.Professional (Human) memories are the best assets and deliver the highest quality. 2.Professional memories are a key asset for MT success. 3.All Memory assets need to be protected and managed. 4.Flow of memories between Professional and Machine must be properly balanced. 5.Dynamic modeling offers significant advantage over static modeling. 6.Continuous business analytics is needed to optimize machine assets. 7.A single cost model per language is needed, independent of MT services/engines. 8.An aggressive yet cautious approach is warranted to go forward. MT Post Editing Project – Key Lessons MT Post Editing does improve productivity and efficiency of a localization supply chain.


Download ppt "100 years of progress and innovation © 2011 IBM Corporation The value of Post Editing - IBM Case Study Frank X. Rojas, Jian Ming Xu, Santi Pont Nesta,"

Similar presentations


Ads by Google