Automating Translation in the Localisation Factory An Investigation of Post-Editing Effort Sharon O’Brien Dublin City University

Assumptions about MT T (MT + PE) < T (Trans)

Do we have proof? Dated studies: Pan-American Health Organisation General Motors European Union 3-4 times faster than translation But: No details given More Recently: Average daily throughput for PE: 5,250 words per day Krings (2001): only thorough, published empirical data on PE rates

MT + CL CL: Relatively young field of research/implementation Consequently: little empirical data

CL improves “translatability” The notion of translatability is based on so-called "translatability indicators" where the occurrence of such an indicator in the text is considered to have a negative effect on the quality of machine translation. The fewer translatability indicators, the better suited the text is to translation using MT. (Underwood and Jongejan, 2001: 363)

Can we prove it - empirically? By using CL rules to eliminate negative “translatability indicators”, post-editing effort of MT output will be lower than for output where negative translatability indicators have not been removed.

Experimental Set-Up Validity! Professional, experienced subjects, native speakers (German) Homogenous backgrounds and level of experience Familiar text (user guide) Familiar working environment Payment for time However: limited number of subjects

Framework of Analysis How do you measure post-editing “effort”? Temporal Technical Cognitive Two sentence types: “Snti” “Smin-nti”

Framework of Analysis Temporal Effort: How much time, in seconds, did it take to post-edit each sentence? Technical Effort: How many deletions, insertions, cut & pastes were made for each sentence? Cognitive Effort: Combined Temporal & Technical Additional measurement: Choice Network Analysis

Analysis Tools IBM Websphere Translog Excel

Translog User Interface

Translog Log File

Results: General Temporal Effort

Temporal Effort: Individual Variation

Temporal Effort by Sentence Type Processing Speed: the total number of source words in each segment divided by the total processing time for that segment

Processing Speed by Sentence Type

Technical Effort by Sentence Type

Technical Effort: Cut & Paste Very little activity! Retyping of entire phrases rather than cutting & pasting Less effort to re-type? Need for training?

Cognitive Effort On average, the elimination of NTIs suggests that PE effort is reduced. However, CNA shows: More edits to some NTIs than to others Even though NTIs have been removed from a sentence, this does not guarantee zero post-editing

High PE Effort Gerund (“ing” form of verb) Ungrammatical Phrase Putting an adjective after the noun Non-finite verb (no tense marked) Slang Misspelling Long Noun Phrase Ellipsis Long Sentence (more than 25 words) Verbs with particles Use of Footnotes Multiple Prepositions Short Segment (fewer than 4 words)

Medium PE Effort Multiple Coordinators Problematic Punctuation Passive Voice Phrase not syntactically complete Use of Personal Pronouns Use of Slash as a separator Ambiguous coordination Use of brackets Proper Nouns Missing “that” in a relative clause

Low PE Effort Abbreviations Demonstrative Pronouns Missing “in order to” Contractions (“Let’s”)

Conclusions Taking into account that no QA was performed on the final texts: On average post-editing can be faster than translation High degree of individual variation On average, removing NTIs reduces PE Effort But some NTIs demand more effort than others

Conclusions Even if all known NTIs are removed, sentences may still require PE effort.

Conclusions Not all CL rules will have equal impact Even if CL is applied, PE effort will not be removed completely Post-editors are still human and still translators…