1. Post-Editing – Professional translation service redefined
Darja Fišer
University of Ljubljana
5 December 2014
Brussels, Belgium

2. Presentation Outline Basic concepts about post-editing
Quality in translation projects
Types of post-editing
Post-editing guidelines

3. MOTIVATION
Why should I care about post-editing?

4. Why PEMT? Increasing demand for PEMT in the market:
increasing volume of short-lived documents
different levels of text quality acceptable
The industry perspective on MT:
to lower productivity prices
to publish more content
to publish in more languages
to publish in less time
The TAUS (2010) survey:
52% companies in the US, EU & Asia provide PE services regularly
74% of the resources they used are freelance translators

5. the big picture
How does MT affect the translation process and the translator?

6. Integrating MT in the translation process
Phase 1:
Translation Memories
Source
text 0%
translated
Translation Memory (TM)
Hybrid x%
Phase 2:
Machine Translation
(MT)
Hybrid
text
100%
translated but with MT errors
Un-translated segments
Phase 3:
Post-editing
Target
text
100%
translated
Post-editor

7. The role of translators in PEMT
The role of PEMT experts:
edit the output
select the adequate corpus
clean up the data so the output is more suitable for the customer
provide constant feedback to improve the system’s performance
the role changes as MT improves
The nature of PEMT projects:
large contents of highly repetitive nature, short-lived, internal use
pre-editing (at the SL level before MT to avoid ambiguous input to the MY system)
post-editing (at the TL level after MT to correct errors in the MT output)

8. basic concepts in pemt
Is post-editing a translation task or a revision task?

9. Post-editing vs. Translation
reviewing an MT text against an original text and correcting any errors in order to comply with a set of quality criteria in as few edits as possible
the set of quality criteria ≠ a personal idea of translation quality
as few edits as possible > to increase the productivity
PE vs. Translation:
Translation: 1 source
PE: 2 sources (the original & the raw MT output):
reject the MT output & translate from scratch (PE closer to Trans than Rev)
correct a lot/a few of errors (PE closer to Rev than Trans)
accept the proposed translation as is (PE closer to Rev than Trans)
PE should be done by a translator, not a monolingual reviewer!

10. Post-editing vs. Revision
PE:
deals with recurring, predictable errors
MT texts put a strain on the post-editing expert, so PE is more cognitively demanding
Revision:
checks for random mistranslations or omissions
human errors more difficult to spot but the texts are easier to read
PE & Revision both require specific skills and should be tackled by translators trained & experienced in the task!
> 100,000 words / 1 month of full-time post-editing

11. the job profile
What skills and qualities do I need to be a good post-editing expert?

12. Skills for post-editing (O’Brien 2002)
Degree in translation or related subjects
Expert in the subject area and target language
Proficient in the source language and contrastive issues
Experience in technical translation/localization
Advanced word processing skills, full key proficiency (search&replace)
Positive, tolerant and open-minded towards MT
Confidence in abilities and technical expertise
Recognition of typical and repetitive MR errors
Ability to use macros and coded dictionaries
Advanced terminology management skills
Background MT knowledge, types of PE and levels of expected quality
Pre-editing skills (controlled language & controlled authoring tools)
Programming skills for automatically correcting errors

13. MT quality
What can I expect from MT and what can clients expect from PEMT?

14. Common MT errors What MT errors to expect:
Depend on the MT system, the content and the language pair used!
error analysis time-consuming but:
crucial to improve the MT system
crucial to raise awareness about the post-editing task
Several error classifications exist (Schäffer, 2003):
Lexical errors (general vocabulary, terminology, polysemy, idioms)
Syntactic errors (sentence analysis, word order)
Grammatical mistakes (tense, number, gender, case, punctuation)
Errors due to defective input (mistakes in the source language)

15. Quality in technical translation and localization
Functionalist’s approach to quality:
the focus is on the customer’s needs and what they pay for
quality is variable and is defined by clients, not the society in general
Fit-for-purpose! (not what trained translators would consider the best)
Quality of MT
standard MT evaluation measures (BLEU, Meteor, NIST, TER):
how close the input is to human quality with a single number
not very reliable in most translation projects
manual quality assessment needed!
crucial for productivity savings & pricing
random strings checked for grammar, terminology and format (grades 1-5)
very specific client’s quality expectations needed! (rapid/full PE)
Quality of PE
MT is used to save costs, so revision of PE texts is usually not done
Crucial to strike a balance between speed and the quality of PE

16. types of post-editing
How much post-editing should I do?

17. Different levels of post-editing
No post-editing
directly published on the internet, with disclaimer
Rapid post-editing
suitable for short-lived documents needed gisting & internal use
min editing, shortest time possible, min no. of changes, to remove blatant & significant errors, no stylistic changes
Full post-editing
leading to human quality, required for texts for publication
max editing, all errors and stylistic changes taken into account (but still in less time than translating from scratch)
Criteria:
the MT system and language pair used
the domain and structure of the text
the use of the final text, the desired quality and the type of readers
the volume of translation and the time available

18. post-editing guidelines
What exactly should I correct and how much?

19. General guidelines for PE
Language- and project-specific guidelines needed for each project!
as short and precise as possible:
a description of the MT system and the source text used
a description of the quality of MT output and the expected quality of the finished translation
scenarios when to discard a useless segment
typical types of errors that need to be corrected
changes to be avoided
terminology issues

20. Guidelines for rapid PE
Read the source segment first
Read the MT suggestion
Make the necessary changes:
Make sure the content of the sentence is accurate
If the terminology is incorrect, don’t spend too much time researching
Don’t post-edit word-order if the sentence can be understood as is
Don’t change style
Don’t replace words with a synonym
Don’t correct grammar mistakes unless the target sentence doesn’t reflect the meaning of the source sentence

21. Guidelines for full PE Always very project-specific
Use the MT suggestion if:
a large piece of the sentence is correct
the raw MT quality is very good with only minor corrections needed
the raw MT quality is not so good but would still be faster to correct it than to translate from scratch
the MT has the correct meaning and is completely understandable
Don’t use the MT suggestion if:
the raw MT doesn’t make any sense and it would take longer to correct it than to translate from scratch
you need a more than a few seconds to understand it
there are errors that would require rearranging most of the text

22. Examples from the guidelines at Microsoft
The 5-10 second evaluation:
the maximum time you should spend evaluating the validity of the MT suggestion
if it is hard to understand already at the beginning, don’t even read the whole sentence, just proceed to translate from scratch instead.
The High-5 & Low-5 rule:
When you detect a long sentence, do the following:
Read the first 5 words. If it’s good, read on until it’s bad, then stop and copy the correct part and continue to translate and forget about reading on.
If the first 5 or 6 words aren’t good, skip to read the last 5 or 6 words. It the last part of the sentence is correct, use it, or just start the whole thing from scratch.
If both first 5 and last 5 words are incorrect, do not carry on reading through the middle to try to identify correct MT segments. Just discard the MT suggestion and proceed to translate from scratch.

23. Post-editing effort and productivity
How hard will post-editing be and how much will I gain?

24. Post-editing effort
Key element to decide if the use of MT is worthwhile or not (Krings, 2001):
Temporal PE effort
Does PEMT save time vs. human translation?
Does PEMT save time vs. TM fuzzy matches?
Depends of the quality of the raw MT output and type of errors!
Cognitive PE effort
How complex and cognitively demanding are the corrections?
Obvious mistakes (gender) vs. ambiguous complex syntactic structures
Technical PE effort
Does PE require to delete, insert, reorder or a all 3?
Measuring PE effort:
temporal: the easiest to measure
cognitive & technical PE: eye-trackers, Translog, Think Aloud Protocols (useful in research, less so in the commercial world)

25. Post-editing productivity
One of the big unknown factors in PEMT projects
new field, so no standard metrics exist
productivity in PE estimated at 4,000-10,000 words/day
many variables to consider:
the quality of raw MT output?
the productivity of translators in general?
the experience of post-editors?
the amount of effort to post-edit fuzzy matches?
inconclusive results:
early studies: show productivity gains up to 3 times compared to HT (Vasconcellos and Leon 1985)
recent studies: productivity gain not always achieved (O’Brien 2006, Guerberof 2008)
commercial users: many claim high productivity gains but don’t make their methodology available
Test before you commit!

26. lets pemt!
PEMT is here to stay

27. Learn & Teach PEMT
Ride the wave!