Applying Machine Translation Metrics to Student-Written Translations Lisa N. Michaud Computer Science Department Merrimack College North Andover, Massachusetts, USA Patricia Ann McCoy Language Department Universidad de las Americas Puebla Puebla, Mexico

Evaluating learner translation Michaud and McCoy2 Translating ancient texts and doing what MT cannot do yet

Criteria for judging translations fluency (is it well-formed?) fidelity (does it convey original meaning?) (Hovy et al., 2002) 3Michaud and McCoy

Multiplicity of translations  In each one of these jobs the professor could have agreed to work 6 hours a day and therefore would not be surpassing the working day hour limit.  In each one of these jobs the teacher could have agreed to work 6 hours per day and therefore he wouldn't be bound by the limits of the working day.  In each of these examples the teaching could have been arranged so that he/she works six hours a day and would not be affected by any workday limitations.  In both of these jobs the professor could have agreed to work six hours daily and therefore he wouldn't be affecting his work shift limit. 4Michaud and McCoy

Multiplicity of translations  In each one of these jobs the professor could have agreed to work 6 hours a day and therefore would not be surpassing the working day hour limit.  In each one of these jobs the teacher could have agreed to work 6 hours per day and therefore he wouldn't be bound by the limits of the working day.  In each of these examples the teaching could have been arranged so that he/she works six hours a day and would not be affected by any workday limitations.  In both of these jobs the professor could have agreed to work six hours daily and therefore he wouldn't be affecting his work shift limit. 5Michaud and McCoy

BLEU 6Michaud and McCoy Hypothesis Multiple References

TERp 7Michaud and McCoy Hypothesis Single Reference PHRASAL EQUIVALENCE or SYNONYM SHIFTSUBSTITUTION INSERTION SAME STEM

TERp alignment and tags 8Michaud and McCoy

Student translation corpus 9Michaud and McCoy

Sentence boundaries 10Michaud and McCoy EXPERT: Due to the fact that previous studies have shown that POMC neurons are linked to obesity, the scientists believe the findings might also lead to developing treatments to control obesity and other metabolic disorders. STUDENT: Past studies have shown that the POMC neurons are involved with obesity. The scientists believe this finding could also help develop treatments for the obese and other metabolic disorders. ORIGINAL: Estudios en el pasado han demostrado que las neuronas POMC están involucradas en la obesidad, por lo tanto los científicos creen que el hallazgo también ofrece la posibilidad de desarrollar tratamientos para controlar la obesidad y otros trastornos metabólicos.

Does TERp agree with an expert? Michaud and McCoy11 Instructor Scores vs Inverted TERp 650 sentences (22%) Pearson Correlation r =

Score distribution 12Michaud and McCoy

Instructor rubric (original) Michaud and McCoy13 10Excellent 9Good 8Satisfactory 0-7 Deficient

TERp scores over the semester 14Michaud and McCoy

Evaluating TERp tags (pilot) Michaud and McCoy15

Future work Michaud and McCoy16

Instructor rubric (revised) Michaud and McCoy Excellent 90Good 80Satisfactory 0-70 Deficient

Modifying the TERp Score 18Michaud and McCoy Hypothesis Single Reference PHRASAL EQUIVALENCE or SYNONYM SHIFTSUBSTITUTION INSERTION SAME STEM

Recognizing false cognates 19Michaud and McCoy Hypothesis Single Reference SUBSTITUTION cynical brazen Source cínico

Extracting mistranslation pairs Michaud and McCoy20 SPANISH DICTIONARY ENGLISH DICTIONARY cynical cínico cynical brazen zonazone