Presentation is loading. Please wait.

Presentation is loading. Please wait.

Dr. Stephen Doherty & Dr. Sharon O’Brien

Similar presentations


Presentation on theme: "Dr. Stephen Doherty & Dr. Sharon O’Brien"— Presentation transcript:

1 Assessing the Usability of Machine Translated Content: A User-Centred Study using Eye Tracking
Dr. Stephen Doherty & Dr. Sharon O’Brien Centre for Next Generation Localisation School of Applied Language & Intercultural Studies Dublin City University

2 Outline Introduction Research Aims Methods Results Conclusions

3 Introduction Increased need for translation
Diversity of content and users Rise in prevalence of machine translation [MT] both off- and online Mixed reports of quality – attitudes and expectations Divergence in R&D – translation studies/computer science Evaluation metrics – human and automatic Our focus here is on usability

4 Research Aims To investigate if there are differences in usability between the English [source language] and the unedited machine translated target languages [FR, DE, SP, JP]. Or in other words: how usable is machine translated content? Adoption of the ISO/TR definition of usability Importance of ecological validity: real materials and users

5 Methods User-centred approach [n = 30]; task driven – ‘new user’ scenario Eye tracking [tobii 1750]: Fixation count and average duration Attentional shifts; percentage time in each window Textual regressions

6

7 Methods Post-task questionnaire; five-point Likert Comprehension
Task completion Potential improvement Future reuse Recommendation Recall

8 Methods Usability Satisfaction Efficiency [task success/task time]

9 Eye Tracking Task time Fixation count and average duration
Lowest for EN [sig. JP] Fixation count and average duration Lowest for EN [sig. JP] for both Attentional shifts; percentage time in each window EN and FR spent most time in task window EN fewest shifts of attention [sig. JP] Textual regressions Raw number and distance: EN and SP [sig. JP] ‘Long’ regressions: JP [sig. all others]

10 Questionnaire Results
Comprehension EN rated highest [sig. for FR and JP] Task completion EN rated highest [sig. for JP] Potential improvement SP & EN rated as needing least improvement, but could still be improved upon Future reuse FR & EN rated highest Recommendation EN rated highest [sig. for JP and DE] Recall EN scored highest [sig. for JP and DE]

11 Usability Results Task completion Efficiency Satisfaction
EN rated highest [sig. for FR, DE, and JP] Task completion EN and SP more successful [sig. JP] Efficiency EN most efficient [sig. JP and DE]

12 Conclusions So, just how usable is raw MT?
Similar results for EN, SP, and FR DE and JP more problematic [MT system] Functionally usable [more than just ‘gisting’] UX best for EN users MT viable for certain pairs Human intervention necessary to ensure best UX

13 stephen.doherty@dcu.ie sharon.obrien@dcu.ie
Questions? This research is supported by the Science Foundation Ireland (Grant 07/CE/I1142) as part of the Centre for Next Generation Localisation (www.cngl.ie) at Dublin City University.

14 Predictors of Positive UX
Satisfied users: comprehension & task time Satisfied users: recommend to others Task completion: textual regressions Cognitive effort: instructions aiding task completion


Download ppt "Dr. Stephen Doherty & Dr. Sharon O’Brien"

Similar presentations


Ads by Google