Presentation on theme: "Dr. Stephen Doherty & Dr. Sharon O’Brien"— Presentation transcript:
1 Assessing the Usability of Machine Translated Content: A User-Centred Study using Eye Tracking Dr. Stephen Doherty & Dr. Sharon O’BrienCentre for Next Generation LocalisationSchool of Applied Language & Intercultural StudiesDublin City University
3 Introduction Increased need for translation Diversity of content and usersRise in prevalence of machine translation [MT] both off- and onlineMixed reports of quality – attitudes and expectationsDivergence in R&D – translation studies/computer scienceEvaluation metrics – human and automaticOur focus here is on usability
4 Research AimsTo investigate if there are differences in usability between the English [source language] and the unedited machine translated target languages [FR, DE, SP, JP].Or in other words: how usable is machine translated content?Adoption of the ISO/TR definition of usabilityImportance of ecological validity: real materials and users
5 MethodsUser-centred approach [n = 30]; task driven – ‘new user’ scenarioEye tracking [tobii 1750]:Fixation count and average durationAttentional shifts; percentage time in each windowTextual regressions
9 Eye Tracking Task time Fixation count and average duration Lowest for EN [sig. JP]Fixation count and average durationLowest for EN [sig. JP] for bothAttentional shifts; percentage time in each windowEN and FR spent most time in task windowEN fewest shifts of attention [sig. JP]Textual regressionsRaw number and distance: EN and SP [sig. JP]‘Long’ regressions: JP [sig. all others]
10 Questionnaire Results ComprehensionEN rated highest [sig. for FR and JP]Task completionEN rated highest [sig. for JP]Potential improvementSP & EN rated as needing least improvement, but could still be improved uponFuture reuseFR & EN rated highestRecommendationEN rated highest [sig. for JP and DE]RecallEN scored highest [sig. for JP and DE]
11 Usability Results Task completion Efficiency Satisfaction EN rated highest [sig. for FR, DE, and JP]Task completionEN and SP more successful [sig. JP]EfficiencyEN most efficient [sig. JP and DE]
12 Conclusions So, just how usable is raw MT? Similar results for EN, SP, and FRDE and JP more problematic [MT system]Functionally usable [more than just ‘gisting’]UX best for EN usersMT viable for certain pairsHuman intervention necessary to ensure best UX
13 email@example.com firstname.lastname@example.org Questions?This research is supported by the Science Foundation Ireland (Grant 07/CE/I1142) as part of the Centre for Next Generation Localisation (www.cngl.ie) at Dublin City University.
Your consent to our cookies if you continue to use this website.