1. IUED Institute of Translation and Interpreting
When is Less Actually More? A Usability Comparison of Two CAT Interfaces

2. When is Less Actually More
When is Less Actually More? A Usability Comparison of Two CAT Interfaces
Overview
Background: Cognitive friction, usability
The study: Research questions, participants, methods
Results
Implications and future directions

3. When is Less Actually More
When is Less Actually More? A Usability Comparison of Two CAT Interfaces
Background
Cognitive friction:
“the resistance encountered by a human intellect when it engages with a complex system of rules that change as the problem changes” (Cooper 2004, 19)
Translators may be exposing themselves to unnecessary cognitive friction by the way they use their tools

4. When is Less Actually More
When is Less Actually More? A Usability Comparison of Two CAT Interfaces
Background
Cognitive friction
Apparent overload of cognitive resources by the amount of information presented on crowded screen
Apparent information underload of cognitive resources on a lean user interface

5. Availability of (adaptive) MT results
When is Less Actually More? A Usability Comparison of Two CAT Interfaces
Background
New forms of cognitive friction may arise as a result of recent developments in modern CAT tools.
Sub-segment matching
Availability of (adaptive) MT results
Suggestions change over the course of the translation (within segments, across segments)
Potential for cognitive friction
(Ehrensberger-Dow & O’Brien 2015; O’Brien et al. 2017; Teixeira & O’Brien 2017).

6. When is Less Actually More
When is Less Actually More? A Usability Comparison of Two CAT Interfaces
The study
Typical translation task
Focus on productivity
Eye-tracking
Keystroke-logging
Post-task questionnaire

7. When is Less Actually More
When is Less Actually More? A Usability Comparison of Two CAT Interfaces
The study - Participants
8 MA students (2nd or 4th semester)
Tools skills (self-rated)
No previous experience
Beginner skills
Basic skills
SDL Trados Studio 2017 2 6
Lilt 1

8. When is Less Actually More
When is Less Actually More? A Usability Comparison of Two CAT Interfaces
The study – Texts and resources
Translation of 2 short comparable texts (see following slide)
Language direction: English  German
Translation Memory (> 6000 Translation units)
Terminology (55 entries)
Machine Translation (Lilt, SDL Language Cloud)
Access to external resources (Online dictionaries etc.)

9. When is Less Actually More
When is Less Actually More? A Usability Comparison of Two CAT Interfaces
The study – Texts
Text type: Academic job announcement from linguistlist.org
Basic familiarity with the type of text (classroom exercises)
Comparable level of difficulty
Text 1
Text 2
Length
208 words
239 words
Number of unique words
136 words
145 words
Vocabulary density
.654
.607
Average sentence length
34.7 words
34.1 words
Readability (Gunning-Fog)
9.7
7.7

10. When is Less Actually More
When is Less Actually More? A Usability Comparison of Two CAT Interfaces
Results – What are we looking at?
Focus on 3 segments per text
Stage of translation
at beginning, middle and end of the text
Segment length:
2 Short segments (segments 1 and 3)
8 to 10 words/segment each
1 long segment (segment 2)
36 words/segment

11. When is Less Actually More
When is Less Actually More? A Usability Comparison of Two CAT Interfaces
Results – What are we looking at?
Time on segment
Number of Keystrokes
Number of Deletions
Number of pauses (> 3s)

12. When is Less Actually More
When is Less Actually More? A Usability Comparison of Two CAT Interfaces
Results
Time on segments (in seconds)
Segment 1
Segment 2
Segment 3
SDL Trados Studio 2017 62
252
Lilt 43
217 36
Percentage of Time on segment (Lilt/SDL Trados)
69%
86%
58%

13. When is Less Actually More
When is Less Actually More? A Usability Comparison of Two CAT Interfaces
Results
Number of keystrokes
Segment 1
Segment 2
Segment 3
SDL Trados Studio 2017 85
341 89
Lilt 35
303 72
Percentage of keystrokes (Lilt/SDL Trados)
41%
89%
81%

14. When is Less Actually More
When is Less Actually More? A Usability Comparison of Two CAT Interfaces
Results
Number of deletions
Segment 1
Segment 2
Segment 3
SDL Trados Studio 2017 8 31 6
Lilt 2 20
Percentage of deletions (Lilt/SDL Trados)
25%
65%
100%

15. When is Less Actually More
When is Less Actually More? A Usability Comparison of Two CAT Interfaces
Results
Number of pauses (>3s)
Segment 1
Segment 2
Segment 3
SDL Trados Studio 2017
6.3
17.7
6.7
Lilt
5.3
15.6
3.9
Percentage of number of pauses (Lilt/SDL Trados)
84%
88%
58%

16. When is Less Actually More
When is Less Actually More? A Usability Comparison of Two CAT Interfaces
Results
Number of brief pauses (>3s and <5s)
Segment 1
Segment 2
Segment 3
SDL Trados Studio 2017
2.3
7.8
3.3
Lilt
2.2
6.5
1.4
Percentage of deletions (Lilt/SDL)
96%
83%
42%

17. When is Less Actually More
When is Less Actually More? A Usability Comparison of Two CAT Interfaces
Results
Number of extended pauses (>5s)
Segment 1
Segment 2
Segment 3
SDL Trados Studio 2017 4
9.8
3.3
Lilt 3
9.1
1.4
Percentage of deletions (Lilt/SDL)
75%
93%
42%

18. When is Less Actually More
When is Less Actually More? A Usability Comparison of Two CAT Interfaces
Results
Heatmap SDL Trados Studio 2017

19. When is Less Actually More
When is Less Actually More? A Usability Comparison of Two CAT Interfaces
Results
Heatmap Lilt

20. When is Less Actually More
When is Less Actually More? A Usability Comparison of Two CAT Interfaces
Results
Key Performance Indicators SDL Trados Studio 2017

21. When is Less Actually More
When is Less Actually More? A Usability Comparison of Two CAT Interfaces
Results
Video Clip showing representative Eye-Tracking data for SDL Trados Studio 2017

22. When is Less Actually More
When is Less Actually More? A Usability Comparison of Two CAT Interfaces
Results
Video Clip showing representative Eye-Tracking data for Lilt

23. When is Less Actually More
When is Less Actually More? A Usability Comparison of Two CAT Interfaces
Results – Summary (SDL Trados Studio 2017)
Lower overall productivity with SDL Trados Studio (across all categories measured)
Participants take time to get used to SDL Trados Studio 2017 interface (number of deletions decreases during task)
Users do not focus on “traditional” Translation Memory results window in SDL Trados Studio 2017

24. When is Less Actually More
When is Less Actually More? A Usability Comparison of Two CAT Interfaces
Results – Summary (Lilt)
Higher overall productivity with Lilt (across all categories measured)
The longer the segment, the smaller the difference in productivity (time on segment)
Changing of MT suggestions is perceived to be disruptive
Fewer pauses in the 3rd segment (result of Adaptive MT?)

25. When is Less Actually More
When is Less Actually More? A Usability Comparison of Two CAT Interfaces
Implications and future directions
Is there a difference in quality?
Participants seem to trust MT suggestions for shorter segments more than for longer segments. Perceived or real difference in quality?
How can this effect translator’s training?

26. When is Less Actually More
When is Less Actually More? A Usability Comparison of Two CAT Interfaces
Literature
Cooper, Alan (2004). The Inmates are Running the Asylum: Why Hi-Tech Products Drive Us Crazy and How to Restore the Sanity. Indianapolis: Sams Publishing.
Ehrensberger-Dow, Maureen & Sharon O’Brien (2015). Ergonomics of the translation workplace: Potential for cognitive friction. Special Issue of Translation Spaces, 4 (1):
O’Brien, Sharon, Maureen Ehrensberger-Dow, Marcel Hasler, & Megan Connolly (2017). Irritating CAT tool features that matter to translators. Hermes Journal of Language and Communication in Business 57.
Teixeira, Carlos S. & Sharon O'Brien (2017/forthcoming). Investigating the cognitive ergonomic aspects of translation tools in a workplace setting. Special Issue of Translation Spaces 6 (1).