ITEM METADATA RECORD
Title: The impact of metadata on translator performance: how translators work with translation memories and machine translation.
Authors: Teixeira, Carlos; R0292799;
Issue Date: 17-Dec-2014
Abstract: This thesis investigates whether and how translation metadata affect translator performance in a workflow that combines suggestions from translation memories and machine translation. The study is based on a translation process experiment with 10 professional translators working from English into Spanish in a workplace setting.

The keystroke logging tools Inputlog and MTeval allowed for the collection of data on translation times and typing effort. BB FlashBack was used for screen and face recording. A Tobii eye tracker was used to identify how the translators shifted their attention between different parts of the screen. The final translations were assessed for quality by two professional reviewers using an error-score system. Finally, interviews were used for eliciting opinions from participants about certain aspects of their performance.

The quantitative data were analysed with mixed-effects linear(ised) regression models. The results show that translation metadata affect translation time and typing effort, and that the effects vary according to the type of translation suggestion (exact matches, fuzzy matches, machine translation). As a complementary finding, the current study identified no significant correlation between the translators’ performances while typing and their performances while translating.

The qualitative data obtained from the interviews show a mismatch between thetranslators’ perceived performance and their measured performance. Theytend to prefer an environment with translation suggestions and metadata, even when this environment does not correspond with better performance. Translators mentioned metadata as a helpful feature in the translationtool, among other reasons because metadata help them adapt their translation strategies more easily according to the suggestion type. Task familiarity was also identified as an important factor affecting translators’ perceptions.

The results obtained in this study suggest the need to advance research on how translators interact with translation tools, with a view to increase not only productivity but also job satisfaction. I hope this thesis has also contributed to the field in terms of the methodology of workplace studies, by presenting some challenges and solutions. An important lesson is the need to find an optimal balance between ecological validity and data validity when conducting translation experiments in realistic scenarios.
Description: This was a double doctorate obtained through an agreement between KU Leuven and Universitat Rovira i Virgili (Tarragona, Spain).
Table of Contents: Chapter 1. Introduction .............................................................................................. 1
1.1. Motivation ............................................................................................................. 1
1.2. Aims and objectives ............................................................................................... 3
1.3. Overall research design .......................................................................................... 4
1.4. Translation metadata .............................................................................................. 5
1.5. Thesis structure ...................................................................................................... 6
Chapter 2. Overview of current translation technologies .......................................... 9
2.1. Translation memory ............................................................................................. 11
2.2. Machine translation .............................................................................................. 13
2.3. TM & MT integration........................................................................................... 16
Chapter 3. Previous studies on translation technology ............................................ 19
3.1. Translation memory ............................................................................................. 19
3.2. Machine translation .............................................................................................. 22
3.3. TM & MT integration........................................................................................... 25
3.4. Ergonomics and usability ..................................................................................... 28
3.5. The human factor ................................................................................................. 30
3.6. Process research methods ..................................................................................... 32
3.7. Conclusion ........................................................................................................... 33
Chapter 4. Methodology ........................................................................................... 37
4.1. Research question ................................................................................................ 37
4.2. Hypotheses ........................................................................................................... 38
4.3. Definitions of variables ........................................................................................ 38
4.3.1. Translation time............................................................................................. 38
4.3.2. Typing effort ................................................................................................. 39
4.3.3. Error score ..................................................................................................... 39
4.3.4. Translation metadata ..................................................................................... 40
4.3.5. Type of translation suggestion ....................................................................... 40
4.4. Pilot study ............................................................................................................ 40
4.4.1. Research question and hypotheses ................................................................. 40
4.4.2. Participants .................................................................................................... 41
4.4.3. Experimental setup ........................................................................................ 41
4.4.4. Data collection .............................................................................................. 43
4.4.5. Results ......................................................................................... 44
4.4.5.1. Time ....................................................................................................... 44
4.4.5.2. Quality .................................................................................................... 46
4.4.6. Discussion ..................................................................................................... 47
4.5. Main experiment .................................................................................................. 49
4.5.1. Background ................................................................................................... 49
4.5.2. IBM projects ................................................................................................. 50
4.5.3. The translation tool ........................................................................................ 51
4.5.4. Participants .................................................................................................... 51
4.5.5. Source texts ................................................................................................... 52
4.5.6. Translation memory....................................................................................... 54
4.5.7. Running the experiment ................................................................................. 55
4.5.7.1. The translation tasks ............................................................................... 55
4.5.7.2. Text presentation .................................................................................... 58
4.5.7.3. Instructions ............................................................................................. 59
4.6. Data collection methods ....................................................................................... 60
4.6.1. Keystroke logging ......................................................................................... 60
4.6.2. Screen recording ............................................................................................ 62
4.6.3. Eye tracking .................................................................................................. 63
4.6.4. Interviews and retrospection .......................................................................... 68
4.6.5. Translation reviews ....................................................................................... 69
4.7. Equipment and software ....................................................................................... 70
4.8. Data analysis ........................................................................................................ 73
4.8.1. Time and typing ............................................................................................ 73
4.8.2. Quality .......................................................................................................... 73
4.8.3. Perception ..................................................................................................... 75
4.9. Ethical considerations ........................................................................................... 76
Chapter 5. Results ..................................................................................................... 77
5.1. General quantitative results .................................................................................. 77
5.1.1. Translation time ............................................................................................. 77
5.1.2. Typing effort ................................................................................................. 79
5.1.2.1. Typing effort, P09 normalised................................................................. 80
5.1.3. Error score ..................................................................................................... 83
5.1.4. Correlation between tasks .............................................................................. 85
5.1.5. Correlation within tasks ................................................................................. 87
5.2. General interview data .......................................................................................... 88
5.2.1. Comparison between quantitative and qualitative data ................................... 89
5.2.1.1. Time ....................................................................................................... 90
5.2.1.2. Comfort vs. effort ................................................................................... 90
5.2.1.3. Errors ...................................................................................................... 91
5.3. Additional information from the interviews .......................................................... 91
5.3.1. Translation vs. revision vs. post-editing ......................................................... 92
5.3.2. The role of translation suggestions ................................................................. 93
5.3.3. The role of metadata ...................................................................................... 94
5.3.4. The perception of machine translation ........................................................... 96
5.3.5. Task familiarity ............................................................................................. 96
5.3.6. Different strategies ........................................................................................ 97
5.4. Quantitative results by suggestion type ................................................................. 98
5.4.1. Translation time ........................................................................................... 100
5.4.1.1. Task (main effect) ................................................................................. 104
5.4.1.2. Suggestion Type (main effect) .............................................................. 104
5.4.1.3. Text (main effect) ................................................................................. 106
5.4.1.4. Task and Suggestion Type (interaction effect)....................................... 106
5.4.1.5. Task and Gender (interaction effect) ..................................................... 108
5.4.1.6. Suggestion Type and Gender (interaction effect) ................................... 110
5.4.1.7. Text and Experience (interaction effect) ................................................ 112
5.4.2. Typing effort (preliminary considerations) ................................................... 113
5.4.3. Typing effort (binary) .................................................................................. 115
5.4.3.1. Task (main effect) ................................................................................. 115
5.4.3.2. Suggestion Type (main effect) .............................................................. 116
5.4.3.3. Task and Suggestion Type (interaction effect)....................................... 118
5.4.4. Typing effort (non-zero cases) ..................................................................... 120
5.4.4.1. Suggestion Type (main effect) .............................................................. 121
5.4.4.2. Task and Suggestion Type (interaction effect)....................................... 123
5.4.5. Error score (preliminary considerations) ...................................................... 125
5.4.6. Error score (binary) ..................................................................................... 127
5.4.6.1. Suggestion Type (main effect) .............................................................. 127
5.4.7. Error score (non-zero cases) ........................................................................ 129
5.4.7.1. Suggestion Type (main effect) .............................................................. 130
5.5. Additional information from eye tracking and screen recordings ........................ 131
5.5.1. Example 1 ................................................................................................... 132
5.5.2. Example 2 ................................................................................................... 138
5.5.3. Example 3 ................................................................................................... 142
5.5.4. Example 4 ................................................................................................... 145
5.5.5. Example 5 ................................................................................................... 147
Chapter 6. Discussion .............................................................................................. 151
6.1. Hypothesis testing .............................................................................................. 151
6.1.1. Hypothesis 1 (H1) ....................................................................................... 151
6.1.2. Sub-hypothesis 1a (H1a) .............................................................................. 151
6.1.3. Hypothesis 2 (H2) ....................................................................................... 152
6.1.4. Sub-hypothesis 2a (H2a) .............................................................................. 152
6.1.5. Hypothesis 3 (H3) ....................................................................................... 153
6.1.6. Sub-hypothesis 3a (H3a) .............................................................................. 154
6.1.7. Conclusion .................................................................................................. 154
6.2. Additional quantitative findings ......................................................................... 156
6.2.1. Metadata ...................................................................................................... 156
6.2.2. Suggestion Type .......................................................................................... 157
6.2.3. Metadata and Suggestion Type (interaction) ................................................ 159
6.2.4. Text ........................................................................................................ 161
6.2.5. Gender (not significant) ............................................................................... 162
6.2.6. Metadata and Gender (interaction) ............................................................... 163
6.2.7. Suggestion Type and Gender (interaction) ................................................... 164
6.2.8. Text and Experience (interaction effect) ...................................................... 165
6.2.9. Non-significant effects ................................................................................. 165
6.3. Interview data .................................................................................................... 167
6.4. Summary ....................................................................................................... 168
Chapter 7. Conclusion ............................................................................................ 171
7.1. Findings ............................................................................................................. 171
7.2. Applicability ...................................................................................................... 172
7.3. Contribution to the field ..................................................................................... 172
7.4. Limitations of the present study .......................................................................... 173
7.4.1. The observer’s paradox ................................................................................ 173
7.4.2. Quantitative data collection methods ........................................................... 175
7.4.3. Qualitative data collection methods ............................................................. 177
7.4.4. Choice of translation tool ............................................................................. 178
7.4.5. Choice of materials ...................................................................................... 179
7.4.6. Data analysis ............................................................................................... 179
7.4.6.1. Quality assessment ............................................................................ 180
7.4.7. The human factor ......................................................................................... 181
7.4.8. Translation instructions ............................................................................... 182
7.4.9. Learning curve ............................................................................................ 183
7.5. Avenues for future research ................................................................................ 183
7.6. Final remarks ..................................................................................................... 186
References ........................................................................................................ 189
Appendices ........................................................................................................ 203
Appendix 1 – Source text used in the Copy task .................................................... 204
Appendix 2 – Source text used in the Scratch task ................................................. 205
Appendix 3 – SourceText31 (used in the Visual and Blind tasks) .......................... 206
Appendix 4 – SourceText42 (used in the Visual and Blind tasks) .......................... 208
Appendix 5 – TranslationMemory31 (used for SourceText31) .............................. 210
Appendix 6 – TranslationMemory42 (used for SourceText42) .............................. 215
Appendix 7 – Translation instructions for the Visual task ...................................... 220
Appendix 8 – Translation instructions for the Blind task ....................................... 221
Appendix 9 – Research participant release form, Main experiment ........................ 222
Appendix 10 – Research participant release form, Interviews ................................ 223
Appendix 11 – Metadata in translation tools .......................................................... 224
Publication status: published
KU Leuven publication type: TH
Appears in Collections:Translation and Intercultural Transfer, Leuven

Files in This Item:
File Status SizeFormat
teixeira_thesis_final.pdf Published 4807KbAdobe PDFView/Open Request a copy

These files are only available to some KU Leuven Association staff members

 




All items in Lirias are protected by copyright, with all rights reserved.