A Study of the Decision-making Behavior of Markers in E-C Sentence Translation Assessment Wen Hui Nie Jianzhong

Abstract Second language testing increasingly used rated tasks in place of objectively scored items. An understanding of the raters’ decision-making behavior is of particular importance for both reliability and validity. This paper is intended to investigate the decision-making behavior of experienced raters with similar background in their assessment of an English-Chinese sentence translation task on the basis of quantitative and qualitative analysis. rating scores and raters’ verbal protocol analysis indicated that raters in this study differed significantly in their decision making processes. Key words: English-Chinese sentence translation; Think-aloud verbal protocol; Decision-making behavior; Inter-rater difference; Translation assessment (1-11)

(2-11)

Current researches on rater behavior Bachman et al. 1995 speaking Brown writing Cumming et al 2002 Lumley & McNamara Lumely Milanovic et al. 1996 Pollitt & Murray Weigle 1994 Weigle et al. 2003 张文霞 2004 (3-11)

Objective To explore decision-making behavior of experienced raters during the process of translation assessment.

Research Design and Methodology (5-11)

Results and discussion RQ1: How is the students’ E-C sentence translation assessed by the raters in terms of scores? Source Sum of Squares df Mean Square F Sig. Between Groups 1.35 5.00 0.27 0.21 0.95 Accuracy Within Groups 67.90 54.00 1.26 Total 69.25 59.00 3.40 0.68 0.67 0.65 Expression 55.00 1.02 58.40 Figure 4.1 a Frequency analysis on the category of accuracy Table 4.3 ANOVA table for the data of table 4.1 Figure 4.1 b Frequency analyses on the category of expression (8-11)

Qualitative judgments 最后评分 RQ2 Are there any remarkable differences among raters in their qualitative judgments to the same translation? RQ3 What elements do these raters focus on while marking these two translation tasks and how do they focus on these elements? (9-11) A model of the decision– making process in translation marking Decide final mark Tentative marking Good First Impression Bad Macro-assessment Micro-assessment 1.Task completion 2.Legibility Qualitative judgments Elements 1.Locating errors 2.Judging gravities of errors 1 In addition to the extremely good or bad performance, raters usually hold different perceptions to the same performance. 2. In E-C sentence translation assessment, rater showed great difference in deciding on the gravities of errors located.

E-C translation elements raters focused on in marking Sentence translation Discourse translation criterion-related elements Comprehension of the source text Mistranslation Omission Incorrect addition Uncontrolled translation Lexical choice Chinese conventions Fluency in expression Stylistic inappropriateness Task completion Error types Error gravity Error frequency non-criterion elements Layout Spelling mistakes Rater’s affective responses Comparison with other scripts criterion-related elements Comprehension of the source text Mistranslation Omission Incorrect addition Obscurity of language Uncontrolled translation Lexical choice Sentence structure Pauses of sentences Coherence of sentences Literary grace Chinese conventions Fluency in expression Succinctness of language Stylistic inappropriateness Global communicative effect Task completion Error types Error gravity Error frequency non-criterion elements Layout Spelling mistakes Rater’s affective responses Comparison with other scripts (10-11)

incomplete investigation of protocols

THANKS ！