1. TSL 3123 LANGUAGE ASSESSMENT
SCORING, GRADING AND ASSESSMENT CRITERIA
(3 hours)

2. LECTURE’S OBJECTIVES
Identify and differentiate the different approaches used in scoring
Objective
Holistic
Analytic
Use the different approaches used in scoring in assessing language

3. OBJECTIVE APPROACH

4. OBJECTIVE SCORING
relies on quantified methods of evaluating students’ writing
analyzing occurrences of linguistic features.
The objective scoring used were categorized as word and sentence counts, syntactic complexity with no attention to error, and syntactic complexity with attention to error.
objective scoring that took error into account were able to discriminate between three essay quality levels.

5. OBJECTIVE SCORING
A sample of how objective scoring is conducted is given by Bailey (1999)
Establish standardization by limiting the length of the assessment: Count the first 250 words of the essay.
Identify the elements to be assessed: Go through the essay up to the 250th word underlining every mistake – from spelling and mechanics through verb tenses, morphology, vocabulary, etc. Include every error that a literate reader might note.
Operationalise the assessment: Assign a weight score to each error, from 3 to 1. A score of 3 is a severe distortion of readability or flow of ideas; 2 is a moderate distortion; and 1 is a minor error that does not affect readability in any significant way.
Quantify the assessment: Calculate the essay Correctness Score by using 250 words as the numerator of a fraction, and the sum of error scores as the denominator: The denominator is the sum of all the error scores.

6. HOLISTIC SCORING

7. HOLISTIC SCORING
A rater makes an overall judgement about the quality of the performance (to rate the overall proficiency level)
A single score is awarded to the writing
Normally this score is on a scale of 1 to 4, or 1 to 6, or even 1 to 10.(Bailey, 1998 : 187).
Each score on the scale will be accompanied with general descriptors of ability.
The example below is of a holistic scoring scheme based on a 6 point scale.
It includes broad descriptors of what a student’s essay reflects for each band.

8. HOLISTIC SCORING METHOD

9. HOLISTIC SCORING
Another type of scoring related to the holistic approach is primary trait scoring (Bailey, 1998).
A particular functional focus is selected based on the purpose of the writing and grading is based on how well the student is able to express that function.
For example, if the function is to persuade, scoring would be on how well the author has been able to persuade the grader rather than how well organised the ideas were, or how grammatical the structures in the essay were.
This technique emphasises functional and communicative ability rather than discrete linguistic ability and accuracy.

10. HOLISTIC SCORING
It is usually used for large scale assessment because raters can be trained and scores can be computed more quickly (Bauer, 1981).
The major disadvantage emerges from the limitations of the single score, which gives no details.
Does not allow raters to distinguish between various aspects of writing such as control of syntax, depth of vocabulary, organization, and so on.
Problematic for second-language writers since different aspects of writing ability may develop at different rates for different L2 learner.

11. ANALYTIC SCORING

12. ANALYTIC SCORING
A rater assigns a score to each of the component for the task.
Example of analytical scoring approach:
The points assigned to each component reflect the importance of each of the components.
Analytic scoring is useful in the classroom because the result can help teachers diagnose students strengths and learning needs.
Components
Weight
Content
30 points
Organisation
20 points
Vocabulary
Language Used
25 points
Mechanics
5 points

13. ANALYTICAL SCORING METHOD

14. ANALYTIC SCORING
it provides more useful diagnostic information about students’ writing abilities.
it is easier to train raters to use analytic scoring schemes
The major disadvantage of analytic scoring is that it takes longer than holistic scoring

15. COMPARING THE THREE APPROACHES
Scoring Approach
Advantages
Disadvantages
Holistic
Quickly graded
Provide a public standard that is understood by the teachers and students alike
Relatively higher degree of rater reliability
Applicable to the assessment of many different topics
Emphasise the students’ strengths rather than their weaknesses.
The single score may actually mask differences
across individual compositions.
Does not provide a lot of diagnostic feedback
Analytical
It provides clear guidelines in grading in the form of the various components.
Allows the graders to consciously address important aspects of writing.
Writing ability is unnaturally split up into components.
Objective
Emphasises the students’ strengths rather than their weaknesses.
Still some degree of subjectivity involved.
Accentuates negative aspects of the learner’s
writing without giving credit for what they can do well.

16. DESIGNING SCORING RUBRICS FOR YOUR CLASSROOM
Rubrics are rating scales - as opposed to checklists-that are used with performance assessments.
They are formally defined as scoring guides, consisting of specific pre-established performance criteria, used in evaluating student work on performance assessments.

17. DESIGNING SCORING RUBRICS FOR YOUR CLASSROOM
Prior to designing a specific rubric, a teacher must decide whether the performance or product will be scored holistically or analytically (Airasian, 2000 & 2001).
specific performance criteria and observable indicators must be identified as an initial step to development.
the use of a holistic or analytic approach to scoring has several possible implications.
teachers must consider first how they intend to use the results. If an overall, summative score is desired, a holistic scoring approach would be more desirable. In contrast, if formative feedback is the goal, an analytic scoring rubric should be used2001).
Other implications include the time requirements, the nature of the task itself, and the specific performance criteria being observed.

18. HOLISTIC RUBRICS
Holistic rubrics are customarily utilized when errors in some part of the process can be tolerated provided the overall quality is high (Chase, 1999).
Nitko (2001) states that use of holistic rubrics is probably more appropriate when performance tasks require students to create some sort of response and where there is no definitive correct answer.
Holistic rubrics are also used when the purpose of the performance assessment is summative in nature.
A template for holistic scoring rubrics is presented in Table 1.

19. HOLISTIC RUBRICS

20. ANALYTIC RUBRICS
Analytic rubrics are usually preferred when a fairly focused type of response is required (Nitko, 2001)
For performance tasks in which there may be one or two acceptable responses and creativity is not an essential feature of the students' responses.
A general rule of thumb is that an individual's work should be examined a separate time for each of the specific performance tasks or scoring criteria (Mertler, 2001)

21. ANALYTIC RUBRICS

22. RUBRICS TO GRADES
It is not a good idea to think of rubrics in terms of percentages (Trice, 2000).
For example, if a rubric has six levels (or “points”), a score of 3 should not be equated to 50% (an “F” in most letter grading systems).
The converting process is more a process of logic than it is a mathematical one.
Trice (2000) suggests that in a rubric scoring system, there are typically more scores at the average and above average categories (i.e., equating to grades of “C” or better) than there are below average categories.

23. RUBRICS TO GRADES
if a rubric consisted of nine score categories, the equivalent grades and categories might look like this:

24. STEPS IN THE DESIGN OF SCORING RUBRICS

25. TUTORIAL TASKS ISL TASK
Based on a stimulus, construct the scoring rubrics in the marking of essay questions to enhance test objectivity.
ISL TASK
Do some background reading on scoring rubrics.

26. REFERENCES
Brown, H. D., & Abeywickrama, P. (2010). Language Assessment: Principles and Classroom Practices.New York, NY: Pearson Education.
Brown, H.D. (2007). Teaching by principles: An interactive approach to language pedagogy. Englewood Cliffs, NJ: Prentice Hall Regents.
Chitravelu, Nesamalar, ELT Methodology: Principles and Practice. Penerbit Fajar Bakti, Sdn, Bhd.
Malaysia Education Blueprint