1. Understanding the Work of the HKEAA:
Mark Processing and Grading

2. Mark processing Some important steps after completion of marking:
Equating of marks of optional parts / papers
Moderation of SBA marks submitted by schools

3. Graded Approach for English Language Papers 1 & 3

4. Why do we need a graded approach?
Adopted in Paper 1 (Reading) and Paper 3 (Listening and Integrated Skills)
Compulsory Part A and either Part B1 (easier) or Part B2 (more difficult)
Cater to candidates with a wide range of ability

5. What are the advantages?
No labeling effect
No need to extend examination time or reduce coverage of the curriculum
Candidates need not indicate choice at registration so possible to catch up before the examination
Statistical equating to ensure that candidates receive comparable scores on the same scale whether they opt for the easier or more difficult part

6. Why is there a capping policy?
Candidates attempting Parts A and B2 will be able to attain the highest level of performance (Levels 5 and 5* / 5**)
Candidates who attempt Parts A and B1 will only be able to attain up to Level 4 in the relevant component
Reason for capping policy is that items in B1 do not assess skills covered by Level 5 descriptors

7. Why is equating necessary?
Easier to get high marks in Part B1 than in Part B2 because the items are easier
1 mark in B1 is not equal to 1 mark in B2
Need to convert the marks to the same scale
Use the marks of Part A as the mediator because all candidates do Part A

8. Example 100 candidates got 36 marks in Part A
50 of them did B1 and 50 did B2
The 50 B1 candidates got 40 marks in B1
The 50 B2 candidates got 32 marks in B2
40 marks in B1 = 32 marks in B2

9. What is equipercentile equating?
Large candidature for English Language (e.g. ~ 80,000 in 2013)
All candidates do Part A and can be rank ordered based on their marks in Part A
B1 marks are statistically adjusted for difficulty to the same level as B2 marks using the percentile ranks of candidates’ performance in Part A and Part B1/B2

10. Equipercentile method of equating
B2 candidates’ ranking in Part B2
40th%
B2 candidates’ ranking in Part A
40th%
B1 candidates’ ranking in Part A
50th%
B1 candidates’ ranking in Part B1
50th% 40 36 36 32

11. Equipercentile equating

12. Is there a fixed conversion factor?
The examination papers for each year will be different from the previous year’s
Therefore no fixed conversion factor
A conversion table will be produced every year for that particular examination paper based on the performance of the candidates in Parts A, B1 and B2

13. How to choose between B1 and B2?
Levels 4-5
Parts A & B2 (more difficult part)
Levels 3-4
Parts A & B1 or
Parts A and B2
Levels 1-3
Parts A and B1 (easier part)

14. How is the paper mark calculated?
For Part B1 candidates:
Part A marks + Part B2 equivalent marks
For Part B2 candidates:
Part A marks + Part B2 marks
Therefore, choosing B1 or B2 makes no difference for mid-range candidates

15. Moderation of SBA Marks

16. Some queries/concerns
Why moderate schools’ SBA marks?
Is it fair to moderate students’ SBA marks based on exam results?

17. Why moderation?
Teachers know their students well and thus are best placed to judge their relative performance within a school
However, they are not necessarily aware of the standards of performance across all schools
Moderation ensures the consistency of assessment standards across schools in order to achieve across-school comparability

18. What are the principles of SBA moderation?
Accept teacher judgments of relative student performance within a school (internal standardisation, student queries)
Moderation is done on school basis (not on individual basis) to standardise across schools

19. Methods of SBA moderation
Statistical
The distribution of a school’s SBA marks for a subject is re-shaped taking into consideration school’s pattern of mark distribution for that subject’s examination papers
Expert judgment
A team of experienced, subject experts reviews samples of schools’ SBA work for a subject

20. Moderation methods Expert Judgment Moderation Statistical Moderation
for subjects with a small candidature or that involve outcomes that are very different from those assessed through the public examination
for subjects in which the SBA and the public examination share a substantial portion of common assessment objectives
Design and Applied Technology
Visual Arts
English Language, Chinese Language
Liberal Studies,
History, Chinese History
Biology, Chemistry, Physics
Science (Combined Science/Integrated Science)
Information and Communication Technology

21. Features of statistical moderation
Internal ranking would not be changed
The statistical moderation method would consider the empirical correlation between examination results and SBA results
Eliminate the effect of strictness and leniency

22. Features of statistical moderation
Sample review to help identify outlier schools
For outlier schools, adjustment will be carried out, if necessary, by considering a number of factors, such as moderated marks based on examination results and sample review findings so as to reflect the actual SBA performance

23. Expert judgment moderation
Review of samples of students’ work
Determination of school performance level on SBA and individual students’ moderated scores
Detection of cases with extreme adjustment and follow-up actions

24. Review of samples of students’ work
Stratified random sampling within each school
District Coordinators / External Assessors review samples with reference to the stipulated assessment criteria to determine the group performance level
More samples from a school may be collected if necessary

25. Result of moderation in 2012 HKDSE
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
much lower
lower
slightly
higher
much higher
2009CE
2010CE
2012DSE as
expected
Mean SBA Adjustment

26. Result of moderation in 2012 HKDSE
0.0%
10.0%
20.0%
30.0%
40.0%
50.0%
60.0%
70.0%
much lower
lower
slightly
higher
much higher
2009CE
2010CE
2012DSE as
expected
Mean SBA Adjustment

27. Result of moderation in 2012 HKDSE
In general, most teachers are aware of the standards and their SBA marks need very little adjustment
Small deviation from the standard is expected due to normal variation of experience and subjective judgment: no systemic bias observed
Moderation provides assurance of fairness to students, parents, and schools

28. Reference
HKDSE Examination: Moderation of School-based Assessment Scores

29. Standards-referenced
Reporting

30. Some queries/concerns
Why the grade awards are different across subjects?
Is this unfair to my subject?

31. Percentage of levels awarded科目
Subject 出席 人數
No. Sat
中文作答
Chinese Version%
考生考獲各等級的百分率
Percentage of levels awarded
5**
5*+ 5+ 4+ 3+ 2+ 1+ U
資訊及通訊科技
Information and Communication Technology
7 918
61.6
0.7
2.9
7.2
25.2
50.9
77.5
93.0
7.0
通識教育 Liberal Studies
69 721
88.0
0.8
3.0
7.7
35.7
72.5
90.8
98.9
1.1
英語文學 Literature in English
451 -
2.7
10.4
25.9
57.6
80.5
92.5
96.9
3.1 數學
Mathematics
必修部分
Compulsory Part
69 299
49.5
4.8
12.1
34.1
57.1
79.7
91.7
8.3
延伸部分（微積分與統計）
Extended Part (Calculus and Statistics)
7 347
27.1
1.4
5.9
15.1
43.4
69.7
86.9
95.1
4.9
延伸部分（代數與微積分）
Extended Part (Algebra and Calculus)
8 091
18.1
12.0
31.4
56.1
79.8
98.0
2.0
音樂 Music
232
8.6
1.7
7.8
19.0
60.8
85.8
97.8
99.6
0.4
體育 Physical Education
681
93.5
0.1
1.6
9.3
37.7
63.9
88.7
11.3
物理 Physics
15 233
2.4
9.6
23.8
48.8
73.0
88.8
97.0
.....
.....
.....

32. Standards-referenced reporting system 24 NSS Subjects
Descriptors and exemplars to indicate standards required for each level
Facilitate users, including tertiary institutions and employers, to set appropriate entrance/ job requirements
3232

33. Standards-referenced reporting system 24 NSS Subjects
Adopted to report candidates' assessment results in all 24 HKDSE Category A subjects
SRR adopted for HKCE English Language and Chinese Language since 2007 33

34. Grading in the HKDSE
Cut scores and grade levels determined so that each level represents a fixed standard of achievement from year to year
Methodology combines psychometric measurement and expert judgment
Psychometric measurements:
i) Monitoring tests for the core subjects
ii) Group ability index (GAI) for elective subjects

35. Grading in the HKDSE
Expert judgments made independently by experienced examiners and experts based on the actual performance of candidates on the exams, using the level descriptors and library scripts for reference
Psychometric cut score recommendations are derived for the reference of the expert panel
No pre-set quota for each level /subject

36. Standards maintenance in 2013
Four core subjects (CEML)
Additional reference made to:
statistical data from the monitoring tests which were administered to a representative sample of 2012 and 2013 DSE school candidates 36

37. Using monitoring tests to maintain standards
Statistically-generated cut scores for reference of Expert Panel 37

38. Group Ability Index (GAI)
for elective subjects
Different Category A elective subjects may attract students of different abilities
Compute an indicator (index) to reflect overall performance in core subjects (ability) of all candidates taking the subject (group), taking into account how the performance in the elective correlates with each of the core subjects
Provided as supplementary statistics, in addition to the level descriptors, library scripts and live scripts, to the subject experts for their expert judgment. Final decision based on performance as reflected in the candidates’ scripts

39. Group Ability Index
The GAI of Subject X for a certain level (e.g. Level 3) or above is defined as: b
Coefficients obtained by regressing scores of Subject X on scores of CEML
n/N
Proportion of candidates in a group attained a particular level (e.g. Level 3) for a core subject 39

40. Grading Methodology and Procedures

41. Who are involved in grading
Each subject sets up its expert panel which comprises:
the chief examiner(s) of different papers,
assistant examiners/markers, if necessary;
SBA supervisors/chief project examiner, if applicable, and
subject manager 41

42. Overview of grading procedures
Familiarisation with standards
Sample script selection
Standardisation
Post-marking exercise
Expert Panel Meeting
Internal Meeting
Public Examinations Board Meeting 42

43. 4. Post-marking exercise
Determine preliminary cut scores ranges for all levels for each exam paper (incl. SBA) with reference to:
the difficulty of live question paper and requirements of marking scheme
inter-paper correlations, mean and standard deviation of current year’s papers in comparison to previous years
item statistics, if available
impression of candidates’ performance when marking live scripts or reviewing students’ SBA work
level descriptors and borderline performance samples in the previous exams 43

44. 5. Expert panel meeting Objective:
Apply expert judgment to arrive at a set of recommended cut scores for different levels for the subject and its components, if applicable
Make reference to marked live scripts falling within / around the recommended cut score ranges and exam statistics 44

45. Expert panel meeting (cont’d)
Also refer to cut scores suggested by statistical means:
for core subjects, the suggested cut scores are derived from the results of the monitoring tests (2013 data in comparison with the 2012 data)
for elective subjects, the suggested cut scores are derived from GAI 45

46. Percentage of levels awarded科目
Subject 出席 人數
No. Sat
中文作答
Chinese Version%
考生考獲各等級的百分率
Percentage of levels awarded
5**
5*+ 5+ 4+ 3+ 2+ 1+ U
資訊及通訊科技
Information and Communication Technology
7 918
61.6
0.7
2.9
7.2
25.2
50.9
77.5
93.0
7.0
通識教育 Liberal Studies
69 721
88.0
0.8
3.0
7.7
35.7
72.5
90.8
98.9
1.1
英語文學 Literature in English
451 -
2.7
10.4
25.9
57.6
80.5
92.5
96.9
3.1 數學
Mathematics
必修部分
Compulsory Part
69 299
49.5
4.8
12.1
34.1
57.1
79.7
91.7
8.3
延伸部分（微積分與統計）
Extended Part (Calculus and Statistics)
7 347
27.1
1.4
5.9
15.1
43.4
69.7
86.9
95.1
4.9
延伸部分（代數與微積分）
Extended Part (Algebra and Calculus)
8 091
18.1
12.0
31.4
56.1
79.8
98.0
2.0
音樂 Music
232
8.6
1.7
7.8
19.0
60.8
85.8
97.8
99.6
0.4
體育 Physical Education
681
93.5
0.1
1.6
9.3
37.7
63.9
88.7
11.3
物理 Physics
15 233
2.4
9.6
23.8
48.8
73.0
88.8
97.0
.....
.....
.....

47. Reference
Grading Procedures and Standards-reference Reporting in the HKDSE Examination

48. Reporting of Results

49. Reporting of HKDSE results
For Category A subjects, results reported mostly on subject basis
Component results reported for:
Chinese Language
English Language
Combined Science

50. Reasons for component reporting
Allow end users to understand the achievement of candidates in
different areas of the subject e.g. reading, writing, listening and speaking skills in language subjects
different half subjects in Combined Science
For selection for studies or employment

51. FAQ on component result and subject result
Why is my subject level lower than the average of individual component levels?

52. The facts
Subject level is based on the overall subject mark which is the combination of all component marks
NOT a simple averaging of component levels achieved

53. The facts
Each level covers a range of marks; a candidate may have a score close to the lower/higher end of the range
Components of a subject carry different weightings; the marks for each component go through a weighting procedure before combining them to arrive at the subject mark

54. Example: HKDSE English Language
Comprises four components:
Component 1: Reading (20%)
Component 2: Writing (25%)
Component 3: Listening and Integrated Skills (30%)
Component 4: Speaking + SBA (25%)

55.    * * *  * *  *  5 4 3 2 1 Level achieved Student A 5 4
Reading
(20%)
Writing
(25%)
Listening
(30%)
Speaking
Subject
Student A * 5 4
Student B  3 5 4 3 2 1    * * *  * * 

56. Thank You