Presentation is loading. Please wait.

Presentation is loading. Please wait.

Understanding the Work of the HKEAA: Mark Processing and Grading.

Similar presentations


Presentation on theme: "Understanding the Work of the HKEAA: Mark Processing and Grading."— Presentation transcript:

1 Understanding the Work of the HKEAA: Mark Processing and Grading

2 2 Mark processing Some important steps after completion of marking:  Equating of marks of optional parts / papers  Moderation of SBA marks submitted by schools

3 Graded Approach for English Language Papers 1 & 3

4 4 Why do we need a graded approach?  Adopted in Paper 1 (Reading) and Paper 3 (Listening and Integrated Skills)  Compulsory Part A and either Part B1 (easier) or Part B2 (more difficult)  Cater to candidates with a wide range of ability

5 5 What are the advantages?  No labeling effect  No need to extend examination time or reduce coverage of the curriculum  Candidates need not indicate choice at registration so possible to catch up before the examination  Statistical equating to ensure that candidates receive comparable scores on the same scale whether they opt for the easier or more difficult part

6 6 Why is there a capping policy?  Candidates attempting Parts A and B2 will be able to attain the highest level of performance (Levels 5 and 5* / 5**)  Candidates who attempt Parts A and B1 will only be able to attain up to Level 4 in the relevant component  Reason for capping policy is that items in B1 do not assess skills covered by Level 5 descriptors

7 7 Why is equating necessary?  Easier to get high marks in Part B1 than in Part B2 because the items are easier  1 mark in B1 is not equal to 1 mark in B2  Need to convert the marks to the same scale  Use the marks of Part A as the mediator because all candidates do Part A

8 8 Example  100 candidates got 36 marks in Part A  50 of them did B1 and 50 did B2  The 50 B1 candidates got 40 marks in B1  The 50 B2 candidates got 32 marks in B2  40 marks in B1 = 32 marks in B2

9 9 What is equipercentile equating?  Large candidature for English Language (e.g. ~ 80,000 in 2013)  All candidates do Part A and can be rank ordered based on their marks in Part A  B1 marks are statistically adjusted for difficulty to the same level as B2 marks using the percentile ranks of candidates’ performance in Part A and Part B1/B2

10 10 Equipercentile method of equating B1 candidates’ ranking in Part B1 50th% B1 candidates’ ranking in Part A 50th% B2 candidates’ ranking in Part A 40th% B2 candidates’ ranking in Part B2 40th%

11 11 Equipercentile equating aflets/HKDSE_SRR_A4_Booklet_Jun2011.p df aflets/HKDSE_SRR_A4_Booklet_Jun2011.p df

12 12 Is there a fixed conversion factor?  The examination papers for each year will be different from the previous year’s  Therefore no fixed conversion factor  A conversion table will be produced every year for that particular examination paper based on the performance of the candidates in Parts A, B1 and B2

13 13 How to choose between B1 and B2? Levels 3-4 Levels 1-3 Levels 4-5 Parts A & B2 (more difficult part) Parts A and B1 (easier part) Parts A & B1 or Parts A and B2

14 14 How is the paper mark calculated?  For Part B1 candidates: Part A marks + Part B2 equivalent marks  For Part B2 candidates: Part A marks + Part B2 marks  Therefore, choosing B1 or B2 makes no difference for mid-range candidates

15 Moderation of SBA Marks

16 16 Some queries/concerns  Why moderate schools’ SBA marks?  Is it fair to moderate students’ SBA marks based on exam results?

17 17 Why moderation?  Teachers know their students well and thus are best placed to judge their relative performance within a school  However, they are not necessarily aware of the standards of performance across all schools  Moderation ensures the consistency of assessment standards across schools in order to achieve across-school comparability

18 18 What are the principles of SBA moderation?  Accept teacher judgments of relative student performance within a school (internal standardisation, student queries)  Moderation is done on school basis (not on individual basis) to standardise across schools

19 19 Methods of SBA moderation A. Statistical  The distribution of a school’s SBA marks for a subject is re-shaped taking into consideration school’s pattern of mark distribution for that subject’s examination papers B. Expert judgment  A team of experienced, subject experts reviews samples of schools’ SBA work for a subject

20 20 Moderation methods Expert Judgment Moderation Statistical Moderation for subjects with a small candidature or that involve outcomes that are very different from those assessed through the public examination for subjects in which the SBA and the public examination share a substantial portion of common assessment objectives Design and Applied Technology Visual Arts English Language, Chinese Language Liberal Studies, History, Chinese History Biology, Chemistry, Physics Science (Combined Science/Integrated Science) Information and Communication Technology

21 21 Features of statistical moderation  Internal ranking would not be changed  The statistical moderation method would consider the empirical correlation between examination results and SBA results  Eliminate the effect of strictness and leniency

22 22 Features of statistical moderation  Sample review to help identify outlier schools  For outlier schools, adjustment will be carried out, if necessary, by considering a number of factors, such as moderated marks based on examination results and sample review findings so as to reflect the actual SBA performance

23 23 Expert judgment moderation  Review of samples of students’ work  Determination of school performance level on SBA and individual students’ moderated scores  Detection of cases with extreme adjustment and follow-up actions

24 24 Review of samples of students’ work  Stratified random sampling within each school  District Coordinators / External Assessors review samples with reference to the stipulated assessment criteria to determine the group performance level  More samples from a school may be collected if necessary

25 25 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% much lowerlowerslightly lower slightly higher much higher 2009CE 2010CE 2012DSE Result of moderation in 2012 HKDSE as expected Mean SBA Adjustment

26 26 0.0% 10.0% 20.0% 30.0% 40.0% 50.0% 60.0% 70.0% much lowerlowerslightly lower slightly higher much higher 2009CE 2010CE 2012DSE Result of moderation in 2012 HKDSE as expected Mean SBA Adjustment

27 27 Result of moderation in 2012 HKDSE  In general, most teachers are aware of the standards and their SBA marks need very little adjustment  Small deviation from the standard is expected due to normal variation of experience and subjective judgment: no systemic bias observed  Moderation provides assurance of fairness to students, parents, and schools

28 28 Reference  HKDSE Examination: Moderation of School-based Assessment Scores KDSE/HKDSE-SBA-ModerationBooklet_r.pdf KDSE/HKDSE-SBA-ModerationBooklet_r.pdf

29 Standards-referencedReporting

30 30 Some queries/concerns  Why the grade awards are different across subjects?  Is this unfair to my subject?

31 科目 Subject 出席 人數 No. Sat 中文作答 Chinese Version % 考生考獲各等級的百分率 Percentage of levels awarded 5**5* U 資訊及通訊科技 Information and Communication Technology 通識教育 Liberal Studies 英語文學 Literature in English 數學 Mathematics 必修部分 Compulsory Part 延伸部分(微積分與統計) Extended Part (Calculus and Statistics) 延伸部分(代數與微積分) Extended Part (Algebra and Calculus) 音樂 Music 體育 Physical Education 物理 Physics

32 32 32 Standards-referenced reporting system 24 NSS Subjects  Descriptors and exemplars to indicate standards required for each level  Facilitate users, including tertiary institutions and employers, to set appropriate entrance/ job requirements

33 33 Standards-referenced reporting system 24 NSS Subjects  Adopted to report candidates' assessment results in all 24 HKDSE Category A subjects  SRR adopted for HKCE English Language and Chinese Language since 2007

34 34 Grading in the HKDSE  Cut scores and grade levels determined so that each level represents a fixed standard of achievement from year to year  Methodology combines psychometric measurement and expert judgment  Psychometric measurements: i)Monitoring tests for the core subjects ii)Group ability index (GAI) for elective subjects

35 35 Grading in the HKDSE  Expert judgments made independently by experienced examiners and experts based on the actual performance of candidates on the exams, using the level descriptors and library scripts for reference  Psychometric cut score recommendations are derived for the reference of the expert panel  No pre-set quota for each level /subject

36 36 Standards maintenance in 2013 Four core subjects (CEML) Additional reference made to:  statistical data from the monitoring tests which were administered to a representative sample of 2012 and 2013 DSE school candidates

37 37 Using monitoring tests to maintain standards Statistically-generated cut scores for reference of Expert Panel

38 38  Different Category A elective subjects may attract students of different abilities  Compute an indicator (index) to reflect overall performance in core subjects (ability) of all candidates taking the subject (group), taking into account how the performance in the elective correlates with each of the core subjects  Provided as supplementary statistics, in addition to the level descriptors, library scripts and live scripts, to the subject experts for their expert judgment. Final decision based on performance as reflected in the candidates’ scripts Group Ability Index (GAI) for elective subjects

39 39 Group Ability Index The GAI of Subject X for a certain level (e.g. Level 3) or above is defined as: n/N Proportion of candidates in a group attained a particular level (e.g. Level 3) for a core subject b Coefficients obtained by regressing scores of Subject X on scores of CEML

40 Grading Methodology and Procedures

41 41 Who are involved in grading  Each subject sets up its expert panel which comprises:  the chief examiner(s) of different papers,  assistant examiners/markers, if necessary;  SBA supervisors/chief project examiner, if applicable, and  subject manager

42 42 Overview of grading procedures 1. Familiarisation with standards 2. Sample script selection 3. Standardisation 4. Post-marking exercise 5. Expert Panel Meeting 6. Internal Meeting 7. Public Examinations Board Meeting

43 43 4. Post-marking exercise 1. Determine preliminary cut scores ranges for all levels for each exam paper (incl. SBA) with reference to:  the difficulty of live question paper and requirements of marking scheme  inter-paper correlations, mean and standard deviation of current year’s papers in comparison to previous years  item statistics, if available  impression of candidates’ performance when marking live scripts or reviewing students’ SBA work  level descriptors and borderline performance samples in the previous exams

44 44 5. Expert panel meeting  Objective:  Apply expert judgment to arrive at a set of recommended cut scores for different levels for the subject and its components, if applicable  Make reference to marked live scripts falling within / around the recommended cut score ranges and exam statistics

45 45 Expert panel meeting (cont’d)  Also refer to cut scores suggested by statistical means:  for core subjects, the suggested cut scores are derived from the results of the monitoring tests (2013 data in comparison with the 2012 data)  for elective subjects, the suggested cut scores are derived from GAI

46 科目 Subject 出席 人數 No. Sat 中文作答 Chinese Version % 考生考獲各等級的百分率 Percentage of levels awarded 5**5* U 資訊及通訊科技 Information and Communication Technology 通識教育 Liberal Studies 英語文學 Literature in English 數學 Mathematics 必修部分 Compulsory Part 延伸部分(微積分與統計) Extended Part (Calculus and Statistics) 延伸部分(代數與微積分) Extended Part (Algebra and Calculus) 音樂 Music 體育 Physical Education 物理 Physics

47 47 Reference  Grading Procedures and Standards- reference Reporting in the HKDSE Examination SE_SRR_A4_Booklet_Jun2011.pdf SE_SRR_A4_Booklet_Jun2011.pdf

48 Reporting of Results

49 49 Reporting of HKDSE results  For Category A subjects, results reported mostly on subject basis  Component results reported for:  Chinese Language  English Language  Combined Science

50 50 Reasons for component reporting  Allow end users to understand the achievement of candidates in  different areas of the subject e.g. reading, writing, listening and speaking skills in language subjects  different half subjects in Combined Science  For selection for studies or employment

51 51 FAQ on component result and subject result  Why is my subject level lower than the average of individual component levels?

52 52 The facts  Subject level is based on the overall subject mark which is the combination of all component marks  NOT a simple averaging of component levels achieved

53 53 The facts  Each level covers a range of marks; a candidate may have a score close to the lower/higher end of the range  Components of a subject carry different weightings; the marks for each component go through a weighting procedure before combining them to arrive at the subject mark

54 54 Example: HKDSE English Language  Comprises four components:  Component 1: Reading (20%)  Component 2: Writing (25%)  Component 3: Listening and Integrated Skills (30%)  Component 4: Speaking + SBA (25%)

55 Level achieved Reading (20%) Writing (25%) Listening (30%) Speaking (25%) Subject Student A * Student B  * * * * *     

56 56 Thank You


Download ppt "Understanding the Work of the HKEAA: Mark Processing and Grading."

Similar presentations


Ads by Google