1. Briefing Session on the 2012 HKDSE English Language Assessment Framework
March 2008

2. Background Third consultation conducted in late 2006
New public assessment of HKCEE English Language first implemented in 2007
Further consultation in the light of the 2007 HKCEE experience
Questionnaire survey conducted in January 2008

3. 2007 HKCEE Experience

4. SBA Implementation Schedule
Year
Options for School
2007
1. Submit SBA marks for feedback; marks included as 15% of the subject result
2. Submit SBA marks for feedback only; marks not included in subject result
3. Not submit SBA marks; exam results to contribute 100% of subject result
2008
2009

5. Implementation of SBA in 2007
No. of schools
Percentage (%)
No. of candidates
Option 1 (Yes)
199 34
31,876 43
Option 2 (Trial)
125 22
20,945 28
Option 3 (No)
254 44
21,388 29
Total
578
100
74,209

6. Moderation Results of Option 1 Schools
The mean of the SBA scores is …
No. of Schools
Percentage (%)
within the range
144
72.4
slightly higher than expected 29
14.6
higher than expected 2
1.0
much higher than expected
slightly lower than expected 21
10.6
lower than expected 3
1.5
much lower than expected

7. Moderation Results of Option 1 Schools
The standard deviation of the SBA scores is …
No of Schools
Percentage
(%)
as expected
179
89.9
slightly wider than expected
wider than expected
slightly narrower than expected 10
5.9
narrower than expected 9
4.5
much wider than expected
much narrower than expected 1
0.5

8. Moderation Effect on Candidates
Mark Adjustment
(% of subject)
No. of candidates
Percentage of candidates (%)
0 (0)
5365 17
1-3 (<1)
19881 62
4-6 (<2)
6237 20
7-9 (<3)
392 1
31,875
100

9. Implication of Moderation Results
Mean and spread of the SBA marks submitted by the majority of schools fall within the expected range
Most teachers have a good understanding of the assessment criteria and can assess their students accurately
Reliability of SBA not an issue
Fairness ensured by statistical moderation

10. Absentee Rate of School Candidates in 1996 CE English Language Speaking Exam
(Syllabus A)
No. Sat
6,623
No. of Absentees
3,058
Absentee Rate
31.6 %
(Syllabus B)
88,201
19,236
17.9%

11. Absentee Rate of School Candidates in 2007 CE English Language Speaking Exam
No. Sat
No. of Absentees
Absentee Rate (%)
Option 1 (Yes)
27,935
3,398
10.8
Option 2 (Trial)
19,307
1,466
7.1
Options 1 & 2
(Yes + Trial)
47,242
4,864
9.3
Option 3 (No)
19,298
3,681
16.0
All Schools
66,540
8,545
11.4

12. Possible Explanations
Students in “Yes” and “Trial” schools had more speaking practice in school because of the SBA
They became more confident with the public exam format, which involves similar speaking tasks – group discussion and individual response
Positive backwash effect of the SBA on the ability to speak English and to interact with peers in different contexts.

13. Effect of SBA on Speaking Exam
No. Sat
Speaking Exam Mean
Moderated
SBA Mean
Option 1 (Yes)
27,804
25.38
27.81
Option 2 (Trial)
19,381
26.00 -
Option 3 (No)
21,293
23.51
All Schools
68,478
24.97

14. Effect of SBA on Speaking Exam
Students in “Yes” schools got higher marks in the SBA than in the speaking exam
Students in “Yes” and “Trial” schools got higher marks in the public speaking exam than those in “No” schools
BUT “Yes” and “Trial” schools may just be better schools

15. Effect of SBA on Speaking Exam
No. Sat
Actual – Predicted Mean Scores
t-value
Option 1 (Yes)
27,804
0.25
-1.97
Option 2 (Trial)
19,381
0.03
-0.21
Option 3 (No)
21,293
- 0.35
2.34*
All Schools
68,478 -

16. Effect of SBA on Speaking Exam
Students in “Yes” schools did better than expected based on prediction from written papers (+0.25)
Students in “Trial” schools did slightly better than expected (+0.03)
Students in “No” schools did significantly worse than expected in the speaking exam (-0.35)

17. Congeneric Measures Model Analysis
Strong evidence that the various components are measuring a single underlying ability
Reliability of components .882 to .702
R > L&IS > W > SBA > S
SBA (.753) more reliable than speaking exam (.702)

18. Inter-paper correlations Correlation Matrix (by listwise case exclusion, N=28,253)
SBA
1.000
.858
.887
.776
.803
.852
.767
.797
.764
.796
.787

19. Survey Results

20. Main focus of questionnaire
Assessment of the modules in the Elective Part
Replacing the speaking exam with SBA
The need for strategic implementation of SBA in 2012 HKDSE
Teachers’ satisfaction with the level of support provided by the HKEAA and the EDB

21. Results: Response Rate
All Schools
Part.
Schools
Option 1 Schools
Option 2 Schools
Option 3 Schools
717
640
199
125
254
422
411
165
109
130
59%
64%
83%
87%
51%

22. Results Responses measured on 6-point scale
3.5 adopted as minimum level of acceptability

23. Assessment of the Modules in the Elective Part
Question
All schools
Option 1 schools
Option 2 schools
Option 3 schools
1. Assessing the modules in the Elective Part of the curriculum through the writing paper is appropriate.
3.7
2. The proposed Paper 2 structure is appropriate.
3.8
3. The proposed weighting of Part A (10%) and Part B (20%) is appropriate.
4.0
4. The question types for the modules in the Elective Part are appropriate.
5. Assessing the modules in the Elective Part of the curriculum through Part B of the SBA is appropriate.
3.9

24. Assessment of the Modules in the Elective Part
Question
All schools
Option 1 schools
Option 2 schools
Option 3 schools
6. The proposed weighting of Part B (5%) is appropriate.
3.6
3.7
3.5
7. The amount and type of effort required of teachers to implement the SBA in HKDSE is reasonable.
2.9
3.1
3.0
2.7
8. The amount and type of effort required of students to complete the SBA in HKDSE is reasonable.
3.4
3.3
9. Students have benefited from the introduction of SBA.
3.9
4.1
10. The effort spent on implementing SBA has been worthwhile.

25. Replacing the Speaking Exam with SBA
Question
All schools
Option 1 schools
Option 2 schools
Option 3 schools
12. School candidates need not take the public speaking examination as their speaking skills can be assessed by means of school-based assessment.
2.7
2.6

26. Teachers’ satisfaction with level of support provided by HKEAA and EDB (1)
13. Resources and support provided by the HKEAA
All schools
01. Handbook provided for the HKCEE English Language School-based Component
4.3
HKCE English Language Examination Introduction to the School-based Assessment Component (with DVD)
03. Training CDRs 1 and 2
4.1
04. HKCEE Chinese Language and English Language School-based Assessment (DVD distributed to parents and schools in October 2006)
3.8
05. HKCEE Chinese Language and English Language Information for Parents on School-based Assessment (leaflet distributed to parents in October 2006)
3.7
06. Statistical moderation of School-based Assessment scores (booklet distributed to schools in June 2007)
07. Group Coordinators System

27. Teachers’ satisfaction with level of support provided by HKEAA and EDB (2)
08. Teachers’ seminars on the implementation of HKCEE English Language SBA component held in September/October
3.9
09. Assessment training sessions held in November (chaired by SBA group coordinators)
4.0
10. Inter-school sharing sessions (chaired by SBA group coordinators)
3.8
11. Professional development courses (conducted by HKU and HKIED)
12. SBA Corner on the HKEAA website (including General Information, List of Recommended Texts and FAQs)
13. The Platform for SBA of CE English Language Exam on the English Campus of HKedCity
4.3
14. The HKU website on SBA Projects

28. Teachers’ satisfaction with level of support provided by HKEAA and EDB (3)
14. Resources and support provided by the EDB
All schools
01. Resource materials
4.0
02. Professional development programmes
03. Grants
4.3
04. School-based Support Services
3.8
05. Collaborative research and and development (“seed”) projects
3.5

29. Need for Strategic Implementation of SBA in 2012 HKDSE
Question
All schools
Option 1 schools
Option 2 schools
Option 3 schools
16. My school’s preparation plan for implementing the SBA for English Language in 2012 HKDSE is on track.
3.5
3.7
3.3
17. The proposed SBA framework for English Language is desirable for implementation in the 2012 HKDSE.
3.6

30. Respondents’ comments (1)
6. The proposed weighting of Part B (5%) is appropriate.
Part B should have a heavier weighting.
Both Parts A and B should each account for 10%
Both Parts A and B should each account for 5%

31. Respondents’ comments (2)
7. The amount and type of effort required of teachers to implement SBA in the HKDSE is reasonable.
Workload for teachers too heavy
Too many assessments to do
More resources needed, especially technical support
More sample tasks provided

32. Respondents’ comments (3)
8. The amount and type of effort required of students to complete SBA in the HKDSE is reasonable.
Reading six texts in three years and having to do four assessments in two years is too demanding.
Hard to monitor time spent by candidates on SBA reading/viewing programme

33. Respondents’ comments (4)
18. State the kind of modification(s) to the English Language framework you think would be desirable for the 2012 HKDSE.
Provide sample tasks
Improve standardization
SBA weighting should be increased/reduced
Use outside examiners for SBA
Keep Speaking exam
Reduce the number of assessments

34. Proposed/Revised Assessment Frameworks
Current proposed framework
Revised proposed framework
Components
Weighting
Requirements
Paper 1 Reading
20%
Paper 2 Writing
30%
25%
Part A
10%
Word limit: 200
Part B
Word limit: 400
15%

35. Proposed/Revised Assessment Frameworks
Components
Weighting
Requirements
Paper 3 Listening & Integrated Skills
30%
Paper 4 Speaking
20%
For private candidates only
10%
For all candidates

36. Proposed/Revised Assessment Frameworks
Components
Weighting
Requirements
SBA
20%
For school candidates only
15%
Part A
Three assessments; group interaction and individual presentation; report three marks
10%
Two assessments; group interaction and individual presentation; report two marks
Part B 5%
One assessment based on the modules in the Elective Part of the curriculum; report one mark

37. Rationale for the Revised Framework
Teachers need to deal with HKCEE, HKALE and HKDSE from 2009 to 2012
Speaking exam retained for school candidates as the majority of teachers do not favour its replacement by SBA
Weighting of SBA component reduced from 20% to 15% and weighting of Part B of Writing paper reduced from 20% to 15% so that 10% can be allocated to the Speaking exam
The total number of SBA marks to be submitted reduced from 4 to 3 to address teachers’ concerns about workload.
Weighting of SBA component reduced to 15%, which is the same as the 2007 HKCEE

38. Schedule for HKDSE Development
Standards-referenced Reporting (SRR)
Production of sample papers and marking schemes
Pilot test of sample papers conducted
Draft of level descriptors and annotated examples completed
Briefing sessions conducted for schools & SRR requirements finalised by Dec 08
SRR booklets with sample papers disseminated to schools by Mar 09

39. Schedule for HKDSE Development
School-based Assessment (SBA)
Assessment criteria, specifications and sample tasks developed
Briefing sessions conducted for schools; SBA requirements finalised by Dec 08
SBA handbook disseminated to schools by May 09

40. Thank You
Questions?