1. Portfolio Committee on Basic Education STANDARDISATION
Dr M Rakometsi
CEO: Umalusi
Prof M Moodley
ASC Member
7 February 2018

2. Overview
Mandate and Regulatory Framework
Framework for Quality Assurance of Assessment
Standardisation of results
Demystifying Myths on standardisation
2017 NSC Standardisation decisions
Conclusion

3. Mandate and Regulatory Framework
Functions of the quality council (QC) regarding quality assurance of assessment
National Qualification Framework (NQF) Act No. 67 of 2008
The QC must develop and implement policy and criteria for assessment for the qualifications on its sub-framework.
The QC must, in respect of quality assurance within its sub- framework, do the following:

4. Mandate and Regulatory Framework
Functions of the quality council (QC) regarding quality assurance of assessment
National Qualification Framework (NQF) Act No. 67 of 2008
develop and implement policy for quality assurance;
ensure the integrity and credibility of quality assurance;
ensure that quality assurance as is necessary for the sub- framework is undertaken;

5. Mandate and Regulatory Framework
Section17A of the GENFETQA Act
(3) The Council must, perform the external moderation of assessment of all assessment bodies and education institutions
(4) The Council may adjust raw marks during the standardisation process

6. Mandate and Regulatory Framework
Section17A of the GENFETQA Act
(5) The Council must, with the concurrence of the Director-General and after consultation with the relevant assessment body or education institution, approve the publication of the results of learners if the Council is satisfied that the assessment body or education institution has—
(i) conducted the assessment free from any irregularity that may jeopardise the integrity of the assessment or its outcomes;
(ii) complied with the requirements prescribed by the Council for conducting assessments;
(iii) applied the standards prescribed by the Council which a learner is required to comply with in order to obtain a certificate; and
(iv) complied with every other condition determined by the Council.

7. Mandate and Regulatory Framework- Cont.
Section18 of the GENFETQA Act
an assessment body—’’.
18(a) must take adequate measures to combat irregularities at assessment and
marking centres and must take adequate security measures to ensure the
confidentiality and integrity of the assessments;
This section read in conjunction with section 42(6) of the Regulations pertaining to the conduct, administration and management of the NSC examination Gazette No 31337, emphasises the importance of keeping confidentiality of all resulting data (written, electronic format and oral)
18(j) may recommend to the Council during the standardisation process that raw
marks be adjusted.

8. Framework for Quality Assurance of Assessment
Quality assurance of external examinations through (18 months cycle):
- Moderation of question papers;
- Moderation and verification of school based assessment (SBA);
- Monitoring the state of readiness to conduct the NSC;
- Monitoring and audit of the selection and appointment of markers;
- Monitoring of the writing of examinations;
- Marking guidelines discussion;
- Monitoring of marking;
- Verification of marking;
- Standardisation and resulting; and
- Certification

10. Objectives of Standardisation
To ensure that a cohort of learners is not advantaged or disadvantaged by extraneous factors other than their knowledge of the subject, abilities and their aptitude.
To achieve comparability and consistency from one year to the next.

11. Why Umalusi Standardises Results and How
International practice – large scale assessment systems;
Standardisation – process used to mitigate the effect of factors other than the learners’ knowledge and aptitude on the learners performance.
Sources of variability – difficulty in question paper, undetected errors, learner interpretation of questions.

12. Why Umalusi standardises results and how cont.
Assumptions – for large populations the distribution of aptitude and intelligence does not change appreciably;
Process of standardisation:
Moderation of question papers;
Review of learner performance against historical performance of candidates in each subject;
Historical average (norm) constructed using past 3 to 5 years data;
Pairs analysis provides further comparisons of raw means;
Statistical moderation of internal assessment.

13. Why Umalusi standardises results and how cont.
Qualitative input meetings;
Reports (Moderator, Chief Marker and Internal Moderator);
Umalusi research (maintaining standards & post exam analysis);
Responsibility of Assessment Standards Committee (ASC) - Committee of Council (independent experts drawn from different HE institutions and research institutes):
Education, Mathematics and Statistics experts – large data
ASC is responsible for setting and maintaining standards;
Observers (SAQA, USAf, Teacher Unions – (Portfolio Committee on Basic Education since 2011).

14. STANDARDISATION WORKSHOP

15. General Principles Applied in the Standardisation of Examination Marks
1. In general no adjustment should exceed 10% of the historical average.
2. In the case of the individual candidate, the adjustment effected should not exceed half of the raw mark obtained by the candidate.
3. If the distribution of the raw marks is below the historical average, the marks may be adjusted upwards subject to limitations.

16. Demystifying Myths on Standardisation
Standardisation is a manipulation of figures to increase learners’ pass rate
Fact 1:
Standardisation is not a process aimed at increasing learners’ pass rate but it is a process aimed at ensuring the equivalence of marks from year to year, regardless of variability of examinations, in the pursuit of a high level of fairness.
Standardisation is an international process which is not unique to South Africa. The overall pass rate is never known at this stage.

17. Demystifying Myths on Standardisation
Fact 1:
In other words an A,B or C symbol should be equivalent to those of previous years and future years regardless of variability in exams.
Candidates’ marks are only adjusted if there is compelling evidence from both qualitative and quantitative reports that the standard of the examination does not compare fairly with the previous examinations.
Candidates examination marks are then adjusted to ensure the candidates are not ,advantaged or disadvantaged due to factors that are beyond their control.

18. Demystifying Myths on Standardisation
Mainly upward or downward adjustment of marks involves the increase or decrease of learners’ marks by the same mark from 0-300
Fact1:
Standardisation is a process guided by principles and guidelines of importance as follows:
No adjustment should exceed half of the raw mark
Adjustment should not exceed 10 percentage points above or below the raw mark
For example, a raw mark of 10 will never exceed 15 and a raw mark of 130 will never be adjusted to more than 160 or less than 100

19. Demystifying Myths on Standardisation
Fact 2:
Mainly upward/downward adjustment does not necessarily mean that all marks from increase/decrease by the same mark.
A mainly upward adjustment can have some marks adjusted downwards while a mainly downward adjusted can have some marks adjusted upwards depending on the candidates performance , in relation to the norm(see Annexure A)
The adjustment is identified as mainly upward/downward ,when most marks are adjusted upward or downward respectively (see Annexure B)

20. Demystifying Myths on Standardisation
The decision to adjust is only determined by the mean of the current cohort
A maximum adjustment is measured from the mean
Fact:
A maximum adjustment is not measured by the mean alone but by the spread of adjustments (see Annexure A)
Decision to adjust is not determined by the mean only as the following factors are also taken into consideration:
The qualitative input reports
The previous five years and the norm (see Annexure C & D)
pass rate
distinction rate
failure rate
As well as the median.

21. Demystifying Myths on Standardisation
Candidates’ final marks are finalised during standardisation
Fact :
Candidates final marks are not finalised at standardisation meetings
NSC resulting consist of two components: the external component (examination) and the internal component (SBA).
The final result/mark of the candidates consists of 75% ,exam and 25% SBA
During the standardisation meeting adjustment are made only on the external component/examination raw mark
(These adjustments are then effected on the examination…sense??) del
The SBA marks are subjected to a process of statistical moderation before being added to the adjusted exam mark in the above ratio.
The underlying assumption is that the learners’ performance in SBA and examination bears a positive relationship

22. 2017 NSC Standardisation Decisions SACAI
Description
Total
2015
2016
2017
Number of subjects presented 27 28
Raw marks 24 16 18
Adjusted (mainly upwards) 2 9 7
Adjusted (mainly downwards) 1 3
Number of subjects standardised:
Description
Total
Number of instructional offerings presented 59
Raw marks accepted 28
Computer adjustment 17
Half computer adjustment 10
Block Adjustment (mainly upwards)
None
Block Adjustment (mainly downwards) 1

23. 2017 NSC Standardisation Decisions IEB
Description
Total
2015
2016
2017
Number of subjects presented 61
Raw marks 49 46
Adjusted (mainly Upwards) 12 7 5
Adjusted (mainly downwards)
none 10
Number of subjects standardised:

24. 2017 NSC Standardisation Decisions DBE
Description
Total
2015
2016
2017
Number of subjects presented 59 58
Raw marks 29 26 38
Adjusted (mainly upwards) 30 28 16
Adjusted (mainly downwards)
none 4
Number of subjects standardised:

25. In conclusion
The ASC, the committee of Council that standardises the results, is an independent body made up of experts from different universities and research institutes in the country.
The ASC standardizes the results of the NSC, NC(V), Nated Courses (N2-N3) and the GETC - offered by the DBE, DHET, IEB, SACAI and Benchmark Assessment Agency.
Equal attention must also be given to other qualifications - not just the NSC.

26. Conclusion
The standardisation process is cumulative - it starts with the appointment of examiners, setting of question papers, moderation of question papers, verification of marking and the so-called standardisation meetings whose processes mark the end of the actual standardisation process.
The full framework of standardisation is an 18 month process encompassing the exam cycle from the setting of the paper to the final standardisation meeting.

27. Conclusion
The final pre-standardisation meeting starts on 15 December and the climax is the open meeting like the one attended by members of the Portfolio Committee on 23 December During pre-standardisation meetings all evidence - both quantitative and qualitative - is considered.
Why is the standardisation data confidential? The standardisation marks are not final because they only include 75% written component and not the 25% of the School-Based Assessment (SBA). Standardisation marks also do not include rules of combination, condonation and language compensation marks.

28. Conclusion
That is why it is dangerous to speculate the outcome of the results based on standardisation outcomes. Any predictions are misleading as the final outcome may be different after taking everything into account.
The marks attained after standardisation do not form the final mark on the subject and the overall pass rate is never known at this stage

29. 12th Conference of the Southern Africa Association for Educational Assessment (SAAEA)
Host: Umalusi
Date: 14 – 16 May 2018
Venue: The Capital Hotel, Menlyn, Pretoria
Theme: Local context in Global context: encouraging diversity in assessment
See you there!!!

30. Thank You!!