1. Portfolio Committee on Higher Education Report on the Quality Assurance of NC (V) L2-4; NATED N2-3 and GETC: ABET L4 3 February 2016 Dr Mafu S Rakometsi

2. WHAT CAN WE LEARN FROM THE NSC RESULTS? Introduction to context, principles, approaches and processes Dr Mafu Rakometsi - CEO of Umalusi

3. Quality Assurance of the DHET, IEB & BENCHMARK November 2015 examination Faith Ramotlhale: Senior Manager Quality Assurance of Assessments

4. Section17 of the GENFETQA Act (5) The Council must, with the concurrence of the Director- General and after consultation with the relevant assessment body or education institution, approve the publication of the results of learners if the Council is satisfied that the assessment body or education institution has— (i) conducted the assessment free from any irregularity that may jeopardise the integrity of the assessment or its outcomes; (ii) complied with the requirements prescribed by the Council for conducting assessments; (iii) applied the standards prescribed by the Council which a learner is required to comply with in order to obtain a certificate; and (iv) complied with every other condition determined by the Council.

5. Regulatory Framework Quality Assurance of Assessment NQF Act Section 27 (h)  The QC must develop and implement policy and criteria for assessment for the qualifications on its sub-framework.

6. Framework for Quality Assurance of Assessment  Evaluation and /or accreditation of assessment bodies;  Periodic inspection of assessment systems;  Ongoing monitoring of assessment systems;  Quality assurance of external examinations through: Moderation of examination question papers, PATs, CATs and ISATs Monitoring and moderation of SBA Monitoring of the conduct, administration and management of assessment and examination processes Moderation of marking Management of concessions and examination irregularities Standardisation of assessment outcomes

7. Moderation of question papers PURPOSE:  To ensure that the question papers are of the required standard;  To ensure that the question papers are relatively: Fair Reliable representative of an adequate sample of the curriculum representative of relevant conceptual domains representative of relevant levels of cognitive challenge

8. Approach DHET:NATED/NC(V))DHET: GETCIEB: GETCBENCHMARK: GETC OffsiteCentralisedOffsite Moderation Process: Question paper development - examiners (assessment body) Internal moderation - Internal moderators (assessment body) External moderation - External moderators (Umalusi) Subsequent moderations and approval

9. Moderation of the question papers  Technical criteria  Internal moderation  Content coverage  Text selection, types and quality of questions  Cognitive demands  Marking memorandum/ guidelines  Language and bias  Predictability Criteria :

10. Question Paper Approval Assessment body No of papers moderated Approval of Papers DHET: GETC26/2654% first moderation 46% second and third moderation IEB: GETC8/888% first moderation 12% second moderation BAA: GETC2/250% first moderation 50% second moderation DHET: NATED N2-N3 N2: 16/16 N3: 20/24 Deviation of 4 Business English papers- late submission and incorrect versions of papers submitted DHET: NC (V) L2-4 L2: 25/25 L3: 25/25 L4: 84/84 (Back-up papers of 2014 used, new papers approved where curricula changed and for 4 new Level 4 subjects)

11. Moderation of the question papers Areas of Good Practice:  No question paper leakages (GETC: IEB, BAA & DHET)  Approved GETC QPs 98% (DHET) and 97% (IEB) compliance ratings.  Some improvement in the internal moderation of the NATED subjects observed.

12. Directives for Compliance Assessment bodies should:  Improve the quality and standard of internal moderation -(DHET, IEB, BAA),  Increase the percentage of question papers approved at first moderation -(DHET, IEB, BAA),  Improve the quality of marking guidelines -(DHET),  Improve the quality of the question papers (GETC) for TRVT4 and WHRT4 – consistently poor quality since November 2013,  Review outdated and/or underspecified curricula for the (NATED) subjects.

13. Directives for Compliance The DHET is required to:  develop assessment frameworks for the NATED N2 and N3 subjects to address amongst others, the cognitive demands and the weighting of topics;  Review its tracking system for question paper moderation to ensure that correct versions of question papers, marking guidelines, internal moderator reports and assessment frameworks are submitted to Umalusi for external moderation, (NC(V) and NATED);  Adhere to its management plan for the setting and submission of question papers for moderation.

14. Internal Assessment (SBA) Definition:  Internal assessment refers to any assessment conducted by the provider, the outcome of which counts towards the achievement of the qualifications: GETC: 50%; NC (V) fundamental subjects 25%; NC (V) vocational subjects 50%; and NATED subjects 40%; Purpose:  Ascertain the degree to which the assessment body is attempting to ensure standardisation across providers,  Ascertain the standard and quality of the tasks,  Establish the extent and quality of internal moderation and feedback, and  Verify the reliability and validity of the assessment outcomes.

15. Internal Moderation - GETC Assessment Body CATs moderated and approved SBA Portfolios DHET5 sets x 26 learning areas On-site moderation: 251 portfolios 6/9 PEDs – 40 centres IEBNoneOn-site moderation: 4/8 Learning Areas 40 portfolios BenchmarkNoneOn-site moderation: 2/2 LAs 20 portfolios

16. Site-Based Assessment: GETC Areas of Good Practice  Year-on-year improvement regarding the administration of SBA, (DHET)  Implemented self-evaluation instrument and visited centres to support and improve implementation of SBA tasks, (BAA).

17. Directives for compliance: GETC  Assessors (educators) do not understand the use of rubrics, and marking guidelines, (BAA, IEB & DHET);  The DHET and IEB should develop a strategy to monitor and support the implementation of the SBA tasks to improve the quality of teaching and learning;  The IEB should revise their User Guides and Facilitator Guides and present these to Umalusi for moderation and approval;  BAA must implement a strategy to ensure that training providers implement all SBA tasks.

18. Internal Assessment (NC (V)/NATED) ISAT (tasks)ISAT (conduct)ICASS NATED 8/9 subjects NC( V) 13/1655/55 sites55/55 subjects Areas of good practice: ISAT  Some colleges had effective systems and procedures in place to ensure the smooth implementation of the tasks. Directives for compliance: ISAT  Defective ISATs should be revised (DHET plan in place),  ISATs should be distributed to colleges in time for effective procurement of resources and implementation,  Enrolments must be in line with available resources to effectively assess candidates’ acquisition of the practical skills required in the workplace,  To improve the quality and standard of marking and scoring,

19. Internal Assessment (NC (V)/NATED) Directives for compliance: ISAT (continued) The DHET must ensure that ISATs are implemented as per the specifications of the specific tasks and that effective moderation of the conduct of the tasks occurs at all levels.

20. Internal Assessment: NC (V)/NATED Directives for compliance/improvement: NC (V) ICASS The DHET should provide training on the internalisation of SAGs to promote effective planning, facilitation and conduct of assessment. Lecturers require training in setting and moderation practices as well as the review of assessments and tasks. Directives for compliance/improvement: NATED ICASS The DHET is required to enforce the implementation of the NATED ICASS guidelines at all sites to ensure that appropriate assessment tasks are developed and marks are correctly converted and captured.

21. Monitoring of Examinations: GETC Directives for compliance (DHET) The DHET should ensure that:  There is adequate and secure storage rooms for the storage of question papers,  The safety and security of examination materials is improved, as some centres keep answer scripts overnight,  All candidates submit proper identification during the writing of examinations. Assessment bodyDHETIEBBAA No exam centres 45156

22. Monitoring of Examination: GETC Directives for compliance (BAA & IEB) Improve the following instances of poor compliance with exam regulations:  Inadequate storage facilities for QPs with poor security arrangements,  Poor administration regarding exam regulations,  Use of cellphones in the examination venues,  Candidates allowed to write without proper identification

23. Monitoring of Examinations: TVET Areas of good practice  Improved systems in the conduct of examinations and measures to ensure the security of examination materials,  Effective training of chief invigilators and invigilators,  Conducive environment for the writing of examinations. Directives for compliance  The DHET should ensure that errata arrive on time at the various examination centres  The DHET should ensure that all examination centres have adequate and reliable security for the storage of examination materials. NC (V) and NATED No. exam centres41

24. Monitoring of Marking Phase: GETC Areas of Good Practice:  Marking centres were generally well organised and suitable for the task, (DHET,IEB,BAA)  Adequate and experienced security personnel at the gates and entrances to the venues, (DHET,IEB, BAA)  Training of marking personnel was generally thorough and effective, (DHET,IEB,BAA) DHETIEBBenchmark No. of marking centres 1111 No. centres monitored 611

25. Monitoring of Marking Phase: GETC Directives for compliance:  The Assessment Bodies should ensure that there is compliance with the following examination regulations:  Availability of the management plans (DHET, IEB & BAA)  Inadequate security measures (DHET, IEB)  Poor systems to manage and report irregularities (DHET)  Administrative mistakes such as sending wrong scripts to centres (DHET),

26. Monitoring of Marking Phase: TVET Areas of good practice: Marking centres provided an environment that was conducive to effective marking and control. NATEDNC (V) No of marking centre monitored7/101/3

27. Monitoring of Marking Phase: TVET Directives for compliance: The DHET must ensure that approved marking guidelines are received in good time at all marking centres, THE appointment of markers should be completed well in advance.

28. Purpose:  Moderation of marking determines the standard and quality of marking and ensures that marking is conducted in accordance with agreed practices  Umalusi engages with the following processes during the moderation of marking: Centralised memorandum discussions Verification of marking (centralised and on-site) Verification of Marking:

29. Areas of good practice:  The panel discussions and the marking of exemplar scripts improved the quality of the marking guidelines (DHET, IEB, BAA). Directives for compliance:  The DHET must ensure that it improves the quality of Afrikaans versions of most marking guidelines,  The DHET should develop a system to ensure that the final memoranda are discussed and cascaded at provincial level,  The IEB should ensure that the marking personnel come prepared at the marking centre,  The BAA should prioritize the training of markers before the actual commencement of marking. Memorandum Discussions - GETC

30. Marking guideline discussions and verification of marking TVET Areas of good practice:  More markers came prepared to the marking centre  Sample marking took place in most subjects  In the majority of subjects there was good adherence to the marking guidelines  In most subjects markers kept notes to enhance the quality of chief marker and internal moderation reports NC (V)NATED N2NATED N3 Marking guideline discussions 38 subjects5/16 subjects moderated 11/20 subjects moderated Verification of marking L2 8/25 L3 6/25 L4 55/76 subjects 12/1619/20

31. Marking guideline discussions and verification of marking TVET (cont) Directives for improvement/compliance: The DHET should: ensure that adequate, suitably qualified and experienced marking staff is recruited and appointed timeously, enforce the attendance of the general training session and the marking guideline discussion by all marking staff, ensure that the correct versions of marking guidelines are distributed and used across the different marking centres, Improve the quality of internal moderation and comply with the 10% internal moderation of all scripts in all subjects, (NC(V)

32. Areas of Good Practice:  Good internal moderation practices (DHET, IEB, BAA).  Good quality and standard of marking, (DHET, IEB, BAA). Verification of Marking: GETC DHETIEBBenchmark On-site moderation  15/26 LA  945 scripts  4/8 LA  83 Scripts  2/2 LA  40 scripts

33. Directives for compliance:  The DHET should ensure that all irregularities are reported by the Internal Moderators and Chief Markers,  The DHET should put measures in place to avoid damaged and/or lost computer discs containing the practical work of candidates in KZN,  The BAA should prioritise the training of markers before the commencement of marking. Verification of Marking

34. Standardisation and Resulting  Provision of GENFETQA Act, 2001 as amended in 2008-– Council may adjust raw marks,  International practice – large scale assessment systems,  Standardisation – process used to mitigate the effect of factors other than learners knowledge and aptitude on the learners performance,  Sources of variability – difficulty in question paper, undetected errors, learner interpretation of questions.

35. Verification of the Resulting Process PlannedStatus Subject structuresSubject structures verified Candidate registrationSystem verified during state of readiness visits Generation of mark sheetsMonitored Capturing of MarksMonitored Standardisation data & Booklets Data sets received and verified Capturing of adjustmentsVerified Statistical moderation & resulting Verified

36. Statistical Moderation Scope of standardisation GETC 2015: DecisionNumber of subjects DHETIEBBM Subjects standardised2682 Raw marks accepted1382 Adjusted mainly upwards 11 -- Adjusted mainly downwards 2 --

37. Statistical Moderation - GETC Areas of good practice:  Booklets and data sets were submitted according to the prescribed timelines (DHET, IEB, BAA).  Standardisation and approval meetings went smoothly.

38. Statistical Moderation - GETC Directives for improvement:  The DHET should develop and implement strategies to improve the poor performance of learners; high number of irregularities; and absenteeism to write the examinations,  The IEB should develop its internal quality assurance of standardisation booklets to ensure that accurate data on graphs and statistics is submitted,  The BAA should document the processes and procedures outlining standard operating procedures w.r.t. capturing and resulting.

39. Statistical Moderation TVET Scope of standardisation 2015: DecisionNumber of subjects NATEDNC (V) Subjects standardised: Engineering Studies : Business Studies 59 4252 Raw marks accepted: Engineering Studies : Business Studies 29 2 122 Adjusted mainly upwards: Engineering Studies : Business Studies 28 2 127 Adjusted mainly downwards12 No decision11

40. Statistical Moderation: TVET Areas of good practice  KS norms replaced by Ogives (NATED) Directives for improvement: The DHET should ensure that:  Standardisation booklets are submitted on time to avoid delays in the standardisation of results,  Accurate data sets and low capture rate is improved to avoid delays in the approval of results,  The consolidated irregularity report is submitted to Umalusi 24 hours prior to the standardisation of results,

41. Conclusion  The quality assurance processes presented above were conducted based on Umalusi criteria. Umalusi uses criteria that are subjected to constant review and refinement, to ensure that they are in line with current trends in assessment and examinations.  From the findings of the quality assurance processes some pockets of good practices were evident but on the other hand, there are still many challenges that need to be addressed.  Significant interventions are required to improve the curriculum, teaching and learning, exposure of learners to a compulsory practical component and assessment for the NATED programmes to make it a meaningful qualification.

42. Conclusion  The outdated and underspecified curriculum and lack of subject assessment guidelines for the NATED programmes creates problems for appropriate quality assurance to be conducted.  The DHET is commendable for the radical actions taken to curb paper leakages in the NATED examinations.  The development and implementation of the ogive method for the first time in 2015 is also worth commendations.  Umalusi takes this opportunity to express appreciation to the department of higher education and training for their concerted efforts in delivering credible examinations in 2015.

43. Thank you!