2 SQA’s Approach to Quality Assurance of Assessment Matthew McCullaghQuality ManagerWelcome欢迎
3 Why are we here?To introduce you to SQA’s New Approach to Quality AssuranceTo help you understand the key changes to Quality Assurance processTo help you become familiar with:The new Quality Assurance CriteriaThe new Confidence StatementsHow visits are planned, conducted and reported
4 Thursday 6th September 2012 9.00 Coffee and Registration Introduction, New Approach to QA Overview, Aims and Objectives of Training DaysPreparation for Qualification Verification Activity, Attitudes and Behaviours, KPMsBreakPreparation for Qualification Verification Activity,Attitudes and BehavioursPlanning for Qualification Verification Activity – the theoryLunchConducting the Qualification Verification Visit – the theoryBreakIntroducing the Quality Assurance CriteriaQuality Assurance Criteria WorkshopClose
5 Friday 7th September 20129.00 Coffee and Registration 9.30 Feedback: The Quality Assurance Criteria Break Decision Making Lunch 1.30 Communicating Decisions 2.00 Theory of Completing Excel Reports/Action Plans 2.30 Break 2.45 Writing effective Verifier Reports 3.45 Case Study Assignment 4.00 Summary and Q & A 4.30 Close
6 SQA’s Approach to QASQA is moving to introducing a new approach to Quality Assurance of qualifications delivered in ChinaIt is based on SQA’sPrinciples of Quality Assurance:Open, fair and transparentAimed at risk reductionProportionateIntelligence ledA shared responsibilityConstantly reviewed and improved
7 What is the new approach to Quality Assurance SQA now operates a transparent, intelligence-led, risk based model for quality assurance of approved centres and potential centres delivering our HND qualifications in China.
8 Timeline of Implementation of New Approach in China Overview to Centre staff in Beijing and XiamenLaunch of new approachOverview to EVs at central verification eventsSpecialist training for Centre staff and EVsImplementation new approach to QASeptember 2011March2012May2012September 2012September
9 New Quality Criteria Five Categories: Management of a centreResourcesCandidate SupportAssessment & VerificationRecords/Data ManagementAll signposted to four QA processesAll pre-rated as High/Medium/Low impactAll supported by possible sources of evidence
10 The New Approach to Quality Assurance: The Four Processes Systems ApprovalQualification ApprovalSystems VerificationQualification Verification
11 High/Medium/Low Impact Criteria All criteria are importantHowever, some are more important than othersThose that are most important are rated as High Impact criteriaThose that are less important are rated as Low Impact criteriaIn between these two are medium impact criteria.Failure to meet a High Impact criteria will have a greater impact on the outcome of the visit
12 Sufficiency of evidence No Systems Verifier or Qualification Verifier can change the impact rating of a criteria.For each criterion you verify, you must decide whether the centre has presented you with:Sufficient evidence to meet the criterionSome, but insufficient evidence to meet the criterionNo evidence to meet the criterion
13 Good Practice and Guidance As part of the verification visit, you should identify good practice that is evident in the centre.You should also give advice and guidance to help centre staff improve the delivery of our qualificationsThis is a mandatory part of your role as a Qualification Verifier
14 Sufficiency of evidence We will support you and help you to understand what sort of evidence centres can provide to meet each criterionOnce you have made your decision regarding the sufficiency of evidence, SQA staff will then calculate our level of confidenceThis will depend on the impact level of the criteria and the sufficiency of the evidence presented
15 Introduction of Confidence Statements In relation to Qualification VerificationConfidence StatementIn relation to Systems VerificationHigh level of confidence in the maintenance of SQA standards within this Verification GroupHigh Level of ConfidenceHigh level of confidence in the systems that support the maintenance of SQA standards within this centreBroad confidence in the maintenance of SQA standards within this Verification GroupBroad ConfidenceBroad confidence in the systems that support the maintenance of SQA standards within this centreReasonable confidence in the maintenance of SQA standards within this Verification Group, although moderate risks exist within the following categories:(insert appropriate category)Reasonable ConfidenceReasonable confidence in the systems that support the maintenance of SQA standards within this centre, although moderate risks exist within the following categories:Minimal confidence in the maintenance of SQA standards within this Verification Group as significant risks exist within the following categories:Minimal ConfidenceMinimal confidence in the systems that support the maintenance of SQA standards within this centre as significant risks exist within the following categories:No confidence in the maintenance of SQA standards within this Verification Group as severe risks exist within the following categories:No ConfidenceNo confidence in the systems that support the maintenance of SQA standards within this centre as severe risks exist within the following categories:
16 How Confidence Statements are calculated Confidence statements are given for the outcome of each Systems Verification and Qualification Verification visit for each of the categories of criteria verifiedThese are based on the sufficiency of the evidence provided to the youA summary statement of confidence is also given, based on the confidence statements for each category
17 What if non-compliances are identified? Where there is No evidence or insufficient evidence to meet a criterion, you must agree with centre staff, during the visit:the action the centre needs to take in order to become compliant.The evidence they need to produceWhere to send this evidenceThe date by which the action must be taken
18 What happens then?SQA will ask you to consider the evidence the centre sends us and ask you to consider whether it is Sufficient or insufficient.Depending on the sufficiency of the evidence, the Confidence Statement will be re-calculated.
19 SanctionsIf the outcome of a verification visit results in a Confidence Statement of Reasonable, Minimal or No Confidence, SQA staff may decide to place a sanction on a centre until our confidence increases to Broad or High.Our confidence may decrease if a centre fails to meet action points agreed during a verification visit.Sanctions will only be used where necessary.
20 Sanctions In relation to Qualification Verification Confidence StatementIn relation to Systems VerificationHigh Level of ConfidenceEntry in Action PlanBroad ConfidenceSuspension of specific qualification certification:by verification groupby qualificationSuspension of approval application by verification groupSuspension of direct certification claim status:Reasonable ConfidenceSuspension of all existing qualification approvalSuspension of centre certificationSuspension of certification for all qualifications with assessed componentsSuspension of qualification approval application - whole centreSuspension of specific qualification approval:Minimal ConfidenceWithdrawal of all existing specific qualification approvalWithdrawal of specific qualification approval:No ConfidenceWithdrawal of centre approval
21 Benefits of new approach for centres Open and transparent to allClear guidance available to verifiersProportionate response where issues are identifiedFocused on Good Practice and improvement as well as compliance
23 SQA’s Approach to Quality Assurance of Assessment David PirnieLead VerifierWelcome欢迎
24 Qualification Verification The Qualification Verifier role activitiesPrepare for verificationPlan verificationCarry out verificationCommunicate verification decisionComplete verification report
25 Preparing for Qualification Verification The Qualification Verifier role activitiesPrepare for verificationPlan verificationCarry out verificationCommunicate verification decisionComplete verification report
26 Being prepared “The best preparation for good work tomorrow is to do good work today”.Elbert Hubbard
27 Preparing for Qualification Verification Before carrying out any activities, verifiers arerequired to have a full understanding of their roleand responsibilities. This involves acquiringknowledge and keeping it up to date.
28 Preparing for Qualification Verification Required knowledge – three key areas:Knowledge of your subject areaCentre roles and centre responsibilitiesQualification Verification
29 Preparing for Qualification Verification Knowledge of your subject area:Have a common interpretation of the Unit Standards including the Evidence Requirements for each unit you have been assigned to verifyKnow how to access Arrangements Documents, Instruments of Assessment and Marking Schemes to gain a full understanding of content
30 Preparing for Qualification Verification Knowledge of centres roles and responsibilities:Centre Co-ordinatorsAssessorsInternal verifiers
31 Preparing for Qualification Verification Knowledge of the Qualification Verifier role:Role ProfileSources of supportVerification policies and proceduresValues
32 QIIPS: Quality, Integrity, Innovation, Partnership & Service Sounds like:“QUIPS” – a dictionary definition:“A clever remark often prompted by the occasion”.
34 SQA Values Quality As custodians of the standards we verify, we continually strive to maintain consistent quality intheir assessment.
35 SQA ValuesIntegrityWe use open and honest communication with internal and external customers (centres), promoting transparency, and building trust with others.We take a standardised approach to verification as laid down by SQA and agreed by consensus within our Qualification Verification team, even if our personal opinions may at times conflict.
36 SQA ValuesInnovationWe view unfamiliar approaches to assessment and verification with an open mind, whilst ensuring quality is maintained.We believe that creativity should be seen not as a threat to quality assurance but as an opportunity for improving upon existing practice.
37 SQA Values Partnership We understand that by working in partnership with SQA centres, we will achieve common goals of excellence and consistency in assessment.
38 SQA ValuesServiceWe maintain a professional approach with SQA centres at all times, regardless of the challenges we might face in carrying out our work.
39 Preparing for Qualification Verification More information on the preparation phase ofQualification Verification can be found in the firstsection of SQA’s New Approach to QualityAssurance: Guidance on Visiting Verification ForHN Qualifications in China
41 Planning for Qualification Verification The Qualification Verifier role activitiesPrepare for verificationPlan verificationCarry out verificationCommunicate verification decisionComplete verification report
42 “Failing to plan is planning to fail”. Alan Lakein
43 Visit Planning: Sampling SQA cannot always verify all HN Units, for allcandidates, on every visit to a centre, therefore asampling strategy has to be adopted.Purpose of sampling:To establish that there is a common interpretation of the standards within each centre and that the assessment and verification systems allow valid, reliable and fair assessment decisions to be made.
44 Visit Planning: Applying a sampling frame SQA is responsible for systematically sampling ona percentre basis:Selected UnitsAssessment decisions and practiceMethods/Instruments of assessmentProblematic/revised unitsLocations where assessment takes placeEvidence of candidates across UnitsDocumented evidence of policies/procedures (implementation)
45 Visit PlanningSQA (China Office) agrees optimum date (s) for visiting verification. This is an agreement between SQA, each centre and each QVSQA selects Units to be verified and communicates this to each centre (copying to SQA China Office)SQA (China Office) confirms to each QV,the centre(s) they should visit, the date of each visit, and the Units they should verifyQualification Verifier contacts centre to confirm visit date(s) and travel arrangements
46 New Approach to Quality Assurance Welcome Back欢迎回来
47 Carry out Qualification Verification The Qualification Verifier role activitiesPrepare for verificationPlan verificationCarry out verificationCommunicate verification decisionComplete verification report
48 Carry out Qualification Verification Discussion pointConsider the verification activities you performfrom the time of arrival until the end of the visit.Contribute to the discussion by letting us knowwhat kind of techniques/approaches you haveused that you have found particularly beneficial.
49 Carry out the visit Prepare yourself for the visit Agree agenda and running order of visitConduct sampling activitiesJudge the sufficiency of evidence
50 Conduct sampling activities Complete Units (all Outcomes complete)Incomplete Units (some Outcomes complete)Evidence not yet internally verifiedAssessment/verification decisionsAssessment and verification practiceInterviews with assessors/verifiers/candidatesReferencing of evidence to standards
51 Introducing the Quality Criteria Introducing the New Approach to QualityAssurance CriteriaIntroducing the Qualification Verification Guidance
52 Introducing the Quality Criteria In your groups, discuss and familiarise yourselves with the criteria you have been allocated.Formulate a list of summary points on flip chart paper to help you explain the purpose of each criterion to your colleague verifiers. The summary should also include possible sources of evidence.In formulating your points, you are asked to use SQA’s New Approach to Quality Assurance: Guidance on Visiting Verification for HND Qualifications in China as your information source.Timings:30 minutes discussion and research20 minutes flip-charting your pointsPlease appoint a spokesperson(s) to present the flip-charted points which will be presented in the first session tomorrow morning.Each group will be assigned a facilitator to answer any questions/queries you may have.
54 The New Approach to Quality Assurance Welcome Back欢迎回来
55 The New Approach to Quality Assurance Introducing the the new approach to quality assurance criteriafeedback
56 Criteria 4.2 4.2 Signed pre-delivery checklist. The centre must provide documented evidence to ensure that assessments are valid, reliable, equitable and fair.Signed pre-delivery checklist.Standardisation meeting minutes, internal audit, review records
57 Criteria 2.32.3Records must be maintained to provide evidence that the centre has sufficient competent staff who have the necessary qualifications, occupational experience and understanding to support the assessment and internal verification of qualifications being offered in the centre.CVs/staff info sheets, CPD records, copies of relevant certificates, list of current assessors and internal verifiers, changes to the deployment of assessors/internal verifiers
58 Criteria 4.84.8Outcomes of External quality assurance must be disseminated to appropriate staff and any action points must be monitored against agreed timescales.Signed distribution list, corrective action log/report, minutes of meetings
59 Criteria 3.23.2Candidates' development needs and prior achievements (where appropriate) must be matched against the requirements of the award. These must then be related to agreed personal action/training plans.APL evidence, skills profile, training needs analysis, assessment plans, reviews
60 Criteria 4.64.6The centre must comply with requests for access to premises, records, information, candidates and staff for the purpose of external quality assurance.Documented procedure for handling QA visits, roles and responsibilities, site selection checklists, permission for SQA QA representatives to obtain access
61 Criteria 2.7 2.7 Current Health & Safety certificates There must be evidence of initial and on-going reviews of accommodation, equipment and reference, learning and assessment materials.Current Health & Safety certificatesitinerary checklists, procurement records, library contents, system for supporting e-assessmentRecords of review
62 Criteria 4.34.3Evidence of candidates' work must be accurately and consistently judged by assessors against SQA's requirements.Assessor reports, records of achievement, standardisation meeting minutes, IV reports, Qualification Verification reports
63 Criteria 4.14.1The centre's documented assessment and verification procedures must be implemented and the subsequent assessment/verification practices reviewed, the results of which must be recorded and actioned.Minutes of assessor/internal verifier meetings, internal audit, review records
64 Criteria 4.7 4.7 Candidate evidence, documented retention policy Candidate evidence must be retained in line with SQA requirements.Candidate evidence, documented retention policy
65 Criteria 4.54.5The centre must take steps to ensure that assessment evidence is the candidate's own work.Candidate disclaimer, Candidate and staff induction materials: assessor/IV roles and responsibilities
66 Criteria 5.65.6Comments/queries about the qualification specification, assessment guidance, qualification verification or related SQA matters must be resolved and recorded.Correspondence file, s, standardisation meeting minutes, notes of action points and resolution
67 Preparing for Qualification Verification The Qualification Verifier role activitiesPrepare for verificationPlan verificationCarry out verificationCommunicate verification decisionComplete verification report
68 Making verification decisions Deciding on the sufficiency of evidence can be likened to making an assessment decisionMaking an assessment decision takes account of the overall strength of the evidence providedOne source of evidence will be sufficient at times. At other times, a number of sources will have to be considered, dependent on each centre’s approach to meeting quality assurance requirements and to some extent the structure of each criterion
69 One evidence source could be sufficient in some cases for this criterion: Candidate’s development needs and prior achievements (where appropriate) must be matched against the requirements of the award. These must then be related to an agreed personal action/training plans.The following are examples only, and do not constitute acomplete list of all possible types of evidence to support theabove criterion.APL/RPL/APA evidenceInitial/diagnostic/formative assessment, skills profiles, training needsanalysesPersonal development/action plans, competence reviews, learneragreement/contracts, assessment plans
70 More than one evidence source is likely to be considered for this criterion: The documented assessment and verification procedures must be implemented and the subsequent assessment/verification practices reviewed, the results of which must be recorded and actioned. The following are examples only, and do not constitute a complete listof all possible types of evidence to support the above criterion.Records of assessmentRecords of internal verification/audits, monitoring of assessor/verifierpracticeReviews of assessment and verification practices
71 The main guiding principles for making verification decisions Decisions should be based:on the sufficiency of evidence relating to each criteriononly on the criteria providedThe evidence examples given in the guidancedocument are just that. They are not a shoppinglist of evidence sources that all centres must have.
72 Awareness of things that could influence on our decisions Attitude of centre staffOwnstandardsOwnexperienceOwnmindset (closed)Our confide-encePrevious centre perform-ance
73 Insufficient/no evidence = action points Sufficient evidence - this means that the centre has provided evidence that fully meets the criterion (no action points required)No evidence – this means that the centre has not provided any evidence in support of the criterion (action points required)Insufficient evidence – this means the centre can provide some evidence in support of the criterion (action points required)
74 Agreeing action points Is there anything wrong with this action point?“You should submit evidence of havingdisseminated the results of Qualificationverification visits to relevant staff”
75 Action points should be SMART Specific: Centres should be clear in terms of the action they need to take to close-off the action point.Measurable: How will SQA or the centre know that the action point has been met, what will be the measure of success?Achievable: The centre must have sufficient time in which to achieve the agreed action. Please factor in time it takes for SQA to receive, process and edit the report and send it to the centre.Relevant: The action must directly relate to the criterionTime bound: Agree a specific and realistic.date for the action to be completed by.
77 New Approach to Quality Assurance Welcome Back欢迎回来
78 The stages of Qualification Verification Prepare for verificationPlan verificationCarry out verificationCommunicate verification decisionComplete verification report
79 The stages of communicating verification decisions Plan to deliver balanced feedbackExplain your verification decisions
80 Plan to deliver balanced feedback Plan to include:Good practiceRecommendationsAction points (where appropriate)Consider:LocationOrder of feedback and emphasisWords and tone
81 Explaining your verification decisions Where action points are given, ensure you:have explored all possible evidence sourcesgather all information to support your decisionscan justify your decisions - clearly relating themto criteria requirementsGive the centre an opportunity to ask questions and clarify points
82 Communicating Decisions Role play exercise:David as a Qualification Verifier feeding back at the end of a Qualification Verification visit to Matthew who is the SQA Co-ordinator.Please note your observations of the following on your observation sheet:The content of the FeedbackAttitudes and behaviours of the Qualification Verifier andthe SQA Co-ordinatorAny possible improvements
83 The stages of Qualification Verification Prepare for verificationPlan verificationCarry out verificationCommunicate verification decisionComplete verification report
84 The stages of completing the Qualification Verification Report Ensure report content matches your feedbackJustify your verification decisionsWrite clear action pointsCheck report prior to submitting to SQA
90 Writing Effective Verifier Reports SMART action pointsReport writing guidance availableccc/Writing for SQA part A.pdfWrite report as soon after visit as possibleChecking content thoroughly
91 Assignment: Report Writing Assignment will be ed to you by Thursday 20th September.You will be asked to:Review a Case StudyComplete a Qualification Verification Visit ReportReturn to by Friday 12th October
92 Assignment: Report Writing SQA will provide you with:Guidance document on how to complete a qualification verification visit reportFeedback on each report to provide supportbyFriday 26th October 2012
93 Summary To introduce you to SQA’s New Approach to Quality Assurance To help you understand the key changes to Quality Assurance processTo help you become familiar with:The new Quality Assurance CriteriaThe new Confidence StatementsHow visits are planned, conducted and reported