Presentation on theme: "Assessment Validation – Why and How Tony Auciello 18 th October 2012."— Presentation transcript:
Assessment Validation – Why and How Tony Auciello 18 th October 2012
Ambition in Action www.sit.nsw.edu.au Session objective /Gain a common understanding with regard to validation requirements for the VET sector and Sydney Institute Assessment validation Assessment Design Competency Mapping
Ambition in Action www.sit.nsw.edu.au What is assessment validation? Process of comparison and evaluation of assessment tools/processes against competency standard/s and bench marks to ensure that assessment tools meet the Principles of Assessment and the Rules of Evidence.
Ambition in Action www.sit.nsw.edu.au Assessment Validation Activities /Assessment Validation activities involve teachers developing a shared interpretation of competency standards and comparing assessment methods and tools. This process is to encourage dialogue between teachers and with industry about assessment practice.
Ambition in Action www.sit.nsw.edu.au Teachers Role in Assessment Validation /Teachers are required to participate in assessment validation activities and record the changes that they make to assessment tools and activities.
Ambition in Action www.sit.nsw.edu.au Assessment Validation Procedure /Assessment Validation involves planning assessment validation activities on an annual basis, conducting assessment validation and improving assessment strategies, processes, tools and practices in a continuous improvement cycle using a risk management approach
Ambition in Action www.sit.nsw.edu.au NVR Standards for the operation of RTOs The new Standards for nationally regulated RTOs (the SNRs), formerly the AQTF standards, require RTOs to systematically validate assessments. 15.5 Assessment including Recognition of Prior Learning (RPL): (a) meets the requirements of the relevant Training Package or VET accredited course; and (b) is conducted in accordance with the principles of assessment and the rules of evidence; and (c) meets workplace and, where relevant, regulatory requirements; and (d) is systematically validated.
Ambition in Action www.sit.nsw.edu.au Industry Engagement about Assessment /The Australian Vocational Education and Training system is “industry-led”. Industry engagement is built into the VET system at a number of levels. For example: /Training Package Development: Industry Skills Councils establish industry led committees to ensure that Training Packages are developed and amended to reflect the needs of industry.
Ambition in Action www.sit.nsw.edu.au NVR Standards for the operation of RTOs /The new Standards for nationally regulated RTOs (the SNRs), formerly the AQTF standards, require RTOs to engage effectively with industry in the preparation of their delivery and assessment strategies, and the development and validation of assessment tools. Standard 15.3 states: /Staff, facilities, equipment and training and assessment materials.... are consistent with the requirements of the Training Package or VET accredited course and... training and assessment strategies and are developed through effective consultation with industry. Standard 16.4 states: /Employers and other parties who contribute to each learner’s training and assessment are engaged in the development, delivery and monitoring of training and assessment.
Ambition in Action www.sit.nsw.edu.au So who is “Industry”? /The former National Quality Council (now known as National Skills Standards Council - NSSC) describes industry as “including, but not limited to, business owners and employers and their representative bodies, employees and unions, and their representative bodies”. (NQC Training Package Glossary, 2009). In our context, industry also includes community organisations, government departments and other RTOs.
Ambition in Action www.sit.nsw.edu.au What are the advantages of validation? /Consistency of interpretation of competency standards and benchmarks /Continuous improvement /Builds confidence in VET products /Customer service /Compliance
Ambition in Action www.sit.nsw.edu.au What do we validate? /assessment methods /assessment tools /assessment procedures /assessment decisions High Risk Units first
Ambition in Action www.sit.nsw.edu.au Underpinning Knowledge When Validating /Principles of Assessment /Rules of Evidence /Employability skills /AQF standards /Training Package/units competence
Ambition in Action www.sit.nsw.edu.au Principles of Assessment /Validity - The process assesses what it claims to assess. Validity requires that: /assessment against the units of competency /assessment of knowledge and skills must be integrated with their practical application /judgement of competence must be based on sufficient evidence. /Flexibility – To be flexible, assessment should: reflect the candidate’s needs, provide for recognition of competencies no matter how, where or when they have been acquired, draw on a range of methods appropriate to the context, competency and the candidate. /Fairness - Fairness in assessment requires: consideration of the individual candidate’s needs and characteristics and any reasonable adjustments that should be applied. /Reliability - Refers to the consistency of the interpretation of evidence and the consistency of assessment outcomes. Reliability requires a standard benchmark of assessor competence and relevant vocational competence.
Ambition in Action www.sit.nsw.edu.au Rules of Evidence /Validity - The process assesses what it claims to assess. Validity requires that: /assessment against the units of competency /assessment of knowledge and skills must be integrated with their practical application /judgement of competence must be based on sufficient evidence. /Currency - Relates to the age of collected evidence. Competency requires demonstration of current performance. /Sufficiency - Relates to the amount of evidence collected. /Authenticity - Relates to ensuring the evidence is from or of the candidate and not another person.
Ambition in Action www.sit.nsw.edu.au Employability skills /Employability skills are the non-technical skills and knowledge necessary for effective participation in the workforce, and can include skills such as communication, self-management, problem solving and teamwork. They are also sometimes referred to as generic skills, capabilities, enabling skills or key competencies.
Ambition in Action www.sit.nsw.edu.au AQF Qualifications /The Australian Qualifications Framework (AQF) is the national policy for regulated qualifications in Australian education and training. It incorporates the qualifications from each education and training sector into a single comprehensive national qualifications framework.
Ambition in Action www.sit.nsw.edu.au AQF Qualifications
Ambition in Action www.sit.nsw.edu.au The future of assessment /all new training packages assessments work-based or simulated environment /COAG directions external validation
Ambition in Action www.sit.nsw.edu.au Clustering /Clustering units that combine to form an integrated work activity/task. /Clustering generic elements across units /Clustering common themes across units
Ambition in Action www.sit.nsw.edu.au Assessment Design Visualise the competency being performed in the workplace Locate BenchmarksIdentify Evidence RequirementsIdentify Assessment Methods Design Tools (instrument +instruction) and map
Ambition in Action www.sit.nsw.edu.au Tips - designing quality assessments Visualise a competent person in the workplace
Ambition in Action www.sit.nsw.edu.au Dimensions of competency /Task skills Performing the task/job to the required standard. /Task management skills (variables) Able to do more than one thing at a time and managing the tasks correctly. /Contingency management skills Responding appropriately to irregularities and breakdowns in routine within a job or workplace. /Job/role environment skills (outcomes) Able to deal with the responsibilities and expectations of the work environment.
Ambition in Action www.sit.nsw.edu.au Examples Assessment methods /Observation /Questioning /Portfolios /Work samples /Third party feedback /Simulations, role plays /Presentations /Self assessment
Ambition in Action www.sit.nsw.edu.au Mapping What is mapping? Why do you need to do this? The Steps in Mapping. 1.Analyse the Unit of competency 2.Select the assessment tool 3.Map the components of the UOC to the assessment tools chosen
Ambition in Action www.sit.nsw.edu.au Analyse the Unit of Competence The following diagram shows the range of information taken into account at this stage of the development process:
Ambition in Action www.sit.nsw.edu.au How to identify assessment methods or evidence gathering techniques
Ambition in Action www.sit.nsw.edu.au Sample Mapping Document
Ambition in Action www.sit.nsw.edu.au Assessment Task Parts of Unit Assessed E=element, PC = perf criteria Part of Unit Assessed S =Skills and K=knowledge Evidence Guide CA= Critical Aspects Simulated case study and written or verbal questions. E1 PC 1.2, 1.3, 1.4, 1.5 E2 PC 2.1, 2.2, 2.3 E3 PC 3.1, 3.2, 3.3, 3.4, 3.5, 3.6 E4 PC 4.3, 4.4 E5 PC 5.1, 5.2 K1-K4 K5-K6 CA2 CA1 Workplace observation or simulated observation E2 PC 2.1, 2.2, 2.3 E3 PC 3.1, 3.2, 3.3, 3.4 E4 PC 4.1, 4.2, 4.4, 4.5 K1-K6 S1-s4 CA1 CA2 Questioning E1 PC 1.2, 1.3, 1.4, 1.5K1-K4 K5-K6 CA2
Ambition in Action www.sit.nsw.edu.au Streamlining /Streamlining / Refining Assessment Validation /less forms
Ambition in Action www.sit.nsw.edu.au Under Review
Ambition in Action www.sit.nsw.edu.au For more information follow the links /Competency Based Assessment PublicationsCompetency Based Assessment Publications /Training Packages @ WorkTraining Packages @ Work /Training.gov.auTraining.gov.au