Measures for a national outcomes based audit model Stakeholder consultations

2 Background COAG: ‘acceleration of national outcomes-based auditing for RTOs’ VQA: work to support achievement of higher standards of teaching, learning and assessment The 4 tasks in developing an outcomes-based audit model: 1. Revision of the AQTF Standards – outcomes based 2. Measures of RTO performance against outcomes Focus of current project and forum 3. Outcomes-based auditing and registering body standards 4. Excellence certification

3 Progress with tasks 1, 3 and 4 Revision of AQTF Standards Focus on achievement of quality skill outcomes by RTOs Extensive consultation process Further refinement of Standards & RTO Guide Outcomes-based auditing To be finalised when outcome measures and data collection methods agreed Revised standards for registering bodies Excellence certification Voluntary process (building on registration requirements) with two certification points: ‘quality committed’ and ‘outstanding'

4 Methodology for outcomes measures project Research Consultations – national and state-based industry, providers and governments Guide and discussion paper Analysis and reporting – set of preferred measures and their management and use Overseen by Steering Committee and feeds into COAG processes

5 Measurement framework Based on review of current measures and contemporary educational indicators Core considerations Continuous improvement & focus on teaching, learning and assessment Sensitive to RTO differences & transparent and auditable & utilising consistent and accessible data Avoids red-tape & relevant Stable & able to detect change over time Net cast wide Not aligned to specific individual essential elements

6 Measures relate to standards The RTO provides quality training and assessment across all of its operations Learner Engagement Measure, Teaching quality measure, Assessment review measure The RTO adheres to principles of access and equity and maximises outcomes for its clients Learner satisfaction measure, Learning outcomes measure, Learner destinations measure, Completion rates measure Management systems are responsive to the needs of clients, staff and stakeholders, and the environment in which the RTO operates Staff engagement measure, Industry and employer satisfaction measure

7 Learner Engagement Measure Provides an index of learner engagement in key areas of RTO delivery & interactions with learners e.g. Was RPL and credit for prior learning provided? Is the environment supportive of learning? Were learners challenged by the program design and learning resources?

8 Teaching quality measure Provides an index of learners’ and graduates’ perceptions of the quality of teaching Examples of statements drawn from similar Higher Education questionnaire include: the teaching staff: provided helpful feedback on my work Motivated me to do my best work Worked hard to make their subjects interesting Made a real effort to understand difficulties I might be having with my work

9 Learner satisfaction measure Provides an index of learners’ and graduates’ satisfaction with the VET experience A complementary measure directed to their satisfaction as consumers, including aspects of the training process and organisation e.g. Experience and outcomes compared to investment in time, fees and income foregone Relevance of training to their industry and work Satisfaction with learning environment, facilities, complaints process

10 Learning outcomes measure Provides a learners’ self-assessment of their learning outcomes e.g. Comparison of actual performance against their own expectations Learner belief that assessment practices were fair, reliable and consistent with students self- assessment of learning outcomes

11 Learner destinations measure Provides a measure of student employment and further learning outcomes e.g. Initial employment, changes in work roles and effect on remuneration Further study

12 Staff engagement measure Provides a measure of RTO staff engagement with the education and training process. The range of possible measures include: Learner feedback on quality of teaching Information on staff turnover or tenure Quality of induction, training and rewards processes Staff perceptions of job enjoyment and inter- staff relations

13 Industry and employer satisfaction measure Provides a measure of employer and industry regulator satisfaction with the quality of training provided, effectiveness of training processes and levels of competence acquired

14 Completion rates measure Currently reported for each jurisdiction as part of the VET Provider collection for publicly- funded RTOs Need to be considered in conjunction with other measures and in context of background of the learner and their communities

15 Assessment review measure Provides a measure of the outcomes of reviews of assessment instruments & processes Assists in providing information on extent to which competency standards evidence requirements are met for compliance purposes

16 Matrix of measures/elements

17 Development of measures With exception of completion rates, current VET national data collections cannot support proposed measures. 4 processes needed Learner and graduate census that build on current student satisfaction and outcomes surveys Staff survey/data from administrative systems Industry regulator and employer census that builds on current employer ‘use and views’ survey Review of assessment instruments/processes

18 Management and use of measures Administered by a central agency or a partially decentralised (to RTOs) approach? Measures will contribute to assessment of compliance with Standards & for RTOs to monitor and continuously improve performance and for benchmarking Need to develop a process to manage dissemination of data derived from application of measures

19 Consultation questions Do the potential measures provide an appropriate basis for assessing the performance of RTOs against new Standards? Additional measures needed? Are the individual measures appropriate and what specific issues do each raise? Are the processes proposed to support the potential measures the most suitable and what specific issues does each raise? What are the possible sensitivities and issues associated with the use and reporting of data associated with the measures?

20 Further comments