Quality Indicators for Learning and Teaching 2014 Survey Managers’ Information Forum 18 July 2014 Dr Andrew Taylor Branch Manager, Higher Education Data and Analysis

Budget Measure Budget measure Upholding Quality – Quality Indicators for Learning and Teaching (QILT) QILT – suite of surveys over the student life cycle, from commencement to employment New, purpose built website to publish results and inform student choice

Indicator development for performance funding AQHE Reference Group, chaired by Professor Ian O’Connor Consultation process: – Discussion papers – Submissions – Roundtables Development of Indicators

Survey burden on students and institutions Centralised administration Employer surveys ad hoc and not well done Need for robust measures of teaching performance Key issues

AQHE Report Key Reference Group recommendations: Centralised administration of the suite of performance measurement instruments under a single third-party provider The suite of instruments to initially comprise the University Experience Survey (UES) and a new Graduate Outcomes Survey (GOS) Scoping study on a new instrument to assess employer needs and satisfaction with graduates Not recommended: Composite Teaching Quality Indicator Collegiate Learning Assessment

MyUniversity website Live in April 2012 Clear, meaningful and transparent information Confidence intervals Subject by institution level data

Amendments made to the Higher Education Support Act 2003 (HESA) in November 2013 Personal information can now be provided to a range of bodies, including – TEQSA – higher education providers and some peak bodies – state governments Safeguards remain – Limited uses – No on-sharing – Provider consent Universities Australia data sharing agreement allows access for member institutions Data sharing

QILT Components Three surveys:  University Experience Survey (UES)  Graduate Outcomes Survey (GOS)  Employer Satisfaction Survey (ESS) Plus new purpose built QILT website to publish results from the surveys QILT contractor selected through tender to administer all elements

Timing University Experience Survey August Graduate Outcomes Survey n/aOctober roundApril and October rounds Employer Satisfaction Survey OctoberJune WebsiteAugust – initial release February and August updates

University Experience Survey 2014 UES administered by GCA/SRC consortium 2015 UES and 2016 UES comprise part of QILT Methodology in line with previous administrations

National results, key items,

National results, scales,

International benchmarking

Graduate Outcomes Survey GOS will provide timely labour market information Review existing AGS instrument to develop new GOS instrument and methodology Reporting methods need review Survey administered in two rounds October and April (following year), though alternative proposals considered 2015 AGS administered by GCA GOS to replace AGS from the second half of GOS and 2017 GOS comprise part of QILT

Employer Satisfaction Survey ESS measures employer satisfaction with generic skills, technical skills and work readiness of graduates Recent graduates provide contact details of direct supervisor Pilot ESS in across 4 universities and 5 fields of education Pilot ESS report available from: Further development of ESS in October-December 2014  instrument and survey methodology to be reviewed and tested on larger scale  universities participate opt-in basis 2014 ESS trial, 2015 ESS and 2016 ESS comprise part of QILT

ESS – key results Overall 92 per cent of employers stated they would be confident in recommending someone with the same qualification from the same university for a similar position to the graduate’s role Overall satisfaction by key factors (skill clusters)  Foundation skills — 95 per cent  Adaptive skills — 89 per cent  Teamwork and interpersonal skills — 96 per cent  Disciplinary skills — 93 per cent  Employability skills — 93 per cent  Enterprise skills — 79 per cent

QILT Website QILT website aims to inform student choice through provision of survey results by institution and course Two phases: 1.Initial release  August 2014 release  Existing data from the AGS and UES, to be provided by department  Limited scope – undergraduates at Table A and B universities 2.Further development  Developed progressively to August 2015  Expanded scope – all institutions and levels of study  Enhanced functionality, e.g. search and comparison functions enhanced  Development and release of mobile app Website updated every six months from August 2014 with latest available QILT survey data

Governance QILT Working Group will provide advice to department on implementation of QILT QILT Working Group largely comprises former members of AQHE Reference Group QILT Working Group will include representation from non-university higher education institutions (NUHEIs)

General methodology Independent administration Random stratified sample Stratification variables are specified for each instrument HEIMS data to be made available for constructing sample frames Additional information for the construction of sample frames, along with student and graduate contact details, to be collected from institutions

Quality assurance Biases and unrepresentativeness minimised Quality assurance is based on achieving required confidence interval, on specific items, for a set proportion of strata UES (entire educational experience) and GOS (further full- time study) – Interval of + or – 7.5 pp, at 90% confidence level – 75% of strata at the institution by subject area level, excluding strata with a population of fewer than six ESS (overall supervisor rating) – Interval of + or – 7.5 pp, at 90% confidence level – 8 out of 10 broad fields of education – 25 responses for each field at each institution Confidence intervals calculated as per 2013 UES

NUHEIs Currently around 85 Non University Higher Education Institutions (NUHEIs) accredited under HESA Limited participation in AGS to date, not in scope for UES NUHEIs in scope for QILT:  2015 ESS  2015 UES (trial of NUHEIs in 2014 UES)  2016 GOS (from October 2015 round) Survey results for NUHEIs to be included on QILT website subject to data being robust

Tender Tender opened 13 June Tender closed 11 July Professor Shirley Alexander, UTS, to assist Department in assessing tenders Expect to announce outcome by the end of July Market testing through tender for triennium

23 CONFIDENCE INTERVAL PAPER - ANU