1. The Go8 Quality Verification System Information and advice for reviewers

2. Introduction This induction presentation includes information about: the current focus on academic standards in Australia; the QVS as a distinctive approach of the Go8; the structure and objectives of the QVS; your role as a QVS reviewer and the activities outside the scope of your role; and your role in evaluating the QVS.

3. The context for the QVS Intensification of interest in setting and monitoring academic standards Regulatory settings introduced from 2011 include a Higher Education Standards Framework These standards are used to audit higher education providers under the Tertiary Education Quality and Standards Agency (TEQSA) regulatory framework. The Higher Education Standards Panel is currently revising the Standards Framework.

4. The context for the QVS The Australian Qualifications Framework (AQF) has been revised to describe each course level qualification in terms of the knowledge and skills that a graduate will possess. Relevant international developments include: -the Tuning Project in Europe to align degree structures (Tuning, n.d.); andTuning Project -the OECD’s Assessing Higher Education Learning Outcomes (AHELO) feasibility study in which Australia is participating (OECD, 2011).AHELO

5. The academic achievement of final year students A key reference point for standards There are clear international trends towards accepting: students’ academic attainment as the primary reference point for monitoring academic standards; and attainment in final year of study as a suitable point for benchmarking. ‘If academic standards are primarily defined by academic attainment, the quality and robustness of process for assessing, grading and reporting individual and group attainment are paramount’. (Richard James, 2010, presentation to the AUQA Auditors Meeting)

6. Aims of the Quality Verification System (QVS) Within the focus on academic standards: to demonstrate the appropriateness of the standards of learning outcomes and grades awarded in Go8 universities; to maintain and improve the academic standards of Go8 universities; to enable comparisons of learning outcomes in similar subjects across Go8 universities; and to promote discussion on good practice in teaching and learning in the Go8 universities.

7. Distinctive features of the QVS The QVS: focuses upon assessment in a sample of final year subjects in undergraduate programs (restricted to assessment tasks amenable to review – e.g. verbal presentations are excluded); concentrates on benchmarking for comparing grades awarded in similar subjects across Go8 universities; is conducted by senior academics in the relevant discipline who will have an understanding of academic standards in leading universities around the world; and is sufficiently flexible to complement other quality assurance mechanisms within Go8 universities to minimise duplication of efforts.

8. Core elements of the QVS The QVS is a process of external, discipline-led, academic peer review of final year undergraduate student outcomes. The QVS will review a minimum of 25 percent of final year assessment requirements (normally at least two core subjects) in undergraduate programs across Go8 universities. The Go8 QVS Reviewers will: o review the appropriateness and comparative quality of the specified learning outcomes, assessment tasks, assessment criteria and assessment processes set in samples of final year subjects in selected fields of education; and o report on the appropriateness of the grades awarded to stratified random samples of student work in these subjects, drawing on their academic judgment and the set of documents provided.

9. The QVS approach Quality verification rather than grade moderation The QVS involves quality verification by external reviewers. It is not moderation of grades by external examiners. Your review will retrospectively verify the appropriateness of grades awarded to a sample of student assessment after their results have been published. Your review will not influence the grades awarded to students enrolled in the subjects you review. In your role as an external QVS reviewer, you are not expected to: design assessment tasks; re-mark student work; or standardise student grades.

10. The QVS since 2011 Pilot (2011 and 2012) The pilot reviewed: Physics History Psychology Accounting Chemistry Economics Philosophy From 2013 From 2013 onwards, the QVS reviews a small number of new fields each year, plus any fields still outstanding from previous years. 2013: Maths, English and Chinese (Mandarin) 2014: Political Science, Sociology and Geology

11. Process steps to ensure robustness and credibility All QVS reviewers will declare any potential conflict of interest (e.g. being involved in collaborative teaching, research or consultancy work with colleagues teaching in the subjects being reviewed) at the outset of the review and in the written report which will be published on the website of the university being reviewed. Reciprocal reviewing arrangements of individual departments/faculties between universities are avoided as far as possible.

12. Sensitive nature of materials for review As a QVS reviewer, you sign a confidentiality agreement with the university being reviewed before commencing the review; you must not disclose any information provided to you for the purpose of the review without written permission of the QVS coordinator in the university being reviewed; you must keep in mind that your report will be discussed in departmental/faculty committees that are likely to have student representatives; you accept that the university being reviewed will own copyright of any material produced in relation to the review; and you will not retain any rights (including copyright and moral rights) in connection with the materials produced for the review.

13. Your role as a QVS reviewer You will be provided with the following materials for the subjects you will review: the subject outline, subject guide and learning objectives; information on how the learning outcomes of this subject relate to the degree program level outcomes; copies of the actual assessment tasks which students have completed (such as exam scripts, specific essay assignments, work- placement requirements, performance-based assessment arrangements, etc); the grading criteria used for each assessment task; and grade nomenclature and categories used within the university (e.g. below pass, pass, credit, distinction and high distinction).

14. Verifying grades awarded For each assessment task, you will also review a randomly selected and de- identified sample of 5% of student assessment at each of the following grade levels or their equivalent – below pass, pass, credit, distinction and high distinction. For subjects with small enrolments, a minimum of 5 items of student work will be reviewed. For subjects with large enrolments, a maximum of 25 items of student work will be reviewed. Where appropriate, stratified samples will include items of student work from all campuses including off-shore campuses.

15. Carrying out a review Within 2 weeks of receiving the materials, you will: review all of the materials you have received; liaise with the person who sent you the relevant material to obtain further information for your review, if necessary; send your report (Attachment A of QVS Guidelines) to the QVS coordinator in the university being reviewed (listed in the next slide); and send a copy of your report to the Go8 Secretariat.

16. QVS coordinators Monash University Patrick Orchard Project Advisor, Office of the Pro Vice- Chancellor (Learning and Teaching) The University of Western Australia Winthrop Professor Grady Venville Dean, Coursework Studies The University of Melbourne Dr Marija Maher, Director, Academic Strategy The University of Queensland Pauline Pavier Executive Officer, Office of the Deputy Vice-Chancellor (Academic) The Australian National University Suzie Alcorn Manager Academic Standards and Quality Office The University of Sydney Professor Marie Carroll Pro Vice-Chancellor (Academic Affairs) UNSW Australia Sonia Powell A/g General Manager, Office of the Pro Vice-Chancellor (Students) The University of Adelaide Jessica Raeburn Learning and Quality Support Officer

17. Key questions for review Are the learning outcomes appropriate for a final year undergraduate subject? Are the learning outcomes comparable to those of final year subjects in similar universities? Are assessment processes and the determination of grades rigorous, equitable for diverse students and fairly conducted based on the material you have been provided? As a senior academic, you have the academic experience and judgment to review the material provided to you. For the purpose of this review, the next three slides suggest issues that you might consider for inclusion in your report.

18. Review of learning objectives To what extent is the information provided about learning objectives clear and sufficient? To what extent are the learning objectives “precise, challenging and complete” (Laurillard 2002: 183)? How do the learning objectives specified for the subject compare with those of final year subjects in similar universities?

19. Review of assessment tasks To what extent are the assessment tasks suitable for the specified learning objectives? Is there sufficient variety and complexity in assessment tasks? To what extent are the assessment tasks timed appropriately? Is the language used in assessment tasks unambiguous, appropriate and inclusive of all students? Are the marking criteria sufficiently clear? How do the assessment tasks and marking criteria compare with those of final year subjects in similar universities?

20. Suggestions for summary comments In your experience, how do the specified learning outcomes and student achievements compare with those of final year subjects in similar universities? Are there key issues which should be brought to the attention of supervising committees in the faculty/department, division or wider university? Are there examples of good practice that might be noted and disseminated more widely as appropriate? (Adapted from the QAA code of practice on external examination)

21. Summary judgement on each unit  The learning outcomes, assessment tasks and assessment processes set for the subject I have reviewed were appropriate. Any recommendations made are for the purposes of enhancement to the subject and its assessment.  The learning outcomes, assessment tasks and assessment processes set for the subject I have reviewed were appropriate. HOWEVER, there are some risks to the future quality assurance of the subject and its assessment, as outlined in my recommendations.  There are immediate concerns or risks relating to the learning outcomes, assessment tasks and/or assessment processes set for the subject I have reviewed. These require immediate action on behalf of the University to prevent reoccurrence in the next review. (Adapted from Report Coversheet for External Examiners, the University of Cambridge)

22. Report cover sheet The Report Coversheet records the extent to which you agree with the grades awarded to the sample of student work you have reviewed. Number of assessments Below pass PassCreditDistinctionHigh Distinction Reviewed in total Agree with grade awarded Believe grade awarded to be unduly high Believe grade awarded to be unduly low

23. Discussion and publication of review summaries Your review report will be received by the QVS coordinator of the university being reviewed and discussed with the relevant Faculty/Department. The QVS coordinator will provide you with the University’s overall response to your report. Review recommendations and schedule of discipline reviews will be published on the websites of the participating universities.

24. Your evaluation of the QVS You will be asked to provide feedback on your experience of being a QVS reviewer: Did you have access to sufficient information for the purpose of this review? What other information would have helped you undertake the review more effectively and efficiently? What is your overall assessment of the effectiveness and usefulness of the process? o What are the best aspects of this process? o What could be improved? Feedback from reviewers (and reviewees) will be to continuously improve the QVS

25. References James, R. (2010). The academic perspective on academic standards: The challenges in making the implicit explicit. Presentation made to AUQA Auditors Meeting at Gold Cost, June 30. Retrieved 24 April, 2011, from The University of Melbourne website at http://www.cshe.unimelb.edu.au/people/james.html http://www.cshe.unimelb.edu.au/people/james.html Laurillard, D. (2002). Rethinking university teaching: A conversational framework for the effective use of learning technologies (2nd ed.). London: Routledge. University of Cambridge (2009). Report coversheet for external examiners. Retrieved April 24, 2011, from the University of Cambridge website at http://www.admin.cam.ac.uk/offices/education/examiners/eecoversheet.pdf http://www.admin.cam.ac.uk/offices/education/examiners/eecoversheet.pdf Tuning (n.d.). Tuning Educational Structures in Europe. Retrieved April 24, 2011, from http://tuning.unideusto.org/tuningeu/ http://tuning.unideusto.org/tuningeu/ OECD (2011). Testing student and university performance globally: OECD’s AHELO. Retrieved April 24, 2011, from the OECD website at www.oecd.org/edu/ahelowww.oecd.org/edu/ahelo QAA (2004). Code of practice for the assurance of academic quality and standards in higher education: External Examining. Retrieved 10 April, 2011, from the Quality Assurance Agency for Higher Education website at www.qaa.ac.ukwww.qaa.ac.uk

26. Contact us If you have any queries, or need more information, please contact The Group of Eight secretariat at +61 (02) 6239 5488.