1. Taught Postgraduate Program Review
On-line briefing for Program Directors and Staff

2. TOPICS Background Key Features Self-Evaluation
Review by External Panel
Review Report

3. 1: BACKGROUND #1 Audit Report November 2010 Recommendation 5
“The QAC recommends that HKUST introduce a system of periodic reviews of all taught programmes, with defined frequency, terms of reference and requirements for action and follow-up on review recommendations.”

4. 1: BACKGROUND #2
Report of Provost’s Task Force on Educational Quality and Learning Outcomes of TPg Programs: March 2013
Recommendation
The Task Force suggests that there should be a comprehensive
review of the TPg programs periodically at a minimum frequency of
five to six years, with defined terms of reference and
requirements for action and follow-up on review recommendations.
The periodic review should include benchmarking against similar
programs, and some external element, such as inviting faculty from other local
institutions as external reviewers and/or employer bodies to review and provide a report on the programs.

5. 2: KEY FEATURES Senate approval of policy: 3 December 2013
TPg programs to be reviewed every 5 years
Individual programs or cognate groups
4 stages:
Internal review and production of SED
External Panel review of documentation and visit
Production of TPg Program Review Report/Action Plan
Consideration of Report and Action Plan
Pilot in 2014 (4 programs); full implementation from 2015

6. 3: SELF-EVALUATION #1 Self-Evaluation Document (SED) - core areas:
Program Delivery, Management, Quality Assurance
The Curriculum, Benchmarking, Quality Enhancement
Students: Intake; Performance; Support
Stakeholder Feedback
Assessment
Resources and Risk
Other Review Components
Plans for Development
Include (in 7) aspects of review
specific to individual programs

7. 3: SELF-EVALUATION #2
Program Director collates documentation and data, and drafts SED
SED to be an evidential-based self-critical evaluation of the relative success of the program during the last five years in regard to the core areas, incorporating data analysis and references to documentation as appropriate.
SED and documentation + data
discussed and analysed by
program teaching team and administrators

8. 4: REVIEW BY EXTERNAL PANEL #1
Panel Visit by end of August to be determined by Program Director
Ideally days, to allow scrutiny of documentation, meetings (see below) and discussion of review outcomes and recommendations
Visit Program
External Review Panel meetings to include:
The Dean (and, where appropriate, the responsible Department Head)
The Program Director, the program teaching team and the program’s administrators
Support, including technical, staff as appropriate
Current students (both full-time and part-time, where appropriate)
The program’s graduates, if available
Local employers of the program’s graduates
The Dean (and, where appropriate, the responsible Department Head) and Program Director at the conclusion of the visit
External Review Panels also may wish to tour the facilities and review the learning resources available to the program and its students

9. 4: REVIEW BY EXTERNAL PANEL #2
Documentation
SEDs to be discussed with External Review Panels during visits. Additionally, the documentation and data that underpin internal review should be made available to External Review Panels during their visits and is likely to include the following:
the program specification, to include the course structure and program and course objectives, as approved by the University;
course outlines/other documentation provided to students for all courses in the academic year of review;
a summary of approved changes to the program since its inception, with the rationale;
the program management structure, including details of the teaching team and committees concerned with the quality assurance of the program;
details of any benchmarking (local or international) of the program and/or the assessments;
the Program Annual Report on Taught Postgraduate Education for each year of the review period, with a critical commentary on actions taken arising from the reports;
Minutes of the Staff Student Liaison Committee and other committees associated with the program, for the year of review and the previous two years;
critical commentaries on feedback from stakeholders during the review period and on the relative success of any actions taken arising from such feedback (to include results of Student Feedback Questionnaires (SFQ), the TPg Exit Survey and any employer feedback; feedback from Staff Student Liaison Committee; and those made as a consequence of the last External Reviewers’ Report where applicables);
a range of assessments and associated marking schemes, model answers and student work from each year of the review period, across a sample of courses;
Intake data, for each year of the review period, with a critical analysis - Section 3 of the Self-Evaluation Document refers;
marks/grades and program award data for each year of the review period, with a critical analysis - Section 3 of the Self-Evaluation Document refers

10. 4: REVIEW BY EXTERNAL PANEL #3
External Reviewers’ Report covers:
Nature of Engagement
Syllabus and Curriculum
Assessment and Academic Standards
Student and Graduate Support and Feedback
Additional Information
Good Practice
Overall Report
Additional comments
To be submitted to AP (T&L) within two weeks of Panel Visit
(forwarded to Program Director/Dean (and responsible Department Head))

11. 5: REVIEW REPORT #1 1. External Subject Specialist and Review Panel
Indicate the extent of engagement with the External Subject Specialist during the review period;
Summarise the major findings of the External Review Panel, with particular reference to any proposed changes to the program, the appropriateness of student assessments and grade/award distributions, and the relative standards of student achievement in relation to international benchmarking.
2. Overall Evaluation and Forward Look
Taking account of the suggestions of the External Review Panel on the self-evaluation document and following their visit, provide a critical commentary on the relative success of the program, with specific reference to strengths and areas for improvement.
3. Recommendations and Action Plan
List the Recommendations arising from the review.
Append an Action Plan to indicate how the recommendations and related actions would be taken forward, the timescales for implementation, milestones to allow monitoring and those responsible for each action.
[example action plan

12. 5: REVIEW REPORT #2
TPg Program Review Reports to be submitted, within four weeks of receipt of External Reviewers’ Reports, to:
Dean (via responsible Department Head where appropriate);
CTLQ and CPS (for comment and subsequent monitoring, as deemed appropriate);
Provost (for information)
Append External Reviewers’ Report and Action Plan
Deans comment on TPg Program Review Reports in School Annual Reports
Program Annual Reports on Taught Postgraduate Education will not be required in the years in which Taught Postgraduate Program Reviews are undertaken
Program Annual Reports in the years following Taught Postgraduate Program Reviews should be accompanied by progress reports on implementation of action plans, indicating reasons for any delay in implementation, where appropriate

13. 6: NEXT STEPS Program Director to:
Identify Panel members (academic/subject specialist from peer institution outside HK preferably with QA experience for example as a Program Director, Head of Department, Dean, etc.; local alumnus or employer; senior HKUST academic from another School primarily to comment on and provide advice on the program’s management and quality assurance arrangements in relation to the University’s procedures and quality assurance framework, and to guide the external members of the Panel accordingly)
Seek Dean’s approval and submit nominations to Associate Provost (T&L) for CTLQ approval
Write to nominees, to offer appointment and agree date for Panel visit (draft letter to be provided by ARO, to include all relevant details, including requirements, fees etc.)
Undertake internal self review (up to 2 months):
Collate and critically analyse documentation and data sets
Draft SED
Arrange meetings of Program Team, undertake further review/analysis and revisions of SED
Send final SED to Panel for consideration, in advance of Panel Visit
Agree program and documentation for visit with Panel Chair and make arrangements for visit, including organising and briefing those who are to meet the Panel