Monitoring and Evaluation Harvey Hurree David /londonmetuni londonmet.ac.uk

Agenda 1 Overview – Monitoring & Evaluation - Harvey Hurree 2 Performance Enhancement Meetings - David Tye 3 Course Evaluation - Harvey Hurree londonmet.ac.uk

The Quality Code (Chapter B8) sets out the following Expectation about programme monitoring and review, which higher education institutions are required to meet. Higher education providers, in discharging their responsibilities for setting and maintaining academic standards and assuring and enhancing the quality of learning opportunities, operate effective, regular and systematic processes for monitoring and for review of programmes. The following Expectation (Chapter A: Expectation A3.3) is also relevant to this topic: Degree-awarding bodies ensure that processes for the monitoring and review of programmes are implemented which explicitly address whether the UK threshold academic standards are achieved and whether the academic standards required by the individual degree-awarding body are being maintained. londonmet.ac.uk

The Indicators Indicator 1 Higher education providers maintain strategic oversight of the processes for, and outcomes of, programme monitoring and programme review, to ensure processes are applied systematically and operated consistently. Indicator 2 Higher education providers take deliberate steps to use the outcomes of programme monitoring and review processes for enhancement purposes. Indicator 3 Higher education providers operate a process to protect the academic interests of students when a programme is closed. Indicator 4 Higher education providers define processes, roles and responsibilities for programme monitoring and programme review and communicate them to those involved. londonmet.ac.uk

Indicator 5 Higher education providers evaluate their processes for programme monitoring and review and take action to improve them where necessary. Indicator 6 Higher education providers make use of reference points and draw on expertise from those outside the programme in their processes for programme monitoring and review. Indicator 7 Higher education providers involve students in programme monitoring and review processes. Indicator 8 Higher education providers enable staff and other participants to contribute effectively to programme monitoring and programme review by putting in place appropriate arrangements for their support and development. londonmet.ac.uk

Course design in line with agreed benchmarks, frameworks and regulations Course Validation Student Feedback Course Performance Data / External Examiner Monitoring Module and Course Modification Course Review

Monitoring & Evaluation Working Group (MEWG) Set up April 2013 Chaired by Dominic Palmer Brown All 4 faculties represented Office of Institutional Effectiveness Academic Registry QEU londonmet.ac.uk

Monitoring & Evaluation Working Group (MEWG) Tasks To consider the effectiveness and value of current monitoring and evaluation procedures To consider the UK Quality Code Expectations in respect of monitoring and evaluation To consider sector good practice in respect of monitoring and evaluation Objectives To recommend a cohesive, timely and effective set of quality monitoring and evaluation practices to the University’s Undergraduate and Postgraduate Committee To identify actions (quick wins) that might be achieved quickly and easily To identify resources that might yield early results or that could deliver impact londonmet.ac.uk

Monitoring & Evaluation Working Group (MEWG) What did it look at? londonmet.ac.uk Procedures Module Logs Module evaluation questionnaires External Examiner reports PEMS Course Committees Course Logs PEMs FAMs Student Engagement Employer feedback Data RPA data DLHE data NSS data

Recommendations New Module and Course Logs with prepopulated data (course logs only). Move from consideration of module performance at an early July PEM to consideration of course performance and modules of concern, at a late July PEM. PEM Chair Report (PCR) of Course and Module logs when final data are available in November. Development of a new Faculty Annual Monitoring Statement (FAMS) that will provide the University with: a) an evaluation of Faculty performance at undergraduate and postgraduate level, b) confirmation of the implementation of the University’s quality assurance procedures, c) information on validation/review/modification/deletion activity and d) an update on enhancement within the Faculty (QEU0022-FAMS). New Course and Module Modification Deadlines – 2 nd February 2015 for modifications that impact on programme planning and two weeks post PEM for modifications that arise from PEM discussions Online course evaluation – pilot in May. londonmet.ac.uk

Monitoring and evaluation process Performance Enhancement Meeting (PEM) in July, then PEM Chair Report in November Faculty UGPG Committee University UGPG Committee Module monitoring data - available from OIE Annual Monitoring Livelink site Supplementary module performance data - Published by Academic Registry Marksheets - Component marks produced by Module Leader External Examiner comments - Module Confirmation Proforma (MCP) - Published by Academic Registry SSB Secretary Results from module questionnaires - from Faculty records External Examiner Annual Report OR External Examiner written commentary in advance of PEM (where the External Examiner cannot attend the PEM) -available from the EE and QEU Livelink sites -Published by Academic Registry/Assessment on EE Livelink site Course (& SSB) monitoring data - Available from OIE Annual Monitoring Livelink site Course Committee minutes - from Faculty records Results from course questionnaires - from Faculty records NSS (previous year data only for July) Supplied by OIE via Faculty senior managers DLHE data - Supplied by OIE in course monitoring data PTES data - Available from Research Office Module Log Completed by Module Leader Course Log Data provided by OIE, Commentary sections completed by Course Leader PEM Chair Report Data provided by OIE, Commentary sections completed by PEM Chair Faculty Annual Monitoring Statement (FAMS) Completed by Associate Dean Scrutiny Forum Boxes on the left denote information type and the documentation into which it feeds. Information is either supplied by the Office of Institutional Effectiveness (OIE) or must be actively sourced by the person responsible for the production of the core document. All Course Logs, PEM Chairs’ Reports, FAMS, and PEM minutes need to be provided to QEU who will hold them on Livelink. Core documentation Information and source

Annual Monitoring for Collaborative Partners Module Level Student Feedback Module Level External Examiner Comment Course Level External Examiner Comment (aggregated from modules) Course Level Performance Data (available from the Office of Institutional Effectiveness (OIE) the Academic Registry, and/or the collaborative partner) Overall summary of good practice/issues to be addressed and annual update on partner activity University evaluation of collaborative partner performance QEU Collaborative Performance Report Module Logs are shared with the Academic Liaison Tutor Faculty UGPG Committee Collaborative Course Log produced by collaborative partner Faculty update on partner activity University UGPG Committee Collaborative Module Log produced by collaborative partner Collaborative Annual Monitoring Statement (CAMS) produced by collaborative partner Partnership Annual Monitoring Statement (PAMS) produced by Faculty Collaborative Partner PEMs Course Level Student Feedback (aggregated from modules) Other relevant course information e.g. employment statistics (available locally) Module Level Performance Data The boxes on the left denote the action and the documentation into which it feeds. Advice is supplied (variously) by the Quality Enhancement Unit (QEU), the Registry and the Office of Institutional Effectiveness (OIE). The boxes on the right denote the forum at which the annual monitoring documents are considered Scrutiny Documentation Information

londonmet.ac.uk Core Documentation for Module, Course, Subject and Faculty Monitoring & Evaluation PEM Chair Report (PCR) Aggregated and individual data provided by OIE, Commentary sections completed by PEM Chair Course Log Data provided by OIE, Commentary sections completed by Course Leader Module Log Completed by Module Leader Faculty Annual Monitoring Statement (FAMS) Completed by Associate Dean

londonmet.ac.uk Review Forums for Module, Course, Subject and Faculty Monitoring & Evaluation Performance Enhancement Meeting (PEM) Faculty UGPG Committee University UGPG Committee PEM Chair’s Report (PCR) July PEM November PEM (Chair only) November Faculty UGPG December/ January University UGPG December/ January University UGPG

londonmet.ac.uk Monitoring & evaluation activity July 2015 ActionResponsibilityDateOriginal date Academic Registry produce & distribute module data to PEM Chairs and Secretaries, for distribution to Module Leaders. Based on module results from standard publication date 25 June with subsequent updates to 10 July. Module Log pro-formas included. Academic RegistryFri 10 July Original date proposed was 1 July; extended to allow for late marks OIE issue data for Course Logs, together with data consolidated at PEM level, plus DLHE and NSS data as available, to PEM Chairs / Secretaries for onward distribution to Course Leaders. OIE also make data available to Academic Registry and QEU. Data based on awards and progression data from standard publication date of 7 July with subsequent updates to 10 July. Course Log proformas included. OIETue 14 July Original date proposed for Course data was 10 July; extended to allow for late awards and progression arising from late marks. Original date for PEM data was 17 July Module Leaders complete Module Logs & distribute to Course Leaders, PEM Chair/Secretary, and Academic Registry Module LeadersWed 15 July Original date proposed was Wed 8 July Academic Registry distribute to External Examiners the relevant course data (not Course Logs) and completed Module Logs as available RegistryFri 17 July Original date proposed was 15 July Course Leaders complete Course Logs & distribute to PEM Chair/Secretary and QEU Course LeadersFri 17 July Original date proposed was 17 July External Examiners unable to attend PEMs feedback any comments to PEM Chair & Secretary, cc Academic Registry External Examiners with Academic Registry support for queries Thur 23 July Original date proposed was 24 July PEMs meet PEM Chairs, Secretaries, and other members Mon 20 to Thur 23 July Original dates proposed July PEM minutes produced, approved and distributed to all PEM Members. It is up to PEM Chairs whether they produce also a draft PEM Chairs Report at this stage; the formal requirement for a PCR is in November. PEM Chairs & PEM SecretariesFriday 31 July Original date proposed was 30 July External Examiner Annual Reports (UG modules/courses only) External Examiners to submit to Academic Registry, who publish via Livelink and circulate to Chairs and designated faculty contacts Fri 4 Sept Original date proposed was 4 Sept

londonmet.ac.uk Course Evaluation New process Consistent practiceCentral accessibilityOnline evaluation Reaching all students of London Met Collaborative students Asking the question Why core questions What questions National Student Survey (NSS )? What about local level questions

londonmet.ac.uk Course Evaluation Pilot 2015 Programme EvaSys Options Hard CopyOnline What courses 25 of the largest from across the Faculties Data Where does it goWho looks at it