1. The External Review Report as Ground for ENQA Board Decision Tibor Szanto Vice-President, ENQA Barcelona, 17 March 2009

2. Some statistics* Externally reviewed agencies: 23 Membership: 40 full + 8 candidate Review reports on ENQA website: 21 (ACQUIN, NQAI missing) Year of report 2005: 3 2006: 4 2007: 7 2008: 7 Lenght of reports: pp.10 (HSV) – 64 (NVAO) * Source: ENQA website (www.enqa.eu, 14-15 March 2009)www.enqa.eu 17.03.2009, Barcelona Tibor Szanto 2

3. Type and co-ordination of review Type A (ENQA memb. crit. / ESG) Type B (also other aspects) statutory functions ECA Code of Good Practice special context (e.g. bi-national character, NVAO) follow-up on previous review Co-ordination national ENQA (+ other non-national) 17.03.2009, Barcelona Tibor Szanto 3

4. National (internat.) context in reports Various approaches: brief history of agency only (in many cases) description of national context description of international context (too) To be considered: background information needed relevance of national (internat) context for judging compliance 17.03.2009, Barcelona Tibor Szanto 4

5. Coverage – ESG in reports Part I – mentioned in some cases, in relation to agency activities/standards Part II discussed mostly under Standard 3.1 (Part III, Use of external QA processes) discussed separately in some cases in some other cases not discussed at all (first reports) Part III – discussed in all reports 17.03.2009, Barcelona Tibor Szanto 5

6. ENQA additional criteria 1 Not really discussed in the reports. ENQA Criterion 3 – Resources (ESG 3.4) Agencies should have adequate and proportional resources, both human and financial, to enable them to organise and run their external quality assurance process(es) in an effective and efficient manner, with appropriate provision for the development of their processes, procedures and staff. 17.03.2009, Barcelona Tibor Szanto 6

7. ENQA additional criteria 2 ENQA criterion 8 - Miscellaneous i. The agency pays careful attention to its declared principles at all times, and ensures both that its requirements and processes are managed professionally and that its judgements and decisions are reached in a consistent manner, even if the judgments are formed by different groups ii. If the agency makes formal quality assurance decisions, or conclusions which have formal consequences, it should have an appeals procedure. The nature and form of the appeals procedure should be determined in the light of the constitution of the agency. iii. The agency is willing to contribute actively to the aims of ENQA 17.03.2009, Barcelona Tibor Szanto 7

8. Presentation of findings in reports Diverse approaches evidence and statements evidence, some reflections, conclusions evidence, analysis, conclusions evidence, analysis, conclusions, recommendations Length, level of elaboration varies. Something is missing in many cases… 17.03.2009, Barcelona Tibor Szanto 8

9. Presentation of findings – the ideal case Best practice (what the Board makes happy:) ESG Parts II and III fully covered Each standard discussed separately Structure (under each standard): a) evidence (description, facts and data) b) analysis (consideration) c) conclusion (judgement on compliance) d) recommendation (if any) Dont forget about ENQA additional criteria! 17.03.2009, Barcelona Tibor Szanto 9

10. Judgements in the reports Cases: separate judgement on each criterion overall judgement too Some examples as to wording: the agency complies with… the criterion has been met the criterion has been complied with sufficiently complies substantially complies criteria are being largely met full compliance has not been achieved not yet fully compliant 17.03.2009, Barcelona Tibor Szanto 10

11. Judgements expected The Agency is with the criterion/ESG standard fully compliant substantially compliant partially compliant non-compliant Overall judgement not required (but can be made if the panel wishes to do so) 17.03.2009, Barcelona Tibor Szanto 11

12. Recommendations expected Best practice: under the relevant criterion PLUS / OR summarised either in the Executive Summary or at the end of the report in a concluding chapter Should stand out, well structured! SHOULD BE! Reviews: accountability + improvement 17.03.2009, Barcelona Tibor Szanto 12

13. The Boards headache 1 The European Standards and Guidelines were not written as a set of binding regulations or requirements and are not a set of compliance criteria to be checked off. (NOKUT Report, 2008. 11.p) The ENQA membership criteria ARE compliance criteria to be checked off … or not? The compliance paradox? 17.03.2009, Barcelona Tibor Szanto 13

14. The Boards headache 2 Are there various levels of compliance? Linguistically-logically: definitely not. (Slightly pregnant) Socio-logically: yes The substantial (social pregnancy) paradox? 17.03.2009, Barcelona Tibor Szanto 14

15. The solution? Wisdom, advice, and good reports needed! 17.03.2009, Barcelona Tibor Szanto 15

16. Thank you for your attention!