1. Paris, 11th of July 2008 Quality Assurance in Higher Education Recognition procedures of agencies Bruno CURVALE Head of international affairs at AÉRES Agence dévaluation de la recherche et de lenseignement supérieur, France ENQA Vice President European Association for Quality Assurance in Higher Education

2. 2 Bruno Curvale, Paris 11th of July 2008 1. Objectives and organisation of the work 2. Reminder: How to become an ENQA full member? 3. The review of the review 4. Questions and issues for the working groups Outline of the presentation

3. 3 Bruno Curvale, Paris 11th of July 2008 To raise awareness about ENQA membership requirements and mechanisms. To think about what makes the quality of an agency review and at how to assess this quality. To include ENQAs members in the ongoing discussion about ENQA membership. To help the board to complement and improve ENQAs documentation and guidelines. To help the board to achieve coherence in the membership requirements and processes. 1. Objectives of the session

4. 4 Bruno Curvale, Paris 11th of July 2008 A presentation aimed at giving a common background to the participants. Work in parallel workshops. Three main topics: Meta-evaluation: objectives, role, constraints and difficulties. Reviewing and complementing the Guidelines for national review of ENQA member agencies. Reviewing and complementing the Briefing pack for review panel members of ENQA coordinated reviews. A debriefing of the workshop sessions. 1. Organisation of the session

5. 5 Bruno Curvale, Paris 11th of July 2008 Coexistence of two mechanisms: The membership application form and the analysis of it The scrutiny of the agencys review report Since 2005 the ESG are at the core of the membership. In consequence the two mechanisms are largely redundant But not completely … The scrutiny of member reviews is taking a pre-eminent part (Cyclical review every 5 years, …) A better linking, or a simplification, could be a desirable objective. 2. How to become an ENQA full member?

6. 6 Bruno Curvale, Paris 11th of July 2008 2. The analysis of the membership application form The criteria 1. Activities 2. Official status 3. Resources 4. Mission statement 5. Independence 6. External quality assurance criteria and processes 6.1 Processes, criteria and procedures pre-defined and publicly available 6.2 Processes expected to include self-assessment, external assessment, publication of a report including outcome, follow-up procedure 7. Accountability procedures 8. Miscellaneous 8.1 Professionalism and consistency of judgment 8.2 Appeal procedure if the matter makes it necessary 8.3 Willingness of the agency to contribute to the aims of ENQA

7. 7 Bruno Curvale, Paris 11th of July 2008 2. What ENQA does? The Membership Committee: consists of 3 members of the Board reports to the Board The role of the Committee To answer the fundamental question: Does the final report of the agencys review provide sufficient, verified evidence that the agency meets the ENQA membership criteria and thereby the European Standards and Guidelines for Quality Assurance in Higher Education (ESG)? The evaluation is desk-based according to a reading of the application and a consideration of what is said and the evidence provided by the applicant. Today, considering that a review of the agency is part of what is needed to become a full member, this option should only lead to a candidate membership.

8. 8 Bruno Curvale, Paris 11th of July 2008 2. The scrutiny of the member review The criteria 1. Activities 2. Official status 3. Resources 4. Mission statement 5. Independence 6. External quality assurance criteria and processes 7. Accountability procedures Integrity of the review process Can we trust this review process that says this agency complies with the ESG?

9. 9 Bruno Curvale, Paris 11th of July 2008 2. What ENQA does? The Review Committee: It is made of 3 members of the Board. It reports to the Board. The role of the Committee To answer two fundamental questions: Does the information provided in the final report of the agencys review satisfy the board that the review was conducted to the required level of independence, integrity and robustness? Does the final report of the agencys review provide sufficient, verified evidence that the agency meets the ENQA membership criteria and thereby the European Standards and Guidelines for Quality Assurance in Higher Education (ESG)? The evaluation is desk-based according to the reading of the review report and a consideration of what is said and the evidence provided by the panel.

10. 10 Bruno Curvale, Paris 11th of July 2008 3. The review of the review The elements of the judgment: What is the Review Committee looking at? To be kept in mind: A review is made up of a process and of outcomes. The reviews forwarded to ENQA normally conclude that agencies comply with the ESG. Review Process FindingsEvidence Ok Not Ok Ok Not Ok Ok

11. 11 Bruno Curvale, Paris 11th of July 2008 3. Evaluation AND meta-evaluation Agency Procedures, behaviour Q Standards Evaluation against ESG 3 Purposes Evaluation of the evaluation National regulations Specific objectives Evaluation of the evaluation Inclusion in the EQAR ENQA full membership ENQA Register Committee Decided … Coordinated … For ENQA membership For other purposes For application to the Register External review Source : Bruno Curvale Full membership of ENQA normally constitutes satisfactory evidence for substantial compliance with the ESG

12. 12 Bruno Curvale, Paris 11th of July 2008 3. The review of the review It is the evaluation of a process of which the quality deals mainly with: The coordination of the review (organised on national basis/by ENQA/…) ENQA provides Guidelines: Guidelines for national reviews of ENQA member agencies. The independence of the reviewers. The evidence and facts that support the assessment. ENQA provides for each panel it coordinates a Briefing pack for review panel members of ENQA co-ordinated reviews. Question: What do we need to know and need to do in order to be sure about the quality of the review process?

13. 13 Bruno Curvale, Paris 11th of July 2008 4. Questions or issues for the working groups Is the word meta-evaluation satisfactory? What are the difficulties and specific constraints of this exercise? Opportunity of purposes and quality. To what extent do the purposes of the agency have to be taken into account when looking at the quality of its activities? What supplementary documentation would be useful when reviewing the review report? The notion of substantial compliance. How to deal with it? The risk of formalism when dealing with a notion like independence. What could be or should be improved in the Guidelines and Briefing pack? …

14. 14 Bruno Curvale, Paris 11th of July 2008 Thank you for your attention

15. 15 Bruno Curvale, Paris 11th of July 2008 Recommendation of the review committee - 1 The report does not provide sufficient evidence that the review was conducted to the required level of independence, integrity and robustness. No conclusion can be drawn as regard the compliance of the agency with the ESG (ENQA membership criteria). Consequences The Board may request further information The self-evaluation document, internal agency documentation, … The Board may want to discuss with the panel in order to get clarification. The application can be considered again if clarification are pertinent. The agency can appeal against the board decision.

16. 16 Bruno Curvale, Paris 11th of July 2008 Recommendation of the Review Committee - 2 The report provides sufficient evidence to conclude that the agency does not adequately comply with ESG (ENQA membership criteria). The Review Committee contradicts the Review Panel. The committee recommends that the Board does not re-confirm or grant the full membership. Consequences The Board may give the agency a candidate membership meaning that the agency has two years to conform to the criteria. In case too much criteria are problematic the review can be declared unacceptable. In case the agency was already at the end of a candidate membership, this membership is terminated and the agency might apply for Associate status. The agency can appeal against the board decision.

17. 17 Bruno Curvale, Paris 11th of July 2008 Recommendation of the Review Committee - 3 The Review Committee finds the review was conducted to the required level of independence, integrity and robustness. The report does not provide sufficient evidence to conclude that the agency does adequately comply with ESG (ENQA membership criteria). The committee recommends that the Board request further information. Consequences The Board request further information: The self-evaluation document, internal agency documentation, … The Board may want to discuss with the panel in order to get clarification. The application is considered again after clarification.

18. 18 Bruno Curvale, Paris 11th of July 2008 Recommendation of the Review Committee - 4 The Review Committee finds the review was conducted to the required level of independence, integrity and robustness. The report provides sufficient, verified evidence that the agency meets the ESG (ENQA membership criteria). The Review Committee recommends the Board confirms or grants the agency ENQA Full Membership. Consequences Normally, the Board follows the recommendation.