1. Agency reviews: purpose and stages of the review process Achim Hopbach

2. The purpose of the review Evaluation, whether or not the agency meets the ENQA membership criteria (and thereby the European Standards and Guidelines for Quality Assurance in Higher Education, ESG)? The board will have to make a yes/no decision based on the review. The level of conformity with the ESG that is expected is substantial compliance, not rigid adherence.

3. The purpose of the review Beware: 2 types of reviews: Type A (ENQA membership criteria / ESG) Type B (also other aspects) statutory functions ECA Code of Good Practice special context (e.g. bi-national character, NVAO) follow-up on previous review Co-ordination: national or ENQA (+ other non-national)

4. The review is based on three principles - The review is an evidence-based process carried out through peer review; -The information provided by the Agency is assumed to be factually correct unless other evidence points to the contrary; -The review is a process of verification of the information provided in the self-evaluation and other documentation and the exploration of any matters which are omitted from that documentation; - The level of conformity with the ESG that is expected is substantial compliance, not rigid adherence.

5. The review report analysis and board decision are based on three principles - The information provided in the review report satisfy that the review was conducted to the required level of independence, integrity and robustness. - The review report provide sufficient, verified evidence that the agency meets the ENQA membership criteria and thereby the European Standards and Guidelines for Quality Assurance in Higher Education (ESG). - Theres no discrepancy between the panels conclusions and the evidence brought forward in its report.

6. Stage 1: prior to the site visit/the agencys self evaluation report (SER) SER shall be available to panel at least 8 weeks before the site visit Panel chair and secretary agree visit schedule with agency at least 6 weeks in advance Lines of inquiry are developed by chair in consultation with panel at least 4 weeks before the site visit Briefing paper by review secretary (background, schedule and lines of inquiry for the review) at least 2 weeks before the site visit

7. Stage 2: The site-visit Briefing meeting the day before the site visit Sharing impressions from documentation Highlighting lines off inquiry Agreement on the schedule Agreement on request for additional documentation Site visit Meetings with key personnel (agencys staff and bodies, panel members) and stakeholders etc. Evaluation of additional information onsite Regular panel meetings, incl. final meeting Final debriefing meeting with agency (optional)

8. Stage 3: writing the report 1 st draft by the review secretary 2 nd draft by the review secretary after consultation of panel Submition to agency for correction of factual errors Final report by review secretary and panel chair Report has to reveal both, evidence for and reasoning behind panels conclusions: Description, analysis, conclusion

9. Stage 3: writing the report Remember: The board will have to make a yes/no decision. Therefore, the conclusions have to be clear! Conclusions: The agency isfully compliant substantially compliant partially compliant non-compliant with the membership criterion. Panel may comment on overall compliance with membership criteria if it wishes

10. Stage 3: writing the report Outline report structure 1 Executive summary 2 Introduction 3 Findings 3.1 ESG 3.1, 3.3 / ENQA criterion 1: Activities 3.2 ESG Section 2 / ENQA criterion 1: External quality assurance processes 3.3 ESG 3.2 / ENQA criterion 2:: Official status 3.4 ESG 3.4 / ENQA criterion 3: Resources 3.5 ESG 3.5 / ENQA criterion 4: Mission statement 3.6 ESG 3.6 / ENQA criterion 5: Independence 3.7 ESG 3.7 / ENQA Criterion 6: External quality assurance criteria and processes used by the members 3.8 ESG 3.8 / ENQA Criterion 7: Accountability procedures 4+ Any sections relating to additional Terms of Reference of the review 5 Conclusion 6 Annexes

11. Stage 4: decision making by the board Review committee conducts the initial scrutiny of the review: each member of the Committee will complete a template (ENQA membership criteria and considerations on the independence, integrity and robustness of the review process) the committee members will combine their analyses to form a conclusion and to decide what recommendation to make to the Board

12. Stage 4: decision making by the board the conclusion and recommendation will fall in to one of five categories: 1. (re)-confirm Full Membership 2. request further information in relation to the findings 3. request further information in relation to the evidence of the review process 4. full membership is not (re-)confirmed -agency is given two years to conform to criteria -agency is loosing membership -applicant is granting Candidate membership 5. review is rejected as unacceptable -agency can reapply within limited period -agency is not granting membership

13. Stage 4: decision making by the board The review report analysis and board decision are based on three principles - The information provided in the review report satisfy that the review was conducted to the required level of independence, integrity and robustness. - The review report provide sufficient, verified evidence that the agency meets the ENQA membership criteria and thereby the European Standards and Guidelines for Quality Assurance in Higher Education (ESG). - Theres no discrepancy between the panels conclusions and the evidence brought forward in its report.

14. Stage 4: decision making by the board Interpreting the membership criteria is challenging (Principle based or rules based?) Thus, transparent motivation of conclusions is of utmost importance in order to facilitate consistent board decisions.

15. Stage 4: decision making by the board Interpretation of membership criteria Challenges: Notions are not as clear as expected. e.g. what does independence mean? Criterion 5 Independence (ESG 3.6): Agencies should be independent to the extent both that they have autonomous responsibility for their operations and that the conclusions and recommendations made in their reports cannot be influenced by third parties such as higher education institutions, ministries or other stakeholders. How can independence be guaranteed?

16. Thank your for your attention Achim Hopbach hopbach@akkreditierungsrat.de www.enqa.eu