ENQA seminar:First external evaluations of quality assurance agencies – lessons learned Panel discussion: Practicalities and challenges of self and external reviews: perspective of the expert team Michael Gerhard Kraft (secretary of the review of AQU Catalunya and academic staff member FH Council (FHR))

2 Review Process (flow chart) ToR SE-report written evidence + ENQA briefing package Telephone conference RT draws up lines of inquiry and timetable Site visit: further evidence, verifying written evidence Report: template / draft / amendments by the agency Final report and state- ment of the Agency sub- mitted to the ENQA Board

3 Lessons learned Terms of reference should state clearly the assignment of the review team (RT) (sole vs. multiple purpose review) Focus should be on the ESG, when additional areas for evaluation are being asked for, one should bear in mind that they might bind/require additional resources and a certain composition of the RT (e.g. teaching staff and research assessment of the Spanish agencies) Conclusions of the RT should distinguish between the different areas of evaluation but should take the consistency of the whole report into consideration Heavy workload particularly for the Chair and the Secretary

4 correspondence and some of my handwriting Lessons learned

5 Quality and comprehensiveness of the self evaluation report is important for the preparation of the RT (principles of self-criticism, objectiveness and openness / not window-dressing!) The agency presented a comprehensive evidence list on the internet, from where the documents were well accessible but it is important to prioritise: maybe the agency should state, which documents it considers particularly important as evidence for its findings in the self-evaluation report. Good overview on the higher education system in which the agency under review operates is important in order to appreciate national peculiarities

6 Lessons learned Take your time! Timetable of the site-visit should provide for sufficient time to reflect and discuss findings of the interviews and to work on the draft report Its all about communication useful to have a preparatory telephone conference, if it is not possible to meet in advance ENQA telephone conference was helpful in addition to written statements of the ENQA Board (clarifying issues) Drafting the report Methodological structure of description, analysis and conclusion is very useful

7 Lessons learned Findings and experiences of RT should be made available to other expert panels and be taken into account for improvement of the ENQA briefing package ENQA has already reworked the briefing package (especially on how to interpret the ESG) One should keep in mind that although the whole undertaking is about ENQA membership, it should be aimed at improvement in character and not control guiding principle of trust Important to distinguish between aspects which are under the control of the agency and those which are due to national legislation should be clearly distinguished in the review report

8 ESG, their nature and interpretation Scope of the ESG for the review The membership provisions of ENQA correspond to part 3 of the ESG incorporating also part 2, in Criterion 1 of the membership provisions. (ENQA Briefing Package) How to interpret the ESG? Principle vs. rules based approach Respecting the diversity of national contexts Generic principle vs. specific requirement One has to be cautious in adapting them: -ESG 3.7: student involvement (as appropriate) -ESG 3.6: Independence: has implications for the areas of: financing, composition of the agency, conflict of interest, QA outcomes, reports, etc. -Publication policies Clear interpretation of the ESG by the ENQA Board should be communicated to the RT however, a certain scope for interpretation will always remain

9 ESG, their interpretation and improvement Question of compliance? Provide the RT with a notion of what -compliance -substantial compliance -partial compliance means. Any areas of total non-compliance were unacceptable, and that some element of compliance would be necessary against all criteria in order to provide an overall judgment based on substantial compliance. Each case would be examined on its merits and at a local basis. (ENQA Board) Reflexive handling of the ESG Self critical, evaluative approach also necessary for the development and improvement of the ESG as well as their interpretation

Thank you very much for your attention!