1. György BAZSA President, HAC ENQA Debriefing/brainstorming meeting for the training of experts for agency reviews Barcelona, 17 March 2009 A possible option: grading scheme for judging compliance with criteria

2. Basic quantitative requirements - of a fine wine: ethanol 9 – 13 % sugar0 – 100 g/l pH3 – 4 - of a nice hostess: bust size80 – 99 cm hip size60 – 65 cm waistline75 – 90 cm - of an excellent chemist: publicationsminimum 50 citationsminimum 200 impact factorminimum 40 But still there is a need for the evaluation of its/her/his quality by experts! Now, what about a good quality assurance agency?

3. i. Membership REGULATIONS OF ENQA (as from 26 September 2008) TITLE III. INVOLVEMENT IN ENQA a) Full Membership Full Membership of ENQA is open to quality assurance agencies that meet the relevant (eight) criteria de- scribed in Annex I Membership Provisions document. b) Candidate Membership Candidate Membership is available to quality assurance agencies that meet the criteria described in Annex I Membership Provisions document. (i.e. substantial compliance with a majority of the criteria)

4. ANNEX 1 – Membership Provisions CHAPTER I. CRITERIA FOR FULL MEMBERSHIP a) Full Membership of ENQA is open to quality assu- rance agencies in the field of higher education from EHEA member states that have been operating and conducting actual evaluation activities for at least two years. Before being accepted as a Full Member, an applicant agency must satisfy the Board that it meets the eight criteria, listed below. The applicant agency will thereby also meet the European Standards and Guidelines for Quality Assurance in the European Higher Education Area (ESG). - (8 or 15 criteria?)

5. b) An applicant agency may apply for Candidate Membership rather than Full Membership in the first instance. The Board will grant such member- ship if it believes -that the applicant demonstrates, through its application, substantial compliance with a majority of the criteria and further believes -that the applicant will be able to demonstrate, through an acceptable external review submitted within no more than two years Candidate Membership, full compliance with the criteria for Full Membership.

6. QAA documents QAA expert report Candidate 2y m. Full 5y member ENQA non

7. ii. Evaluation of the compliances by the expert panel Dictionary and stages of compliance Compliant: in accordance with a specified body or rules (usually used in combination): Fully (compliant): entirely or wholly Substantially (compliant): of ample or considerable amount, quantity, size, etc. Partially or partly (compliant): in part, to some extent or degree, not wholly; being such in part only, not total or general, incomplete Non-compliant: failure or refusal to comply, as with a law, regulation, or term of a contract.

8. ENQA National reviews (quide): summarised conclusion stating whether the agency is - fully or substantially compliant; - partially compliant; or - non-compliant.

9. ESU The study visit team believes that objectivity is best served by using a three level marking system. - Compliance, where QA implements the ESG in an effective manner; - Partial compliance, where QA practices are based on the correct interpretation of the ESG, positive aspects have been identified but the manner of implementation is not effective enough. - No compliance, where ARACIS practices are based on misinterpretations of the ESG …;

10. EUA - fully compliant - substantially compliant - partly compliant - (non-compliant) Summary (in the case of ARACIS): substantially compliant

11. March 2009!: Guidelines for external reviews of ENQA member and applicant agencies - fully compliant: the Agency adheres to the letter (literal interpretation) and spirit (intent) of the criterion/ESG standard; - substantially compliant: if the criterion/ESG standard is not fulfilled to the letter, the Agency might still be considered substantially compliant if the principle/spirit of the stipulated criterion/ESG standard is followed in practice; - partially compliant: the Agency does not fully or substantially comply with the criterion/ESG standard; - non-compliant: the Agency does not comply with the criterion/ESG standard.

12. The way in which the membership criteria/ESG are to be interpreted by the review panel and the Board A possible option: grading scheme for judging compliance with different criteria EvaluationGrading Fully compliant + + + Substantially compliant + + / - Partially compliant + / - - Non-compliant / - - - The grading is proposed by the review panel

13. The Review panel will need to exercise its judgement, in the light of all the evidence before it, and the intentions of the ENQA membership criteria / ESG, as to whether substantial compliance is being achieved, taking account of the circumstances in which the Agency is required to operate (see section 3.6.2). There is little point in adopting a hard line position in respect of compliance with the ESG if, by doing so, trustworthy and credible agencies are prevented from gaining Full membership of ENQA through no fault of their own.

14. There may, as a result, be occasions where a strict reading of the ENQA membership criteria / ESG would indicate substantial rather than full compliance with one or more standards, but the panel may come to the conclusion that this should not jeopardise an overall judgement of compliance. This may occur when national legislation requires a different approach, or where an alternative way of achieving an objective is used, or because of other justifiable reasons.

15. Where an agency is found to be either partially compliant (+ - -) or non-compliant (- - -) with a criterion, the reason for this should be explained. Full or substantial compliance may be impossible for some agencies, owing to restrictions placed on them by the very nature of their work and/or legislation in place in their country(ies) of operation. When considering such cases, the ENQA Board will take mitigating circumstances such as these into account.

16. A possible option for 15 criteria: grading scale for granting ENQA membership Decided by the Board Proposed by the review panel PointsLimit Overall grading? Membership granted 45 – 41 1 partly c. 0 non-c. Fully ? Full member 40-39-38 Board decides 37-36-35 2 partly c. 1 non-c. Substan- tially ? Candidate member 34-33-32 Board decides 31 31 ---NonNon-member

17. Point Fully 3+ Subst 2+ Partly 1 Non 0 Sum Full member 41 12210 41 11400 41 13101 !!! 41 1302 !!!0 41 Full or candidate member 40 39 38 111 3 038 1032038 9510 8700 Candidate member 37 36 35 1111236 1022136 9411 8520 78 !!!00 37

18. Point Fully 3+ Subst 2+ Partly 1 Non 0 Sum Candidate or non-member 34 33 32 10 023 !!! 32 9132 8322 8241 7512 7350 66203232 Non-member 31 831 !3 ! 31 823 !2 ! 31 815 !1 ! 31

19. iii. Further proposals - renumbering of ENQA criteria - public list of data - publicity of ENQA evaluation - QA vs. QP

20. 11 Executive summary 2+- Glossary 32 Introduction (giving contextual information: reason(s) for the commis- sioning of the review; the place of the agency in the quality assurance structure of its jurisdiction; the main functions; the engagement of the agency with the ESG); 43 Findings (each membership criterion should be discussed separately) 4.13.1 4.23.2 4.33.3 4.43.4 4.53.5 4.63.6 4.73.7 4.83.8 5+4 Any sections relating to additional Terms of Reference of the review 65 Conclusion and recommendations (?) 76 Annexes (Key pieces of evidence – i.e. extracts from legislation, policies and procedures etc. – may be added as appendices)

21. Further proposals (out of scope of the title): ENQA should prepare a list of compulsory data and information about QAA to be posted permanently on member agencys homepage (partly in English). Application, self-evaluation report, date of site visit and composition of visiting panel should be posted on the QAAs homepage like on ENQA homepage. QA vs. QP: quality assurance is crucial, but qua- lity of the product (learning outcomes etc.) is the main aim. Should we move to QP direction?

22. Thank you for your kind attention!

23. ENQA Board decisions: (see Scrutiny form) 3. Full members mandatory cyclical review in each 5 year: - full membership re-confirmed for another 5 years or - candidate membership granted in order to conform with full membership criteria in 2 years or - neither full, nor candidate membership is granted?