1. Room for expertise? ENQA-workshop Assessing educational quality: Knowledge production and the role of experts Session 2. The quality Contract: Agency regulations meet expert panels Oslo, February 14-15 2008 Kirsten Hofgaard Lycke University of Oslo, Norway

2. 3 questions 1Is there room for expertise in NOKUTs audits? 2 Do the experts use their expertise? 3 Can we trust the experts?

3. 3 answers 1 Yes! 2 Yes! 3Yes!

4. 1 Is there room for expertise in NOKUTs audits? Yes! Inherent dilemmas in the audit system makes it dependent on professional judgment 2 examples

5. Audit system components Purpose for audits Mandate for expert panel Criteria for evaluating quality systems Institutional documents Site visit by expert panels Report from panel Approval by NOKUT Board

6. Example: The NOKUT criteria 1.How the work on educational quality is made an integral part of the institutions strategic work 2.How the objectives for the institutions work on quality are defined 3.How work on quality is linked to steering and management at all levels of the organisation 4.How work on quality is organised in routines and measures that ensure broad participation, with defined distribution of responsibility and authority for the various stages of the work 5.How the institution retrieves and processes such data and evaluative information as are necessary to order make satisfactory assessments of the quality of all study units, and how this information is accumulated at higher levels, including the top level of the institution 6.How analysis of the information and assessment of goal achievement in the work on quality are systematically provided for 7.How the institution uses the results of work on quality as a basis for decisions and measures with a view to securing and further developing quality of studies 8.How work on quality is made to contribute to resource management and priorities at the institution (human resources, infrastructure, service) 9.How the system ensures a focus on the total learning environment and the active participation by students in work on quality and total learning environment 10.How an annual Quality Report to the board of the institution gives a coherent overall assessment of educational quality at the institution and an overview of plans and measures for continued work on quality

7. A closer look at criteria! 1How the work on educational quality is made an integral part of the institutions strategic work 2How the objectives for the institutions work on quality are defined Standards or focus of attention?

8. The example indicates that There IS room for – and need for – the use of expertise!

9. 2 Do the experts use their expertise? Yes! What kind of expert knowledge is used? When is expertise used? –Basis for decisions –Decision making processes –The report

10. What kind of expert knowledge is used? Experientially based competencies (high score) Attitude/motivation (high score) Disciplinary knowledge (low score)

11. Basis for decision making Order of priority: 1.Interviews at site visit 2.Panel discussions 3.Mandate and criteria 4.…. 5.…. 6.Literature on evaluation 7.Courses/seminars run by NOKUT

12. Decision making processes Important process characteristics: The purpose of the audit as a basis Panel discussions to reach full concensus or agreement Emphasis on statements from the institution representatives Experts well prepared for the audit Confident panel leadership

13. 3 Can the experts be trusted? Yes! Credibility is related to: Legitimacy Reputation Loyalty

14. Legitimacy threatened if the audit process: is superficial and does not run a thorough control of the system is conducted at random or dependent on the panel members does not add new knowledge or insight

15. Reputation Ambivalent statements about reputation! NOKUT appointments have high status among my colleagues My colleagues are indifferent to NOKUT My institution make use of my experience as auditor My institution does not encourage ac staff to take NOKUT appointments

16. Loyalty Communality in perspectives on criteria and process Postive attitude to the expert role (100%) Served on many panels (88% on 3 or more audits)

17. Conclusion NOKUT can depend on the experts NOKUT might learn a lot from the 10 – 20% who have other views Expertise or professional judgment?

18. Contact and references: Kirsten Hofgaard LYCKE Professor Institute for Educational Research Faculty of Education University of Oslo P.O.Box 1092 Blindern 0317 Oslo, Norway Email: k.h.lycke@ped.uio.nok.h.lycke@ped.uio.no Fax: +4722854250 Aas, Gro Hanne et. al: Quality Tellers. Paper to EAIR Forum 2006, Rome Italy Lycke K H: Perspectives on quality assurance in Higher Education in Norway. Quality in Higher Education 2004;10 (3):219-230) Lycke, K H: Sakkyndig for NOKUT: Hva sier de selv? Paper to NOKUT May 2007. Unpublished.