1. 1 Quality Assurance in Italian Universities Gianfranco Rebora, University Cattaneo – LIUC (Italy) Matteo Turri, University of Milan (Italy) EURAM 2010 Back to the Future 19-22 May 2010 – Tor Vergata, Rome

2. 2 This paper aims to review the main events in the development of evaluation activities in Italian universities from 1993 to the present day, to explain the reasons of poor results coming from a relevant collective effort to find some relevant evidence also in the optics of pubblic management and governance

3. 3 QA/ Evaluation Not a very specific concept… … comprehends different approaches, methodologies and practices referred to definition, development and assessment of quality, aiming to improve the ability of institutions, staff and students to meet HE goals (which are debated not fixed beforehand)

4. 4 2007 – 2009 Bologna Process Stocktaking Report 2009 Stage of development of external QA systemStage of development of external QA system3 Level of student participation in QA 2 Level of international participation in QA 2 Report from working groups appointed by the Bologna Follow-up Group to the Ministerial Conference in Leuven/Louvain-la-Neuve 28-29 April 2009

5. 5 Why do results seem poor? (of QA in Italian experience) 1993- 2009: Evaluation and QA activities and practices have been growing at a fast rate But results seem poor: -Accountability ? -Improvement ?

6. 6 IDEA METHODS USE BODIES Inertia, opportunism and unforeseen or undesidered effects Components of an evaluation system Institutional and Organisational Impact: Knowledge- Learning Improvement Accountability Conceptual framework

7. 7 The critical point: How evaluation is used The “actual use” according to Patton (1997, p.20) is the best way to understand the value of evaluation activities and the efforts dedicated to it. We considered: Generating knowledge Improvement Accountability non-use, given that in some cases no use is made of evaluation output

8. 8 The Italian QA story: three stages 1993-1998: start up 1999 – 2006: a surfeit of information and very few results 2007-2009: stalemate

9. 9 Bodies Kind of use 1993-1998 Start up 1999-2006 Information diffused 2006-2009 Stalemate Ev. Units Instituted with the aim of verify “the administration of state funding, productivity in teaching and research” Charged to develop the assessment techniques established by CNSVU and to make an annual report Continuing activities OVSU Collects extensive information - CNVSU Replaced the OSVU: -Collecting data - consultancy and assistency to ministerial requirements Merger with CIVR Anvur: instituted by law in 2006, delay prolonged to 2010 CIVR In charge of research evaluation (exercise 2001-03) Starts new exercise 2004-08 (2010)

10. 10 How QA has been used: an overview Kind of use1993-1998 Start up 1999-2006 Information diffused 2006-2009 Stalemate Knowledge Accountability Improvement No use Growing and diffused knowledge This use of evaluation “causes embarrassment” because it provides justification for university government actions that cannot be taken ” Path of improvement has been interrupted (in research field) No use of evaluation prevails – risk is avoided

11. 11 1999-2006: a surfeit of information but very few results Universities interpreted this greater autonomy by increasing the number of degree courses and adopting opportunist behaviour such as increasing the number of competitions for lecturing positions, at times dealing with career opportunities in an underhand way

12. 12 This university was admitted…

13. 13 2007-2009: stalemate 2007: a law decided the merger of the CNVSU and CIVR and the setting up of the ANVUR 2008: the new government modified the set-up of Anvur 2009: the new rules are on the way of final approval, but the Agency is not yet beginning its activity 2010: a new research evaluation exercise is now starting (2004-2008): CIVR is still in charge of it Universities must now manage relevant cuts in their budgets with the risk that future rules about evaluation will increase the pressure for compliance and conformity

14. 14 Lessons learned 1.To emphasize (to stress) the USE (of evaluation) helps to understand things 2.The field of Higher education is open /very sensitive to accept evaluation 1.International (European) drive 2.Availability of core competences 3.Culture/past experiences of research 3.Governance matters 1.University system governance 2.Institution level 4.Two different vision are in conflict: 1.Administrative/ bureaucratic 2.Professional

15. 15 3. Governance problems have negative impact on use of evaluation –University system level –HE Institutions level

16. 16 At University system level An administrative/ bureaucratic approach prevails on a more substantial and professional one Minimum quality requisites have a very formal intepretation Lack of resources/budget and political events influence the continuity of QA practices: es. CIVR/VTR after 2006

17. 17 At HEIs level A weak type of governance like the one in Italian universities is disinterested in evaluation because it involves making decisions that no one has the strength to make: Several rectors saw evaluation as a stimulus and tool for governance but then had difficulty in finding the necessary consensus for re-election A sudden reduction in the autonomy of evaluation units is however quite common after the election of a new rector (MINELLI E, REBORA G., TURRI M. (2008). How can evaluation_fail? The case of Italian universities. QUALITY IN HIGHER EDUCATION. vol. 14)

18. 18 Mismatch between macro level initiatives in QA and micro level experienced needs We can remove it by: –establishing a threshold in order to prevent the diffusion of weak higher education initiatives –abolishing rules that impose specific organisational patterns and limit strategies of differentiation –providing rules that university leaders (rectors, deans and the various coordinators of teaching activities) can use in order to validate and strengthen their strategic choices and their government structure –promoting autonomy and a more competent professional approach of central bodies and agencies operating in the field of HE

19. 19 How QA has been used: an overview Kind of use1993-1998 Start up 1999-2006 Information diffused 2006-2009 Stalemate Knowledge PredominantExtended: Most frequent useMost frequent use Accountability the attempt to use quantitative data to influence decision - making and the allocation of funds has failed Decisions are postponed Improvement Lack of involvement of universities in self evaluation procedures that could generate learning VTR: stimulated various disciplines to discuss and define their criteria of excellence for research Lack of involvment continues VTR was interrupted No use abundance of bulky and generic documents which are not used, as most teaching staff are unaware that they exist