1. Finn Hansson MPP/CBS fh,lpf@cbs.dk The Peer Review and Dialogue Evaluating the research organization in dialogue. Finn Hansson Copenhagen Business School Paper presented at the AEA 2009 Conference in Orlando, Florida

2. How to do research evaluation in organizations? Learning and/or control Evaluation influence on research organisations Evaluation between control and learning Research leadership and research evaluation Research evaluation in changing organisational surroundings, Deficits in quantitative methodologies used to evaluate research Corrected by the use of an extended version of the peer review system Need of new methodologies: Characterization (Laredo) Finn Hansson MPP/CBS fh,lpf@cbs.dk

3. The peer review system has to change due to new challenges to research: New public management policy framework for research: Peer reviews used in research assessment systems (i.e. RAE) More and more and cross- and transdisciplinary research ( Organizational changes in science and research discussed asTriple Helix, Mode 1 and 2 research ), More and more fluid project based or intermediate research organizations, Demands for dissemination/communication of research.

4. Changing the peer review system From the distant anonymous system to the new modified: A modified peer review integrate verbal communication, self evaluation reports, research documents, site visits and interviews The reviewers are no longer anonymous persons but known faces like in the participatory or dialogue model for research The evaluation takes shape of a formative evaluation The formative peer review is used more and more by science policy organizations and agencies but our knowledge of how it influences the classic peer review is limited. 2 cases, one university, one national science policy. Finn Hansson MPP/CBS fh,lpf@cbs.dk

5. Case 1: Evaluation of research based activities in four cross disciplinary research priority areas at the University of Copenhagen BioCampus research priority area comprises core biotechnological research as well as ethical, cultural and social implications of the developments in biotechnology and biomedicine. BioCampus has comprised biotechnological research in the natural and health sciences. It has also examined the ethical, cultural and social implications of developments in biotechnology and biomedicine, including gene therapy, stem cell research and lifestyle-related illnesses. Body and Mind research priority area aims to examine the correlation between the function of the brain and conscious mental life. The research has included such disparate disciplines as molecular biology, psychiatry, psychology and philosophy, and one of its goals was to boost research into serious neurological and psychiatric disorders such as Alzheimer’s disease and schizophrenia. Religion in the 21st century research priority area covers studies of how religions influence society and individuals in present time. The research priority area has focused on four main areas, each with its own core areas, religion, society and law, religion between conflict and reconciliation, religion in transformation and religious knowledge and knowledge about religion. Europe in transition research priority area focuses on the political, economic, legal and cultural changes and transitions, which we see in Europe. The Research priority area is interdisciplinary. It will address issues that transverse different areas such as political sciences, history, social sciences, law, arts and cultural studies, geography and Romance and Eastern European languages. Each program was set up with a steering committee and a budget of 3 million Euros to be spread over the four years. Finn Hansson MPP/CBS fh,lpf@cbs.dk

6. Case 1: The evaluation approach University demanded: focus on organizational experience and learning and the experience with cross disciplinary and cross faculty collaboration, The method: External evaluation, based on four self-evaluation reports, written by the steering group chairperson with documentation prepared by the university administration Evaluator and the expert group evaluated the self-evaluation reports, wrote questions to the follow up interviews and analysed the transcribed interviews.

7. Case 1: Some results Two natural science dominated areas had difficulties in integrating researchers from social science and humanities, All areas met limits to their leadership from the university system (faculty, departments) when organizing cross disciplinary projects Funding was seen as too low for medicine and natural science All areas met resistance from faculty and departments when trying to establish cross disciplinary teaching programs Finn Hansson MPP/CBS fh,lpf@cbs.dk

8. Case 2: Mid-term Evaluation of Large-scale Programmes Norway Functional Genomics in Norway - FUGE Research in functional genomics at the international level to enhance expertise in basic biological, medical and marine research, as well as to promote innovation and industrial development in Norway.FUGE AQUACULTURE - An Industry in Growth - HAVBRUK Research to promote aquaculture as a sustainable, market-oriented, profitable growth industry. HAVBRUK Nanotechnology and New Materials - NANOMAT Research to promote Norway as a world leader in research in selected areas relating to new materials and nanotechnology.NANOMAT Climate Change and its Impacts in Norway - NORKLIMA Research to enhance knowledge about the climate and promote a better interface between different types of research to give society a stronger platform for dealing with changes in the global climate. NORKLIMA Optimal Management of Petroleum Resources - PETROMAKS Research to facilitate the utilisation of petroleum reserves to the benefit of society.PETROMAKS Clean Energy for the Future - RENERGI Research to generate clean energy systems for the future.RENERGI Core Competence and Growth in ICT - VERDIKT Research to place Norway at the forefront of ICT development and the application of ICT- based knowledge to innovation and interaction.VERDIKT Funding: 200 million § a year for 10 years, Full time adminstrative manager, part time research leadership. Finn Hansson MPP/CBS fh,lpf@cbs.dk

9. Case II: The evaluation approach The Research Council demanded: Focus on organizational experience and learning and the experience with cross disciplinary and cross faculty collaboration, The method: Scandinavian external evaluation group, evaluation based on self- evaluation reports, written by the steering group chairperson and the support persons from The Research Council, - and the expert group evaluated the self-evaluation reports, interviewed a large number of key people, wrote questions to the follow up interviews, analysed the interviews.

10. Case 2: A few results This type of large, long term strategic research programs needs a much better coordination by the Research Council, as well as a more secure long term finance planning. The individual programs need more administrative support from the council and especially more and stronger research leadership. The programs has been visible international in the EU but not much in Norway – need for better planning and coordination organization in the council. Finn Hansson MPP/CBS fh,lpf@cbs.dk

11. Reflections on the dialogue peer review model as an evaluation method When evaluating complex research organizations, the challenge to the evaluation lies in focussing the evaluation on some of the very important aspects for the best possible organization of successful research programs, without relying only on evaluation of indicators of productivity like end results, articles or other products, but on organizing, leadership, networking, communication.

12. Finn Hansson MPP/CBS fh,lpf@cbs.dk The peer review dialogue model Strength: flexible handling of the difficult question of how evaluations discern and measure quality Can fulfil some of the often-heard wish from researchers to have a dialogue with the peers doing a review or an evaluation of research activities, Dialogue during interviews and presentations Weakness: Can the evaluator act as a peer? Will the organization refrain from demanding a traditional performance based evaluation?