Presentation on theme: "Technopolis 1 Forskningsrådsevalueringen - metoder, utfordringer, resultat Erik Arnold Technopolis EVA September 5 2002."— Presentation transcript:
Technopolis 1 Forskningsrådsevalueringen - metoder, utfordringer, resultat Erik Arnold Technopolis EVA September 5 2002
Technopolis 2 The evaluation team Panel members … Pieter Drenth, Amsterdam/ALLEA Anders Flodström, KTH Jecqueline Godet, CNRS Philippe Laredo, CSI Ecole des Mines Ben Martin, SPRU Frieder Meyer-Krahmer, FhG-ISI Erkki Ormala, Nokia Arie Rip, Twente (chair) The research team... Erik Arnold, James Stroyan, Paul Simmonds, Sarah Teather, Ben Thuriaux, Alina Östling, Technopolis (leader) Johan Hauknes, Marianne Broch, Per Koch, Heidi Wiig, STEP Stefan Kuhlmann, Sybille Hinze, FhG-ISI Egil Kallerud, Randi Søgnen, Liv Langfeldt, Magnus Gulbrandsen, NIFU Barend van der Meulen, Frank van der Most, Twente
Technopolis 3 Contents … Why have research councils? Was the research council reform a good idea? Is RCN doing a good job? What are the research policy challenges? What next?
Technopolis 4 RCN’s mandate (§2) - and that’s just the written bit RCN shall develop its research policy and administer grants for research, based on guidelines drawn up by the government and parliament. RCN shall further provide advice as a basis for the development of the government’s general research policy. RCN shall Initiate research, which promotes the development of Norwegian industry and society Contribute to the development of knowledge of humanity’s common problems, especially problems related to the environment and development Support researcher-initiated research, among other things by acting as a supplementary source of funding for the institutions, which perform basic research Work to achieve a good balance between long-term basic research and use-oriented research Work to ensure that user concerns are taken into account in applied research Promote international research co-operation Promote quality, efficiency and relevance in the research system Initiate and use the results of evaluations of research and research institutions Take strategic responsibility for the research institute sector in Norway Promote the exploitation of the results of research by the state, industry and the general public
Technopolis 5 Our mandate (leaving out the details …) The evaluation of the Research Council of Norway is to cover the period from its establishment on 1 January 1993 to 31 December 2000, both dates inclusive. It shall give an overall evaluation of the Research Council in the light of the principal objectives laid down in report to the Storting St. meld. nr. 43 (1991-92), recommendation to the Storting Innst. S. nr. 231 (1991-92), and the statutes (articles of association) of the Research Council. The evaluation shall analyse the connection between the Research Council’s framework conditions, organisation and instruments, and the objectives laid down for its activities. Assessments shall be empirically grounded, among other things in the experiences of central groups of actors in the Ministries, research institutions, the commercial sector and the Research Council itself. In the light of this analysis, the evaluation shall consider what the Research Council’s framework conditions should be, how the Research Council ought to be organised, and what steps the Council itself should take, so that one may be as well equipped as possible to meet the future challenges confronting Norwegian research.
Technopolis 6 Defining the field of observation Collecting data in the field Janus is the god of evaluation. He is a judge, not a researcher Structuring Observing Analysing Judging Choosing evaluation questions, effects to be evaluated, defining criteria Defining indicators or suitable alternatives Analysis of data Analysis of causality and attribution of effects Judging effects individually Formulating a synthetic judgement
Technopolis 7 Given that evaluation tools are poor, we look for convergence Railway Research Group, KTH
Technopolis 8 Detailed mapping of work steps onto terms of reference
Technopolis 9 Our evaluation leant heavily on domain theory. We made this explicit, focusing on the idea of innovation systems.. Framework Conditions Financial environment; taxation and incentives; propensity to innovation and entrepreneurship; mobility Education and Research System Professional education, training Higher education and research Public sector research Industrial System Large companies Mature SMEs New, technology- based firms Intermediaries Institutes; Brokers Consumers (final demand) Producers (intermediate demand) Demand Banking, venture capital IPR & information Innovation & business support Standards and norms Infrastructure Political System Government Governance RTD policies
Technopolis 11 In the judging phase, we also tried to take account of current changes - some of which go beyond theory Drivers Growth in Mode 2 production Changes in the nature of technologies ‘Hyphen-technologies’ Dematerialisation Appropriation via intellectual property Changed industrial organisation of knowledge production Globalisation De-integration Acceleration: concurrent science Changed social contract Increased relevance The state as a growing user Trends Global knowledge markets Emergence of superuniversities Reorganisation of the Research Institute sector End of the ‘3-hump model’ Industrial ‘observatories’ and new PPPs at the university/industry interface IPR as a constraint as well as an income generator
Technopolis 12 Results. The research council reform was a good idea in 1993. It still is The old structure was not only messy, but was unable to cope with the newer policy challenges, such as the Main Target Areas The issues in the Norwegian research system are about adequate research funding and the role of research in economic and social development development Changes in the way knowledge is produced and used require both holism and diversity in research policy Some other countries are moving towards more integrated solutions - Denmark, England, Finland …
Technopolis 13 History matters. The merger was a triumph of political turf over policy vision. The forces that destroyed the action areas won again NAVF RSFRMFRHFRNF NMF RCN KS BF NT IE MU MH NTNFNLVF NFFR NORAS
Technopolis 14 RCN’s three ‘steering levels’ became a battlefield in the early days and continue to provide means to fragment the Council Executive Board Division Boards (6) Programme & Discipline Boards Director General Division Directors (6) Strategy, Admin Divisional Staff Government Ministries
Technopolis 15 The organisation and its context lead to rigidities Vertical links are weak No-one is present at more than one level Links to overall strategy are hard to forge Higher levels tend to bless what goes on below, rather than engaging strategically Structures are locked in, owing to their tight and detailed links with external funders Little or no slack is available for change-agent or arena roles There is quite a lot of administrative diversity, but some of it is in the wrong places
Technopolis 16 RCN does a solid job where it can, but has more responsibility than authority Making progress towards its six (summarised) goals Administration is cost-effective Starvation rations do not encourage dynamism The sector principle reinforces the barriers between the Divisions RCN needs to change its character from planning-system to research and innovation policy arena cp FUGE, Demo 2000, OG21...
Technopolis 17 Goal fulfilment is almost surprisingly good. Testimony to the power of muddling through? RCN shall produce useful national and sectoral research policy advice to the government, based on an holistic national perspective RCN shall fund research to meet social and industrial needs, taking account of users’ needs and promoting the uptake of results RCN shall fund the high-quality basic and applied research needed in the national system of knowledge production, seeking to integrate the two as far as is appropriate while securing the place of basic research RCN is tasked with strategic responsibility for the research institute sector in Norway RCN is tasked with promoting the interaction of Norwegian knowledge production with the international knowledge production system RCN shall use appropriate and efficient processes (including evaluation) and organisational structures in performing its tasks Improving Yes, but … Q/A and procedures in place for both. Integration? Well done. Consequences? Exploiting EU. Integration into world research Use of diversity should be more strategic
Technopolis 18 RCN’s ability to act as an holistic council reflects its funding
Technopolis 19 Norwegian research policy faces major challenges The ‘value-creation gap’ (cp Norman, Reve) Raising national R&D/GDP, especially through the creation of new industry Move from the so-called ‘three-hump model’ with a strict division of labour among universities, institutes and ‘users’ to one where different modes of knowledge production interpenetrate … … and to take the consequences by extending and modernising the roles of the institutions Find an appropriate balance between the number of researchers and the amount of ‘bottom-up’ funding available in the system. This is really a question about funding, not about basic research
Technopolis 21 Governments have cut SND and walked away from innovation policy. IE’s budget fell 16% 93-00, and has been cut by another 150 MNOK
Technopolis 22 Norway needs a stronger Innovation Agency function, cp TEKES. Getting the needed political understanding is hard, in a rich country
Technopolis 23 What are the functions that would need to be encompassed in an holistic research council? Strategic Programmes, Institutes and Infrastructures Strategic Innovation Agency Response mode/ ‘free’ research Absorptive Capacity Thematic R&D areas Strategy and Foresight SND
Technopolis 24 The struggle to integrate disparate cultures will continue to be hard Structural issues Cultures Contested nature of the social contract Sectoral principle and organisation Few convincing examples Integrated approaches Change agencies Capture of principal-agent systems by client communities Research Community Education Ministry Research Council Industry Ministry Innovation Agency Industrial Community
Technopolis 25 After the evaluation, political path-dependency re-asserts itself Those who already ‘knew’ the answer before we started work carried on telling everyone what they already ‘knew’ The ‘basic research’ faction still wants NAVF back. (Perhaps the collective amnesia of this faction is liked with its average age…!) Industry’s representatives want NTNF back My good friend Hans Skoie’e campaign against a single council continues. It’s about time he was awarded a medal - at least for persistence! The education minister seems to have blocked the idea of raising research and innovation to the national (prime ministerial) level Politically understandable … … but this has important negative symbolic value, cp Finland, Sweden, Ireland The idea of continuing with a single council is accepted … but what that actually means remains fluid