1. Monitoring R&D as a Methodological Tool for Impact Studies Stig Slipersæter Norwegian Institute for Studies in Research and Higher Education Centre for Innovation Research

2. 2 Background Lack of international standards for output and impact indicators Small steps to integrate outcome data relevant for impact studies into Norway's national R&D statistics

3. 3 Monitoring R&D 1. An example on monitoring 2. An example on use of data 3. Lessons learned

4. 4 Example: Research institutes 65 independent research institutes Annual budget about 1 billion US $ Approximately 7000 employees Monitoring scheme run by NIFU on behalf of the Research Council of Norway Purpose: Accountability and policy analysis Integrated with regular R&D statistics

5. 5 Input to research activities Financial resources split on: – Government funds – Normal research contracts: Programme allowances from research council Public sources, split on various ministries and their agencies, local and regional authorities etc. Domestic industry, split into three categories Contracts from abroad, split into seven categories Detailed data on human capital

6. 6 Output from research activites Indicators for output: – Enlarged set of variables for printed dissemination Articles in scientific journals Different kinds of reports Books and chapters in books Articles in magazines of professional associations Popularization – Patents applied for and granted – Spin-offs – Partcipatons at conferences, seminars etc

7. 7 Indicators for diffusion In- and outgoing mobility Shared positions (with universities, industry and others) Cooperative research projects Supervising of students Visiting students Visiting scholars (in- and outgoing)

8. 8 Advantages of the scheme Details on income An extended set of variables on outcome Variables on different diffusion processes

9. 9 Utilising the institute monitor Users evaluation of applied social research at 31 research institutes Analysis of financial data showed: – 12 institutes serving regional and local authorities – 19 institutes serving national authorities and international organisations How is differences in customer base affecting the outputs of the institutes?

10. 10 Differences in written output

11. 11 What purposes were R&D used for? User survey shows two main categories of use: 1. Instrumental: New facts and policy support 2. Conceptual: New ideas, concepts, methods and ways of understanding Output from national and academically oriented institutes considered being of largest instrumental value for users

12. 12 Lessons learned for indicators 1 There are many ways of transferring scientific results: Broad set of transfer mechanisms should be included Use of output is far from obvious: Impact can come in unexpected ways Ask the user of scientific results what is important before making the indicator.

13. 13 Lessons learned for indicators 2 When setting up a system of indicators, consider: A broad set of types of output Include at least these groups in the process: 1. Producers of output (i.e. scientists) and the ones reporting (i.e. administrators) 2. Users of output 3. Users of indicators The importance of time series