1. Version 9.2.2003 VTT TECHNOLOGY STUDIES Evaluating the societal impacts of Public research organisations: A (belated) paradigm shift in the making Kaisa Lähteenmäki-Smith and Kirsi Hyytinen VTT Technology Studies

2. Version 9.2.2003 VTT TECHNOLOGY STUDIES Presentation The project on societal impact evaluation: research questions, methods, participating organisations and their differentiated working environments, steering processes and R&D ambitions Evaluation model for societal impacts of publicly funded R&D organisations Some key findings and conclusions and their implications for the evaluation model

3. Version 9.2.2003 VTT TECHNOLOGY STUDIES Description of the project: questions How are publicly funded R&D organisations adjusting their evaluation practices to the emerging ‘management by results’-based accountability demands set by the government? How can different evaluation practices and methods be used for this purpose in assessing the societal impacts of R&D activities? What are the similarities and differences, as well as lessons to be drawn between policy sectors (best practices, challenges, bottlenecks…)? What are the implications of impact assessment on administrative practices, including steering processes, in respect of governmental actors and public research organisations?

4. Version 9.2.2003 VTT TECHNOLOGY STUDIES Description of the project: organisations MULTI-SECTOR TECHNOLOGY R&D INSTITUTE (VTT) is national research institute in technology working in close interface with both the business sector and policy-making. DEFENCE SECTOR R&D INSTITUTE: The Finnish Defence Forces Technical Research Centre RESEARCH INSTITUTE IN THE AGRICULTURE AND FOOD SECTOR, MTT Agrifood Research Finland TWO POLYTECHNICS – one WITH ITS SUBSTANTIVE FOCUS IN THE AREA OF BUSINESS MANAGEMENT, ANOTHER MULTIDISCIPLINARY POLYTECHNIC

5. Version 9.2.2003 VTT TECHNOLOGY STUDIES Motivations of impact assessment Internal and external motives : Change in management and administrative culture Demand for the accountability and result-driven planning Spread of a service-oriented culture – customer orientation Demand for transparency and accountability Compatibility and integration of policy objectives Selection of sectoral objectives and consolidation of sector objectives with inter- connected sectors Selection and compatibility of horizontal goals Demand for organisational development Identification of demands and multiple development challenges Supporting strategic planning – contributes to steering and priority-setting, allows for operational freedom in developing core competences and activities Prioritizing the activities

6. Version 9.2.2003 VTT TECHNOLOGY STUDIES National approach to accountability National and international dimensions of effectiveness Customer-orientation: are the needs of the clients met? Future orientation: are the visions and strategic targets set meeting the needs of the changing nature of society and of the policy environment? Cost effectiveness Socio-economic and societal dimension: Have the main external costs and effects been taken into consideration?

7. Version 9.2.2003 VTT TECHNOLOGY STUDIES THE EMERGENT FRAMEWORK OF IMPACT EVALUATION

8. Version 9.2.2003 VTT TECHNOLOGY STUDIES Key concepts 1/3 - output, outcome, impact, effectiveness The preservation of healthy oat grain compounds in the baking process The healthy food product Lower cholesterol levels Improved national health  longer life-expectancy  economic effects Output Outcome Impact Effectiveness

9. Version 9.2.2003 VTT TECHNOLOGY STUDIES Kohdealueen ulkopuolella T&k- vaikutukset Key concepts 2/3 - impact typology IMPACTS ANTICIPATED In target area Useful Detrimental Useful Detrimental UNANTICIPATED Outside target area Other impacts R&D impacts Other impacts R&D impacts ( Mickwitz 2004)

10. Version 9.2.2003 VTT TECHNOLOGY STUDIES Key concepts 3/3 -impacts through time IMMEDIATE networking improved R&D efficiency patent applications publications prototypes INTERMEDIATE partnership-based cooperation new/ improved products, services and processes company growth improved company competitiveness cost savings higher employment strengthened expertise technology transfer standards, norms support for decision making public discourse ULTIMATE improved industry competitiveness higher investment better safety improved quality of life promotion of regional development improved awareness -1 0 1 2 4 6 8 10 15 YEARS, OR MORE IMPACTS Benefits to economy and society Benefits to project participants (Tassey 2003)

11. Version 9.2.2003 VTT TECHNOLOGY STUDIES Dimensions of impacts and final beneficiaries 1.Economy, technology and commercialisation 2.Knowledge, expertise and human capital 3.Networking and social capital 4.Decision making and public discourse 5.Social and physical environment Own organisation Customer Society Public R&D-organisation

12. Version 9.2.2003 VTT TECHNOLOGY STUDIES Evaluation model: some criteria Relevance Quantifiable nature Reliability Easy availability Dynamics of change and trends Strategic feedback loop

13. Version 9.2.2003 VTT TECHNOLOGY STUDIES How to get information about impacts and effectiveness through evaluation Ex ante evaluation Preparation phase Execution phase Concluding phase Ex post evaluation Stakeholder and client ’inputs’ into evaluation cycle Follow-up studies Client satisfaction surveys Interactive discussion fora Mid-term evaluation

14. Version 9.2.2003 VTT TECHNOLOGY STUDIES Impact assessment as a learning and steering tool Rationales for public support MOTIVES Market failure Transaction costs Risks and uncertainty Policy statements OBJECTIVES Public safety New enterprises and workplaces Welfare services Policy implementation ORGANIZATIONAL CAPACITY as the means (for doing ’the rights things’) Transparency, permanence, flexibility, predictability, legitimacy, justness Using the knowledge about changes amongst the stakeholders ADDED VALUE OF R&D To the organisation To the customer For society Definition of the new rationales Allocation of the resources Prioritizing Definition on the strategy Feedback