Presentation on theme: "2014.02.16 Agnė Paliokaitė, Senior Policy Researcher, Public Policy and Management Institute, Lithuania COHESION POLICY SUPPORT TO INNOVATION IN LITHUANIA:"— Presentation transcript:
Agnė Paliokaitė, Senior Policy Researcher, Public Policy and Management Institute, Lithuania COHESION POLICY SUPPORT TO INNOVATION IN LITHUANIA: Lessons Learnt for Evaluation Presentation for DIRECTORATE-GENERAL REGIONAL POLICY "EVALUATION NETWORK MEETING" Brussels, 14 April 2011
ROADMAP Aims and challenges Approach Conclusions Added value, strengths and limitations Insights, based on 4 studies carried by PPMI in 2010: - Two system level evaluations for the Knowledge Economy Forum and the Prime Ministers Office; - ERAWATCH country report; - SF indicators system evaluation for the Ministry of Finance (RTDI measures case study).
WHY SYSTEM EVALUATION? System (portfolio) evaluation – evaluating policy portfolios, not individual programmes. Retrospective: new political will put innovation high on the political agenda in 2009/2010. New ideas - need for revisiting the incrementally developed policy mix. Prospective: need for rethinking the future priorities in the context of Progress Strategy Lithuania 2030 and the new Structural Funds period.
AIMS OF SYSTEM EVALUATION To analyse the extent to which SF funded innovation policy portfolio/mix reflects specific conditions and levels of the National Innovation System (NIS). To analyse how the financial proportions fit to the policy agenda (the preferred routes). To present preliminary insights on effectiveness in achieving set targets. To draw conclusions on governance & monitoring system.
sdafasdfasdfasdf EVALUATION FRAMEWORK Hypotheses about bottlenecks Conclusions 1. Innovation system health: market, capability, institutional, network, system, and governance failures 2. Intervention logic and policy mixes 3. Extent to which outputs and results are achieved, critical factors Innovation Policy and Governance development Based on: Arnold E. Evaluating research and innovation policy: a systems world needs systems evaluations, Research Evaluation, volume 13(1), 2004 Relevance (Are we doing the right things?) Effectiveness (Are we doing things right?)
CHALLENGES AND LIMITATIONS Timing: low absorption of funds at the time of evaluation (most measures started operation in ). Small scale evaluations. Hence, inability to apply quantitative approach. A moving object: innovation policy and governance reform (LIS , SITA); changes in the system of SF objectives. Inability to rely on the system of quantitative indicators. Innovation policy specific: M&E exceptionally difficult for innovation programmes: inherently qualitative and diffuse nature of innovation benefits. Long cause-effect chain.
QUALITATIVE APPROACH DATA COLLECTION Semi structured interview programme with stakeholders and target groups (~30 in total); Desk research: literature review, secondary and administrative data; Expert panels (focus groups); Triangulation principle applied for avoiding subjectivity and partiality of the data as well as guaranteeing impartial conclusions. ANALYTICAL TOOLS Assessment of the innovation system and the RTDI policy mix using thesystem failures framework; Logical models and reconstruction of the policy intervention logic; Meta-analysis of previously carried out studies and analysis of trends in the theoretical debate; Data integrating methods: scenarios and road-mapping; Comparative analysis / benchmarking of other countries experience; Risk analysis, critical factors and analysis of policy options.
sdafasdfasdfasdf RESULTS 1: INTERVENTION LOGIC Younger researchers More researchers Public R&D infrastructure quality and access to business Higher R&D collaboration between public and private sectors Higher public R&D potential and capacity Higher researchers mobility Higher private sector R&D capacity and potential Higher value added in the economy Higher private R&D investments Better qualified researchers More and better researchers in public sector Better innovation support services Stronger clusters More research ers in business Better private R&D infrastru cture More business R&D projects ESFERDF
sdafasdfasdfasdf Firms NGOs HEIS, PRIs Strengthening public R&D system Public private R&D collaboration- Investments in private R&D base Investments into productivity R&D in business, 162.2m Clusters & innovation support services 92.7m Researchers in business: 9.3m RTDI Networks 6.23m Direct support to companies, 732.4m (MoE): -Access to capital ( 415m ) - process inovelties (118.2m) -E-business, investments into production technologies Valleys, national complex programmes 678,6m (MoES) Source: PPMI, Knowledge Economy Forum, 2010 Heavily expanding and versatile, but linear logic persists. Mainly follows two routes: (1) to strengthen public R&D base, and (2) to invest in R&D in R&D performing firms. Lack of critical mass to implement some objectives placed high on political agenda (e.g. R&D collaboration). RESULTS 3: POLICY MIX ROUTES
EVALUATION OF SF MONITORING SYSTEM Quantitative (statistical analysis) as well as qualitative (logical models and consensus building activities). SMART framework (specific, measurable, achievable, timed..) HORIZONTAL EVALUATION VERTICAL EVALUATION ~ 1000 indicators ~ 150 indicators
INDICATOR LEVEL OF ACHIEVEMENT BEFORE 2015 REMARKS 1 OBJECTIVE: TO STRENGTHEN PRIVATE AND PUBLIC R&D BASE PRIVATE INVESTMENTS (million EUR) - RLOW OBSTACLES FOR PRIVATE INVESTMENTS, THUS THIS INDICATOR CAN ONLY BE APPLIED AT IMPACT LEVEL R&D CENTRES CREATED AND FUNCTIONAL – RHIGHNO THREATS NUMBER OF R&D BASE DEVELOPMENT PROJECTS– P MEDIUM INDICATOR ACHIEVED WILL BE TWICE LOWER AS PLANNED, HOWEVER THIS DOES NOT REFLECT THE REAL DECREASE OF ALLOCATED RESOURCES (-300 MILLION EUR FROM THE VALLEYS TO FINANCIAL ENGINEERING MEASYRES). 2 OBJECTIVE - TO INCREASE PUBLIC SECTOR R&D EFFECTIVENESS AND ACCESSIBILITY TO COMPANIES NUMBER OF GENERAL WORK PLACES CREATED IN THE R&D SECTOR - R MEDIUMNEW MEASURES ARE BEING CREATED NUMBER OF COOPERATION CONTRACTS SEIGNED BETWEEN PUBLIC AND PRIVATE SECTOR INSTITUTIONS- R HIGHNO THREATS NUMBER OF R&D PROJECTS- PLOW NEW MEASURES TO ENSURE ACHIEVEMENT OF THE INDICATOR VALUE ARE BEING CREATED 3 OBJECTIVE: TO INCREASE R&D ACTIVITY IN PRIVATE SECTOR PRIVATE INVESTMENTS (million EUR) - RHIGHNO THREATS NUMBER OF R&D PROJECTS (R&D ACTIVITY IN COMPANIES) – P HIGHNO THREATS 4 OBJECTIVE – TO INCREASE BUSINESS AND SCIENCE COLLABORATION, INTENSIFY THE KNOWLEDGE FLOWS NEW BORN TECHNOLOGY INTENSIVE COMPANIES- R HIGHTHIS IS AN IMPACT LEVEL INDICATOR R&D AND INNOVATION ENVIRONMENT IMPROVEMENT PROJECTS- P HIGHNO THREATS
KEY CONCLUSIONS 1.Structural gap: Lack of innovation absorptive capacity in business and society; limited local market: the key barrier to knowledge intensive firms. 2.Policy myopia 1: Excessive focus on supply side measures and on supporting the winners can be contradictory to the systemic characteristics of NIS 3.Policy myopia 2: Quantitative targets will be met 99 percent, but it does not mean achievement of qualitative objectives. 4.Risk-averse approach to implementation due to limited capacity to evaluate innovation projects. 5.Hypothesis: only a minor part of economy benefits from innovation measures. Financially marginal soft measures are important for behavioral additionality: project pipeline building, innovation brokering
Governance allowing quality ideas entering the market: o Boosting capability to develop RTDI policy, strengthening project and programme level intelligence; novel approaches to funding; a stronger involvement of users in evaluation and funding. Policy as a discovery process: Promoting innovative, risky, flexible, bottom-up approaches; project pipeline building. Empowering people to innovate (bottom-up), and demand side: procurement, regulation, clusters along the value chain, networks around societal problems. RECOMMENDATIONS FOR INNOVATIVE POLICY
STRENGTH AND LIMITS OF APPROACH Strengths: Focus on the NIS bottlenecks as opposed to the mechanical transfer of policy models that may not be the most relevant for the NIS. Allows for internal coherence and looking beyond the quantitative input/output indicators. From macro to micro level analysis (focus on important details). Limitations of qualitative approach: lack of hard data and evidence (e.g. as opposed to counterfactual analysis) for tracing the real change and explaining obtained effects. Object for the following evaluations. Recommendation for following evaluations: look for behavioural additionality (knowledge spillovers, changes in innovation process related behavioural patterns, interaction additionality, etc.), quantifying impact of networks
THANK YOU FOR ATTENTION! Agnė Paliokaitė Senior Policy Researcher Public Policy and Management Institute