1. What Lies Beneath… avoiding the risk of undervaluation Luke Georghiou PREST, Manchester Business School http://www.mbs.ac.uk/PREST

3. Proposition 1 Present techniques, and particularly surveys, focus on predictable and linear effects while missing or recording only perfunctorily a series of other effects which may be of equal or greater value.

4. [Behavioural additionality] can help explain that the effects of policy interventions may be greater than those perceived through a narrower input-output lens, and that the range of effects and the means by which they are achieved are complex mechanisms requiring substantial effort for successful policy design.

5. Proposition 2 Even where effects are properly identified and/or calculated there is a tendency to underestimate the contribution of public intervention.

6. Historical context -co-evolution… Evaluation approaches tend to co-evolve cumulatively with policy development, eg –1970s modification of peer review to extend criteria –1980s interest began in collaborative R&D programmes –1990s rise of: performance indicators emphasis on knowledge transfer indicators Institutional evaluation using programmatic approach –2000s interest in; evaluation of system capabilities eg national or regional systems aggregate or interactive effects of policies (“policy mix”) effect of “soft” policy tools such as foresight strategic and persistent effects of public support (“behavioural additionality”)

7. Evaluation and situating the object in its context When we come to evaluate support for industrial R&D (or other research) we encounter key problem Project fallacy –Confusion in timing and scope between the unit of research and the contractual entity –Research impacts are often cumulative over series of projects –Effects of research policies result from an interaction between the measure and the strategy of the research performer

8. Project fallacy Key problem of “project fallacy” in which policymaker assumes that a contract is equivalent to a project –In practice contracted work is often only part of a longer and broader project Real deliverables ContractReal project Contract deliverables

9. EUREKA Initiative as Example Evaluation approach since mid-1990s based on “Continuous and Systematic Evaluation” –Final reporting requirements replaced by short impact survey –Follow-up after one and three years if market effects reported

10. EUREKA CSE

11. Typical question (1)

12. Typical question (2)

13. Gives some useful results eg turnover effects - skewed returns % of turnover accounted by % of firms in EUREKA Initiative

14. But several problems… Inaccurate completion of survey forms –Trivial but important – confusing currency units –Non-trivial and also important - What to measure, when to measure, how to interpret all dependent upon the respondent’s underlying model of innovation – implicit or explicit Large proportion of benefits dealt with only by means of importance rating with no model of how they interact or translate to measurable effects

15. EUREKA 2006 Changing methodology – keeping CSE as supplementary approach Main effort on panel carrying out case studies of high impact projects – use skew as basis for purposive sample Using methodology developed by PREST in evaluation of Japanese National R&D Programme for Medical and Welfare Equipment using Beta method as starting point

16. Iceberg model Sales of innovative product Reduced process costs Licence income Firm strategy, organisation and method learning Use of technology in other parts of the business New contacts/networks & prestige Employment, competence & training Spillovers to non-participants User and social benefits

17. Effects on firm strategy Projects normally aligned with firm strategies but also create pathways to transform them –Movement from R&D services/consultancy to manufacture –Assisting privatised state enterprises to become commercially viable –Improvements in organisation and methods induced by the project, including working at a higher technological level frequently internalised by the firms.

18. Use of technologies in other parts of the business  Neglected aspect of benefits Opening new technological options through follow-on projects Application of technologies developed to exploit new opportunities, potentially with turnover effects equal to or greater than in the originally intended application.

19. Contacts, networks and prestige  New contacts and networks generated by projects  Mobilising effect on large sectors eg AUTOSAR network in the automotive industry which emerged from ITEA EAST EEA.  Prestige effects of the EUREKA label  helpful to give smaller firms a foothold in the market and visibility to investors.

20. Employment Quantitative versus qualitative important here –In quantitative treatment need to consider jobs induced or displaced in value chain –Qualitative can include upgrading of competences, hiring higher proportion of scientists and engineers –Also elimination of dangerous or unpleasant work

21. Spillovers Using Jaffe’s classifications: –Knowledge spillovers appear weak and rare because SMEs have focussed niche strategies and large consortia working in parallel with competitors –Market spillovers very strong – quick measure is to look at payback time for customers –Network spillovers usually exploiting complementarities of equipment or equipment/service

22. User benefits Improved environments, health, safety or simply from better value for money and functionality in everyday objects

23. Bringing it to reality… Benefits come as a package and much is lost in disaggregation 3 cases

24. Case study 1- Σ!1692 Sanifogger Lynx Award for humidifiers that keep food fresh in supermarket storage and refrigerated cabinets Contronics turnover from €700k in 1999 to €3.8M in 2005 almost all from project Transfer of technology to vehicle air-conditioning cleaner led to orders for 10k units p.a. (contact to new partner through EUREKA Secretariat) Organisational learning included introduction of quality manager in Contronics and market repositioning in UK partner Pendred leading to €375k sales 16 jobs created in partners, 10 in sub-contractors Customer benefit of 50% reduction in wastage with 6 months payback for fruit/veg and 4 week payback for meat – estimate of €42M p.a. customer benefit EUREKA label providing substantial marketing and visibility effects

25. Case study 2 – Σ!1872 P3D Developed passive surveillance systems for tracking aircraft 200km around airports ERA is SME privatised 1994 and using EUREKA project in R&D based shift from military to commercial orientation ERA and its spin-offs 30-50% of total ERA's sales in period of 2002-4, total value of €5.4-9.0M over 3 years. Fourteen new jobs in ERA and 20 in partner firms performing outsourced manufacture Chairing international standards setting group Reduced costs by half compared with conventional systems Fundamental contribution to safety

26. Case study 3 - Σ!2275 Visualix Gendex Dental Systems developed powerful digital system for high resolution dental radiography Sales, since 2002, worth €28.6M to produce 8,635 units –20% of firm’s total 2005 sales 3 new jobs created, 2 in R&D now developing future projects using extended knowledge base and IPR platform Eliminates film development and liquid disposal for dentists giving them 2-4 year payback and more time with patients Patient gets immediate image and possibility of better treatment decisions

27. Additionality – what difference does the intervention make? Input additionality – are resources being spent on desired target? Output additionality – what proportion of outputs result from particular intervention? Behavioural additionality – what difference in behaviour results from the intervention –Concept formulated mid-1990s to help explain consistent evaluation findings –Rooted in question of how support interacts with strategies and capabilities of funded organisations –Looks closely at mode of delivery of support for research –Emphasis on persistent changes –OECD project explored measurement issues in industrial grant schemes

28. Effects during grant cycle Awareness application Contract Monitoring Research Post-project support Failed/ non- applicants stimulated IPR & collaboration rules adopted Feedback Research direction or linkages changed Exploitation Route changed

29. EUREKA contribution to effects of its projects Public funding –Role of public funding in enabling, extending or accelerating the work stressed –No success cases without public funding and some not returned to EUREKA because funding no longer available –Advice from funding agencies helpful to smaller or less experienced firms and large firms spoke very positively of the role of cluster offices. Collaboration –Complementary skills brought together, often in vertical structure, linking users and suppliers –In clusters “pre-competitive” configurations, establishing de facto standards or sharing cost of developing technologies to be applied in different sectors –Often the initiative of EUREKA that brought together these networks

30. Under-evaluation = Under-investment Our failure to appreciate the full extent of both the private and the social returns to R&D is a key reason for the international decline in government investment in research It is possible, though not proven, that industry itself may underestimate its own benefits from R&D, particularly under shareholder pressure to show short term profits

31. Conclusions (1) “Standard” questionnaire based evaluation approaches are never a sufficient means to assess effects however sophisticated an analysis is subsequently applied –Nonetheless they can be a useful first step towards identifying case studies High impact cases can be a good approximation to total effects in an R&D initiative –but not for other instruments where effects are more evenly spread (eg information and advisory services Behavioural additionality perspective opens the way to a much more comprehensive appreciation of effects –Ultimately we are concerned as much with building capacities as with short-term impacts Evaluators need to probe what lies beneath the surface if they are to succeed in capturing and measuring the value created

32. Conclusions (2) Evaluators need to probe what lies beneath the surface if they are to succeed in capturing and measuring the value created This allows us to address some of the key policy issues of today