1. Prof. Anthony Arundel Australian Innovation Research Centre, University of Tasmania, Australia and UNU-MERIT, Maastricht, the Netherlands History and design of the European Community Innovation Survey (CIS) questionnaire

2. 1. Introduction

3. Responses to the CIS from over 200,000 enterprises Conducted every two years (7 surveys so far) CIS 2014 under development

4. Uses of the CIS-1 Policy relevant indicators used in the 2013 European Innovation Union Scoreboard: 1.Non-R&D innovation expenditures 2.Percent of SMEs that innovate in-house 3.Percent of SMEs that collaborate on innovation 4.Percent of SMEs with product or process innovations 5.Percent of SMEs with organisational/marketing innovations 6.The share of total sales from innovative products

5. National performance on the Innovation Union Scoreboard, 2013

6. Benchmarking In-depth analyses Policy uses of innovation surveys: Benchmarking motivates, but in- depth analysis is required to know what to do.

9. Effects on performance 1.CIS output indicators -Change in annual turnover or employment over three years -Innovation sales share - Crude estimate of productivity 2.Data linkage to administrative data for profits, productivity, production etc. 3.Panel data: can examine effect over time on innovation sales share

10. 2. R&D versus innovation

11. The origins – R&D surveys First attempts to measure R&D: –1917: First R&D survey, with further experimentation up until the late 1930s. –1953: First large American R&D survey. –1963: First international R&D survey. –1981: OECD considers R&D data quality and international comparability acceptable.

12. R&D focus of early innovation research Are innovation activities a supplement to R&D? (perspective of Frascati Manual) or Is innovation is a unique activity that may or may not involve R&D?

13. 13

14. June 2008 14

16. 16 What is going on? There is a lot more than R&D – other factors contribute to economic development and productivity improvements. –Technology purchases (adoption as in the fishing sector) –Organisational innovation –Innovative activities that do not require R&D

17. Ways of innovating without R&D Technology adoption Minor modifications or incremental changes, including use of engineering knowledge Imitation including reverse engineering Combining existing knowledge in new ways

18. Percent of firms innovating without R&D 45.4% of innovative firms in this sample did not perform R&D Of R&D performing firms, 63% reported innovations that did not require R&D

19. Method of innovation (from least to most advanced in-house capabilities) for product/process innovation and for organisational innovation

21. R&D versus innovation R&D is largely an activity of manufacturing firms – much rarer in services –Innovation data are a better measure of innovation activities in services Definition of R&D (requirement for scientific or technological novelty) is confusing for firms in services –What about development without research?

22. 22 Definition of Innovation An innovation is the implementation of a new or significantly improved product (good or service), process, new marketing method, or new organisational method. Source: 3 rd revision, Oslo Manual, OECD –Innovation is NOT invention. –It does NOT require creative effort – adoption of new manufacturing technology is an innovation. –It does NOT need to be commercially successful.

23. Percent of innovative firms in Europe that do not perform R&D Innobarometer 2007: 52.5% of innovative firms did not perform R&D. CIS-3 (2002- 2004): 50% did not perform R&D

24. Countries: Germany, Italy, Belgium, Netherlands, Luxembourg, Ireland, Denmark, Norway Percentage of Innovative and R&D Performing Firms by Size: CIS-1 Estimates for 8 Countries Combined 0 10 20 30 40 50 60 70 80 90 100 < 2020-99100-249250-499500-9991000-1999> 2000 Number of Employees Innovators R&D performers

26. Innovative status of an enterprise in the CIS An innovative firm has introduced one or more types of innovations within a defined period of time (3 years). Innovative status can depend on the maximum degree of novelty of at least one of the firm’s innovations: –‘World first’ – often based on in-house creative effort. –New to the firm’s market: could be based on creative effort (reverse engineering, engineering improvements) or bought from another firm. –Only new to the firm: purchased off the shelf.

27. Innovation as a creative versus diffusion activity New product/process resulting from large investment in in-house R&D New technology bought ‘off the shelf’ Purchased technology is adapted to firm’s needs Maximum Creative effort Minimum creative effort

28. R&D status by change in turnover 2004 to 2006 Results confirmed in an econometric model that controls for size, sector, country, total innovation expenditures, innovative capabilities. Source: Arundel, Bordoy & Kanerva, 2008

29. 3. History of the Community Innovation Survey (CIS)

30. First innovation surveys Object based: collect data on specific innovations identified in trade journal advertisements or other sources. –Townsend, 1981, Kleinknect (various studies) De Bresson & Murray (1984): stratified random survey of all Canadian firms (not just R&D performers), asked for descriptions of three most important innovations. –Defined innovation as “any new or improved product which has withstood the trial of the market and generated a return on investment, or a new or improved process for commercial production. By new we mean new to Canada”.

31. Subject-based innovation surveys The focus is on the firm or enterprise, with questions about its innovation activities. First surveys in the 1970s in Canada, limited to R&D performing firms. Advantages over an object approach: –Can collect information on all types of innovations –Can cover process and other types of innovations.

32. Experimental innovation surveys of the 1980s MIT (United States) & Fraunhofer (Germany) surveys only sent to R&D performing firms. Scandinavian survey: conflicting definitions, product innovations defined as deriving from ‘R&D projects that resulted in marketable new products”. Ifo (Germany) referred to new or improved products and processes. Dutch, French and Italian surveys asked if the firm had introduced an innovation that was new to the firm or its local region.

33. First Oslo Manual (1992) “ The core task is to integrate an understanding of the R&D contribution with an account of the non-R&D inputs to the innovation process” –Innovation is a supplement to R&D What differentiates a change from an innovation are “elements of novelty and significance” –(similar wording to the Frascati Manual) But, definition of an incremental innovation is ambiguous about the need for R&D: “an incremental product innovation is an existing product whose performance has been significantly enhanced or upgraded.

34. CIS-1 (implemented in 1993) CIS-1 definition of a product innovation “A significant innovation is a newly-marketed product whose intended use, performance, characteristics, technical construction, design, or use of materials and components is new or substantially changed.” “An incremental innovation is an existing product whose technical characteristics have been enhanced or upgraded.” –Whether or not R&D is required is left undefined.

35. CIS-2 to CIS-4: removing ambiguity over R&D CIS-2 (implemented in 1997): –Asked respondents ‘who developed’ their innovations and included the option ‘mainly other enterprises or organisations’ CIS-3 (implemented in 2001): –Changed definition of product and process innovations to state that they only needed to be ‘new to your enterprise’. CIS-4 (implemented in 2005): –Requirement for R&D for collaboration removed, with definition of collaboration changed to ‘joint R&D and other collaboration projects’

36. Survey Observation period Main changes or additions compared to the previous survey CIS-11990-1992 - CIS-21994-1996 Added questions on who developed product and process innovations. Cooperation question extended to cover more than R&D. Deleted questions on sources of new technology, technology transfer outside the enterprise, appropriation methods, and product life cycles. CIS-31998-2000 Changed question on the innovation sales share to the share of sales from unchanged, new-to-firm and new-to-market products. CIS-42002-2004 Added questions on three types of organizational innovation and two types of marketing innovation, plus questions on the effects of organizational innovation. Under product innovation included separate questions for goods and services. Asked about three types of process innovations. CIS-20062004-2006 Frequency increased to every two years. Implemented cognitive testing for all question changes and additions. CIS-20082006-2008 Increased coverage of organizational and marketing innovation. One-page module for questions of high policy interest, with the first module on environmental innovation. CIS-20102008-2010 Separate question on expenditures for design. Module on creativity and skills. CIS-20122010-2012 Module on strategies and obstacles to growth. Reintroduced a modified version of the CIS-1 appropriation question. New q uestions on public procurement and innovation

37. But, the forces for R&D might be back, with a proposal to combine the CIS and R&D surveys!

39. 4. The CIS today

40. The European CIS is a ‘general’ innovation survey that covers a large number of topics, none of them in particular depth. –Exception: knowledge sourcing

41. Limitations of the CIS Cross-sectional survey –only a few countries have panel data: –Norway, Germany, Tasmania (not a CIS survey but has some similar questions) As a general rule, does not include questions on rare activities –These questions collect little information of value but add to the response burden

42. Who answers the CIS? Ideally, the CEO or R&D manager

43. Who might use the CIS? Academics Businesses Policy analysts

44. Three different users Business managers –Need timely data for benchmarking against other firms A general failure of large surveys, but feasible for smaller surveys. Academics –Need access to micro data for testing innovation theory (many data access limitations) Policy community (main target) –Need results that are directly relevant to policy issues (rarely provided by academics); often rely on indicators and descriptive analyses but econometrics are useful if properly explained.

45. Definition of innovative enterprises At least one product, process, marketing or organisational innovation in the preceding three years. Only needs to be ‘new to the firm’ – includes simple technology adoption. –Criticized for being too broad

46. Indicator for ‘how’ enterprises innovate: Spain EU average

47. Finland versus Portugal

48. Main players in the CIS Eurostat (statistical agency of the European Union) coordinates the design of the CIS questionnaire and makes recommendations for the survey methodology. CIS Working Group: All European countries that implement the CIS are members. OECD and the European Commission also attend. They assess the revised questionnaire from the Task Force and can request changes. Decision to accept based on consensus. National Statistical Offices: Responsible for implementing the CIS in their country. By EU law, must provide specific indicators to Eurostat; can voluntarily provide full data for the Eurostat Safe Centre and anonymized data.

49. CIS Task Force

50. Topics covered in the CIS Firm’s main markets Product and process innovation –Who developed –Innovation sales share (output variable) –Novelty (world-first, Europe first, country first) –Innovation activities & expenditures –Public support –Sources of information –Types of collaboration partners Organisational and marketing innovations Barriers to innovation (drivers in CIS-2014) Innovation objectives (not asked in every CIS) Module of questions of policy interest

51. Percent of UK firms reporting cost factors as of high importance as barriers to innovation (CIS-4) Source: D’Este et al, 2012

52. Proposed new question on barriers for non-innovative enterprises

53. Importance of innovation drivers

54. Modules CIS 2008: –Environmental innovation CIS 2010: –Creativity and skills (including how the firm encourages creativity) CIS 2012: –Strategies and obstacles for growth (meeting enterprise goals) –Appropriation methods (also used in CIS-1) CIS 2014: –Improved questions on environmental innovation, including importance of different drivers

55. Required & optional CIS questions Many questions required by EU law (EC regulation 1450/2004) Optional questions include: –Novelty (world first) of product and process innovations –Innovation module –Questions on public procurement, appropriation National statistical offices can and do add other questions of high relevance to their own policy issues.

56. Keeping the CIS relevant CIS questions used in benchmarking need to stay the same, but new questions and topics are needed every year to attract academic researchers and evidence for topical policy questions. In response, since CIS 2008 the questionnaire includes a module of one-off questions that can change in every CIS.

57. System for continuous improvement of the CIS 1.Cognitive testing of all new questions & major changes 2.Quality reports for each CIS survey -non-response rates for specific questions - Experiences of National Statistical Offices 3.Research by NSOs on methodology - Effect of combining R&D and innovation surveys - How respondents understand key concepts 4.Bi-annual surveys of NSOs - Survey method (online, mailed, face-to-face) - Post survey methods for improving data quality 5.Surveys of policy users - What innovation survey data do they use 6.Database of academic papers that use the CIS - Which questions do they use and do not use - New questions that academics would like for their research

58. Cognitive testing Face-to-face interviews with firm managers –Goal to ensure that all questions are understood as intended by all respondents, and –Respondents can provide accurate responses Introduced on a consistent basis in 2004 (recommendation of report on CIS-4).

59. Logical error

60. Logical error due to question placement

61. Accuracy of responses (not all problems can be solved)

62. Two questions in one

63. Item non-response rates: use of quality reports to track progress Improvements to question design: –Decreased item non-response rates for the innovation sales share question from approximately 25% to less than 3% by CIS-4 –Decreased item non response rates for the question on innovation expenditures from 36% to approximately 15% by CIS-4.

64. Research by National Statistical Offices Norway tested the effect of combining the R&D and CIS surveys –Found small declines in number of innovative firms. Netherlands compared responses to online and mailed surveys. –Found much higher share of innovators to online surveys (70%) compared to mailed surveys (50%)

65. NSO survey For every CIS, survey NSOs on a range of relevant questions Topics include: – issues with new questions in the previous survey –survey methods, particularly the use of online surveys –New questions that may have tried and which might be of interest to all NSOs

66. NSO survey results

68. Software for online surveys 24 countries provided an online option (80%)

69. Possible questions for removal from CIS 2014 (% of countries)

70. Number of NSOs that added a non-standard question to their national 2012 CIS survey

71. Surveys of policy users Infrequent because expensive –Last large scale survey in 2005 –New survey planned for 2014 Interested in the types of indicators that policy makers find useful If CIS results have influenced policy

72. 2005 Policy survey results R&D indicators CIS indicators

73. Academic users Small-scale surveys of10 to 20 leading academics, based on number of publications using CIS data. Ask which types of new questions they would like, which questions perform poorly in analyses, etc. Maintain database of publications using the CIS.

74. Results for 2012 incomplete Academic use of CIS data

75. Results for 2012 incomplete

76. 5. Conclusions: Key lessons for a useful innovation survey

77. 10 recommendations: not all of which have been fully implemented in Europe 1.Collect data on question quality, including item non-response rates, ‘not relevant’ response rates, number of respondents who contact you for better explanations, etc. 2.Cognitively test all new questions and major changes to existing questions (50 interviews) 3.The working group to design innovation questionnaires should include representatives from both statistical offices and user communities.

78. 4.The addition or deletion of CIS questions should be backed by empirical data on data quality and on the relevance of the question to users. 5.Allow room for experimentation with new questions – perhaps by province? 6.System for continuous improvement. 7.Add new questions in each CIS – keep it interesting for respondents and for users.

79. Recommendations that have not been fully implemented in Europe 8.Provide academics with timely access to the data. 9.In return for data access, insist that academics provide a ten page evaluation of the relevance of their research for policy. 10.Develop a strong interface between the policy community and the National Statistical Office to encourage the uptake of CIS results for policy uses.

80. And two more…. Create a panel data set (possibly a sub- sample) Provide data linkage to other data, especially for outcome analyses

81. Thank-you! For questions or more information: a.arundel@utas.edu.au Detailed CIS reports available for last 5 CIS surveys.