Prof. Anthony Arundel Australian Innovation Research Centre, University of Tasmania, Australia and UNU-MERIT, Maastricht, the Netherlands History and design of the European Community Innovation Survey (CIS) questionnaire

1. Introduction

Responses to the CIS from over 200,000 enterprises Conducted every two years (7 surveys so far) CIS 2014 under development

Uses of the CIS-1 Policy relevant indicators used in the 2013 European Innovation Union Scoreboard: 1.Non-R&D innovation expenditures 2.Percent of SMEs that innovate in-house 3.Percent of SMEs that collaborate on innovation 4.Percent of SMEs with product or process innovations 5.Percent of SMEs with organisational/marketing innovations 6.The share of total sales from innovative products

National performance on the Innovation Union Scoreboard, 2013

Benchmarking In-depth analyses Policy uses of innovation surveys: Benchmarking motivates, but in- depth analysis is required to know what to do.

Effects on performance 1.CIS output indicators -Change in annual turnover or employment over three years -Innovation sales share - Crude estimate of productivity 2.Data linkage to administrative data for profits, productivity, production etc. 3.Panel data: can examine effect over time on innovation sales share

2. R&D versus innovation

The origins – R&D surveys First attempts to measure R&D: –1917: First R&D survey, with further experimentation up until the late 1930s. –1953: First large American R&D survey. –1963: First international R&D survey. –1981: OECD considers R&D data quality and international comparability acceptable.

R&D focus of early innovation research Are innovation activities a supplement to R&D? (perspective of Frascati Manual) or Is innovation is a unique activity that may or may not involve R&D?

13

June

16 What is going on? There is a lot more than R&D – other factors contribute to economic development and productivity improvements. –Technology purchases (adoption as in the fishing sector) –Organisational innovation –Innovative activities that do not require R&D

Ways of innovating without R&D Technology adoption Minor modifications or incremental changes, including use of engineering knowledge Imitation including reverse engineering Combining existing knowledge in new ways

Percent of firms innovating without R&D 45.4% of innovative firms in this sample did not perform R&D Of R&D performing firms, 63% reported innovations that did not require R&D

Method of innovation (from least to most advanced in-house capabilities) for product/process innovation and for organisational innovation

R&D versus innovation R&D is largely an activity of manufacturing firms – much rarer in services –Innovation data are a better measure of innovation activities in services Definition of R&D (requirement for scientific or technological novelty) is confusing for firms in services –What about development without research?

22 Definition of Innovation An innovation is the implementation of a new or significantly improved product (good or service), process, new marketing method, or new organisational method. Source: 3 rd revision, Oslo Manual, OECD –Innovation is NOT invention. –It does NOT require creative effort – adoption of new manufacturing technology is an innovation. –It does NOT need to be commercially successful.

Percent of innovative firms in Europe that do not perform R&D Innobarometer 2007: 52.5% of innovative firms did not perform R&D. CIS-3 ( ): 50% did not perform R&D

Countries: Germany, Italy, Belgium, Netherlands, Luxembourg, Ireland, Denmark, Norway Percentage of Innovative and R&D Performing Firms by Size: CIS-1 Estimates for 8 Countries Combined < > 2000 Number of Employees Innovators R&D performers

Innovative status of an enterprise in the CIS An innovative firm has introduced one or more types of innovations within a defined period of time (3 years). Innovative status can depend on the maximum degree of novelty of at least one of the firm’s innovations: –‘World first’ – often based on in-house creative effort. –New to the firm’s market: could be based on creative effort (reverse engineering, engineering improvements) or bought from another firm. –Only new to the firm: purchased off the shelf.

Innovation as a creative versus diffusion activity New product/process resulting from large investment in in-house R&D New technology bought ‘off the shelf’ Purchased technology is adapted to firm’s needs Maximum Creative effort Minimum creative effort

R&D status by change in turnover 2004 to 2006 Results confirmed in an econometric model that controls for size, sector, country, total innovation expenditures, innovative capabilities. Source: Arundel, Bordoy & Kanerva, 2008

3. History of the Community Innovation Survey (CIS)

First innovation surveys Object based: collect data on specific innovations identified in trade journal advertisements or other sources. –Townsend, 1981, Kleinknect (various studies) De Bresson & Murray (1984): stratified random survey of all Canadian firms (not just R&D performers), asked for descriptions of three most important innovations. –Defined innovation as “any new or improved product which has withstood the trial of the market and generated a return on investment, or a new or improved process for commercial production. By new we mean new to Canada”.

Subject-based innovation surveys The focus is on the firm or enterprise, with questions about its innovation activities. First surveys in the 1970s in Canada, limited to R&D performing firms. Advantages over an object approach: –Can collect information on all types of innovations –Can cover process and other types of innovations.

Experimental innovation surveys of the 1980s MIT (United States) & Fraunhofer (Germany) surveys only sent to R&D performing firms. Scandinavian survey: conflicting definitions, product innovations defined as deriving from ‘R&D projects that resulted in marketable new products”. Ifo (Germany) referred to new or improved products and processes. Dutch, French and Italian surveys asked if the firm had introduced an innovation that was new to the firm or its local region.

First Oslo Manual (1992) “ The core task is to integrate an understanding of the R&D contribution with an account of the non-R&D inputs to the innovation process” –Innovation is a supplement to R&D What differentiates a change from an innovation are “elements of novelty and significance” –(similar wording to the Frascati Manual) But, definition of an incremental innovation is ambiguous about the need for R&D: “an incremental product innovation is an existing product whose performance has been significantly enhanced or upgraded.

CIS-1 (implemented in 1993) CIS-1 definition of a product innovation “A significant innovation is a newly-marketed product whose intended use, performance, characteristics, technical construction, design, or use of materials and components is new or substantially changed.” “An incremental innovation is an existing product whose technical characteristics have been enhanced or upgraded.” –Whether or not R&D is required is left undefined.

CIS-2 to CIS-4: removing ambiguity over R&D CIS-2 (implemented in 1997): –Asked respondents ‘who developed’ their innovations and included the option ‘mainly other enterprises or organisations’ CIS-3 (implemented in 2001): –Changed definition of product and process innovations to state that they only needed to be ‘new to your enterprise’. CIS-4 (implemented in 2005): –Requirement for R&D for collaboration removed, with definition of collaboration changed to ‘joint R&D and other collaboration projects’

Survey Observation period Main changes or additions compared to the previous survey CIS CIS Added questions on who developed product and process innovations. Cooperation question extended to cover more than R&D. Deleted questions on sources of new technology, technology transfer outside the enterprise, appropriation methods, and product life cycles. CIS Changed question on the innovation sales share to the share of sales from unchanged, new-to-firm and new-to-market products. CIS Added questions on three types of organizational innovation and two types of marketing innovation, plus questions on the effects of organizational innovation. Under product innovation included separate questions for goods and services. Asked about three types of process innovations. CIS Frequency increased to every two years. Implemented cognitive testing for all question changes and additions. CIS Increased coverage of organizational and marketing innovation. One-page module for questions of high policy interest, with the first module on environmental innovation. CIS Separate question on expenditures for design. Module on creativity and skills. CIS Module on strategies and obstacles to growth. Reintroduced a modified version of the CIS-1 appropriation question. New q uestions on public procurement and innovation

But, the forces for R&D might be back, with a proposal to combine the CIS and R&D surveys!

4. The CIS today

The European CIS is a ‘general’ innovation survey that covers a large number of topics, none of them in particular depth. –Exception: knowledge sourcing

Limitations of the CIS Cross-sectional survey –only a few countries have panel data: –Norway, Germany, Tasmania (not a CIS survey but has some similar questions) As a general rule, does not include questions on rare activities –These questions collect little information of value but add to the response burden

Who answers the CIS? Ideally, the CEO or R&D manager

Who might use the CIS? Academics Businesses Policy analysts

Three different users Business managers –Need timely data for benchmarking against other firms A general failure of large surveys, but feasible for smaller surveys. Academics –Need access to micro data for testing innovation theory (many data access limitations) Policy community (main target) –Need results that are directly relevant to policy issues (rarely provided by academics); often rely on indicators and descriptive analyses but econometrics are useful if properly explained.

Definition of innovative enterprises At least one product, process, marketing or organisational innovation in the preceding three years. Only needs to be ‘new to the firm’ – includes simple technology adoption. –Criticized for being too broad

Indicator for ‘how’ enterprises innovate: Spain EU average

Finland versus Portugal

Main players in the CIS Eurostat (statistical agency of the European Union) coordinates the design of the CIS questionnaire and makes recommendations for the survey methodology. CIS Working Group: All European countries that implement the CIS are members. OECD and the European Commission also attend. They assess the revised questionnaire from the Task Force and can request changes. Decision to accept based on consensus. National Statistical Offices: Responsible for implementing the CIS in their country. By EU law, must provide specific indicators to Eurostat; can voluntarily provide full data for the Eurostat Safe Centre and anonymized data.

CIS Task Force

Topics covered in the CIS Firm’s main markets Product and process innovation –Who developed –Innovation sales share (output variable) –Novelty (world-first, Europe first, country first) –Innovation activities & expenditures –Public support –Sources of information –Types of collaboration partners Organisational and marketing innovations Barriers to innovation (drivers in CIS-2014) Innovation objectives (not asked in every CIS) Module of questions of policy interest

Percent of UK firms reporting cost factors as of high importance as barriers to innovation (CIS-4) Source: D’Este et al, 2012

Proposed new question on barriers for non-innovative enterprises

Importance of innovation drivers

Modules CIS 2008: –Environmental innovation CIS 2010: –Creativity and skills (including how the firm encourages creativity) CIS 2012: –Strategies and obstacles for growth (meeting enterprise goals) –Appropriation methods (also used in CIS-1) CIS 2014: –Improved questions on environmental innovation, including importance of different drivers

Required & optional CIS questions Many questions required by EU law (EC regulation 1450/2004) Optional questions include: –Novelty (world first) of product and process innovations –Innovation module –Questions on public procurement, appropriation National statistical offices can and do add other questions of high relevance to their own policy issues.

Keeping the CIS relevant CIS questions used in benchmarking need to stay the same, but new questions and topics are needed every year to attract academic researchers and evidence for topical policy questions. In response, since CIS 2008 the questionnaire includes a module of one-off questions that can change in every CIS.

System for continuous improvement of the CIS 1.Cognitive testing of all new questions & major changes 2.Quality reports for each CIS survey -non-response rates for specific questions - Experiences of National Statistical Offices 3.Research by NSOs on methodology - Effect of combining R&D and innovation surveys - How respondents understand key concepts 4.Bi-annual surveys of NSOs - Survey method (online, mailed, face-to-face) - Post survey methods for improving data quality 5.Surveys of policy users - What innovation survey data do they use 6.Database of academic papers that use the CIS - Which questions do they use and do not use - New questions that academics would like for their research

Cognitive testing Face-to-face interviews with firm managers –Goal to ensure that all questions are understood as intended by all respondents, and –Respondents can provide accurate responses Introduced on a consistent basis in 2004 (recommendation of report on CIS-4).

Logical error

Logical error due to question placement

Accuracy of responses (not all problems can be solved)

Two questions in one

Item non-response rates: use of quality reports to track progress Improvements to question design: –Decreased item non-response rates for the innovation sales share question from approximately 25% to less than 3% by CIS-4 –Decreased item non response rates for the question on innovation expenditures from 36% to approximately 15% by CIS-4.

Research by National Statistical Offices Norway tested the effect of combining the R&D and CIS surveys –Found small declines in number of innovative firms. Netherlands compared responses to online and mailed surveys. –Found much higher share of innovators to online surveys (70%) compared to mailed surveys (50%)

NSO survey For every CIS, survey NSOs on a range of relevant questions Topics include: – issues with new questions in the previous survey –survey methods, particularly the use of online surveys –New questions that may have tried and which might be of interest to all NSOs

NSO survey results

Software for online surveys 24 countries provided an online option (80%)

Possible questions for removal from CIS 2014 (% of countries)

Number of NSOs that added a non-standard question to their national 2012 CIS survey

Surveys of policy users Infrequent because expensive –Last large scale survey in 2005 –New survey planned for 2014 Interested in the types of indicators that policy makers find useful If CIS results have influenced policy

2005 Policy survey results R&D indicators CIS indicators

Academic users Small-scale surveys of10 to 20 leading academics, based on number of publications using CIS data. Ask which types of new questions they would like, which questions perform poorly in analyses, etc. Maintain database of publications using the CIS.

Results for 2012 incomplete Academic use of CIS data

Results for 2012 incomplete

5. Conclusions: Key lessons for a useful innovation survey

10 recommendations: not all of which have been fully implemented in Europe 1.Collect data on question quality, including item non-response rates, ‘not relevant’ response rates, number of respondents who contact you for better explanations, etc. 2.Cognitively test all new questions and major changes to existing questions (50 interviews) 3.The working group to design innovation questionnaires should include representatives from both statistical offices and user communities.

4.The addition or deletion of CIS questions should be backed by empirical data on data quality and on the relevance of the question to users. 5.Allow room for experimentation with new questions – perhaps by province? 6.System for continuous improvement. 7.Add new questions in each CIS – keep it interesting for respondents and for users.

Recommendations that have not been fully implemented in Europe 8.Provide academics with timely access to the data. 9.In return for data access, insist that academics provide a ten page evaluation of the relevance of their research for policy. 10.Develop a strong interface between the policy community and the National Statistical Office to encourage the uptake of CIS results for policy uses.

And two more…. Create a panel data set (possibly a sub- sample) Provide data linkage to other data, especially for outcome analyses

Thank-you! For questions or more information: Detailed CIS reports available for last 5 CIS surveys.