1. Innovation surveys: design, implementation, lessons learnt Micheline Goedhuys

2. June 11 2008DEIP, Amman June, 10-12 20082 Why do we need to measure innovation? Scarcity of data in general, on innovation in particular Lack of policy tools for benchmarking Insufficient monitoring and evaluation of policies Nature of innovation calls for firm-level information

3. June 11 2008DEIP, Amman June, 10-12 20083 Structure of session: 1. Conceptual background 2. Experiences with innovation surveys 3. Methodological aspects 4. Use of innovation survey data 5. Opportunities and limitations to innovation data collection: key issues

4. June 11 2008DEIP, Amman June, 10-12 20084 1. Conceptual background linear view that science, research and discovery underlie innovation (science push) innovation measured by science indicators:  R&D  engineers  patenting  bibliometrics, publications, citation indices surveys (USA, 1960s) collecting R&D, patent data;

5. June 11 2008DEIP, Amman June, 10-12 20085 1. Conceptual background End 1980s, 1990s ‘activity approach’: investigating the ‘black box’ innovation results from interaction firm-market, learning, feedback (chain-link model of Kline and Rosenberg 1986) need for indicators capturing non-R&D activities and incremental change development of surveys asking firms about their innovation process

6. June 11 2008DEIP, Amman June, 10-12 20086 1. Conceptual background Harmonisation of survey efforts in the ‘Oslo Manual’, 1992, 1997, 2005 basis for Community Innovation Surveys innovation is measured as : an activity (R&D, industrial design, acquisition of machinery, external technology, training) and an output (introduction of product or process innovations) Features: new-to-the-firm, significant improvements

7. June 11 2008DEIP, Amman June, 10-12 20087 2. Experiences : CIS CIS-1: 1990-92; first regional effort to collect innovation data; 13 European countries, CIS-2: 1994-1996; 17 countries CIS-3: 1998-2000; more firms, more questions, services, organisational change, 29 countries CIS-light: 2000-2002, limited set of questions, 18 countries CIS-4: 2002-2004: 29 countries, organisational innovation and effect CIS-2006: 2004-2006; 29 countries; no data available yet CIS-2008: 2006-2008

8. June 11 2008DEIP, Amman June, 10-12 20088 2. Experiences : Latin America Need of information to monitor the impacts of economic reforms (trade liberalisation, privatisation, deregulation, FDI,etc). Source: Crespi, 2007

9. June 11 2008DEIP, Amman June, 10-12 20089 2. Experiences : Latin America Specific nature of innovation in Latin American countries:  Importance of incremental innovation; organisational and marketing innovation;  Importance of innovation embodied in machinery and equipment (dissemination)  Less private and more informal R&D  Fragmented flows of information

10. June 11 2008DEIP, Amman June, 10-12 200810 2. Experiences : Latin America Need for changes to the survey instrument : Bogotá Manual to complement OSLO Manual.  from innovations to firm-level innovative activities and technology efforts  human resources, capabilities  enlarged data need on organisational, delivery and design innovations lack of centralised agency, different questionnaires and sampling methodologies

11. June 11 2008DEIP, Amman June, 10-12 200811 2. Experiences : Latin America Second wave of Innovation surveys: 10 countries; 2000-2001; Argentina, Brazil, Chile, Cuba, Ecuador, Mexico, Panama, Peru, Trinidad & Tobago and Uruguay more of uniformity but without common questionnaire and sampling methodologies revision of Bogotá Manual and Annex to Oslo Manual (2005).

12. June 11 2008DEIP, Amman June, 10-12 200812 2. Experiences : Latin America Third wave of Innovation surveys: 5 countries; 2003-2005; Argentina, Brazil, Chile, Colombia and Uruguay  A lot of exit and a core group of countries with “consolidated” routines (but still with institutional problems and financial issues).  ECLAC-RICYT-OAS network (2006) to create a harmonized “core” questionnaire (plus access to micro-data)

13. June 11 2008DEIP, Amman June, 10-12 200813 2. Experiences : Asia, Africa In Southeast Asia: Malaysia (3), Taiwan (1), Singapore (1), Thailand (2), China, India… In Africa: South Africa (2) Planning to conduct an innovation survey in 20+ countries (NEPAD survey)

14. June 11 2008DEIP, Amman June, 10-12 200814 2. Agenda: ongoing debate to design innovation surveys to the context of developing countries concept of innovation : organisational, packaging, delivery, design innovations, waste management techniques, … trade off between country/regional design and benchmarking options increasing policy relevance inclusion of services and resource-based sectors

15. June 11 2008DEIP, Amman June, 10-12 200815 3. Methodological aspects : questionnaire content Basic information: name, location, industry, ownership, year established… Firm performance: sales, employment, … Innovation activities: Investment, Training, intra- mural and external R&D, …and expenditures Innovation outputs (product/process/organisational) Sources of information for innovation Cooperation for innovation Government policy or incentives affecting innovation

16. June 11 2008DEIP, Amman June, 10-12 200816 3. Methodological aspects : questionnaire content Objectives, goals or reasons for innovating Impact of innovations on firm performance Obstacles to innovation

17. June 11 2008DEIP, Amman June, 10-12 200817 3. Methodological aspects Organisation: national statistics agency, MOST, universities, consultants Reference period: 2 or 3 years (mostly 3) Participation: voluntary, compulsary (in Latin America)

18. June 11 2008DEIP, Amman June, 10-12 200818

19. June 11 2008DEIP, Amman June, 10-12 200819

20. June 11 2008DEIP, Amman June, 10-12 200820 3. Methodological aspects Survey modalities: postal, PTEF follow up, personal interview, telephone interview, online questionnaire, CATI Sector coverage: initially manufacturing, increasingly services, resource based industries Firm size: cutoff points: 5, 10, 20 or 50 workers

21. June 11 2008DEIP, Amman June, 10-12 200821 4. Use of innovation surveys by academics and researchers  Innovation and firm performance  Identify determinants/constraints to innovation  Innovation strategies  Regional and country studies  Industry studies  Innovation patterns over time  Developing innovation indicators: measurement issues

22. June 11 2008DEIP, Amman June, 10-12 200822 4. Use of innovation surveys for policy making:  Indicators for benchmarking  Mapping innovation ; innovation in new sectors  Assessing trends  Monitoring specific policy instruments

23. June 11 2008DEIP, Amman June, 10-12 200823  Uses 20 Indicators Cross-country comparisons, industry comparisons changes over time consensus on policy action  uses CIS based indicators % SMEs with in-house innovative activities % SMEs that collaborate on innovation total innovation expenditures as % sales % new-to-market products/sales % new-to-firm products/sales 4. Example: European innovation scoreboard

24. June 11 2008DEIP, Amman June, 10-12 200824 5. Opportunities and limitations Innovation surveys have become key research inputs of modern innovation studies (Crespi, 2007)

25. June 11 2008DEIP, Amman June, 10-12 200825 5. Opportunities and limitations Heterogeneity across questionnaires and methodologies remains and is even on the rise due to broadening concept of innovation, scope, … Lots of country studies, little cross-country comparisons in developing countries This limits the use of survey data as benchmarking tool (e.g. Crespi, 2007)

26. June 11 2008DEIP, Amman June, 10-12 200826 5. Opportunities and limitations On the questionnaire:  Need for the development of harmonized guidelines with a core set of questions  Optional policy-relevant questions can be added for policy monitoring Methodology for country benchmarking:  Preferably common sampling methodology: size cut-off point, industry coverage, …  Compulsory common (length of) reference period, and participation mode

27. June 11 2008DEIP, Amman June, 10-12 200827 5. Opportunities and limitations Dissemination of non-aggregated micro-data is crucial Assessing trends: need for panel data Need of involvement of stakeholders from the start Need for strong coordination mechanism

28. June 11 2008DEIP, Amman June, 10-12 200828 Useful links: For a download of the CIS-4 questionnaire: http://www.berr.gov.uk/files/file9688.pdf Oslo Manual: http://epp.eurostat.ec.europa.eu/cache/ITY_PUBLIC/OSLO/EN/OSL O-EN.PDF Bogotá manual: http://www.ricyt.edu.ar/interior//difusion/pubs/bogota/bogota_eng.pdf NEPAD study: http://www.nepadst.org/doclibrary/pdfs/innopolicy_aug2004.pdf