1. Maastricht Economic and Social Research and Training Centre on Innovation and Technology (UNU-MERIT) and FH Aachen, University of Applied Sciences Science, Technology and Innovation (STI) Indicators for Evaluation Professor Norbert Janz DEIP Montevideo 30 March 2009

2. DEIP Montevideo 2009 - Indicators 30 March 2009 - 2 STI Indicators Measuring Innovation: Manuals upon Manuals Typology of Indicators Innovation Surveys Use of Innovation Data

3. DEIP Montevideo 2009 - Indicators 30 March 2009 - 3 Measuring Innovation: Manuals upon Manuals OECD Manuals  Frascati manualHow to measure R&D  Oslo manual How to measure innovation  Canberra manualHow to use human resource data to measure innovation  Patent StatisticsHow to use patent data to manualmeasure innovation Non-OECD Manuals  Bogota manualHow to measure innovation in Latin American countries  Santiago manualInternationalisation of R&D …  NEPAD studyHow to measure innovation in African countries

4. DEIP Montevideo 2009 - Indicators 30 March 2009 - 4 Frascati Manual History of the manual  First edition 1963, last revision 2002 Scope of the manual  R&D statistics Sampling approach  Business enterprises, government, non-profit, higher education, hospitals/health care  Survey of all enterprises known or assumed to perform R&D  Recently, samples of service industries

5. DEIP Montevideo 2009 - Indicators 30 March 2009 - 5 Frascati Manual: Main Indicators R&D personnel  Data on head counts, full time equivalents (FTE)  Classified by occupation, qualification  Breakdown by sector (Intramural) R&D expenditure  Current costs, capital expenditure  Breakdown by sources of funds  Domestic: GERD, National: GNERD Other indicators  GBOARD (Governmental Budget) Remark: National, regional, sectoral aggregates

6. DEIP Montevideo 2009 - Indicators 30 March 2009 - 6 Frascati Manual: GERD by Region in Europe

7. DEIP Montevideo 2009 - Indicators 30 March 2009 - 7 Frascati R&D Definition Basic Research  New knowledge without application or use in view Applied Research  New knowledge with practical aim or objective Experimental Development  Existing knowledge directed to new applications or improving applications substantially

8. DEIP Montevideo 2009 - Indicators 30 March 2009 - 8 Oslo Manual History of the manual  First edition 1992, last revision 2005 Scope of the manual  Innovation statistics Sampling approach  Business enterprises (at least 10 employees)  Stratified random sample (some census)  Repeated cross-section, some panel surveys

9. DEIP Montevideo 2009 - Indicators 30 March 2009 - 9 Oslo Manual: Main Indicators Innovative / innovating firms Types of innovative activities  Intramural and extramural R&D  Acquisition of machinery etc.  Other Preparations by type of activity Innovation expenditure Impact of innovation  Sales with product innovation Objectives of and Obstacles to innovations Linkages in innovation  Information sources  Co-operation Remark: Shares of firms using sampling weights

10. DEIP Montevideo 2009 - Indicators 30 March 2009 - 10 Oslo Manual: Protection Methods by Type Source: Eurostat, Statistics in Focus 91/2007

11. DEIP Montevideo 2009 - Indicators 30 March 2009 - 11 Oslo Innovation Definition Product innovation  NSI characteristics or intended uses Process innovation  NSI production / delivery methods  Techniques, equipment, software Marketing innovation (recently)  NSI design, packaging, placement, promotion, pricing Organisational innovation (recently)  NSI business practice, workplace organisations or external relations

12. DEIP Montevideo 2009 - Indicators 30 March 2009 - 12 Oslo Innovation Definition 2 Period under review  Sampling period: often more than one year What is new?  New to the firm: Innovation or diffusion?  New to the market: Regional, national, global?  New to the world: Who knows that? Again, what is new?  New: How new to be new?  Significantly impr.: What is significantly?

13. DEIP Montevideo 2009 - Indicators 30 March 2009 - 13 Canberra Manual History of the manual  First edition 1995 Scope of the manual  Human Resources in Science and Technology (HRST) Sampling approach  Person (individual) in household surveys, population censuses and administrative records

14. DEIP Montevideo 2009 - Indicators 30 March 2009 - 14 Canberra Manual: HRST in Europe

15. DEIP Montevideo 2009 - Indicators 30 March 2009 - 15 Patent Statistics Manual History of the manual  First edition 1994, last revision 2009 Scope of the manual  Patent indicators Sampling approach  Patent derived data mainly using patent databases

16. DEIP Montevideo 2009 - Indicators 30 March 2009 - 16 EU Member States EPO Patent Applications 2003

17. DEIP Montevideo 2009 - Indicators 30 March 2009 - 17 Patent Statistics Manual: Main Indicators Number of patents (patent counts)  Patents filed (applications)  Patents granted or registered  Classified by country, region, industry, institutions, inventors, technology field Citation based indicators (weighted patent counts)  Backward and forward citations  Current Impact Index: Patents of the last 5 years cited this year  Citation Performance Index: Number of patents in the most highly cited Patent Values

18. DEIP Montevideo 2009 - Indicators 30 March 2009 - 18 EU Member States EPO Patent Applications 2003

19. DEIP Montevideo 2009 - Indicators 30 March 2009 - 19 Bogota Manual History of the manual  Regional manual, first edition 2001 Scope of the manual  Innovation statistics for Latin America and the Caribbean Countries Sampling approach  Similar to Oslo recommendations  More detailed industry strata (ISIC 3 or 4)

20. DEIP Montevideo 2009 - Indicators 30 March 2009 - 20 Bogota Manual: Main Indicators Innovation focus  Innovation process instead of result  Innovation system approach more pronounced Innovation efforts  Embodied and disembodied technology Innovation results and innovation goals Innovation funding (sources of funds) Innovation linkages  Frequencies by type, agent, institution  Degree of satisfaction Innovation policy assessment  Knowledge of institutions and programs  Assessment of programs

21. DEIP Montevideo 2009 - Indicators 30 March 2009 - 21 NEPAD Innovation Survey Design History of the NEPAD study  First publication, 2004  Intergovernmental committee, 2007 Scope of the study  Policy relevant innovation surveys  Implementation in Africa  Training Module Additional ore more detailed innovation indicators  Learning process  Innovation-related policies: importance, impact

22. DEIP Montevideo 2009 - Indicators 30 March 2009 - 22 What Is an Indicator? “Indicator Indicator” or “Greater Honeyguide”

23. DEIP Montevideo 2009 - Indicators 30 March 2009 - 23 What Was an Indicator again? Indicator  Latin “indicare”: to indicate, to show, to be a sign of, to give notice of  Tool indicating facts/information in general not directly measurable Economic indicator  An economic indicator is a statistic about the economy allowing analyses of economic performance and predictions of future performance  Leading, lagging and coincident indicators

24. DEIP Montevideo 2009 - Indicators 30 March 2009 - 24 Typology of Indicators Traditional vs. new innovation indicators  R&D survey/Patent statistics based indicators  Innovation survey based indicators etc. Input, Throughput and Output Indicators  Linear view of innovation  Input Process Output Simple, Complex and Composite Indicators  Number of indicators involved and how they are combined

25. DEIP Montevideo 2009 - Indicators 30 March 2009 - 25 Traditional and New Indicators Traditional Innovation Indicators  Innovation indicators related to R&D: Number of R&D employees Amount of R&D expenditure  Innovation indicators related to patents: Number of patents granted Number of patent applications New Innovation Indicators  Innovation indicators related to broader concepts of innovation  Literature based innovation indicators

26. DEIP Montevideo 2009 - Indicators 30 March 2009 - 26 Inputoriented Indicators R&D based input indicators  R&D personnel: share in total personnel  R&D expenditure: share in GDP (Broader) innovation based input indicators  Innovation expenditure: share in GDP  ICT expenditure: share in GDP Patent indicators  Patent applications, grants, stock: per population

27. DEIP Montevideo 2009 - Indicators 30 March 2009 - 27 R&D Expenditure Source: DST South Africa (2007)

28. DEIP Montevideo 2009 - Indicators 30 March 2009 - 28 R&D Personnel Source: DST South Africa (2007)

29. DEIP Montevideo 2009 - Indicators 30 March 2009 - 29 Patent Applications Source: OECD Patent Statistics Manual, 2009

30. DEIP Montevideo 2009 - Indicators 30 March 2009 - 30 Output Indicators Direct output indicators  Scientific publications: (weighted) number of articles, pages  Product, process, organizational, marketing innovations: share of firms  Innovative sales: share in total sales Indirect output indicators  Changes in profits, costs, productivity, employment, market shares

31. DEIP Montevideo 2009 - Indicators 30 March 2009 - 31 Scientific Publications Source: OECD, STI Scoreboard 2007

32. DEIP Montevideo 2009 - Indicators 30 March 2009 - 32 Product Innovations Source: OECD, STI Scoreboard 2007

33. DEIP Montevideo 2009 - Indicators 30 March 2009 - 33 Innovative Sales Source: OECD, STI Scoreboard 2007

34. DEIP Montevideo 2009 - Indicators 30 March 2009 - 34 And another Typology of Innovation Indicators Simple Indicators  Building mostly on a single variable  e.g. share of innovative firms in an industry etc. Complex Indicators  Combining variables, but not aggregating  e.g. share of firms with linkages to universities in innovative firms etc. Composite Indicators  Attempt to aggregate variables  e.g. degree of openness  Most prominent: IQ (Intelligence Quotient)  Mostly forgotten: Exam grade

35. DEIP Montevideo 2009 - Indicators 30 March 2009 - 35 Complex Innovation Indicator: Effects of Innovation Source: Eurostat, Statistics in Focus 113/2007

36. DEIP Montevideo 2009 - Indicators 30 March 2009 - 36 Complex Innovation Indicator: Sources of Information Source: Eurostat, Statistics in Focus 81/2007

37. DEIP Montevideo 2009 - Indicators 30 March 2009 - 37 Composite Indicators Definition of composite indicator  Individual indicators compiled into single index  Ideally based on a model  Measuring multidimensional concepts which cannot be measured with a single indicator  E.g. competitiveness OECD Handbook of Constructing Composite Indicators, 2008

38. DEIP Montevideo 2009 - Indicators 30 March 2009 - 38 Composite Indicators: Pros and Cons (OECD Handbook)

39. DEIP Montevideo 2009 - Indicators 30 March 2009 - 39 EIS 2008: Innovation Performance ModerateinnovatorsInnovationfollowersInnovationleadersCatching-upcountries

40. DEIP Montevideo 2009 - Indicators 30 March 2009 - 40 Dimensions of Composite Innovation Indicators EIS 2008

41. DEIP Montevideo 2009 - Indicators 30 March 2009 - 41 Comparing Countries with Composite Indicators: EIS 2006

42. DEIP Montevideo 2009 - Indicators 30 March 2009 - 42 Comparing Estonian and Slovenian Sub-Indexes Examples for Hugo Hollanders, 2009. Thanks Hugo!

43. DEIP Montevideo 2009 - Indicators 30 March 2009 - 43 Composite Indicators Revisited “Everything should be made as simple as possible, but no simpler.” (Albert Einstein, 1934)

44. DEIP Montevideo 2009 - Indicators 30 March 2009 - 44 Characteristics of a Good Survey Target and frame population  Target population is well defined  Frame population has good coverage: only minor undercoverage and overcoverage Stratification and Sampling  Stratification criteria are observable in frame and target population  Representative sampling: random sample  Unbalanced sampling is well motivated

45. DEIP Montevideo 2009 - Indicators 30 March 2009 - 45 Characteristics of a Good Survey Questionnaire  Questions are based on research questions  Questions are not suggestive  Questions uses diction of the correspondent  Definitions are short and near to the questions  Complicated questions are accompanied by a list of good examples  As few open questions as possible  Scales are balanced  Information asked is available  Questions should be answerable by one person  Questionnaire is pretested

46. DEIP Montevideo 2009 - Indicators 30 March 2009 - 46 Characteristics of a Good Survey Data cleansing  Data consistency has been checked Non-response  Key questions are asked to non-respondents in an additional survey  Non-response bias is tested for Expansions  Weighting factors are based on sampling probabilities  Weighting factors are adjusted for non-response  Weighting factors are possibly adjusted for non- response bias  Missing values / items are properly imputed

47. DEIP Montevideo 2009 - Indicators 30 March 2009 - 47 Use of Innovation Survey Data Users of Innovation Survey Data Political Use: Innovation Reporting Academic Use: Innovation Analyses Requirements for Innovation Surveys

48. DEIP Montevideo 2009 - Indicators 30 March 2009 - 48 Users of the Innovation Surveys: Case of Germany Main external users:  German federal government (esp. STE)  European Commission, OECD  Participating Firms Other external users:  Federal state governments (larger states)  Industrial associations  Academic research (universities, non-profit research institutes, incl. PhDs) Internal users:  Contract research  Academic research (incl. PhDs)

49. DEIP Montevideo 2009 - Indicators 30 March 2009 - 49 Aspects of German Innovation Survey Annual innovation survey  Bi-annually extended questionnaire with focus on a special topic (8-16 pages)  Bi-annually short questionnaire on core indicators (4 pages) Panel innovation survey  Sampling the same set of firms every year  Bi-annually adjustment of the sample Cutting sample for firm failure etc. Extending sample with newly formed firms Data links at micro level  Patent databases (German and European)  Database of R&D subsidies

50. DEIP Montevideo 2009 - Indicators 30 March 2009 - 50 Innovation Reports for Policy Innovation reporting  Indicator reports (annually)  Background reports (biannually)  Sectoral reports (annually) Expert Commission Research and Innovation  Expert Report on Research, Innovation and Technological Performance  Studies on the German Innovation System European reporting  European Innovation Scoreboard (EIS)  and much more

51. DEIP Montevideo 2009 - Indicators 30 March 2009 - 51 Indicator Reports Users  German federal government (esp. BMBF)  Published printed and online Reporting strategy  Short report (16-20 pages)  Highlighting main developments Main contents:  Development of innovation indicators over time (CIS-type core indicators)  Projections for the current and following year (innovation expenditure, innovation intensity)

52. DEIP Montevideo 2009 - Indicators 30 March 2009 - 52 Cost Reduction through Process Innovations

53. DEIP Montevideo 2009 - Indicators 30 March 2009 - 53 Sectoral Reports Users  Participating firms: printed version  Others (time delay): internet version 21 sectoral reports (only in German)  12 for manufacturing industries  8 for service sector industries  1 for mining, energy, water supply Brief contents (4 pages):  Ranking of industries  Development of innovation indicators  Benchmarking of best-practice firms (biannually)

54. DEIP Montevideo 2009 - Indicators 30 March 2009 - 54 Sectoral Reports: Example of Automotive Industry

55. DEIP Montevideo 2009 - Indicators 30 March 2009 - 55 Expert Commission Annual Expert Report  Immediate and intermediate need for action  Core topics this year  Structure and Trends: Education, R&D, Innovation, SMEs, Formations, Patents, Publications Studies on the Innovation System  R&D and Knowledge intensive sectors  Innovation behaviour and finance  International R&D of German firms  and 10 more

56. DEIP Montevideo 2009 - Indicators 30 March 2009 - 56 Expert Commission Report: Example

57. DEIP Montevideo 2009 - Indicators 30 March 2009 - 57 Expert Commission Report: Example 2

58. DEIP Montevideo 2009 - Indicators 30 March 2009 - 58 Academic Research: Questions Determinants of innovation behaviour  Technological opportunities depending on absorptive capacity Determinants of innovation success  Continuous R&D activity or R&D department  Co-operation esp. with customers Innovation and employment  Product innovation creating  Process innovation reducing, but less clear  Skill bias of technological change, esp. for the service sector

59. DEIP Montevideo 2009 - Indicators 30 March 2009 - 59 Academic Research: Examples Internationalisation of R&D-co-opetition An empirical analysis of the effects of patents and secrecy on knowledge spillovers Persistence of innovation Capital control, debt financing and innovation activity Employment effects of different innovation activities... could be completed with 75-100 other topics

60. DEIP Montevideo 2009 - Indicators 30 March 2009 - 60 Scientific Use File Micro-aggregation Factual anonymisation  Multiplicative errors: e.g. turnover, employees  Intensities, rates: e.g. innovation expenditure  Truncation: i.e. upper limits for extreme values, e.g. innovation intensity to 35%  Grouping: i.e. range (ordinal) instead of value, e.g. innovative sales between 5 and 10 %  Aggregation: i.e. less detailed classification, e.g. location of innovation partners  Withholding information: e.g. some variables for banks, insurances

61. DEIP Montevideo 2009 - Indicators 30 March 2009 - 61 Scientific Use File (2) Availability of data  All waves with 3 years time delay  Possibility to build a panel  Contract specifying Non-profit academic research Research topic Names of researcher Access to original micro data at surveying institution or statistical office

62. DEIP Montevideo 2009 - Indicators 30 March 2009 - 62 Education Use File Complete anonymisation  Artificial data set  Generated by statistical re-sampling  No large firms contained Availability  Single cross-sections 1999 and 2000  Contract specifying use  Download from ftp-server Teaching purpose  Econometric courses on micro data  Courses on empirical economics of innovation

63. Maastricht Economic and Social Research and Training Centre on Innovation and Technology (UNU-MERIT) and FH Aachen, University of Applied Sciences How to summarize this?