Maastricht Economic and Social Research and Training Centre on Innovation and Technology (UNU-MERIT) and FH Aachen, University of Applied Sciences Science, Technology and Innovation (STI) Indicators for Evaluation Professor Norbert Janz DEIP Montevideo 30 March 2009

DEIP Montevideo Indicators 30 March STI Indicators Measuring Innovation: Manuals upon Manuals Typology of Indicators Innovation Surveys Use of Innovation Data

DEIP Montevideo Indicators 30 March Measuring Innovation: Manuals upon Manuals OECD Manuals  Frascati manualHow to measure R&D  Oslo manual How to measure innovation  Canberra manualHow to use human resource data to measure innovation  Patent StatisticsHow to use patent data to manualmeasure innovation Non-OECD Manuals  Bogota manualHow to measure innovation in Latin American countries  Santiago manualInternationalisation of R&D …  NEPAD studyHow to measure innovation in African countries

DEIP Montevideo Indicators 30 March Frascati Manual History of the manual  First edition 1963, last revision 2002 Scope of the manual  R&D statistics Sampling approach  Business enterprises, government, non-profit, higher education, hospitals/health care  Survey of all enterprises known or assumed to perform R&D  Recently, samples of service industries

DEIP Montevideo Indicators 30 March Frascati Manual: Main Indicators R&D personnel  Data on head counts, full time equivalents (FTE)  Classified by occupation, qualification  Breakdown by sector (Intramural) R&D expenditure  Current costs, capital expenditure  Breakdown by sources of funds  Domestic: GERD, National: GNERD Other indicators  GBOARD (Governmental Budget) Remark: National, regional, sectoral aggregates

DEIP Montevideo Indicators 30 March Frascati Manual: GERD by Region in Europe

DEIP Montevideo Indicators 30 March Frascati R&D Definition Basic Research  New knowledge without application or use in view Applied Research  New knowledge with practical aim or objective Experimental Development  Existing knowledge directed to new applications or improving applications substantially

DEIP Montevideo Indicators 30 March Oslo Manual History of the manual  First edition 1992, last revision 2005 Scope of the manual  Innovation statistics Sampling approach  Business enterprises (at least 10 employees)  Stratified random sample (some census)  Repeated cross-section, some panel surveys

DEIP Montevideo Indicators 30 March Oslo Manual: Main Indicators Innovative / innovating firms Types of innovative activities  Intramural and extramural R&D  Acquisition of machinery etc.  Other Preparations by type of activity Innovation expenditure Impact of innovation  Sales with product innovation Objectives of and Obstacles to innovations Linkages in innovation  Information sources  Co-operation Remark: Shares of firms using sampling weights

DEIP Montevideo Indicators 30 March Oslo Manual: Protection Methods by Type Source: Eurostat, Statistics in Focus 91/2007

DEIP Montevideo Indicators 30 March Oslo Innovation Definition Product innovation  NSI characteristics or intended uses Process innovation  NSI production / delivery methods  Techniques, equipment, software Marketing innovation (recently)  NSI design, packaging, placement, promotion, pricing Organisational innovation (recently)  NSI business practice, workplace organisations or external relations

DEIP Montevideo Indicators 30 March Oslo Innovation Definition 2 Period under review  Sampling period: often more than one year What is new?  New to the firm: Innovation or diffusion?  New to the market: Regional, national, global?  New to the world: Who knows that? Again, what is new?  New: How new to be new?  Significantly impr.: What is significantly?

DEIP Montevideo Indicators 30 March Canberra Manual History of the manual  First edition 1995 Scope of the manual  Human Resources in Science and Technology (HRST) Sampling approach  Person (individual) in household surveys, population censuses and administrative records

DEIP Montevideo Indicators 30 March Canberra Manual: HRST in Europe

DEIP Montevideo Indicators 30 March Patent Statistics Manual History of the manual  First edition 1994, last revision 2009 Scope of the manual  Patent indicators Sampling approach  Patent derived data mainly using patent databases

DEIP Montevideo Indicators 30 March EU Member States EPO Patent Applications 2003

DEIP Montevideo Indicators 30 March Patent Statistics Manual: Main Indicators Number of patents (patent counts)  Patents filed (applications)  Patents granted or registered  Classified by country, region, industry, institutions, inventors, technology field Citation based indicators (weighted patent counts)  Backward and forward citations  Current Impact Index: Patents of the last 5 years cited this year  Citation Performance Index: Number of patents in the most highly cited Patent Values

DEIP Montevideo Indicators 30 March EU Member States EPO Patent Applications 2003

DEIP Montevideo Indicators 30 March Bogota Manual History of the manual  Regional manual, first edition 2001 Scope of the manual  Innovation statistics for Latin America and the Caribbean Countries Sampling approach  Similar to Oslo recommendations  More detailed industry strata (ISIC 3 or 4)

DEIP Montevideo Indicators 30 March Bogota Manual: Main Indicators Innovation focus  Innovation process instead of result  Innovation system approach more pronounced Innovation efforts  Embodied and disembodied technology Innovation results and innovation goals Innovation funding (sources of funds) Innovation linkages  Frequencies by type, agent, institution  Degree of satisfaction Innovation policy assessment  Knowledge of institutions and programs  Assessment of programs

DEIP Montevideo Indicators 30 March NEPAD Innovation Survey Design History of the NEPAD study  First publication, 2004  Intergovernmental committee, 2007 Scope of the study  Policy relevant innovation surveys  Implementation in Africa  Training Module Additional ore more detailed innovation indicators  Learning process  Innovation-related policies: importance, impact

DEIP Montevideo Indicators 30 March What Is an Indicator? “Indicator Indicator” or “Greater Honeyguide”

DEIP Montevideo Indicators 30 March What Was an Indicator again? Indicator  Latin “indicare”: to indicate, to show, to be a sign of, to give notice of  Tool indicating facts/information in general not directly measurable Economic indicator  An economic indicator is a statistic about the economy allowing analyses of economic performance and predictions of future performance  Leading, lagging and coincident indicators

DEIP Montevideo Indicators 30 March Typology of Indicators Traditional vs. new innovation indicators  R&D survey/Patent statistics based indicators  Innovation survey based indicators etc. Input, Throughput and Output Indicators  Linear view of innovation  Input Process Output Simple, Complex and Composite Indicators  Number of indicators involved and how they are combined

DEIP Montevideo Indicators 30 March Traditional and New Indicators Traditional Innovation Indicators  Innovation indicators related to R&D: Number of R&D employees Amount of R&D expenditure  Innovation indicators related to patents: Number of patents granted Number of patent applications New Innovation Indicators  Innovation indicators related to broader concepts of innovation  Literature based innovation indicators

DEIP Montevideo Indicators 30 March Inputoriented Indicators R&D based input indicators  R&D personnel: share in total personnel  R&D expenditure: share in GDP (Broader) innovation based input indicators  Innovation expenditure: share in GDP  ICT expenditure: share in GDP Patent indicators  Patent applications, grants, stock: per population

DEIP Montevideo Indicators 30 March R&D Expenditure Source: DST South Africa (2007)

DEIP Montevideo Indicators 30 March R&D Personnel Source: DST South Africa (2007)

DEIP Montevideo Indicators 30 March Patent Applications Source: OECD Patent Statistics Manual, 2009

DEIP Montevideo Indicators 30 March Output Indicators Direct output indicators  Scientific publications: (weighted) number of articles, pages  Product, process, organizational, marketing innovations: share of firms  Innovative sales: share in total sales Indirect output indicators  Changes in profits, costs, productivity, employment, market shares

DEIP Montevideo Indicators 30 March Scientific Publications Source: OECD, STI Scoreboard 2007

DEIP Montevideo Indicators 30 March Product Innovations Source: OECD, STI Scoreboard 2007

DEIP Montevideo Indicators 30 March Innovative Sales Source: OECD, STI Scoreboard 2007

DEIP Montevideo Indicators 30 March And another Typology of Innovation Indicators Simple Indicators  Building mostly on a single variable  e.g. share of innovative firms in an industry etc. Complex Indicators  Combining variables, but not aggregating  e.g. share of firms with linkages to universities in innovative firms etc. Composite Indicators  Attempt to aggregate variables  e.g. degree of openness  Most prominent: IQ (Intelligence Quotient)  Mostly forgotten: Exam grade

DEIP Montevideo Indicators 30 March Complex Innovation Indicator: Effects of Innovation Source: Eurostat, Statistics in Focus 113/2007

DEIP Montevideo Indicators 30 March Complex Innovation Indicator: Sources of Information Source: Eurostat, Statistics in Focus 81/2007

DEIP Montevideo Indicators 30 March Composite Indicators Definition of composite indicator  Individual indicators compiled into single index  Ideally based on a model  Measuring multidimensional concepts which cannot be measured with a single indicator  E.g. competitiveness OECD Handbook of Constructing Composite Indicators, 2008

DEIP Montevideo Indicators 30 March Composite Indicators: Pros and Cons (OECD Handbook)

DEIP Montevideo Indicators 30 March EIS 2008: Innovation Performance ModerateinnovatorsInnovationfollowersInnovationleadersCatching-upcountries

DEIP Montevideo Indicators 30 March Dimensions of Composite Innovation Indicators EIS 2008

DEIP Montevideo Indicators 30 March Comparing Countries with Composite Indicators: EIS 2006

DEIP Montevideo Indicators 30 March Comparing Estonian and Slovenian Sub-Indexes Examples for Hugo Hollanders, Thanks Hugo!

DEIP Montevideo Indicators 30 March Composite Indicators Revisited “Everything should be made as simple as possible, but no simpler.” (Albert Einstein, 1934)

DEIP Montevideo Indicators 30 March Characteristics of a Good Survey Target and frame population  Target population is well defined  Frame population has good coverage: only minor undercoverage and overcoverage Stratification and Sampling  Stratification criteria are observable in frame and target population  Representative sampling: random sample  Unbalanced sampling is well motivated

DEIP Montevideo Indicators 30 March Characteristics of a Good Survey Questionnaire  Questions are based on research questions  Questions are not suggestive  Questions uses diction of the correspondent  Definitions are short and near to the questions  Complicated questions are accompanied by a list of good examples  As few open questions as possible  Scales are balanced  Information asked is available  Questions should be answerable by one person  Questionnaire is pretested

DEIP Montevideo Indicators 30 March Characteristics of a Good Survey Data cleansing  Data consistency has been checked Non-response  Key questions are asked to non-respondents in an additional survey  Non-response bias is tested for Expansions  Weighting factors are based on sampling probabilities  Weighting factors are adjusted for non-response  Weighting factors are possibly adjusted for non- response bias  Missing values / items are properly imputed

DEIP Montevideo Indicators 30 March Use of Innovation Survey Data Users of Innovation Survey Data Political Use: Innovation Reporting Academic Use: Innovation Analyses Requirements for Innovation Surveys

DEIP Montevideo Indicators 30 March Users of the Innovation Surveys: Case of Germany Main external users:  German federal government (esp. STE)  European Commission, OECD  Participating Firms Other external users:  Federal state governments (larger states)  Industrial associations  Academic research (universities, non-profit research institutes, incl. PhDs) Internal users:  Contract research  Academic research (incl. PhDs)

DEIP Montevideo Indicators 30 March Aspects of German Innovation Survey Annual innovation survey  Bi-annually extended questionnaire with focus on a special topic (8-16 pages)  Bi-annually short questionnaire on core indicators (4 pages) Panel innovation survey  Sampling the same set of firms every year  Bi-annually adjustment of the sample Cutting sample for firm failure etc. Extending sample with newly formed firms Data links at micro level  Patent databases (German and European)  Database of R&D subsidies

DEIP Montevideo Indicators 30 March Innovation Reports for Policy Innovation reporting  Indicator reports (annually)  Background reports (biannually)  Sectoral reports (annually) Expert Commission Research and Innovation  Expert Report on Research, Innovation and Technological Performance  Studies on the German Innovation System European reporting  European Innovation Scoreboard (EIS)  and much more

DEIP Montevideo Indicators 30 March Indicator Reports Users  German federal government (esp. BMBF)  Published printed and online Reporting strategy  Short report (16-20 pages)  Highlighting main developments Main contents:  Development of innovation indicators over time (CIS-type core indicators)  Projections for the current and following year (innovation expenditure, innovation intensity)

DEIP Montevideo Indicators 30 March Cost Reduction through Process Innovations

DEIP Montevideo Indicators 30 March Sectoral Reports Users  Participating firms: printed version  Others (time delay): internet version 21 sectoral reports (only in German)  12 for manufacturing industries  8 for service sector industries  1 for mining, energy, water supply Brief contents (4 pages):  Ranking of industries  Development of innovation indicators  Benchmarking of best-practice firms (biannually)

DEIP Montevideo Indicators 30 March Sectoral Reports: Example of Automotive Industry

DEIP Montevideo Indicators 30 March Expert Commission Annual Expert Report  Immediate and intermediate need for action  Core topics this year  Structure and Trends: Education, R&D, Innovation, SMEs, Formations, Patents, Publications Studies on the Innovation System  R&D and Knowledge intensive sectors  Innovation behaviour and finance  International R&D of German firms  and 10 more

DEIP Montevideo Indicators 30 March Expert Commission Report: Example

DEIP Montevideo Indicators 30 March Expert Commission Report: Example 2

DEIP Montevideo Indicators 30 March Academic Research: Questions Determinants of innovation behaviour  Technological opportunities depending on absorptive capacity Determinants of innovation success  Continuous R&D activity or R&D department  Co-operation esp. with customers Innovation and employment  Product innovation creating  Process innovation reducing, but less clear  Skill bias of technological change, esp. for the service sector

DEIP Montevideo Indicators 30 March Academic Research: Examples Internationalisation of R&D-co-opetition An empirical analysis of the effects of patents and secrecy on knowledge spillovers Persistence of innovation Capital control, debt financing and innovation activity Employment effects of different innovation activities... could be completed with other topics

DEIP Montevideo Indicators 30 March Scientific Use File Micro-aggregation Factual anonymisation  Multiplicative errors: e.g. turnover, employees  Intensities, rates: e.g. innovation expenditure  Truncation: i.e. upper limits for extreme values, e.g. innovation intensity to 35%  Grouping: i.e. range (ordinal) instead of value, e.g. innovative sales between 5 and 10 %  Aggregation: i.e. less detailed classification, e.g. location of innovation partners  Withholding information: e.g. some variables for banks, insurances

DEIP Montevideo Indicators 30 March Scientific Use File (2) Availability of data  All waves with 3 years time delay  Possibility to build a panel  Contract specifying Non-profit academic research Research topic Names of researcher Access to original micro data at surveying institution or statistical office

DEIP Montevideo Indicators 30 March Education Use File Complete anonymisation  Artificial data set  Generated by statistical re-sampling  No large firms contained Availability  Single cross-sections 1999 and 2000  Contract specifying use  Download from ftp-server Teaching purpose  Econometric courses on micro data  Courses on empirical economics of innovation

Maastricht Economic and Social Research and Training Centre on Innovation and Technology (UNU-MERIT) and FH Aachen, University of Applied Sciences How to summarize this?