ICT IMPACT INDICATORS: LINKING DATA FROM DIFFERENT SOURCES Partnership Event, Geneva Wednesday May 28 th 2008 An EU initiative with 13 National Statistics Offices Tony Clayton UK Office for National Statistics

Developing Impact Indicators Agenda Background to ICT indicator development Limits to surveys – what we actually measure Impact assessment; evidence from firms Using firm level data to develop indicators – the advantages …. and the challenges to overcome

Background to ICT indicators OECD ‘diffusion’ model, based on linear view Readiness; ability of firms, individuals to adopt technology Intensity / use; measures of the proportion of firms, individuals who adopt, and the amount of use Impact; changes in behaviour, and in economic structure and performance as result of use

Background to EU ICT indicators E- Europe indicators to 2005 focused on use Citizens -access or ownership of technology -use of internet, and applications Enterprises -access to or ownership of technology -investment in networks / availability of broadband -use of internet and e-business links Government -services available on line and degree of take-up -investment in specific areas, e.g. education Impact? -Only e-commerce, and some individual metrics, to represent changed behaviour resulting from use

Background to EU ICT indicators i2010 more focused on impacts EU Information space -development of broadband availability, take-up, speed and pricing -multiplatform access in firms Advanced services -focused on household / individual uses of range of internet services Security -incidence of problems and solutions -attitudes to risk / trust and confidence Impact -ICT investment, use and impact on growth / productivity ICT innovation -ICT R&D and effects on business transformation

Background to ICT indicators ‘Impacts’ now need to reflect innovation in ICT use Innovation no longer a linear process -depends on networks / feedback / co-creation ICT as ‘general purpose technology’ -major impacts on economic performance based on use -dependent on range of complementary investments Innovative uses particularly strong in services -majority of ICT investment in market services -returns to IT investment strongest in services -.. where growth, productivity and innovation are hardest to measure Pervasive presence of ICT -impacts too important not to be quantified, as part of economic measurement

Limits to surveys Measuring ICT investment is hard to do Much existing data doesn’t reflect ICT properly as capital -macroeconomic treatment of software investment is inconsistent between countries -firm level source data misses out large parts of ‘own account’ IT software investment – because the respondents can’t quantify it -as ICT service purchases, and IT enabled services, substitute for ‘own investment’, can we pick up the effects? -(Finland results for this study suggests productivity gains from IT outsourcing will ensure this grows) Measurement differences can be very significant …..

Limits to surveys … and macro data * 2002 Economist Feb 2006

Limits to surveys What else have we learned about indicators? Surveys about ‘self assessed’ impacts not reliable -assessment of payback from ICT can be inconsistent -links to actual outcomes from production surveys do a better job Questions on technology less useful than what is done with it -technology changes, and respondents don’t always understand it -effects on behaviour may be a better guide to economic effects ICT indicators are sensitive to structural differences -industry characteristics can be stronger than national variations -member state estimates may not be comparable, in spite of similar methods.

Who we are 8 NSIs directly involved in developing research, metadata and method - UK, Sweden, Finland, Denmark, Italy, France, Austria, Netherlands Slovenia involved through Ljubljana University 4 more joined to use method and contribute to analysis - Germany, Norway, Czech Republic and Ireland Academic support from Free University of Amsterdam

What we’ve done Firm level analysis within countries, linking surveys on ICT use, output, productivity, IT investment, skills, innovation … Core variables from common EU ICT use survey ‘Add-on’ analysis in countries with extra survey data Built metadata system to aggregate data for industries / countries consistently Country / industry indicators built using survey data Able to bring in other data from National Accounts Analysis across different environments New method for indicators

From firm data to macro indicators What we’ve done ….. Firm level analysis within countries, linking surveys on ICT use, output, etc to identify … -core variables from common EU ICT use survey -‘add-on’ analysis in countries with extra survey data Built metadata framework to aggregate this data for industries / countries on consistent basis -country / industry indicators built using survey data -able to bring in other data from National Accounts -analysis across different environments -new method for constructing indicators

From firm data to macro indicators Industry / country aggregates help We can aggregate to comparable intermediate aggregate data -metadata used to generate completely comparable aggregated data by country / industry / firm size / ownership / market characteristics etc.. -then we can analyse at this level From ‘industry / country’ analysis we get -ability to do analysis across different national policy environments. -ability to bring in data not available at firm level -quantification of effects at macro level – £billion more persuasive than a regression coefficient!

Researcher Policy Question Research Design Program Code Publication Network Metadata Network members Cross-country Tables NSOs Provision of metadata. Approval of access. Disclosure analysis From firm data to macro indicators

Assemble the data -put production survey, ICT use survey and business register datasets in single research environment Construct the metadata -describe each survey in standard terms; variable names, format, range, -test linkages between units in surveys using register identifiers Run analytical code on datasets, using metadata to access and interpret -code uses SAS or STATA operating within NSI -produces results / aggregates which can be shared -disclosure checking before use (can build into code) What NSIs have to do to join ….. From firm data to macro indicators

ICT metrics Variables for most countries‘Add-on’ variables Firm uses computers % workers using computers Firm uses internet % of workers with access to internet Firm has website Firm orders via internet (or EDI etc) % of orders (or value) Firm sells via internet (or EDI etc) % of sales (or value) Firm has intranet % of workers with access to intranet Firm has fast internet access % of workers with access to fast internet access Use of electronic business processes Mobility of access to technology Mobility of workers accessing technology IT investment (or expenditure) at firm level From firm data to macro indicators

Contextual variables / complements to IT Variables for most countries‘Add-on’ Variables Firm size Firm age Employment Payroll Capital Stock Ownership (local or international) Outsource ratios (measured as purchases / sales or similar) Measures of labour ‘quality’ (measured in terms of pay per person) Industry characteristics (growth, entry / exit, firm share turbulence) Multinational organisation of firms (from FDI and ownership data) Employee skills (from linked employer – employee datasets) Changes in organisation (from some local surveys, or CIS) Measures of innovation (from CIS) From firm data to macro indicators

Measures of Impact LevelsGrowth rates Productivity measured as - Gross sales / employee (vs industry) - V.A. / employee (vs industry) Productivity measured in terms of Multi-Factor Productivity (for countries which have firm level data on investment, IT investment, labour and other inputs) Productivity change measured as - Gross sales / employee (vs industry) - V.A. / employee (vs industry) Growth rates measured for: - Gross sales (deflated by industry PPI) - Market share of IT using firms in industry - Value added (double deflated where possible) - Employment, or share of employment in industry From firm data to macro indicators

countryElecomMexElecDISTRFINBUTOT SWE NOR FIN GBR NLD SLO 5776 GER 75 CZE AUT ITA IRE From firm data to macro indicators …and what it can deliver (cont) (example; index of fast-internet enabled workers)

From firm data to macro indicators …and what it can deliver (cont) (example; index of purchases made through the internet) countryElecomMexElecDISTRFINBUTOT NOR FRA AUT GER 119 SWE FIN CZE IRE GBR NLD ITA SLO.1121

From firm data to macro indicators …and what it can deliver (cont) (example; index of sales made through the internet) countryElecomMexElecDISTRFINBUTOT NOR AUT FRA GER.9..9 CZE FIN IRE NLD GBR SWE ITA SLO.1131

Impact evidence from firms What microdata contributes to indicators From micro analysis we get -clear evidence on how inputs to innovation affect behaviour, competition and performance at level where decisions are made -pointers to policy ’levers to pull’ in terms of firm level incentives -limited ability to compare drivers of performnce between countries Advantages and disadvantages -firm level analysis is flexible, fast, and uses ‘all the data’ in terms of variation / degrees of freedom -but international comparisons of microdata analysis are vulnerable to differences in surveys / definitions / data treatments

Impact evidence from firms Earlier firm level analysis showed ….. On productivity impact -IT investment and use associated with higher productivity measured through labour productivity or MFP (for a limited number of countries) -Productivity of IT effects stronger in US owned firms, in multinationals, in younger and more innovative firms, and in services On IT investment and CT deployment -links between communications infrastructure and firm investment IT are significant (for UK at least) -broadband using workers have a significant separate effect on productivity, over and above IT investment

Impact evidence from firms What this study has shown from firm level analysis includes ….. On productivity impact -consistency across 12 countries in productivity effects of core ICT use metrics for manufacturing firms -much more difference in services; UK NL, France and Scandinavian analyses show significant positive effects, elsewhere mixed picture On IT investment and CT deployment -broadband is a good indicator – or predictor - of firm level purchased investment -broadband using workers still have a significant separate effect on productivity in countries where use is greatest -‘joining up workers’ has the biggest effect in differentiated services, where much of the investment is in systems and people

Impact evidence from firms What firm level analysis has shown continued ….. On Innovation -evidence that connected workers are more likely to innovate -.. From UK that firms which get innovative ideas from third parties are more likely to invest in broadband connections -.. From NL / SE that ICT using firms are more likely to produce innovation outputs, given other inputs, and that productivity effects associated with ICT are strongly influenced by this -=> it’s what firms change with ICT that counts, not what they spend On ICT and skills -higher skills levels increase returns from ICT investment, or from ICT use, whether we use direct measures of skills, or wage measures

Impact evidence from firms What we’ve shown from firm level analysis continued ….. On organisation -confirmation that some business process links can be associated with significant, positive, productivity effects, in some countries -some of the organisation / maturity variables can be expressed in terms of a composite index … but our surveys need more work On flexibility / specialisation -Finnish evidence that flexibility associated with mobile access to IT has significant productivity benefits -Finnish evidence that outsourcing of IT services is associated with worthwhile labour productivity advantages

Impact evidence from new indicators What we show from industry / country level analysis includes….. On ICT use -Evidence that broadband connected employees provide a strong productivity indicator across the contributing countries, over the period Looks to be significant across industries, even though firm level effects are uneven On ICT use as driver of competition -Cross – EU evidence that higher ICT use, and productivity effects is associated with more dynamic competition -Consistent with US conclusion by Brynjolfson et al, that ICT ‘enterprise architecture’ enables firms to replicate successful business models

Country / industry data shows broadband gains:

..and industry split shows differential impacts : …..industry splits tend to support UK microdata results showing - IT capital matters most in manufacturing - Broadband enabled employees matter more in differentiated services

Advantages Micro based macro indicators can: Use data available at both firm and industry level -industry indicators compiled ‘bottom up’ from linked surveys -combined with sources from National Accounts or Labour Market -aim to include innovation indicators in next round of work Build comparisons across countries -firm level data can’t be shared by most NSIs, and are different -metadata approach can iron out differences before analysis Allow for structural differences between countries -disaggregation possible on completely comparable basis -only need to write analytical code once It works! -all 13 countries in this project have produced outputs, despite very different systems

Challenges Micro based macro indicators need: Initial investment in metadata and systems -work to set out dataset and variable descriptions from key surveys -must be able to bring datasets together within each NSI in a common research environment, to enable them to be linked Systems to control confidentiality within each NSI -research / linking framework must meet national non-disclosure requirements, but still allow firm level data to generate required outputs Agreement on confidentiality controls for merged dataset -agreement across NSIs on the level of detail which can be used for research and analysis, even if it doesn’t meet national quality standards -agreement on what detail can go into the public domain. Persistence! -Work in this project produced lots of data which needs simplifying!