1. Beyond surveys: the research frontier moves to the use of administrative data to evaluate R&D grants
Oliver Herrmann Ministry of Business, Innovation & Employment of New Zealand

2. Overview
Survey data vs administrative data
Micro data research in New Zealand
Using administrative data for evaluation: an R&D example
Impact evaluation, causality and methods
Evaluation findings
Limitations

3. Survey data (Last Millennium)
Empirical research and evaluation have relied on survey data.
Using a sample of individuals from a population with a view towards making statistical inferences about the population.
Examples: Current Population Survey (US), Annual Survey of Hours and Earnings (UK), Business Operation Survey (NZ)
Political science, sociology, and economics were all “revolutionized” using survey data sources.
Empirical research and evaluation have relied on survey data
The methodology: using a sample of individuals from a population with a view towards making statistical inferences about the population.
Examples: Current Population Survey (US), Annual Survey of Hours and Earnings (UK), Business Operation Survey (NZ)
During the second half of 20th century: development of modern survey methods and of statistical techniques for analysing survey data.
Enormous insights in the fields of political science, sociology, and economics were all revolutionized using survey data sources.
Lessons: include knowledge and additions of qualitative perspective

4. Administrative data (New Millennium)
Governments create comprehensive administrative data that cover socio-economic behaviour from education, earnings, workplace to family composition, health and retirement.
Administrative data are highly preferable to survey data because
full population files are generally available and offer much larger sample sizes,
administrative data have a longitudinal structure that enables evaluators to follow individuals over time and address many critical policy questions,
administrative data do not suffer from high rates of non-response, attrition, and under-reporting.
Overcome the linear problem

5. New Zealand’s Hotbed of Microdata Research
Statistics NZ has undertaken a number of projects that integrate data supplied by different government agencies
The Integrated Data Infrastructure (IDI) is a comprehensive database with longitudinal microdata about individuals, households, and firms
The IDI pulls together a range of administrative and survey-based data sources (financial performance, employment, merchandise trade, business practices, government assistance).
The IDI allows for the investigation of previously unanswerable questions. Researchers and evaluators have access to answer research, policy, and evaluation questions to support informed decision making.
The Integrated Data Infrastructure (IDI) is a comprehensive person- and firm-level database for Government researchers to utilise
The IDI pulls together a range of administrative and survey-based data sources, resulting in a rich and multi-dimensional
In addition to contextual and other information

6. Using administrative data for evaluation: An example - publicly funded R&D
Innovation is well known to be an important driver of economic growth and investments in R&D are among the factors that drive innovation.
Governments encourage business investment in R&D, with the aim of correcting or alleviating two main market failures:
Difficulties by firms to fully appropriate the returns to their investment.
Difficulties in finding external finance, in particular for small start-up firms.

7. Policy objectives of R&D grants
Increase and enhance the R&D.
Increase the economic performance.
In New Zealand there are different types of government R&D subsidies
assistance to build R&D capability
assistance for R&D projects

8. Impact evaluation = causal inference
Identify the causal effect of a policy or intervention.
We did program X, and because of it, Y happened.
Use the results to decide whether the programme should be continued/terminated/expanded/modified
Y happened because of X, not for some other reason. Thus it makes sense to think that if we did X again in a similar setting, Y would happen again
In a more research-friendly universe, we’d be able to observe a single firm (called company ICT) after government both gave and didn’t give R&D assistance:
Ytreated ICT - Yuntreated ICT

9. Evaluation problem (1) In reality, we observe one outcome only
The outcome we do not observe is called the counterfactual.
In order to estimate the impact of treatment, we need to estimate the counterfactual outcome Is the solution to get more observations? Then we can calculate:
Average(treated) - Average(untreated)
But what if there’s an underlying difference between the treated and untreated (selection bias)?

10. Evaluation problem (2)
Simplest world : impact is common across all individuals/firms
Impact is different across individuals/firms but doesn’t affect participation/selection => variation in impact has few policy implications
Impact is different across individuals/firms and this affects participation/selection => implications for policy and for unbiased estimation

11. Non or Quasi-experimental approaches
We use non-experimental approaches to build a counterfactual - selection on observables (matching) - selection on unobservables (Difference in differences)

12. Control group and eight matching variables
Employment and change in employment
Total productivity and change in total productivity
Capital intensity and change in capital intensity
Exporting industry
Firms in group
Firm age
R&D activity
Ownership status

13. Actual and counterfactual outcomes
Changes in performance of assisted firms to matched similar New Zealand firms.
The additional impact
result of the public subsidy alone
Comparing actual and counterfactual outcomes.

14. Evaluation findings
Even before receiving R&D grants assisted firms have higher sales, are larger, more capital intensive, exporting and undertaking R&D.
Significant impact on economic performance of the firms due to the R&D subsidies.
“Capability Building” has a positive impact on employment growth, sales, value added, and productivity.
Counterintuitive finding: No impacts for “Project Funding” !
Positive impacts for small firms.
Positive impacts for firms that had not undertaken R&D two years prior to receiving their assistance.
No impacts for large firms and no impacts for prior R&D performers.

15. Limitations and related evaluations
Data limitations (4 years after receiving assistance)
Previous evaluations (case studies and surveys) showed very positive outcomes.
Before/After comparison (completion of the grant) does not inform about any additional impact.
Most previous evaluations have overestimated the economic impact.
Our methods address this question by comparing the performance of an assisted firm to a matched similar firm.

16. Additional slides

17. Potential outcomes framework (Rubin, 1974)

18. Main identification problem is the lack of a counterfactual response

19. Matching
Match each treated participant to one or more untreated participant based on observable characteristics.
After matching treated to most similar untreated, subtract the means, calculate average difference

20. Issues
Selection/participation depends only on observable characteristics; it does not depend on unobservables
Difficult to match on many observables X => condense all observables into one “propensity score,” match on that score.
Need good data

21. Difference in differences Before – after : Growth in outcomes for assisted firms
Firm with assistance
Outcome, Y (e.g. sales)
Time t1 t2
Zero growth O Y0 Y1 Y'

22. Difference in differences Comparison of outcomes post treatment Y(b) – Y(d)
Firm with assistance
Outcome, Y (e.g. sales)
Time t1 t2 O Y0 Y1 Y' b
Control firms without assistance a d c

23. Difference in differences = Growth in sales for assisted firms – Growth in sales for control firms
Firm with assistance
Outcome, Y (e.g. sales)
Time t1 t2 O Y0 Y1 Y' b e
Control firms without assistance a d c