1. Impact analysis and counterfactuals in practise: the case of Structural Funds support for enterprise Gerhard Untiedt GEFRA-Münster,Germany Conference: Improving Evaluation Methods 6th Conference of Evaluation of Cohesion Policy Warsaw, Poland 30.11.2009 – 01.12.2009

2. Content 1.Overview 2.Direct Aid to Enterprises 3.Evaluation process 4.Counterfactual Analysis 5.Data requirements 6.R&D support to firms in Thuringia 7.Conclusions

3. 2. Direct Aid to enterprise Direct grants or subsidies to reduce the (user) cost of capital to build up and modernise the enterprise capital stock to induce more R&D investment at the enterprise level Target variables: higher investment, higher R&D more employment higher productivity better long-term development Economic Rationale: Theory of the firm, R&D and public goods

4. 2. Direct Aid to enterprise Source: DG Regional Policy (2009), own calculations Cohesion Policy 2000-2006, including n+3: 163 billion euro Approximately 33 billion euro as direct aid to firms Distribution of Structural Funds by economic categories, in %

5. 3. Evaluation Process An ideal evaluation process can be looked at as a series of three steps (Fay,1996) Microeconometric evaluation The impacts of the measures on the individual firm should be estimated Macroeconomic evaluation It should be examined if the impacts are large enough to yield net social gains if all spillover effects and side-effects are taken into account Effectiveness- or cost-benefit analysis It should be examined if this is the best outcome that could have been achieved for the money spent

6. 4. Counterfactual Analysis Potential-Outcome Approach Measuring the causal effect of direct aid to enterprise on some outcome variable Notation: Y 1 i : outcome if enterprise i received grants Y 0 i : outcome if enterprise i did not get support D i : indicator (0,1) signals enterprise i did (1) or did not (0) receive direct aid Y i : Y 0 i + D i (Y 1 i – Y 0 i ) X : set of firm characteristics (branch, number of employees, turnover, etc.) Causal effect for enterprise i :

7. 4. Counterfactual Analysis Fundamental problem of evaluation: Y 1 is observed for those who received direct aid Y 0 is given for those who did not receive direct aid Y 1 and Y 0 can never be observed for one firm at the same time

8. 4. Counterfactual Analysis Treatment effects and selection bias Population Average treatment effect (ATE) Average treatment effect of the treated (ATT) Policy relevant effect is given by (5) But second term in (5) is not observable Main task: Identification of an unbiased estimator

9. 4. Counterfactual Analysis Only if the outcome is not effected whether a firm received direct aid or not the population average of the non-receiving firms can be used as an unbiased estimator This assumption is normally not fulfilled if non-experimental data is used. Firms are not randomly applying for subsidies and / or are not chosen randomly by the authorities Selection: a)Observables: Branch, Age, Profitability etc. b)Unobservables: motivation of the entrepreneur, administration decision rules etc.

10. 5. Data requirement Data availability is crucial to perform an impact analysis by counterfactual methods Cross-sectional data (one time survey) Panel data (the same firms are surveyed several times) Panel data allow more advanced methods to be used In most EU Member States detailed databases at the firm level that include a treatment variable (direct aid) are not at hand Need to generate data by specific surveys Costly and time consuming

11. 5. Data requirement Firm-specific variables for the statistical analysis: Outcome variables: investment, R&D spending, employment, productivity, Exogenous variables: Branch, Size, Age, share of intermediate inputs, capital stock, legal form Treatment variable (policy impact): EU-public funding or total sum of overall public funding in the establishment

12. 6. R&D Support in Thuringia One time survey of Thuringian firms concerning their R&D behaviour and investment support Performed in 2004, covering the period from 2001-3 Cross-section data for 1484 establishments 284 firms in the dataset received R&D subsidies (either regional, national or supranational EU-wide) Information concerning firm characteristics and performance variables (employment, investment, innovation performance etc.) different measures for enterprise support including R&D-support

13. 6. R&D Support in Thuringia Data Method R&D-Survey Linear Regression Propensity-Score-Matching Selection-Model Difference-In-DifferenceNot applicable Method-of-Matching Difference- In-Difference Not applicable

14. 6. R&D Support in Thuringia Propensity Score Matchting Steps involved in the analysis: 1.Estimating the propensity score for each unit in the sample (estimate the individual probability that a firm received support, based on observable firm characteristics) 2.Matching the units using the estimated propensity scores 3.Assess the quality of the matching 4.Estimate the impact and its standard errors

15. 4. Counterfactual Analysis Target variables: General innovativeness (binary dummy for patent application between 2001-2003) Total number of patents, Number of R&D employees R&D Expenditures and Intensity (=R&D expenditure as share of total turnover) Observable firm Characteristics: Age, Export and Import quota, legal status, Branch, Ownership, Number of employees etc.

16. 4. Counterfactual Analysis Means of control variables differ significantly between treated and untreated firms before the matching procedure is applied. Afterwards matched firms are identical in their observables!

17. 4. Counterfactual Analysis Matching Results with ATT = Average Treatment Effect on the Treated:

18. 4. Conclusions Significant share of the Structural Fund is spend on direct aid to firms Little is known at the micro (firm) level about the impact Counterfactual analysis could act as a standard instrument But: data requirements are high! And better data could help! Significant gains from these impact analysis can be expected