1. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 1 The Fundamental Evaluation Problem and its Solution

2. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 2 1.Evaluation Problem 2.Treatment Effects 3.Selection Bias 4.Solution Approaches Hagen, Tobias und Bernd Fitzenberger (2004), Mikroökonometrische Methoden zur Ex-post-Evaluation, in: Hagen, Tobias und Alexander Spermann, Hartz-Gesetze – Methodische Ansätze zu einer Evaluierung, ZEW-Wirtschaftsanalysen, 74, S.45-72

3. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 3 Example: Evaluation of programs of active labour market policy (e.g. job creation measures)  Investigation of a program‘s effect on a certain outcome variable (e.g. the employment probability)

4. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 4 Measurement of the program‘s success by the share of individuals entering employment during a certain period of time is not sufficient. Problem: The causal effect of the program is not measured – employment take-up could happen without program participation

5. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 5 Causal effect: Employment probability of participants versus employment probability of participants had they not participated Problem: „counterfactual situation“ – participants cannot simultaneously be non- participants!

6. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 6 Solution: Estimation of the hypothetical employment probability participants in case of non- participation by using the employment probability for non-participants. Use of a „comparison or control group“, to be able to estimate the success of participation.

7. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 7 In case participants and control group differ with respect to observable or non- observable characteristics that do have an influence on outcome variables Selection Bias

8. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 8 Question: What is the effect of a program on the outcome variable y? y 1 : Outcome variable in case of participation y 0 : Outcome variable in case of non-participation C : Dummy variable, set to 1 in case of participation

9. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 9 The actual observed outcome variable for an individual i results from: The program effect is:

10. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 10 Problem: It is not possible to calculate an individual causal effect. No individual can be in two different states of participation at the same point in time.

11. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 11 However, it is possible to estimate The mean effect of participation on the group of participants („Average Effect of Treatment on the Treated“ – ATT). The mean effect of participation expected for an individual drawn randomly from a population of participants and non-participants („Average Treatment Effect“ – ATE).

12. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 12 E[y 1 ] only is observable for participants and E[y 0 ] only is observable for non- participants.

13. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 13 Case differentiation: 1.Participants and non-participants („control group“) differ neither with respect to observed nor to unobserved characteristics  consistent estimates of expected value of the outcome variable using the sample mean:

14. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 14 WhereasT: Number of participants NT: Number of non- participants

15. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 15 2.Participants and non-participants differ in regard to observed and unobserved characteristics  Selection Bias  Difference of sample means does not lead to consistent estimators

16. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 16 X-Heterogeneity: Heterogeneity of the treatment effect that can be explained by differences in observed variables. U-Heterogeneity: Heterogeneity of the treatment effect that can be explained by differences in unobserved variables.

17. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 17 Definition homogeneous treatment effect: Treatment has the same effect on individuals with different observed attributes, i.e. no X-heterogeneity. Measure has the same effect on individuals with different unobserved attributes, i.e. no U-heterogeneity. Treatment effect is identical for all individuals and ATT=ATE.

18. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 18 X-Heterogeneity / U-Heterogeneity:  Heterogeneous treatment effect  Selection bias 1.Selection bias due to observed variables 2.Selection bias due to unobserved variables

19. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 19 „selection on observables“ „selection on unobservables“ Regression Methods „Propensity- Score- Matching“ Difference-in- Difference- Estimators (DiD) Selection Models Instrumental Variable Approaches (IV)

20. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 20 1.Social experiments 2.Natural experiments 3. Quasi experiments 4. Non experimental statistical / econometric methods

21. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 21 1. Social Experiments Example: Introduction of a care budget Basic Information (Arntz/Spermann, ZEW DP 04-84): 7 sites in East and West Germany goal: 2000 participants; assigned randomly to 1000 in the program & 1000 in the control group duration: 2005-2008

22. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 22 Matching transfer plus case management Treatment: Outcome: Duration in home care Life satisfaction Quality of care Home Care arrangements

23. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 23 Discussion of biases: Randomization bias Treatment dropout bias Control group substitution bias Attrition bias General equilibrium effects Preliminary results: Swiss Journal 2006

24. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 24 No randomization bias No structural change of participants and non- participants due to the fact that they participate in a social experiment

25. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 25 No Treatment Group Dropout Bias R=1  T=1 R=0 indicator variable for control group R=1 indicator variable for program group T=0 indicator variable for non-receipt of program T=1 indicator variable for program receipt Persons who were assigned to the program group have to receive the program

26. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 26 III) No Control Group Substitution Bias R=0  T=0 Persons of the control group do not participate in comparable programs

27. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 27 No General Equlibrium Effects No indirect effect of the program that could change the direct effect of the program

28. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 28 No Attrition Bias Program or control group members may not get lost during the experiment

29. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 29 2. Natural Experiments See ZEW-DP 07-001 (Boockmann/Zwick/Ammermüller/Maier) on „Eingliederungszuschüssen an ältere Arbeitnehmer“ (Hartz I-III evaluation) See later presentation on Difference-in- Difference estimators

30. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 30 3. Quasi Experiment  Spermann/Strotmann (ZEW DP 05-68) about the „ Targeted Negative Income Tax (TNIT) experiment“ in Mannheim Main features: 1.Target group: Means-tested unemployed 2.Employee subsidy (earnings supplement) 3.Time-restriction which varies between household types

31. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 31 Example for Quasi Experiments: Evaluation Design for TNIT Study Site-randomized control group in the same local labour market Program district in the northern part of Mannheim Comparison district in a comparable southern part on Mannheim Program and comparison group are comparable due to t-tests on important observables

32. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 32 Example for Quasi Experiments: Treatment, Outcome, & Identification Information about the potential earnings supplement in case of participation in the private labour market Check by survey: Did the program group understand the program (proxy for receipt of treatment) Result: Program group understood the basic idea Treatment:

33. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 33 Example for Quasi Experiments: Treatment, Outcome, & Identification Participation: available Income: available Hours of work: not available Duration of jobs after time limit: not available Outcome:

34. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 34 Identification: Is the selection on observable assumption plausible? HIT (1997) and HIST (1998) set up criteria for comparison group data quality: 1.Same data source for program and comparison group: fulfilled 2.Program and comparison group reside in the same local labour market: fulfilled 3.Data contain a rich set of covariates: only partly fulfilled, do not observe individual employment history and pre-program data Quality checks for matching are not feasible Solution: We restrict to Probit and Tobit, Propensity Score Matching confirmed results

35. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 35 If one accepts the plausibility of the selection on-observables assumption,  Then average marginal effect Probit estimation of the program dummy could be interpreted as ATE

36. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 36 Results: Descriptive Statistics

37. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 37 Probit Models for overall employment (average marginal effects)

38. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 38 Notes: Administrative data, Mannheim 2000, p-values in parentheses. ***/**/* indicate statistical significance at the 1, 5 and 10 percent level, respectively.

39. The Evaluation Problem Alexander Spermann, University of Freiburg, 2007/2008 39 4. Non-experimental Methods  „ Matching“  Instrumental Variables  Panel Data  etc.