1. Departamento Nacional de Planeación República de Colombia Departamento Nacional de Planeación República de Colombia

2. Departamento Nacional de Planeación República de Colombia Third Meeting of the Social Policy Network Impact evaluation of Empleo en Acción: Lessons and relevance for Colombia Manuel Fernando Castro Director of the Public Policy Evaluation Division National Planning Department Colombia Buenos Aires, November, 2004

3. Departamento Nacional de Planeación República de Colombia 1.Background 2.Empleo en Acción 3.Political economy of the program and the evaluation 4.Direct Support to Employment Program -PADE- 5.Lessons learned from Empleo en Acción Contents

4. Departamento Nacional de Planeación República de Colombia 1.Background 2.Empleo en Acción 3.Political economy of the program and the evaluation 4.Direct Support to Employment Program -PADE- 5.Lessons from Empleo en Acción evaluation Contents

5. Departamento Nacional de Planeación República de Colombia 1. Background Between 1999 and 2001, Colombia experienced the deepest economic crisis over 70 years:  In 1999 GDP decreased by 4,2%  Unemployment reached levels over 20% within the main cities and 40% among the poorest population  Income of the poorest households decreased by 21%  Population under poverty line grew from 50% in 1997 to 60% in 2000, and under indigenous line from 18,1% to 23,5% in the same period Under these conditions, Empleo en Acción was designed and implemented as a part of a social safety net

6. Departamento Nacional de Planeación República de Colombia 1.Background 2.Empleo en Acción 3.Political economy of the program and the evaluation 4.Direct Support to Employment Program -PADE- 5.Lessons learned from Empleo en Acción Contents

7. Departamento Nacional de Planeación República de Colombia 2. Empleo en Acción: the program Objectives: -Provide temporary work to non qualified labor force to mitigate the effects of the economic crisis among the poorest (Sisben 1 y 2) -Improve income temporarily to 300.000 unemployed - Build and repair community infrastructure in low income urban areas

8. Departamento Nacional de Planeación República de Colombia 2. Empleo en Acción: the program Characteristics: - Workfare program - Small projects (between U$15.000 and U$115.000) to be executed in a 5 month period, in pipe water or sewage network, road reparation, community centers, sport center, street lighting, etc. - Pays labor work (maximum 50 people per project) and a percentage of the raw supplies - Monthly p ayments of US$70 per beneficiary, for a maximum term of 5 months - Projects to be executed by departments, municipalities, public facilities companies, NGO’s, independent contractors, organized community, etc.

9. Departamento Nacional de Planeación República de Colombia 2. Empleo en Acción: the program Characteristics: - Financed through a multilateral loan (IADB and World Bank) for US$ 300 millions - Resources allocation: -80% to municipalities with more than 100.000 people (78 cities) -20% to municipalities with less than 100.000 people (other urban areas) - Co financed projects for raw supplies purchase: -Cities with more than 500.000 people until 20% of supplies -Municipalities with less than 500.000 people until 60% of supplies

10. Departamento Nacional de Planeación República de Colombia 2. Empleo en Acción: the evaluation Considering Empleo en Acción was an innovative intervention in Colombia’s labor market, the government decided to evaluate its impacts, specifically: 1. Direct impacts on beneficiaries and their families: Labor market participation Improvement and protection of human capital (labor skills) Targeting (unemployed, informal employees, less productive workers) 2. Indirect impacts on neighbors of project’s influential areas: Benefits derive from infrastructure developments Potential substitution effect of public and private investment

11. Departamento Nacional de Planeación República de Colombia 2. Empleo en Acción: the evaluation –Experimental evaluation, with a base line before the initiation of the projects and two follow ups to treatment and control groups X X X Base line – survey:: - Applicants (beneficiaries and non beneficiaries) - Eligible population in treatment areas 2nd follow up survey: - Applicants (beneficiaries and non beneficiaries) - Eligible population in treatment areas 1st follow up, survey to applicants Short-term effects Medium-term effects Initial situation – Due to the lack of randomized selection of beneficiaries, it was necessary to use a quasi-experimental methodology.

12. Departamento Nacional de Planeación República de Colombia 2. Empleo en Acción: some short-term impacts 1.Was household income modified? Yes. 39% (US$16 per month) and the percentage was higher for women and youth beneficiaries 2.Was labor supply of beneficiaries increased? Yes. 36% (8,6 additional hours per week) with significant differences between small and large municipalities 3.Was there substitution of labor work of other members of the household? There was no crowding out of other members labor effort, nor transfers.

13. Departamento Nacional de Planeación República de Colombia 4.Was household consumption increased? Yes. It increased between 6,6% and 9% in food such as milk and meat, and also in hygiene supplies 5.Did it produce effects on health and education? No. They were not expected (short term exposure). Validate initial hypothesis 6.Did it arrive to targeted population ? Yes. Self-targeting scheme reached population with the lowest income expectations and the fewer labor experience - 80% of beneficiaries below poverty line - 30% below indigenous line 2. Empleo en Acción: some short-term impacts

14. Departamento Nacional de Planeación República de Colombia 1.Background 2.Empleo en Acción 3.Political economy of the program and the evaluation 4.Direct Support to Employment Program -PADE- 5.Lessons learned from Empleo en Acción Contents

15. Departamento Nacional de Planeación República de Colombia 3. Political economy of the program and the evaluation Political and institutional non anticipated factors were added to the complexity of a program of this nature. This generated: 1.Disparities between program design and its implementation 2.Modifications on the evaluation’s methodology and schedule 3.Policy decision not foreseen

16. Departamento Nacional de Planeación República de Colombia Disparities between program design and its implementation 3. Political economy of the program and the evaluation DesignedImplemented Target municipalities80% large 20% small 39% large 61% small Selection of beneficiariesRandomizedNot randomized Project cycle (average)90 daysOver 200 days Employment period5 months3,9 months Connection to local projectsAdditional- complementary Substitutes % financing of raw suppliesBetween 20% and 60% Run out 10 months before the end of the program

17. Departamento Nacional de Planeación República de Colombia DesignedImplemented Approved projects 2000 -20036.0003.724 Implemented projects 2001 – 20045.0002.788 Spent resources (millons US $)540 max188,8 - Colombian government (IADB/WB)20087,7 - Local financed340101,1 Direct employments generated300.000 (5 months) 211,307 (3.9 months) Beneficiaries per project (average)5055 Disparities between program design and its implementation 3. Political economy of the program and the evaluation

18. Departamento Nacional de Planeación República de Colombia DesignedImplemented TypeExperimentalQuasi-experimental Sample selectionRandomizedNon randomized Projects sample size538160 Base line gatheringBefore initiation of projects After initiation of the projects, and before 1st payment Schedule: -Evaluation beginning -Results -1er sem 2001 (Pastrana’s gov) - January 2003 (Uribe’s gov) - July 2004 3. Political economy of the program and the evaluation Modifications on the evaluation’s methodology and schedule

19. Departamento Nacional de Planeación República de Colombia Policy decisions taken or not taken affected: 1. Program operation: Design was not followed : –In spite of delays detected on projects cycle, the administration did not make any changes to accelerate project’s approval process (bottle neck) –Program targeting on small cities in detriment of large cities was not fixed Change in the administration generated: –Learning losses due to changes in the technical team –Delays in the evaluation’s procurement process –Questioning program's objectives and pertinence 3. Political economy of the program and the evaluation

20. Departamento Nacional de Planeación República de Colombia Policy decisions taken or not taken affected: 2. Program and evaluation stability – Since the new administration took place in August 2002, it was decided to: Finish Empleo en Acción (Workfare program), and Replaced it with a Wage Subsidies Program – PADE- - It was not a fully informed decision. -Available evidence showed the program's complexity, operational problems, and lost of opportunity to face the economic crisis, among others. –Monitoring results were taking into account to take those decisions, while impact results came later on – Nevertheless, it was decided to continue with the evaluation. Procurement scheme contributed to this purpose. 3. Political economy of the program and the evaluation

21. Departamento Nacional de Planeación República de Colombia 1.Background 2.Empleo en Acción 3.Political economy of the program and the evaluation 4.Direct Support to Employment Program -PADE- 5.Lessons from Empleo en Acción evaluation Contents

22. Departamento Nacional de Planeación República de Colombia Objectives: Create incentives for formal employment generation in private sector through temporary reduce labor costs Increase formal employment as a share of the total, income and employability of beneficiaries Characteristics: Subsidizing part of minimum wage for a 6 month period to small, medium and large companies for each additional low income worker hired (Sisben 1 y 2) Nearly 60.000 subsidies through a pilot in 25 cities. 4. PADE: the program

23. Departamento Nacional de Planeación República de Colombia –Evaluation of a pilot. Expansion of the program will depend upon evaluation results. –Quasi-experimental methodology with a base line and one follow up –Tools: Face to face surveys (companies and people) Face to face interviews to program agents and operators –As an alternative intervention in labor market, the evaluation is intended to gather information on: Program’s efficacy to generate new jobs Substitution effect Replacement of informal to formal jobs Human capital accumulation Measuring beneficiaries income and employability change 4. PADE: the evaluation

24. Departamento Nacional de Planeación República de Colombia 1.Background 2.Empleo en Acción 3.Political economy of the program and the evaluation 4.Direct Support to Employment Program -PADE- 5.Lessons learned from Empleo en Acción Contents

25. Departamento Nacional de Planeación República de Colombia 5. Lessons learned from Empleo en Acción Evaluation cycle most consider political cycle -Impact evaluation was late: -Short term impacts showed positive results -Nevertheless, they were not use to take policy decisions -Political factors as well as procurement and budgeting process are not minor issues. They should be anticipated -The delays on impact results diminished the value of this evaluation -Unforeseen factors, as well as, externalities demand flexibility form evaluators (design, methodology and operation ) -Even though randomization is highly desirable, it has several implementation problems (political economy)

26. Departamento Nacional de Planeación República de Colombia 5. Lessons learned from Empleo en Acción We took advantage of the situation : -Evaluation has improved debate on labor protection issues -Colombia will have evaluation results of 2 alternative labor market interventions -Against all odds Empleo en Acción showed there was no labor substitution, self-targeting was a key factor. -Increments in households consumption showed that Empleo en Acción was an effective social protection mechanism -In the future comparisons among Empleo en Acción and PADE long term effects will allow government to make more informed policy decisions

27. Departamento Nacional de Planeación República de Colombia 5. Lessons learned from Empleo en Acción - In addition… a new practice to test pilots was introduced: -This scheme allow more informed policy decisions related to programs expansion, suppression or modification (i.e. PADE, Familias en Acción) -Contributed to the definition of a multi-annual evaluation agenda -Increased opportunity and relevance of evaluation results for resource allocation

28. Departamento Nacional de Planeación República de Colombia National Planning Department Colombia www.dnp.gov.co