1. ESF AND THE ROMA IN SLOVAKIA Results from a study on the territorial distribution of Roma-relevant ESF projects (2007 – 13) and their impact on Roma communities in Slovakia Jakob Hurrle, Charles University Prague; Andrey Ivanov, UNDP BRC

2. 1. Context

3. Focusing on results… Few important questions: –How to define result? How to measure it? –What kind of data is needed? Where to get it? Most of all, how to move from monitoring the aggregate status of the population to project- level outcome monitoring and evaluation –To assess effectiveness of interventions –To assess the efficiency of funding (ESF and other) Those questions are part of the Roma pilot project: tools and methods for evaluation and data collection funded by DG Regional Policy

4. The M&E chain InputOutputOutcomeImpact IntermediateFinal Financial, physical resources Goods and services produced by inputs (classrooms built, textbooks provided) Access to, use of, and satisfaction with services (enrolment, repetition, dropout rates) Effect on dimension of well- being (literacy)

5. Example: Employment generation project Inputs: Number of training courses, expenditures for re- qualification, number of presentations, number of attendees Outputs: Number of people who passed re-qualification course Outcome: Number of former unemployed who found jobs Impact: Registered increase of HH incomes, change in poverty rates Plus Sustainability: duration of the job Positive externalities: Reduced drop-our rates of children at risk, reduced societal fragmentation Intervention A B C Intervention seen by A is not what is seen by C

6. From theory to practice: ESF How the structural funds perform in reality in regards Roma in the case of Slovakia? Does the system and the existing procedures allow following the M&E chain? If not, what is missing? What can be improved for the next programming period?

7. Assumptions behind the study Improving the situation of Roma requires a combination of 1.Political will 2.Resources 3.Conceptual framework of intervention 4.Implementation procedures – the nitty-gritty of the project cycle Usually we focus on (1) and (2), less often on (3) and almost never on (4)

8. Overall question: Is the ESF programme an effective tool to address the problems of the Slovak Roma on the labour market? Are the stated goals and principles of the ESF programme appropriate to address the targeted problems? Did the programme realization happen in accordance with these goals and principles? Did the distribution of the resources correspond with basic principles for the usage of public resources? Answering these question leads to new, more specific questions

9. A number of specific questions… Does the territorial distribution of the projects match the distribution of the Roma population? What share of Slovakia´s Roma population was reached? To what extend were the invested resources indeed relevant for the needs of the members of marginalized Roma communities? What are the experiences of project owners and what impact did the projects have on beneficiaries? Are the project results sustainable? All this fits into the desired outcome level M&E

10. 2. Methodology

11. 2.1 Data analysis2.2 Case study Prešov Region Combination of two approaches:

12. 2.1 Data analysis Three data sources: Key data from all Roma relevant projects Sample of 298 Roma-relevant projects created on basis of analysis of project documentation Data set of Atlas of Roma Communities (2004)

13. Data analysis - key questions: What does Roma-relevant mean? Method: Analysis of project documents Which types of projects are Roma-relevant? Method: Analysis of project documentation in sample Which communities where targeted? Method: Matching of project data and data from Atlas of Roma Communities (2004)

14. 2.2 Case study Prešov Region Focus on training and employment projects in a structurally disadvantaged region with a high share of Roma 8 projects visited Attempt to create a sample that represents variety of project types (different types of project owners, ethnic projects / mixed groups of beneficaries, urban / rural, open and closed settings)

15. Case study – key questions What are the experiences of project owners? How are the projects perceived by the direct beneficaries and other members of the targeted communities? How do the experiences of different project types and settings differ? Which factors affect the quality and sustainability of results?

16. 3. Roma-relevant projects in the Slovak ESF programme

17. National projects 71 % Out of these Roma- relevant: 8,7 % Calls for proposals 29 % Out of these Roma- relevant: 13,4 % 1 011 506 634 contracted Out of this Roma relevant: 185 142 009 (18 %)

18. Roma-relevant projects within the five ESF priority axes Axis 1: Supporting Employment Growth Roma-relevant: 22 % Axis 2: Social inclusion Roma-relevance: 86 % Axis 3: Bratislava-region programme Roma-relevance: 35 % Axis 4: Building capacities and improving the quality of the public administration No Roma-relevant projects Axis 5: Technical support

19. Priority axis 1: Employment 80 projects of total value 32,840,855, average size 410,511 Project owners tend to be commercial companies Roma are often just one target group among many (e.g. 1 working place) Unclear how many targeted but even a project with 1 Roma ticked as MRC relevant Focus often on investments into existing staff – Roma are disadvantaged due to their exclusion from formal labour market

20. Priority axis 2: Social inclusion. 2.1 and 2.2 574 (570) of total value 113,870,162 (48,891,967) Action 2.1: Social field work (69% or 59 %) 491 (488) projects of total value: 78,951,068 (28,972,874) Ethnically defined beneficaries, relative small average budgets sizes 160,796 (59 371) ) Action 2.2: Employment, training (31% or 41%) 83 (82) projects of total value 34,919,094 (19,919,094) ; Very various groups of beneficaries. Average budget size: 420,712 (242 916)

21. Territorial aspects a) Does the territorial distribution of the projects match the distribution of the Roma population?

22. 99,2 – 62,2 % 61,9 – 35,6 % 35,4 – 21,6 % 21,4 – 10,7 % 10,4 – 4,3 % 4,3 – 0,01 %

23. Territorial distribution of costs Examples: RegionNumber of Roma Number of beneficaries Cost per beneficiary Share of Roma reached Trenčín4 325 (0,72 %) 8 433282 195 % Nitra25 437 (3,59%) 5193602 20,42 % Prešov85 697 (10,77 %) 61 343185 71,58 % Košice89 364 11,61 % 34 568178 38,68 % Slovakia total284 566 (5.29%) 148 637199 52 %

24. Costs per Roma-inhabitant (in Euro, by district, calculated on basis of sample)

25. Features of targeted municipalities (villages only) All municipalities in Roma Atlas (2004) ESF-targeted municipalities (all priorities) ESF-targeted municipalities (action 2.1) ESF-targeted municipalities (action 2.2) Average size1137146714491536 Share of Roma 17,2 %28,9 %31,616,4 % Unemployme nt rate (2004) 21 %24 %26 %17 %

26. ESF projects in smaller-than- average villages (sample-based) Number of municipalities included in Atlas of Roma communities Percentage of municipalities reached by Roma- projects Percentage of municipalities reached (priority 2.1) Percentage of municipalities reached (priority 2.2) All villages in Atlas (2004) 98120 %16 %3 % Less than 300 inhabitants 15011 %9 %3 % 300 – 700 inhabitants 28715 %14 %1 % 700 – 1135 inhabitants 19322 %19 %3 %

27. Level of segregation and unemployment Working index: 1.Location of settlement 2.Type of settlement 3.Distance of the settlement 4.Physical barrier 5.Land ownership 6.Access to electricity 7.Public light 8.Garbage collection in Roma settlement Locations are on a rank between 0 (no segregation) to 15 (extreme level of segregation and underdevelopment)

28. Segregation and underdevelopment of targeted municipalities RegionAll municipalities All targetedAction 2.1Action 2.2 Trenčín1,011,75 2,00,0 Nitra1,051,601,661,44 Prešov2,392,512,891,59 Košice2,872,842,952,98 Slovakia2,142,372,531,90

29. Key findings from regional case study

30. Roma participants of trainings are only in exceptional cases succesful on formal labour markets Assesment of projects´ b

31. Major findings A large number of applications is based on almost identical texts suggesting massive copy/paste The data base doesnt allow even basic outcome evaluation (outcomes impossible to detect) Complexity of administrative procedures makes grant owners dependent on (expensive) consultancy services; various forms dont generate meaningful data Even though Roma are highly overrepresented among the unemployed, a tiny fraction of projects under priority axis Supporting Employment Growth were labeled (on questionable grounds) as MRC relevant Information on the actual ethnic composition of projects´ target groups is missing, which makes it impossible to determine how many Roma are reached

32. The broader context The current setting does not allow robust outcome evaluation of ESF projects –Number of Roma beneficiaries is vague –Outcomes are not clearly defined and cannot be quantified Cost-benefit analysis is impossible (only costs/inputs and accounted for) The system is heavily skewed towards social work that is supposed to provide basic social assistance but does little to lift people out of poverty

33. Recommendations Simple changes: –Modify the application forms so that they generate meaningful data –Applications should be encouraging focus on outcomes and allow for M&E Integrate the project level information in processable data management system (currently information is in pdf files) Modify the TSP workers functions and TORs so that they are part of a local level data generation system for generating project outputs and outcome related data