Andrey Ivanov, Senior Policy Advisor, Human Development and Roma Inclusion cluster, UNDP BRC

Basic typology  Monitoring (the process)  Evaluation (of the results)  Intermediary or final  Using indicators  Input  Output  Outcome  Impact Applied at different levels  Of the National strategy  Of the Action Plans  Of Individual interventions Monitoring what determines the kind of data and the kind of indicators used

 Having a National Strategy drafted is the beginning, not the end. It needs to be matched by  National Action Plans (usually covering 2 years periods and regularly updated)  Local action plans  Sector specific and integrated projects  In the case of the strategy, for M&E we need  Clear targets – numerical expression of the objectives  Adequate indicators – the definition of the target (how do we measure whether the objective was reached)  Quantitative baseline – the starting point against which the progress/regress is be quantified (the value of an indicator at t o )  Milestones – intermediary targets on the way to the general target to keep track of progress (the value of the indicator at t 2, t 4, t 6 )  The lower you go, the higher the chances for real inclusion of Roma in the process

 National strategy  Long-term change in the situation of the target group  Difficult to attribute results (but not impossible)  National action plan  Closer link between inputs and outcomes  Clear objectives (that are the strategy’s milestones  Local action plans  Direct link to project outputs  Clear territorial dimensions  Individual interventions  Counterfactual possible although difficult

 What targets for individual priority areas?  Roma specific or general?  What baseline?  2004? 2011? 2013?  What source of data?  Data availability determines the indicators or the other way around?  What milestones?  The link to individual OPs

 Politically sensitive (incl. misuse of data for political purposes)  Legal (data protection) or ethical considerations (privacy and fear of stigma) constrains  Insufficient attention to comparability across countries, sub-regions, ethnic groups  The crucial question: what to put in the denominator of an indicator?  The nightmare answer: whatever serves the purpose…

 Self-identification  Outside (‘imposed’) identification  By non-Roma  By Roma  Combined (multi-stages) – used in the surveys of UNDP (2004 and 2011) and of FRA (2011)  Crucial decision to be made: are we addressing “all Roma” – or “Roma at risk of marginalization”? The answers is both politically and policy loaded.

The data set of Roma vulnerable to marginalization generated from the UNDP/WB regional survey that is part of EU Roma Pilot Project funded by DG REGIO and from FRA Roma Pilot Survey:  Monitoring fundamental changes possible (but not short-term fluctuations). Suitable for National Strategy evaluation  Most indicators have a base-line populated by data from the survey conducted in 2004 by UNDP  The “best game in town” (because it’s the only one…) Caveats:  Still a survey (a sample is always a sample)  Expensive, provides data on “Roma vulnerable to marginalization” – and not on “Roma in general” Other options  Roma boosters in HBS  Longitudinal surveys

 Be pragmatic - don’t be obsessed by (don’t ask) unanswerable questions like “Who’s Roma?”  But don’t dilute the task of Roma inclusion either  Give priority to socio-economic status  But still keep ethnic identity and specifics in sight  Stick to territorial characteristics  Most of the vulnerable Roma live territorially in separate (segregated) communities  Territorial mapping of those communities is possible  Once a detailed map of Roma-dominated communities is available, it will be possible to correlate ethnic characteristics with territorial tags (individual’s address)  This will allow monitoring a standard set of indicators for a population living in an area with ***% of Roma

 Makes possible to identify the absolute number of the population and not only a percentage  It can be an option solving the problem of individual respondents refusal to declare ethnicity in the census or to declare different one  Less susceptible to political fluctuations  Is more comprehensive in terms of social inclusion (targeting vulnerability per se)  It grasps the marginalized, visibly excluded segment of the Roma population  Actually reflects the fundamental logic of inclusion (including the excluded, not those included already)  Is best for ensuring that control groups (non-Roma living in the same area) are also included

 One approach cannot serve all purposes  Apply different data sources for different planning frameworks  National Strategy – EU-wide survey (representative of… - a matter of political compromise)  National Action Plans – territorially-focused mapping  Individual interventions – project outcome evaluation  Integration of the three levels requires clear milestones in strategies and action plans

 Integrate the monitoring functions into the entire implementation chain of the strategy  Don’t rely on one source of data and give priority to territorial approaches  Include clear milestones in National Strategies that would serve as a link to the National action plans and OPs  Compete the entire vertical planning and M&E architecture (strategy  plan  call for proposals  interventions)  Go beyond poetry in Operational Programs evaluation building the latter bottom up  Be aware: keeping evaluations vague means keeping them fake