1. Lesson 8: Effectiveness Macerata, 11 December Alessandro Valenza, Director, t33 srl

2. Agenda  What does Effectiveness mean?  Assessing Effectiveness  Multicriteria analysis

3. Effectiveness (definition 1) The extent to which the development intervention’s objectives were achieved, or are expected to be achieved, taking into account their relative importance. Note: Also used as an aggregate measure of (or judgement about) the merit or worth of an activity, i.e. the extent to which an intervention has attained, or is expected to attain, its major relevant objectives efficiently in a sustainable fashion and with a positive institutional developmental impact. (OCSE DAC)

4. Effectiveness (definition 2 ) The term effectiveness has many possible meanings. The most common definition identifies effectiveness with “achievement of objectives”. This leaves open the definition to the different meanings of “objectives”. Objectives can be expressed quantitatively in terms of expected output or results. The effectiveness is evaluated simply by comparing what has been obtained with what had been planned: outputs and results indicators are all is needed. (European Commission – DG REGIO EVALSED GUIDE)

5. EFFECTIVENESS: WHAT TO EVALUATE ? EUROPEAN COMMISSION Quality: effectiveness is evaluated by comparing results with quality standards. A bility of a given action to produce a desired change: comparing what is observed after the action has taken place with what would have happened without the action. One needs data that allow recovery of the counterfactual situation. OCSE - DAC Effectiveness assesses whether the results outlined in the logframe are delivered and if they are likely to produce the expected objective. Evaluating effectiveness should include assessment of how women and men benefit from the results brought by the project

6. Evaluation Questions To what extent were the originally defined objectives of the development intervention realistic? To what extent have the (direct) objectives of the development intervention been achieved in accordance with the (adjusted, if applicable) target populations? What are the (concrete) contributions of interventions for achieving the objectives of the development intervention? What factors were crucial for the achievement or failure to achieve the project objectives so far (indication of strengths and weaknesses, e.g. the monitoring and evaluation system)? What is the quality of development-policy, technical planning and coordination ?

7. Information and Data on OUTCOME Indicators Quantitative -Financial -Physical -Procedural Qualitative -Opinions on the level of achievement -Perceptions on satisfaction Methods Literature review Interview Dialogic interview Community interview Project visit Focus group Case study Survey

8. Different way for collecting information

9. How to analyse the information? Quality: parameters and standard Economic / Financial aspects: Cost and Revenues Satisfaction: expectations against the perception Opinions: majority/ minority Physical: achieved against targets (logframe ) Time/ environmental protection: savings with regard to the situation before the intervention (business as usual) Counterfactual: Situation with/ without intervention

10. Objective or Objectives? - Normally a project has more than one objective even if not too many (no,more than 3 - OCSE DAC). - For example a new road construction can make the connection: 1.Safer 2.Cheaper 3.Faster

11. Multicriteria Tool used to compare several interventions in relation to several criteria. Multicriteria analysis is used also in the ex ante evaluation for comparing proposals. It can also be used in the ex post evaluation of an intervention, to compare the relative success of the different components of the intervention. Finally, it can be used to compare separate but similar interventions, for classification purposes. Multicriteria analysis may involve weighting, reflecting the relative importance attributed to each of the criteria. It may result in the formulation of a single judgment or synthetic classification, or in different classifications reflecting the stakeholders' diverse points of view. In the latter case, it is called multicriteria-multijudge analysis. (from EVALSED)

12. Process Step 1 define criteria Step 2 scoring or ranking Step 3 weighting Step 4 aggregating

13. STEP 1: setting criterion Criterion 1: Financial Performance Criterion 2: Procedural Criterion 3: Physical realisation Project 1 Project 2 Project 3 Project 4 Project n…

14. STEP 2: Score or rank for judgment -It is needed to find a way to appraise the project according to different aspects since we used different measuring units for different aspects of different process -We can opt for: - A) Scoring: by assigning a numeric value to different “interval” of performance. For example 3 for “above average” –1 for “on line with average” – 3 for “below average” - B) Ranking: we simply order the different projects according to their performance from the first to the last

15. STEP 2: Scoring Criterion 1:Criterion 2:Criterion 3: Project 1111 Project 2021 Project 3311 Project 4330 Project n…

16. STEP 3: Establishment of weight If some Criteria is more important than others it shall be given more importance. To do it we simple apply a multiplication factor > 1 (e.g. 1,5). Some criteria may have such importance that they have to be singled out. This is the case for criteria determined by a veto threshold (For example “Physical” if some project has 0 performance, it is excluded by the analysis).

17. STEP 3: Apply the weight Criterion 1: ( * 1,5) Criterion 2:Criterion 3: Project 11,511 Project 2021 Project 34,511 Project 44,53out Project n…

18. STEP 4: Aggregate the score Criterion 1: ( * 1,5) Criterion 2:Criterion 3: Total (with weight) Total (without weight) Project 11,5113.53 Project 202133 Project 34,5116,55 Project 44,53out 6 Project n…

19. WORK OUT: SME INCUBATOR Criterion 1: Economic (average increase of turnover) Criterion 2: New Jobs Criterion 3: Satisfaction for quality service Total Project 1 Project 2 Project 3 Project 4

20. WORK OUT DATAS Project1234 Economic80%70%30%70% Physical5010305 QualityHighLowHighMedium LowMediumHigh Quality levels

21. Apply weight A Economic* 1,5 PhysicalVeto = N. of job < 10 B Physical* 2 QualityVeto for “Low”

22. See you www.t33.it a.valenza@t33.it