1. 1 Workshop EVALUATION PROCESS – PROGRAMMES AND PROJECTS Waterford 10 TH April 2006. Dr Jim Fitzpatrick Managing Director Fitzpatrick Associates 10 Lad Lane Dublin 2 Tel: 01-6763200 Fax: 01-6767972 E-mail: jfitzpatrick@fitzpatrick-associates.com www.fitzpatrick-associates.com

2. CONTENT 2  Nature/basis of comments  Types of evaluation  Evaluation as a “process”  Key Steps in doing evaluation  Methodologies in practice  Common problems  Specific cross-border issues  Some “tips”

3. NATURE/BASIS OF COMMENTS 3  Wide practical experience, esp. programme-level  EU-co financed, other  Related work  INTERREG and other cross-border  All external evaluation

4. 4 THE PROJECT CYCLE Source: 2004 PCM Manual

5. EVALUATION CYCLE AND FOCUS 5 Evaluation questionEx-AnteInterim/Mid-TermPost mid-term Rationale *** ** Relevance****** Effectiveness****** Efficiency****** Impact****

6. A PROCESS, NOT JUST A TECHNIQUE 6 Managing the team Stakeholder Involvement Research Analysis Time, resources, budget Client, user relations EVALUATION A Balancing Act

7. TASKS IN TYPICAL EVALUATION PROCESS 7 1.Establish/Understand Context  who is the “client”?  why being done?  any specific use intended?  what kind of evaluation is needed? 2.Obtain/Prepare/Agree Brief (ToR)  is there one?  is it clear?  write one?  is it agreed? 3.Prepare Work Plan (Proposal)  overall approach (i.e. how interpreting brief, how going about it)  analytical framework (i.e. overall logic)  methodology/techniques (e.g. CBA, CEA, MCA)  work programme (i.e. the data, data collection)

8. 8 4.Evaluation Team/Resources (budget)  no. of people/person days  types of people  necessary expertise (e.g. on technical aspects) 5.Doing the Evaluation  implement method/work programme  client relations  manage team  deal with unexpected issues 6.The Output/Report/Schedule  meet how often, how many, when?  nature of report e.g. length? style? nature?  presentations? EVALUATION TASKS CONTINUED

9. EVALUATION METHODOLOGY IN PRACTICE 9  trying to establish if intervention did (or will) make a difference  so “with-without” scientific method at its core  formal quantitative techniques very desirable, but very different  MCA/scoring, weighting and ranking most used  others useful are:  before v after (time-series)  places that do, don’t have (“control group”)  “expert” opinion  views of stakeholders  always need some framework for answering the evaluation questions (samples available)

10. COMMON PROBLEMS IN PRACTICE 10  poor initial projects/programme design  inability to control for external influences  poor/unavailable indicators (too few, too many, not really capturing essence of intervention)  lack of consensus about purpose of evaluation  “scope creep”

11. PARTICULAR ISSUES IN INTERREG CONTEXT 11 “Evaluation meets cross-border co-operation!”  different member state/regional contexts  different national attitudes towards evaluation  projects/programmes relatively small  interventions often “soft” in nature, harder to evaluate  cross-border evaluation teams needed  language/communication problems  more costly (travel, telecoms, translation, time)

12. SOME PRACTICAL TIPS 12  ensure programme/project planning are good (monitoring, evaluation considered at outset)  take time to get shared understanding of what’s happening  ensure there is some kind of method/framework being used  indicators – use “sensibly”, and note they are the fuel of monitoring/evaluation, they are not it themselves  cross-border work needs cross-border teams  remember it’s a process, not just a technique

13. QUESTIONS AND ANSWERS 13 LOOKING FORWARD TO HEARING BOTH THE QUESTIONS AND THE ANSWERS!