1. It’s the learning, not the result which counts most in evaluation Randall Pearce THINK: Insight & Advice The 7 th Australasian Better Boards Conference 5 July 2013

2. Evaluation – What is it? ‘A systematic way of answering questions about projects, policies and programs’ 1.Is it needed or worthwhile? 2.Is it having an effect? 3.At what cost? 4.How could it be improved? 5.Are there better alternatives? 6.Are there any unintended consequences?

3. NFP Evaluation – What it isn’t

4. Who evaluates?

5. Why do (or don’t) they evaluate? Source: New Philanthropy Capital

6. What do they gain? Source: New Philanthropy Capital

7. Who should evaluate? AdvantagesDisadvantages Internal Better overall & informal knowledge Less threatening/known to staff Less costly May be less objective Evaluation could be a part-time responsibility May not be trained in evaluation External More objective Able to dedicate time and attention Greater evaluation expertise Needs to learn about organisation and culture Unfamiliar to staff More costly

8. When to conduct evaluation? Form:Stage of ProgramPurpose: Proactive Evaluation Program Start Program Completion Before a program starts To synthesis information to inform program design Clarificative Evaluation During program development To clarify the program design and how it operates Interactive Evaluation During program delivery To improve program delivery. Involves stakeholders in the evaluation. Monitoring Evaluation Once the program is in a settled stage To monitor program progress for accountability and improvement Impact Evaluation During or after program implementation To assess what has been achieved, to learn and be accountable Source: K Roberts (adapted from Owen and Rogers, 2006)

9. Dispelling myths Theory of change? Not needed because the evaluator will reconstruct the logic of the actual program, not the theoretical model: Foundational activities Activities Outputs Immediate outcomes Intermediate outcomes Long-term outcomes Organisational goals

10. Dispelling myths Mountain of data? Most data is just information…we are looking for insight into what it means Historical data is more valuable than a mountain of current data Your evaluator should identify the few ‘dashboard’ measures that you will need to evaluate Once an evaluation has been conducted you can use the dashboard forever

11. Dispelling myths A wad of cash? Think of what is at stake versus the internal budget allocation – any activity with a value in excess of $200K should be evaluated Governments and foundations often allow for 10% to be spent on evaluation There are many ways that NFPs can reduce the cost of evaluations

12. Using the results of evaluation Share them…as widely as you can Some evaluators will agree to write a summary which protects the egos of those involved Action learning/research is a participative approach based on a four part cycle of: taking action, measuring the results of the action, interpreting the results and planning what change needs to take place for the next cycle of action The best projects conclude with a Summit workshop

13. Beyond program impact evaluation

14. Learning along the way Documentation Documents success and failures Summary of key documents in one place Timeline/sequence of events Isolates key measures for the future Supports performance appraisal for staff and board Helps orient staff, volunteers and contractors

15. Learning along the way Full cost accounting Full costs and expenses need to be calculated to arrive at the true financial picture Need to include: Budget allocation Cash donations In-kind services Pro-bono services

16. Learning along the way Full value assessment Captures all non-financial outputs in addition to financial information For example, while social media produces a host of measures, there are no financial equivalents as there are in traditional media (i.e. TARPs) Need to identify data sources for year-on-year comparison in future

17. Learning along the way Organisational behaviour and Governance Qualitative research reveals issues around organisational behaviour and governance which can impact outcomes Project governance can be examined independent of personalities to pinpoint areas for change/improvement

18. Learning along the way Relationship building The evaluation process has been described as ‘cathartic’ by key players Helps diffuse tensions that build up during a campaign Provides stakeholders a voice/builds goodwill for the future Aids communication ‘across the political/media divide’

19. Over to you… Questions

20. For more information, contact: Randall Pearce +61 2 9358 6664 randall.pearce@thinkinsightadvice.com.au NOTE: For a copy of this presentation, please provide your business card at the end of the session or email