1. X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v MEASUREMENT SYSTEM FOR CHAPLAINCY SERVICES James Noble London, May 2015 1

2. X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT CONTENTS 2 1 Context 01 2 Recap on theory of change (principles) 02 3 Review of the chaplaincy theory of change 03 4 Measurement principles 04 5 Developing a measurement framework for chaplaincy projects 05 6 Iizuka! 06 7 Questions / discussion 07

3. X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT WHY IS MEASURING IMPACT IMPORTANT? Saves staff time Influences the debate on “what works” Improved services Raises profile Motivates staff / volunteers Taken from: Rickey, B, Lumley, T and Ni Ogain, E. (2011) A Journey to Greater Impact. New Philanthropy Capital. Helps secure funding 3

4. X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT WHAT ARE THE CHALLENGES OF IMPACT MEASUREMENT? Lack of skills Cost Complexity Risk of weak results Lack of confidence in value of soft outcomes Risk of negative impact on service delivery Lack of technology Wary of losing sight of ethos 4

5. X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT CONTEXT

6. X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT NPC: TRANSFORMING THE CHARITY SECTOR 6 NPC works at the nexus between charities and funders Charity SectorFunder Increasing the impact of charities eg, impact-focused theories of change Strengthening the partnership Eg, collaboration towards shared goals Increasing the impact of funders eg, effective commissioning Consultancy Think tank

7. X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT “IMPROVING YOUR EVIDENCE” Develop the evaluation capability and expertise of providers who offer services and interventions to offenders. 7 http://www.clinks.org/support/evaluation-and-effectiveness @imp_evidence

8. X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT RESEARCH WITH TIER 1S

9. X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT RECAP ON THEORY OF CHANGE

10. X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT COMPONENTS OF NPC’S APPROACH TO MEASURING SOCIAL IMPACT 10 Map your theory of change Prioritise what you measure Choose your level of evidence Select your sources and tools Effective measurement framework developed Strategic vision Leadership Case for impact measurement

11. X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT a)articulate how you think a service or intervention will have an impact; b)and test your thinking against the evidence you have available. 11 The essence of impact measurement:

12. X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT Teaching assistants (TAs) trained to provide 1-1 support Process for identifying children & families who will benefit Pupils eat the breakfasts Workplans better tailored to individual pupils needs Engagement and enjoyment in maths lessons generally More confident in Maths Maths results amongst targeted pupils show improvement Minimum of one hour of 1-1 tuition with TA per week Inputs Activities Intermediate outcomes Final outcomes Description of resources deployed and targeting process -Attendance records -Feedback from pupils, teachers & parents - Pupil behaviour measures (hard and soft) - Teachers perceptions - Mock test scores Improved Key Stage test scores LOGIC MODEL FOR A SCHOOL BREAKFAST CLUB -Description of activities -Log of contact time -Example work plans Healthy breakfasts available Focus on activities/games to enthuse pupils Pupils engage & enjoy a) tailored learning; b) group activities Outputs Targeted pupils attend and continue to attend Behaviour improves Accountability line

13. X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT VALUE OF THE THEORY OF CHANGE APPROACH 13 Give an account of impact. By this we mean providing a logical reason, or set of reasons, for why your products/service could have impact on outcomes. NESTA: Standards for Impact Investing There should be a clear description of the service, including (i) the outcomes it aims to deliver, and (ii) who it is for NOMS Commissioning Intentions Organisations applying for Level 1 need to demonstrate emerging evidence, clearly explaining the logic or the theory behind their work. Project Oracle: What are the Standards of Evidence? Which, if any, of the following techniques have you used in the last year?

14. X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT NESTA / SRU / PROJECT ORACLE STANDARDS OF EVIDENCE Level 1: You can describe what you do and why it matters logically, coherently, and convincingly Level 2: You capture data that shows positive change but you cannot confirm you caused this Level 3: You can demonstrate causality using a control or comparison group Level 4: You have one + independent replication evaluations that confirm these conclusions Level 5: You have manuals, systems and procedures to ensure consistent replication and positive impact (fidelity) 14 http://www.nesta.org.uk/publications/standards-evidence-impact-investing

15. X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT REVIEW OF THE CHAPLAINCY THEORY OF CHANGE http://www.clinks.org/support-evaluation-and- effectiveness/demonstrating-outcomes

16. X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT MEASUREMENT PRINCIPLES http://www.clinks.org/support/evaluation-and-effectiveness

17. X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT Specific Aim (middle of triangle) Outcome(s)Outcome indicator(s) Data collection method(s) When and by whom How and for whom will the data be used RUN THE PROGRAMME AND COLLECT THE ASSOCIATED EVIDENCE 17 http://www.ces-vol.org.uk/Resources/CharitiesEvaluationServices/Documents/Monitoring%20and%20evaluation%20framework.pdf

18. X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT IMPACT MEASUREMENT TIPS 1.Use theory of change or logic model to articulate how your project or programme works 2.Make sure it’s informed by existing evidence/research 3.Focus on the immediate, short-term effects you have 4.Collect good quality qualitative evidence. Don’t cherry-pick 5.What administrative data is available? Only collect new quantitative data if you have to. 6.Try to measure the progress of service users longitudinally. Consider sampling rather than measuring everyone. 7.Consider your opportunities to measure the counterfactual 8.Publish and promote what you have learned http://www.clinks.org/sites/default/files/Introduction%20to%20evaluation_final_0.pdf 18

19. X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT INTRODUCTION TO THE CURRENT PROJECT

20. X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT IMPACT READINESS FUND 20

21. X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT CONTENT

22. X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT FORMAT

23. X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT REQUIRED FEATURES Monitor changes in outcomes between sessions: Internal and external Intuitive and easy to use Visual and numeric outputs. Exportable to Excel. Different views and levels of access High-level access to sign-off entries from low level users Personal data needs to be stored securely Flexible, adaptable Possible to quickly register an individual and add detailed information later Software should be usable on tablet and desktop platforms Unlimited users Link mentors, mentees and area administrators Off-line data entry 23

24. X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT 24

25. X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT 25

26. X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT 26

27. X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT THOUGHTS?

28. X AXIS LOWER LIMIT UPPER LIMIT CHART TOP Y AXIS LIMIT v James.noble@thinknpc.org NPC - title of the document (can be changed under view/header and footer. Add date here if needed.28