Contribution Analysis Made Do-Able Rod Laird Organisation 18 September 2007 Euston, London Mike Hendricks, Ph.D.

Contribution Analysis Made Do-Able Rod Laird Organisation 24 September 2007 Manchester Mike Hendricks, Ph.D.

What is “Contribution Analysis”? Many people care about the performance of your project Two questions : (1) Did things improve? and (2) Did you cause it? The second question is much harder to answer But audiences typically don’t care – they still want your best answer “Proof”, “conclusive evidence”, and “causal attribution” aren’t the appropriate standard; too scientific, rigorous A more appropriate, real-world standard of evidence is emerging “Reasonable confidence”, “convincing evidence”, “plausible association” Amassing this standard of evidence is called contribution analysis

How Will We Learn About Contribution Analysis Today? * Explore these cutting-edge issues together, as colleagues * Consider six steps, with multiple options for each: 1. Ensure that your project has been fully implemented 2. Gather quantitative data from participants (only) 3. Gather quantitative data from participants and others 4. Gather qualitative data about your project 5. Deliberately seek “plausible alternative explanations” 6. Assess your project’s contribution * Discuss each step and option, with examples when possible * You plan a contribution analysis for your own situation * Presentations, discussions, feedback on your ideas

Step #1: Ensure that your project has been fully implemented An essential first step = Is your project really there? Describe in detail how your project is supposed to be operating: * Detailed narrative * Logic model Describe in detail how your project is actually operating: * Inputs * Activities * Outputs * Be able to report, in some detail, everyone’s experiences Compare the actual reality to the original plans

Essential Components of a Project ACTIVITIES  What the project does with inputs to fulfill its mission feeding and sheltering homeless families providing job training educating teachers about signs of child abuse counseling pregnant women OUTPUTS  The volume of work accomplished by the project number of classes taught number of counseling sessions conducted number of educational materials distributed number of hours of service delivered number of participants served OUTCOMES  Benefits or changes for participants during or after project activities new knowledge increased skills changed attitudes or values modified behavior improved condition altered status INPUTS  Resources dedicated to or consumed by the project money staff & staff time volunteers & volunteer time facilities equipment & supplies

Inputs, Activities, Outputs, and Outcomes An INPUT is often expressed as a NOUN – the name of a person, place, thing, or concept. For example, staff, volunteers, clients, facility, food, money, time. An ACTIVITY is often expressed as a GERUND – an “-ing” verb. For example, feeding homeless people, training people for jobs, counseling pregnant women, mentoring teenagers. An OUTPUT is often expressed as the VOLUME of work accomplished by the project. For example, the number of people fed, the number of training sessions held, the number of persons trained, the number of women counseled, the number of teenagers mentored. An OUTCOME often describes a CHANGE in persons you work with. For example, new knowledge, increased skills, changed attitudes, modified behaviors.

Inputs through Outcomes: The Conceptual Chain

Step #2: Gather quantitative data from participants (only) Can we find persuasive evidence that: 1. Outcomes changed after your efforts 2. Outcomes changed at the appropriate time after your efforts 3. Outcomes changed a lot after your efforts 4. Outcomes changed as your efforts ebbed and flowed 5. The relevant outcomes changed, but not others 6. Statistical analyses show that your efforts were important 7. Outcomes changed in different sites where you worked 8. Outcomes changed each time you phased-in your efforts 9. Outcomes changed most where your efforts were most intense

Step #3: Gather quantitative data from participants and others Can we find persuasive evidence that: 1.Outcomes changed after your efforts, but not elsewhere 2.Outcomes changed at appropriate times, but only after your efforts 3.Outcomes changed as you switched your efforts on and off in different sites 4. For a group, outcomes departed from what was expected 5. For individuals, outcomes departed from what was expected

The graph below compares the ratings given by respondents who did see a Specialist Domestic Violence Police Officer and those who didn’t, in relation to the support they received from the Police following the initial report of an incident.

Step #4: Gather qualitative data about your project 1.Gather rich descriptions of your project and participants: * In-depth journalism, detective work, medical diagnosis * Case studies, modus operandi, most significant change, success case methods 2.Examine the pattern of results within your logic model (program theory/logical framework/theory of change/outcome map, outcome chain, outcome sequence chart): * Study data on inputs, activities, outputs, and outcomes * Do the patterns support or dispute a plausible association?

Step #5: Deliberately seek plausible alternative explanations “Other reasons the outcomes may have changed” May seem strange, but very important for your credibility And we do see other possibilities fairly often Four main steps: 1. Identify the most likely possibilities 2. Seek evidence for the contribution of each one 3. Seek evidence against the contribution of each one 4. Compare and draw conclusions Ideally, design your research up-front to test other possibilities

Step #6: Assess your project’s contribution Pull together multiple lines of evidence * About your project * About plausible alternative explanations Include both quantitative and qualitative information Seek consensus of informed outsiders * Jury deliberations * NIH consensus-building panels * Foundation allocation committees * Others, especially in the UK context?