Copyright © 2014 by The University of Kansas Refining the Program Intervention Based on Research

Copyright © 2014 by The University of Kansas What do we mean by refining the intervention? Refining the intervention refers to the process of making your work more effective by using the results of your evaluation to adjust and improve the initiative Key questions you should consider: – What are we seeing? – What does it mean? – What are the implications for improvement?

Copyright © 2014 by The University of Kansas Why should you use your evaluation research to refine the intervention? The primary purpose of evaluation is to ensure that your intervention is having the intended effect, and make adjustments if something is not working

Copyright © 2014 by The University of Kansas When should you refine the intervention? Regularly throughout the life of an intervention Once a year is typical There are other specific times when an evaluation may be especially helpful When it’s clear that what you’re doing isn’t working When participants are dropping out at a high rate Between sessions a time-limited or sequential program When funders or participants ask you to adjust some aspect of your program When funding or other resources are reduced When the goal or issue changes

Copyright © 2014 by The University of Kansas Who should be involved in refining the intervention? Participants Staff members People who are directly or indirectly involved in supporting the work This might include educators, government officials, health professionals, employers, funders, etc. Since their contribution is needed to make changes successful, it is important that they have input into the planning of those changes Those who led and participated in the evaluation

Copyright © 2014 by The University of Kansas Evaluating an initiative should look at three main areas: Process Impact Outcomes Each aspect of the intervention builds on what comes before. In order to have the impact you want, you have to put together and run your program well (process). If your process didn’t go properly, then you haven’t really conducted the program you planned for. How do you refine an intervention based on research?

Copyright © 2014 by The University of Kansas How do you refine an intervention based on research? Process An evaluation of your process compares what you planned to do with what you actually did Process encompasses logistics and program implementation Community participation Community assessment Program planning Staff hiring and/or volunteer recruitment Staff and/or volunteer training Outreach to and recruitment of potential participants Implementation strategy Evaluation strategy Timelines and benchmarks

Copyright © 2014 by The University of Kansas Impact The impact is the effect it had on the environmental conditions, events, or behaviors that it aimed to change Generally, a program aims only to influence one or more particular behaviors or conditions – risk or protective factors An evaluation of the risk and protective factors you targeted, will tell you if you made the desired impact How do you refine an intervention based on research?

Copyright © 2014 by The University of Kansas How do you refine an intervention based on research? Outcomes The outcomes of a program are the changes that actually took place because of it The goal of an intervention is usually not just a change in behavior (risk/protective factors), but the changes in community health that occur as a result of that change For instance, the impact of a tobacco control program is that people stop smoking. However, the intended outcomes are reduced rates of heart disease, lung cancer, and other smoking-related diseases in the population If your intervention is having the desired outcome, you should still strive to make the intervention better and more effective Ask yourself if there are additional risk and protective factors you could target, or areas of the program that could be expanded or strengthened?

Copyright © 2014 by The University of Kansas Outcomes If your intervention did not result in the outcomes you intended, consider what you might need to change Are there risk and protective factors that would be more relevant? Are there better ways to communicate your program’s message? Were there parts of the program that worked less effectively than others? Were there problems in how the program was delivered? Was it too formal? Were there unaddressed language barriers? Was the program culturally and contextually adapted for your situation? In evaluating the results of the program, it helps to work backwards to determine where the intervention went off track, starting with outcomes and working back to process to determine what area of the intervention needs to be adjusted for better results How do you refine an intervention based on research?