The State & Community Intimate Partner Violence (IPV) & Perinatal Depression (PD) Resource Development Project: Evaluating Your Efforts Health Resources and Services Administration & Social Solutions International, Inc. January 2013 IPV/PD: A Comprehensive Approach

Webinar Speakers and Guests Moderator: Keisher Highsmith, DrPH Director of Special Initiatives and Program Planning and Evaluation HRSA/Maternal and Child Health Bureau Speaker: Kristen Stier, MA Research Associate Social Solutions International, Inc. Speaker: Jenny Namur Karp, MPH President Social Solutions International, Inc. IPV/PD: A Comprehensive Approach

Objectives Explain the purpose of evaluation research Introduce types of evaluations, specifically process evaluations and outcome evaluations Provide information on the challenges and benefits of program evaluation IPV/PD: A Comprehensive Approach

Toolkit Goals – Evaluating Your Efforts Introduce simple types of evaluations, how to collect information and feedback, and what to do once information is collected Provide two tools and multiple resources to assist you in evaluating your efforts IPV/PD: A Comprehensive Approach

Evaluation Basics Jenny Namur Karp Social Solutions International, Inc. January 2013 IPV/PD: A Comprehensive Approach

Why Evaluate? Understand, confirm or increase the impact of products of services on clients Improve service delivery mechanisms to be more efficient and less costly Verify that you are doing what you think you are doing Produce data or verify results that can be used for public relations and promoting services in the community Produce valid comparisons between programs to decide which should be retained Fully examine and describe effective programs for duplication elsewhere IPV/PD: A Comprehensive Approach

Role of Evaluation Research Accountability Assess program quality and continue quality improvement in the program Assess attainment of program goals Determine program effectiveness Success of new efforts Enable program duplication IPV/PD: A Comprehensive Approach

Evaluation Research: True/False #1 # 1: Evaluation research is about producing boring, scientifically generated data with useless conclusions. – False: More recently evaluation has focused on utility, relevance and practicality at least as much as scientific validity. IPV/PD: A Comprehensive Approach

Evaluation Research: True/False #2 # 2: Evaluation research is about proving the success or failure of a program. – False: This implies that implementing the perfect program means never having to assess or critique it’s impacts. Continual feedback and program adjustment is beneficial to all. IPV/PD: A Comprehensive Approach

Evaluation Research: True/False #3 # 3: Evaluation research is very difficult and time consuming and cannot be done without outside experts. – False: Many of you probably partake in evaluation research regularly even if you do not know it. Part of the goal today is for you to realize the benefits of evaluation research and how to use it to make the most impact with your program. IPV/PD: A Comprehensive Approach

Specific Evaluation Questions What are your program goals? What are your target outcomes? What are the steps to achieve the outcome? What are the anticipated results? During and after program you will learn real results What is the analysis and action plan? Meaning what will you analyze and how Who are the person or people responsible? Who is responsible for what goal/outcome? What is the time frame? Will you complete the evaluation weekly, monthly, annually ? IPV/PD: A Comprehensive Approach

Process Evaluation IPV/PD: A Comprehensive Approach

Process Evaluation Monitor the implementation process of an intervention Determine whether or not the intervention is delivered as planned (fidelity; manual-based interventions) Document challenges to the project, barriers encountered, and strategies used to address those barriers (case studies) Determine whether or not the project is delivering the intervention to the intended target IPV/PD: A Comprehensive Approach

Elements of Process Evaluation A thorough process evaluation should include the following elements: Description of the program environment and supplying data. Description of the process used to design and implement the program. Description of program operations, including any changes in the program. Identification and description of intervening events that may have affected implementation and outcomes. Documentation such as meeting minutes, reports, memorandums, newsletters, etc. IPV/PD: A Comprehensive Approach

Sample Process Evaluation Methods and Measures Conduct observations at project meetings, site visits Conduct focus groups with participants and staff re: their perceptions of the project Track the number of clients served; their background characteristics Track services received, time in program, type of discharge IPV/PD: A Comprehensive Approach

Questions Answered with Process Evaluations Are we implementing the program as planned? What aspects of the program are strong? Which ones are weak? What can we do to continue to strengthen our program? Can we improve program efficiency? Are there unexpected effects? Did problems arise? IPV/PD: A Comprehensive Approach

Benefits of Process Evaluations Accountability To better understand the day-to-day function of your program Program development and improvement To help others set up similar services or networks IPV/PD: A Comprehensive Approach

Outcome Evaluation IPV/PD: A Comprehensive Approach

Outcome Evaluations Determine the effects of participation in the project on the participants Determine the factors (individual and programmatic) related to favorable/unfavorable outcomes Determine the relationship between “dose” (amount of the time in the program, number of services received) and “response” (outcomes observed) IPV/PD: A Comprehensive Approach

Elements of Outcome-Based Evaluation? Outcome = Impact on End User Impact = Changes in: – Behavior – Attitude – Skills – Knowledge – Condition/ Status IPV/PD: A Comprehensive Approach

Continuum of Program Outcomes Condition/ Status Behavior Skills Knowledge Community connections Social networks Attitudes/ Values Perceptions/ Feelings Satisfaction with service Participation/ Use of service Awareness of service Note that changes in individuals (outcomes) usually occur in this order, from bottom to top. The most difficult changes are those at the top of the list. IPV/PD: A Comprehensive Approach

Sample Outcome Evaluation Methods Self-Assessment of participants and/or trainers Participant demonstration of knowledge and skills Questionnaires and Surveys of participants and trainers Interviews with participants and trainers Focus groups with participants and trainers Assess products developed IPV/PD: A Comprehensive Approach

Questions Answered with Evaluation Questions What did you do? Who did you do it to? How many people did you serve? What did they get (or not get)? What effect did it have on their attitudes and behavior? Can other programs use this information? How does this information influence other programs? Were there unexpected difficulties? IPV/PD: A Comprehensive Approach

Benefits of Outcome-Based Evaluations Useful as a planning tool; requires needs assessment Clarifies the purpose of the program/service Keeps staff and stakeholders focused on ultimate goals Stimulates discussion of issues Helps keep implementation on track; milestones are identified Indicates when changes to program are needed Energizes staff by demonstrating the real, human impact their work produces and by stressing common purposes and goals Enhances program staff’s knowledge and understanding of how to interpret data IPV/PD: A Comprehensive Approach

Additional Benefits of Outcome-Based Evaluations Gives insight into why and how program services are used Assists in fundraising and grant writing by providing statistics on results Provides empirical evidence that what you are doing is what you intended to do Quantifies the anecdotes and success stories Identifies effective programs/services Enables programs to focus on what is most successful as well as to improve what is not Demonstrates the ways your program contributes to solving community problems IPV/PD: A Comprehensive Approach

Designing an Evaluation IPV/PD: A Comprehensive Approach

Key Considerations in Designing the Evaluation For what purposes is the evaluation being done? Who are the audiences for the information from the evaluation? What kinds of information are needed to make the decision you need to make and/or enlighten your intended audiences? From what sources should the information be collected? How can that information be collected in a reasonable fashion? When is the information needed? What resources are available to collect the information? IPV/PD: A Comprehensive Approach

Sample Data Collection Remember, this will be project-specific Background info (demographics, education, employment, income, residential status) Family & social support Health status & use of health services Criminal behavior history History of trauma and violence What else can you think of that is relevant to your program? IPV/PD: A Comprehensive Approach

Typical Challenges Encountered in Conducting Evaluations Seems too difficult Not thought out ahead of time Lack of integration of project and evaluation- related aims/activities Lack of communication or misunderstandings between project and evaluation, often from differences in training, orientation, goals Evaluation requirements viewed as intrusive or burdensome by project staff Interviews are key, but don’t just interview the successes. Fear of being graded or judged as performing poorly by the evaluators IPV/PD: A Comprehensive Approach

Concluding Thoughts Clearly, evaluation takes planning The effort put into evaluating your programs and planning the evaluation as early as possible serves the ultimate goal of effective programming and helping your clients as best as you can, always Evaluation is a continuing process The time added is repaid in improved programs and improved communities IPV/PD: A Comprehensive Approach

Creating a Culture of Evaluation An ideal culture of evaluation would encompass several values, including: Action-oriented – a culture that actively seeks to problem solve. Teaching-oriented – evaluations that can be used and understood by non- technicians Inclusive and non-hierarchical – solutions to problems involve a diverse range of participants Source: Trochim, William M.K. (2006, October 20). An Evaluation Culture. Retrieved from IPV/PD: A Comprehensive Approach

Creating a Culture of Evaluation, cont. Self-critical – a culture that recognizes its limitations Interdisciplinary – multi-disciplinary research groups are beneficial Forward looking – a perspective of seeking where evaluation can be helpful, not just when problems are apparent Open and democratic – evaluation data should be accessible to all Source: Trochim, William M.K. (2006, October 20). An Evaluation Culture. Retrieved from IPV/PD: A Comprehensive Approach

Resources Resources for Methods in Evaluation and Social Research This website contains a catalogue of free evaluation tools on the Internet US Department of State, Bureau of Educational and Cultural Affairs, Office of Policy and Evaluation s/index.htm IPV/PD: A Comprehensive Approach

A Comprehensive Approach for Community-Based Programs to Address the Intersection of Intimate Partner Violence & Perinatal Depression: Evaluation of Your Efforts Kristen Stier, MA Social Solutions International, Inc. IPV/PD: A Comprehensive Approach

Evaluating Your Efforts Evaluation Can Take Many Forms: Assess how well your IPV/PD initiative is working Gather feedback, data, and information about your initiative and its activities that you can share with funders and members of your community. Define success for your community. IPV/PD: A Comprehensive Approach

Why is Evaluation Important? Assists in securing money and in-kind resources for the initiative Garners community support for and involvement in the initiative Helps to overcome resistance to the initiative Generates ideas about how the initiative can be more effective IPV/PD: A Comprehensive Approach

How Can Success Be Measured? Telephone survey - an opportunity to measure community opinion regarding various issues related to your initiative. Survey of goals -- a paper-pencil survey distributed to community members by mail that asks them to rate the importance and feasibility of your community-based initiative’s goals. Survey of satisfaction -- a paper-pencil survey distributed to members of the initiative that asks them to rate their satisfaction with issues such as the leadership, community involvement, and planning of your community-based initiative. IPV/PD: A Comprehensive Approach

How Can Success Be Measured, cont. Behavior survey(s) -- a paper-pencil survey distributed annually to your target population to assess their knowledge, attitudes and behaviors in relation to your goal. (For example, you may want to survey women who utilize your IPV/PD services to gauge their knowledge, attitude and behaviors). Interviews with key participants -- semi- structured focus group interviews that clarify important events in the life of the initiative and illustrate the value that the initiative has added to the community. Survey of outcomes -- a paper-pencil survey that asks initiative members, funders, and outside experts to comment on and rate the significance of changes made in the community. IPV/PD: A Comprehensive Approach

Measuring Success, cont. Goal attainment reports -- reports that identify progress toward proposed changes identified in the action plan. On-line documentation system -- an Internet- based system that gathers information regarding the daily activities of the initiative, including community actions, community changes, and services provided. Community-level indicators -- information relevant to your initiative that is collected from local, state- or regional-level sources. (For example, you may want to gather statistics about rates of IPV or PD in your state or region that might already be available and track any changes that occur throughout the years). IPV/PD: A Comprehensive Approach

Post Assessment Tool Have there been improvements in awareness towards addressing IPV/PD among staff at the organization? Is IPV/PD a part of new staff training at the organization? Are there guidelines and procedures in place for referring clients who are experiencing IPV/ PD? Do staff routinely refer participants to other programs or organizations for IPV/PD services? Has the organization been able to build a referral and/or partnership network? Does the physical environment (clinic/office) have culturally competent posters, brochures or awareness materials related to IPV/PD? Does the organization utilize National Standards of Care Guidelines for IPV and PD? IPV/PD: A Comprehensive Approach

Contact Us Jenny Namur Karp, MPH Social Solutions International, Inc Georgia Avenue Suite 201 Silver Spring, MD Phone: Fax: Kristen Stier, MA Social Solutions International, Inc Georgia Avenue Suite 201 Silver Spring, MD Phone: Fax: IPV/PD: A Comprehensive Approach

Additional Contact Information For more information, questions, or comments about The State & Community Intimate Partner Violence (IPV) & Perinatal Depression (PD) Resource Development Project, please contact: Keisher Highsmith, Dr.P.H. Project Director Department of Health and Human Services Health Resources and Services Administration Maternal and Child Health Bureau Division of Healthy Start and Perinatal Services IPV/PD: A Comprehensive Approach