Presentation is loading. Please wait.

Presentation is loading. Please wait.

Beginning with the End in Mind: What We Know about Performance Measurement for VMF Programs If you have not already done so, please connect to the audio.

Similar presentations


Presentation on theme: "Beginning with the End in Mind: What We Know about Performance Measurement for VMF Programs If you have not already done so, please connect to the audio."— Presentation transcript:

1 Beginning with the End in Mind: What We Know about Performance Measurement for VMF Programs If you have not already done so, please connect to the audio portion of this Webinar: 1. Dial 1-877-668-4490 2. Enter the access code 622 071 435 # 3. Enter your unique participant ID number displayed on your screen OR choose the option to let the program call you back!!!

2 Housekeeping Phones are muted Ask questions by sending a chat message to All Participants using the chat panel on the right side of your screen

3 Whos on the Webinar Representing Programs and Organizations from 44* States 175 Attendees

4 Beginning with the End in Mind: What We Know about Performance Measurement for VMF Programs Outcomes You will: a.Learn about Performance Measurement for Veteran and Military Family Programs (VMFP) b.Discover methods to measure outputs and outcomes with available tools c.Learn about VMFP data sources to help measure outputs and outcomes d.Explore best practice examples in measuring outcomes from Equal Justice Works

5 Webinar hosts: Education Northwest Hello and Welcome from CNCS! Koby Langley Corporation for National and Community Service

6 *Service to VMF *Creating Service/ Volunteer Opportunities for VMF Education and Certification Employment Access to Benefits Youth Mentoring Transportation Disaster Preparedness Wellness and Other Support Services *Community Coordination * Overarching activities of programs in SAA Specific activities Mentioned in SAA The Serve America Acts Influence on CNCS VMF Performance Measures

7 Session Presenters Dr. Chris Spera, CNCS Director of Research and Evaluation Kerry OBrien, Director of Federal Programs and Strategic Initiatives Equal Justice Works Kathryn Gravely, Program Manager, Federal Programs and Strategic Initiatives, Equal Justice Works

8 Key National Service SAA Issues Areas by Program

9

10 Wellness/Other Supports and Services by Program (n=59)

11 Performance Measurement for Veteran and Military Family Programs What is performance measurement? A systematic process of tracking outputs and outcomes based on the goals of your program, as articulated in your programs logic model What is a logic model? A logical chain of connections showing what the program is to accomplish

12 Why is Performance Measurement important for CNCS? Assess the individual and collective results of programs and continue to enhance program effectiveness Expand CNCSs ability to account for the combined contributions of all organizations making use of our resources

13 Why is Performance Measurement important for VMF Programs? Assesses the individual and collective results of VMF programs by tracking outputs and outcomes related to improving Military family strength and resilience Quality of life for veterans, i.e.,: Transition to civilian life Education attainment Job/vocational skills Behavioral health

14 Performance Measurement for VMFP: Collecting Data on Outputs and Outcomes Tools to measure outputs: Intake forms, attendance and tracking sheets to keep count Tools to measure outcomes: Pre, mid-term and post program surveys designed to measure increases in specific areas over time, as a result of participation in the program, focus groups, interviews Where to find instruments, tools and resources for performance measurement: http://www.nationalservice.gov/resources/npm/home search by focus area

15 Performance Measurement for Veteran and Military Family Programs Important point: Performance measurement is NOT evaluation -Unlike performance measures, evaluations estimate the impacts of programs by comparing the outcomes for individuals receiving a service or participating in a program to the outcomes for similar individuals not receiving a service or not participating in a program. - Code of Federal Regulations for CNCS http://www.gpo.gov/fdsys/pkg/CFR-2010-title45-vol4/pdf/CFR-2010-title45-vol4- part2522-subpartE-subjectgroup-id1784.pdf

16 Performance Measurement for Veteran and Military Family Programs Performance measures track outputs and outcomes based on the goals of your programs logic model A logic model shows the chain of connections illustrating plans to meet community needs; the theory of change INPUTS OUTPUTS OUTCOMES Program investments Activities Participation Short MediumLong-term What we invest: People, time Service provided: # of in-take & referrals Service recipients: # of returning veterans receiving referrals What results/changes: Veterans placed in employment prep programs; then employed Need: access to employment resources

17 Outputs for Veteran and Military Family Programs Defining outputs Amount of service provided (e.g., people served, products created, or programs developed) through the service intervention of your program Common outputs for VMF programs # of veterans or active Service members and families served # of veterans or active Service members engaged in service provision through CNCS programs Others?

18 Outputs for Veteran and Military Family Programs – Program Examples Veterans Helping Veterans Now # of veterans referred to service providers that can meet their needs Metro Community College, Omaha, Nebraska # of people who received outreach from VISTAs regarding support needed by veterans returning to college CA Conservation Corps # of veterans employed as crew members

19 Outcomes for Veteran and Military Family Programs Defining outcomes Reflect the changes in attitudes/beliefs, knowledge/skills, behavior, or conditions of individuals that occur in the organizations, communities, or environment as the result of the services provided Common outcomes for VMF programs Increase in educational attainment Increase in vocational skills and credentials (e.g., professional licensure) Increase in employment outcomes Others?

20 Outcomes for Veteran and Military Family Programs– Program Examples Veterans Helping Veterans Now # of veterans receiving services as result of referral who then gain employment (for example) due to services received Metro Community College, Omaha, Nebraska # of veterans succeeding in college due to support of informed college staff and faculty CA Conservation Corps # of veterans who completed the program and gained key skills for future employment

21 Sources of Data Collection Self-reported data through national service members and volunteers (e.g., # of veterans served) Self-reported data from veterans through surveys (e.g., percentage of veterans served that obtain employment post- service) Public use and administrative data sources (e.g., data from local VA organizations) Observation – outside observers making ratings, counting behaviors

22 Self-Report Data from Service Members Key Question: Who are you serving? Self-reported data through national service members and volunteers Example: # of veterans served Example: # of military families served Important to know penetration rate (i.e., percent of veterans served in relation to total size of veteran population in your community) VA data source, called VETPop, has publicly available data source on veteran population by zip code

23 Self-Report Data from Beneficiaries Key Question: What outcomes are you achieving? Potential Outcomes: Increase in educational attainment Increase in vocational skills and credentials (e.g., professional licensure) Increase in employment outcomes Data Sources: Self-report survey data from veterans/military Public use data sources

24 Public Use Data Sources of Interest Demographic Data on Population Veterans Population (VETPoP) DMDC (DOD) – demographics report Public Use Survey Data Health (Wellness) Survey of Army Families (SAF) Air Force Community Assessment Survey National Survey of Veterans

25 Public Use Data Source of Interest Public Use Survey Data (Cont.) Health (Behavioral Health) Survey of DoD Health Related Behaviors MHAT Screenings Sexual assault survey Employment VBA Vocational Rehab Survey Homelessness VA/DoL homeless counts

26 Administrative Data Sources of Interest Administrative Data Homelessness DoL VETS data on HVRP data Education Data on GI bill usage (VA) Vocational Rehabilitation data Employment Transition Assistance Program data Local data sources from DVOPs, LVERs, etc.

27 Tools and Resources for Performance Measurement http://www.nationalservice.gov/resources/npm/home

28 Equal Justice Works Kathryn Gravely Program Manager Kerry OBrien Director of Federal Programs

29 Equal Justice Works Overview Mission –Equal Justice Works is a 501(c)(3) nonprofit organization dedicated to creating a just society by mobilizing the next generation of lawyers committed to equal justice AmeriCorps Program –40 full-time lawyer members in 10 states. Most members serve for two years –360 minimum-time law student members in 48 states. Most members serve 300 hours in the summer –Placement sites are legal aid organizations and courts working with low-income residents, including veterans

30 Equal Justice Works: Veterans Theory of Change Summary Need: Veterans have complex legal problems that keep them homeless and in poverty Intervention: Lawyers and law students, in conjunction with veterans service infrastructure, resolve legal problems Evidence: 2012 Evaluation, Studies show that those with lawyers fare better than those without Outcome: Lives of veterans are improved, increased income, access to health care, better housing

31 Equal Justice Works: Veterans Theory of Change Need –Low-income and homeless veterans face complex legal problems related to disability benefits, fines, warrants, housing, employment, debt, and family issues –Lawyers are uniquely equipped to resolve such problems Data Documenting Need –2011 V.A. CHALENG report; Research articles on veterans & need for lawyers; HUD, HHA and Veterans Benefits Administration statistics; PTSD and TBI studies (e.g. 3 of top ten unmet needs of homeless vets are legal needs – CHALENG)

32 Equal Justice Works: Veterans Theory of Change Intervention –Lawyers and law students will resolve the legal problems of low-income and homeless veterans, leading to improved lives: higher incomes, better health, safe and secure housing and family stability Intended Outcome –Improve the lives of veterans Evidence-basis for Intervention –Impact of 2010-2012 Veterans Pilot Project –Studies that lawyers make a difference

33 Equal Justice Works: Outputs vs. Outcomes Overarching Goal –Through legal assistance, improve the lives of veterans Outputs: Amount of Service Provided –Example: Assist 30 veterans with benefits claims Outcomes: Reflect the Changes that Occur –Example: 20 veterans win benefit cases, providing access to health care and $360,000 in additional yearly income

34 Equal Justice Works: Outcomes IncomeEmploymentHousingOther VA Disability Benefit cases won $ benefits received % increase in income Fines/Penalties waived # of Minor Criminal Records Sealed or Expunged # Employed after legal barrier removed % increase in wages # Vets housed after legal barrier removed # Vets remaining in housing Access/better access to health care Discharge upgrades (leads to income, housing, access to health care) Wills for the dying

35 Equal Justice Works: Data Collection What we could say in 2010 –1,500 veterans served –Compelling stories about individual cases What we could not say in 2010 –Outcomes of cases –Impact on veterans served –Specific information on veterans served (e.g., demographics, legal issues addressed, etc.) –Which sites were performing well/needed assistance to meet goals

36 Equal Justice Works: Data Collection Today, we can say: –1,500 veterans served –Members completed 226 thorough case evaluations –Filed 91 disability benefits petitions or appeals –Thus far have won 29 of these cases, securing $642,668 in back benefits and $430,933 in yearly benefits for their veteran clients –114 benefits cases are still pending –Successfully able to remove a barrier to employment for 127 veterans served –Demographic information specific to veterans served (e.g., era of service, age, gender) –Number of veterans served for each particular legal issues, both on macro and micro levels

37 Equal Justice Works: Data Collection Today, we can say: –Fines or other penalties waived or reduced for 78 veterans served, totaling $36,771 –Assisted 19 veterans in getting their criminal matters dismissed or expunged –Assisted 118 veterans in repairing their credit –Of the 1,416 veterans served, 740 were homeless or at risk of becoming homeless, and 739 were veterans with a disability –77% were male and 23% were femalemeaning that Fellows served a high number of female veterans, as 10% of veterans in the U.S. are female

38 Equal Justice Works: Data Collection Today, we can say: –20% of those receiving services served in the Vietnam era, and 23% served in Iraq and Afghanistan –The remaining veteran clients served in all other eras of service, including pre-Vietnam, the Gulf Wars and during peacetime –The majority of veterans served were either between 26 and 35 years of age or 55 and older (216 and 420 served, respectively)

39 Equal Justice Works: Data Collection How?

40 Equal Justice Works: Data Collection 2011 –Conducted interviews and focus groups about outcomes 2012 –Implemented new optional measures early 2012 –Brought supervisors together in person – October –Outside evaluation completed – December 2013 –Data resulted in stronger application for funding – Feb –Implemented refined measures & made them mandatory in March 2013 –Report submitted May 1 substantially better

41 Equal Justice Works: Data Collection Collection Format: –Was: Custom-built web application Cost an hourly charge every time data needs changed Was getting expensive Output was Excel spreadsheet –Now: Google Drive - Forms Program manager and coordinator designed, refined Output is Excel spreadsheet Drive.google.com – Create Forms

42 Equal Justice Works: Data Collection in Google Forms

43 Equal Justice Works: Great Stories –Equal Justice Works AmeriCorps Legal Fellow Rochelle Richardson –Homeless Persons Representation Project in Baltimore, MD –Many years experience as an employee of the VA –Reexamine the case of Mr. A, a homeless veteran with a service- connected mental disability whose claim for veteran's disability benefits had already been denied –Rochelle obtained records that had not previously been reviewed and petitioned to reopen the claim and submit further evidence –The petition was successful & the claim was granted in March 2013 –The client will receive a retroactive one-time payment of $50,000 and $3,000 a month in benefits

44 Equal Justice Works: Lessons Learned Define intended outcomes early Align reporting with what funders require Have performance measures and data collection tools ready at the beginning of program year Have a steering committee of on the ground experts to consult Keep refining and improving measures to more accurately reflect program impact

45 Equal Justice Works: Next Steps Figure out how to track and capture long-term outcomes –Example: Did increased income result in better housing, better health, better outcomes for children of veterans served? Comparison Evaluation –Compare veteran outcomes in our program with national statistics or other programs Big Vision –Randomized control trial

46 Performance Measurement for Veteran and Military Family Programs: Re-Cap Performance Measurement: –Tracks outputs and outcomes based on the goals of your program to meet community needs Outputs: –Amount of service provided (e.g., veterans) through the service intervention of your program Outcomes: –Reflect the changes that occur as the result of the services provided (e.g., veterans employed)

47 Questions??? Dr. Chris Spera CNCS Kerry OBrien Equal Justice Works Kathryn Gravely Equal Justice Works

48 To Continue this Discussion…and Others Veterans and Military Families Knowledge Network www.nationalserviceresources.org/veterans http://bit.ly/VetsNet

49 Join Us for the Next Webinar in this Series June 12, 2013 Making a Difference: What We Know about Using an Evidence-based Approach Register: http://tinyurl.com/c8gp8ll

50 Thank you! Please complete the post-webinar survey


Download ppt "Beginning with the End in Mind: What We Know about Performance Measurement for VMF Programs If you have not already done so, please connect to the audio."

Similar presentations


Ads by Google