1. Some Practical Tips for Measuring Financial Success Dr. Angela Lyons University of Illinois Program Evaluation I: Setting the Stage for Outcomes-Based Success Presented by Dr. Angela Lyons University of Illinois October 2009

2. Motivation: Recent Financial Challenges Rising debt Low savings rates Home foreclosures Bankruptcies Unemployment Consumers are in financial trouble!!!

3. Reality Check! How much trouble are we in? 2009 U.S. deficit = $1.4 trillion

4. The Million Dollar Question At the end of the day, does financial education make a difference?

5. What have we learned so far? What does current research tell us? Why is evaluation important? What’s in it for you and your organization? What’s in it for your clients?

6. On the front lines with the evaluation experts…. “What do we mean by evaluation? “What are we really trying to measure?” “How do we define program success?” “How do we know if participants are improving?” “What financial outcomes and indicators should we be using?” “What constitutes a successful, or even acceptable, evaluation?”

7. Take stock of who you are – What is “your vision?” Conduct a needs assessment. Collect baseline information from your target audience. Identify your “signature program(s).” Identify your program objectives. Be realistic! Create an evaluation action plan. What do you want to accomplish? At the end of the day, what do you want to show? Getting Started: Thinking like an evaluator…. (Program Planning Guide)

8. Setting realistic expectations for program participants. Choosing appropriate outcomes and indicators based on participants’ financial situation or other external constraints. Identifying participants’ individual financial needs and applying appropriate educational interventions. Finding “the teachable moment.” How will you define “program success?”

9. What is an outcome-based evaluation? Outcomes are benefits to clients from participating in the program. What do you want your participants to know or be able to do when they have finished the program? Outcomes are usually in terms of enhanced learning and improved behaviors. Outcomes are often confused with program outputs or units of service (e.g., number of clients who went through the program).

10. The Logic Model A picture of the program. Simple representation of the program “theory” or “action” which explains the program and what it is to accomplish. Shows relationships between inputs, outputs, and outcomes.

11. The Logic Model (conti.) INPUTS OUTPUTSOUTCOMES Resources used to develop the program are called inputs. Time and money are the most common inputs needed to implement educational programs. If inputs are invested into the financial education program, then learning opportunities will be created for the target audience. The created educational materials, services, and opportunities are called the program outputs. Changes in participants’ perceptions, knowledge, and behavior that represent real impact in their lives. The benefits derived by the participants from the program are called outcomes.

12. University of Wisconsin - Extension http://www.uwex.edu/ces/pdande/evaluation/evallogicmodel.html

13. Impact Hierarchy of Outcomes Planned End-Result or Improved Socio- Economic Condition Changes in Behavior/Practices Changes in Knowledge, Attitudes, Skills, and Aspirations (KASA) Changes in Perceptions and Levels of Satisfaction

14. Another useful framework…. Transtheoretical Model of Behavior Change (TTM) TTM integrates major psychological theories into a theory of behavior change. Used to identify the state at which individuals are ready and able to change their financial behaviors. Appropriate educational interventions are then tailored to meet individual’s specific needs at that particular stage.

15. 5 Stages of Change Precontemplation Individual not ready to take action and change behavior in the immediate future. Rarely seeks help and rarely uses information. Contemplation Individual is getting ready to take action and intends to change behavior in next 6 months. Open to education. Preparation Individual is ready to take action and intends to change behavior in next 30 days. Practices behavior by taking small steps towards the goal. Seeks information and support, but often concerned that changing behavior may be too difficult and they may not succeed.

16. 5 Stages of Change (conti.) Action Individual changes behavior and maintains behavior for at least 6 months. Believes they can change. Can control “triggers” that cause them to relapse into old behaviors. Has a support system to get them through challenging times. Maintenance Individual has changed behavior and it has lasted for more than 6 months. May relapse into old behaviors, but can overcome temptations so that behavior becomes permanent. Can assess the conditions under which relapse might occur. Can establish successful coping strategies.

17. Example: Financial Practice Do not plan to do Plan to do in the next month Plan to do in the next 2-6 months Have been doing for 1-6 months Have been doing for more than 6 months Set short and long-term financial goals.  Save money regularly.  Use a spending plan to track income and expenses.  Maintain sufficient balances in bank account(s).  Pay bills on time each month.  Review bills each month for accuracy.  Comparison shop before making purchases.  Pay off credit card balances in full each month. 

18. Identifying Program Objectives Objectives should be: Specific Measurable Achievable and observable Reasonable Time specific S.M.A.R.T. objective statements should clearly define what you want to achieve with your program. They should list the end outcomes the program intends to affect or change.

19. Writing objective statements First-time home buyer education program The objectives of this program are to: Develop first-time home buyers’ ability to shop for the lowest mortgage interest rate. Teach first-time home buyers how to save money for closing costs. Teach first-time home buyers how to assess affordable housing. Debt reduction education program The objectives of this program are to: Develop participants’ ability to identify needs and wants separately. Develop participants’ ability to control “wants” to reduce expenditures. Develop participants’ ability to avoid impulse and emotional spending.

20. Achieving your objectives: Selecting appropriate indicators General Indicators (objective and subjective): Number of programs, participants, etc. Knowledge gains Changes in attitudes and satisfaction Changes in skills and confidence Changes in intended and actual behaviors Specific Indicators (objective): Actual dollar changes (reduce debt, increase savings) Development of financial plans Changes in spending habits Building or rebuilding credit reports and credit scores

21. Creating Your “Evaluation Road Map” Putting it all together….

22. Signature Program Program Objective 1 Indicator 1a Indicator 1b Indicator 1c Program Objective 2 Indicator 2a Indicator 2b Indicator 2c Program Objective 3 Indicator 3a Indicator 3b Indicator 3c Final program outcomes

23. Measurement error and validity of indicators.  Financial knowledge  Confidence level  Financial behavior EXAMPLE: Which of the following are valid indicators of behavior change?  Participant opened a bank account.  Participant increased savings.  Participant avoided bankruptcy.  Participant did not default on mortgage payments. A few words of caution when selecting indicators….

24. “What is driving this financial education movement? Why is it so important? What are we ultimately trying to address? Is it reducing the poverty gap in this country? Between those that have and those that don’t have. And it’s widening. And those at the bottom end of the spectrum….what we’re asking them is to build wealth. And at the same time, what we’re asking people in this country who make $20,000 or less is: ‘Absent us raising your wages in this country, we’re asking you to build wealth, to participate in IDA programs. We’re asking you to save with the little amount of money you’re making. We’re asking you to reduce your debt burden, learn how to manage your money, and clean up your credit history with the little amount of money you’re working with. And we want you to get from point A to point B with all those constraints.” Source: Lyons, A. C., Palmer, L., Jayaratne, K.S.U., and Scherpf, E. (2006). "Are We Making the Grade? A National Overview of Financial Education and Program Evaluation.” The Journal of Consumer Affairs, 40(2), 208-235. One non-profit administrator commented….

25. A few additional “words of wisdom” Avoid indicators that may be subject to biases.  Social desirability  Norms and “rules of thumb”  Misperceptions and over-optimism  Memory distortion and recall bias Avoid indicators that may ask sensitive or personal info.  Non-response bias  Program attrition  Self-selection  Low response rates (e.g., follow-ups)

26. Environmental factors can affect outcomes.  Unexpected life events  Program incentives (e.g., rewards, special benefits, enrollment programs)  Individualized financial advice or “coaching” Psychological factors.  Inherent motivation  Ability  Attitudes

27. There is no “magic set” of outcomes or indicators. The list is endless…. Increases in savings. Decreases in debt. Maintaining a regular budget. Comparison shopping. Increases in new accounts opened. Improved credit scores. Improved communication with spouse/partner/parents about finances. Other common indicators?

28. How do these indicators change for various target populations? Youth? Underserved? Adults? Members of your organization?

29. What is your signature program (e.g., course, workshop, educational materials, initiative, campaign)? How are you going to define program success? At the end of the day, what do you want to show? Who is your target audience(s)? What is your delivery method(s) (e.g., in-person, telephone, Internet)? What financial and non-financial resources are available to you for evaluation? Who will use the evaluation and how will it be used? ACTIVITY: Your Evaluation Action Plan – Part A

30. Summary: 4 Steps to Creating a Successful Evaluation Action Plan Step 1: Define objectives of the program. At the end of the day, what do you want to show? Who will be the target audience? Who will use the evaluation and how? Step 2: Select appropriate outcomes, indicators, and methods. What is the most appropriate evaluation format? What types of questions will the evaluation seek to answer? What types of indicators will be used to show impact?

31. Step 3: Identify resources and constraints. What financial and non-financial resources are available? Are there others who can help? What is the program timeline? Given constraints, what can you realistically do? Step 4: Analyze and disseminate findings. How will data be analyzed? What do you hope to learn from findings? What are the potential impacts? How will the results be used and disseminated?

32. Where do we go from here? Evaluation resources at your fingertips

33. U of I Center for Economic and Financial Education http://www.cefe.illinois.edu/

34. Educational Tools for Evaluation http://www.cefe.illinois.edu/tools/evaluation.html

35. Checklist of evaluation resources Program Planning Guide Evaluation Action Plan Evaluation Road Map Evaluation Reading List Sample Evaluations NEFE ® Financial Education Evaluation Toolkit http://www.cefe.illinois.edu/tools/evaluation.html

36. University of Wisconsin-Extension http://www.uwex.edu/ces/pdande/evaluation/index.html

37. Cornell University Extension http://staff.cce.cornell.edu/administration/program/evaluation/evalrefs.htm

38. Penn State Extension http://www.extension.psu.edu/evaluation/

39. Program Evaluation II: Creating Your Evaluation Toolkit More to come soon….

40. Contact Information Dr. Angela Lyons Associate Professor Director, U of I Center for Economic and Financial Education University of Illinois Phone: 217-244-2612 E-mail: anglyons@illinois.edu

41. Questions