Presentation is loading. Please wait.

Presentation is loading. Please wait.

Presented by Cindy Ofstead and Jane Mahoney January 2019

Similar presentations


Presentation on theme: "Presented by Cindy Ofstead and Jane Mahoney January 2019"— Presentation transcript:

1 Presented by Cindy Ofstead and Jane Mahoney January 2019
Basics of Evaluation Presented by Cindy Ofstead and Jane Mahoney January 2019

2 Basics of Evaluation Learning Objectives:
Learn the difference between counting activities and evaluating outcomes Understand the importance of measuring the effectiveness of your programs Define the broader goals of your programs Discover four program features to measure Learn how to create a good survey

3 Counting Activities vs. Evaluating Outcomes
Counting activities shows how much work is done, how many meals served, how many people “reached,” etc. Evaluating outcomes is measuring effectiveness –what difference the program/service made to the people served Increasingly, we need to explain why our programs exist and provide evidence that they should continue and grow.

4 Define Broader Goals WHAT SHOULD PROGRAMS ACCOMPLISH?
EXAMPLE: The goal of the Nutrition Program is more than just supplying a meal. Other goals of the Nutrition Program are: Increase access to nutritious food Increase knowledge about nutritious food

5 Define Broader Goals Other goals of the Nutrition Program cont.
Increase skill in obtaining and preparing nutritious food Increase opportunities for social interactions and reduce isolation Increase opportunities for engagement - volunteering

6 Define Broader Goals (more examples)
EXAMPLES FROM OTHER PROGRAM AREAS: Advocacy – more than advocating for older people Empower people to be advocates Increase exercise of advocacy Healthy Aging – more than holding classes Improve self-management of health Reduce falls, use ER less, etc.

7 Define Broader Goals (more examples)
Caregiver/Dementia – more than offering respite and adaptive equipment Reduce caregiver burden Increase practice of self care Increase understanding of dementia Decrease stigma in community **How do we document that we met these other goals of the program?

8 Moving Towards an Evaluation Approach
Evaluating outcomes is not always easy Need tools to measure effectiveness Need training on how to do this well Training, information and materials will be provided through ACE meetings, webinars and website

9 Moving Towards an Evaluation Approach
Please provide feedback on how useful this is Please share your examples of what has worked for your programs Please let us know what additional tools are needed

10 Four Features to Measure
Education – Were learning objectives met? Was awareness raised? Was knowledge gained? e.g. nutrition education, dementia awareness, advocacy training Tools: Pre-post test of knowledge or awareness

11 Four Features to Measure
Service – Did something improve for the customer? e.g. reduced caregiver burden, increased socialization Tools: Pre-post measure of experience or conditions

12 Four Features to Measure
Behavior Changes – Are healthy behaviors now present? e.g. regular exercise routine, preventive services utilized, eating healthy foods Tools: Pre-post survey about behaviors or practices

13 Four Features to Measure
Customer Service – Was the customer satisfied with the interaction? This is a baseline across all program areas: if the customer is unhappy they won’t participate and won’t encourage others to participate. Tools: Customer Satisfaction Survey

14 Customer Satisfaction Survey
A Customer Satisfaction Survey measures how happy or unhappy the customer was with an experience or service. It is a fixed set of relatively simple questions It should be anonymous Customers should complete with no assistance from staff It should be returned via mail or web

15 Important Considerations
Questions need to be simple and straightforward You want to hear from happy and unhappy people Your population may or may not have access to a computer or the internet

16 Avoid Bias Be as neutral as possible so respondents do not feel pressure to answer in a false-positive manner. The following helps to avoid bias: Provide a means of anonymous response Clearly state confidentiality assurances and follow them Always pre-pay return postage Provide translation options

17 Avoid Bad Questions Make sure questions are accessible to all (language, cultural issues) Make sure questions are clear and unambiguous Avoid “leading” questions Prevent interviewer interpretation or influence on responses

18 The Alarm Bells… Anything that predisposes a positive or negative answer Anything that makes dishonest or ambiguous answers more likely Anything that makes responses hard to interpret CINDY: These will all spring to life when we look at concrete examples that illustrate each problem.

19 The Alarm Bells… Anything that could exclude or intimidate those who are not happy Anything that excludes people who may need additional services not currently offered **The principles of the Customer Satisfaction Survey apply to all surveys and evaluation tools.

20 Taken from real surveys!
Examples of Good & Not-So-Good Questions Taken from real surveys!

21 Category Gaps Do you live: In a city In a small town On a farm?
What if you live in a suburb or in the country but not on a farm?

22 Try this instead… Where do you live? In a city or suburb
In a small town or village In a rural area outside town/village boundaries **Make sure there is an option that fits all situations.

23 Multiple Questions in One
The switchboard staff spoke with me courteously and transferred me to the appropriate person for help. Yes No What if the switchboard staff was rude but I was transferred to the proper person where I received help?

24 **Ask one thing in each question.
Try this instead… The receptionist spoke with me courteously. Yes No The receptionist transferred me to the appropriate person for help. **Ask one thing in each question.

25 Misfit Questions: When No Answer Fits
What prompted you to come to the Aging Unit in the first place? Information and Assistance Long-Term Care Options Counseling Benefit Specialist Eligibility Determination What if I came for caregiver support or to sign up for home delivered meals?

26 Try this instead… What services have you used at the Aging Unit?
Nutrition Caregiver Services Benefit Specialist Healthy Aging Programs Other None **List the programs you particularly want to track, but add “other” and “none” as a catch-all.

27 Another example… How did you hear about the Aging Unit?
Friend/relative Agency newsletter Local newspaper Radio ad Other _________________ **Again, choose the specific resources you want to track and add “other” for anything not listed.

28 Confusing Questions Do you feel that from the time you contacted the Agency and when the Agency contacted you to arrange an appointment/referral was reasonable? Yes No CINDY: What are they trying to ask???

29 Try this instead… How would you rate the amount of time that passed until you discussed your situation with the staff person? Very long Long Neutral/Don’t know/No opinion Short Very short **Ask the question in a clear manner, with a non-biased rating scale for answer choices.

30 Leading Questions Questions worded in a way that tell respondents exactly what you want to hear. Respondents may feel compelled to “make you happy” or to avoid offending you. Respondents may comply so they are not alienating themselves from additional services.

31 Examples of Leading Questions
Check “Yes” or “No” for each statement: I liked the service I received. ___Yes ___No I am comfortable visiting this agency. I would recommend this agency to a friend. ___Yes ___No

32 Try this instead… How satisfied or dissatisfied were you with the service you received? Very satisfied Satisfied Neutral Dissatisfied Very dissatisfied

33 How comfortable or uncomfortable did you feel visiting this agency?
Very Comfortable d. Uncomfortable Comfortable e. Very Uncomfortable Neutral How likely are you to recommend this agency to others? Very Likely d. Unlikely Likely e. Very Unlikely

34 Tips to Avoid Leading Questions
Keep questions as neutral as possible to elicit a true response. Use a rating scale to allow for a wider range of choices. Their answer may not be a definitive Yes or No. Include both the positive and negative when writing your question so you are not pushing a positive response. CINDY:

35 Consequences of Bad Questions
No response – left it blank i.e. they couldn’t figure out how to answer it Text responses to check-box questions i.e. they wrote in an answer because their honest answer wasn’t listed as an option Ambiguous responses i.e. answers do not make sense because question didn’t make sense CINDY:

36 Consequences of Bad Questions
Response errors i.e. checking more than one age category Quitting the survey early too frustrated to finish too many questions that don’t make sense too few questions may not give an opportunity to give a true picture about their whole experience

37 Consequences of Bad Questions
Misleading positive responses i.e. they gave the answer they thought you wanted Throwing the whole survey away! If there are too many errors, ambiguous responses or incomplete surveys, participants may just give up on it.

38 In Summary Work needs to be done to improve the questions used in surveys Scrutinize your survey questions to see if they meet these criteria: Clear and easy to understand Open to full range of responses All possible responses are accounted for Allow for negative responses

39 Good questions lead to surveys that provide helpful information

40 Customer Satisfaction Tools
Caregiver Customer Satisfaction Survey Home Delivered Meals Satisfaction Survey Congregate Meals Satisfaction Survey **More tools will be forthcoming **Send specific requests to Jane Mahoney, GWAAR

41 Coming Soon! Watch for the next presentation on Evaluating Program Success, where we will talk about more tools including: Pre and Post Test Evaluation Survey Follow-up Interview Scale Measurement Tools

42 Contact information: Jane Mahoney, OAA Consultant (608) Evaluate this webinar!


Download ppt "Presented by Cindy Ofstead and Jane Mahoney January 2019"

Similar presentations


Ads by Google