1. Presented by Cindy Ofstead and Jane Mahoney January 2019
Basics of Evaluation
Presented by Cindy Ofstead and Jane Mahoney
January 2019

2. Basics of Evaluation Learning Objectives:
Learn the difference between counting activities and evaluating outcomes
Understand the importance of measuring the effectiveness of your programs
Define the broader goals of your programs
Discover four program features to measure
Learn how to create a good survey

3. Counting Activities vs. Evaluating Outcomes
Counting activities shows how much work is done, how many meals served, how many people “reached,” etc.
Evaluating outcomes is measuring effectiveness –what difference the program/service made to the people served
Increasingly, we need to explain why our programs exist and provide evidence that they should continue and grow.

4. Define Broader Goals WHAT SHOULD PROGRAMS ACCOMPLISH?
EXAMPLE: The goal of the Nutrition Program is more than just supplying a meal. Other goals of the Nutrition Program are:
Increase access to nutritious food
Increase knowledge about nutritious food

5. Define Broader Goals Other goals of the Nutrition Program cont.
Increase skill in obtaining and preparing nutritious food
Increase opportunities for social interactions and reduce isolation
Increase opportunities for engagement - volunteering

6. Define Broader Goals (more examples)
EXAMPLES FROM OTHER PROGRAM AREAS:
Advocacy – more than advocating for older people
Empower people to be advocates
Increase exercise of advocacy
Healthy Aging – more than holding classes
Improve self-management of health
Reduce falls, use ER less, etc.

7. Define Broader Goals (more examples)
Caregiver/Dementia – more than offering respite and adaptive equipment
Reduce caregiver burden
Increase practice of self care
Increase understanding of dementia
Decrease stigma in community
**How do we document that we met these other goals of the program?

8. Moving Towards an Evaluation Approach
Evaluating outcomes is not always easy
Need tools to measure effectiveness
Need training on how to do this well
Training, information and materials will be provided through ACE meetings, webinars and website

9. Moving Towards an Evaluation Approach
Please provide feedback on how useful this is
Please share your examples of what has worked for your programs
Please let us know what additional tools are needed

10. Four Features to Measure
Education – Were learning objectives met? Was awareness raised? Was knowledge gained?
e.g. nutrition education, dementia awareness, advocacy training
Tools: Pre-post test of knowledge or awareness

11. Four Features to Measure
Service – Did something improve for the customer?
e.g. reduced caregiver burden, increased socialization
Tools: Pre-post measure of experience or conditions

12. Four Features to Measure
Behavior Changes – Are healthy behaviors now present?
e.g. regular exercise routine, preventive services utilized, eating healthy foods
Tools: Pre-post survey about behaviors or practices

13. Four Features to Measure
Customer Service – Was the customer satisfied with the interaction?
This is a baseline across all program areas: if the customer is unhappy they won’t participate and won’t encourage others to participate.
Tools: Customer Satisfaction Survey

14. Customer Satisfaction Survey
A Customer Satisfaction Survey measures how happy or unhappy the customer was with an experience or service.
It is a fixed set of relatively simple questions
It should be anonymous
Customers should complete with no assistance from staff
It should be returned via mail or web

15. Important Considerations
Questions need to be simple and straightforward
You want to hear from happy and unhappy people
Your population may or may not have access to a computer or the internet

16. Avoid Bias
Be as neutral as possible so respondents do not feel pressure to answer in a false-positive manner. The following helps to avoid bias:
Provide a means of anonymous response
Clearly state confidentiality assurances and follow them
Always pre-pay return postage
Provide translation options

17. Avoid Bad Questions
Make sure questions are accessible to all (language, cultural issues)
Make sure questions are clear and unambiguous
Avoid “leading” questions
Prevent interviewer interpretation or influence on responses

18. The Alarm Bells…
Anything that predisposes a positive or negative answer
Anything that makes dishonest or ambiguous answers more likely
Anything that makes responses hard to interpret
CINDY:
These will all spring to life when we look at concrete examples that illustrate each problem.

19. The Alarm Bells…
Anything that could exclude or intimidate those who are not happy
Anything that excludes people who may need additional services not currently offered
**The principles of the Customer Satisfaction Survey apply to all surveys and evaluation tools.

20. Taken from real surveys!
Examples of
Good & Not-So-Good
Questions
Taken from real surveys!

21. Category Gaps Do you live: In a city In a small town On a farm?
What if you live in a suburb or in the country but not on a farm?

22. Try this instead… Where do you live? In a city or suburb
In a small town or village
In a rural area outside town/village boundaries
**Make sure there is an option that fits all situations.

23. Multiple Questions in One
The switchboard staff spoke with me courteously and transferred me to the appropriate person for help.
Yes No
What if the switchboard staff was rude but I was transferred to the proper person where I received help?

24. **Ask one thing in each question.
Try this instead…
The receptionist spoke with me courteously.
Yes No
The receptionist transferred me to the appropriate person for help.
**Ask one thing in each question.

25. Misfit Questions: When No Answer Fits
What prompted you to come to the Aging Unit in the first place?
Information and Assistance
Long-Term Care Options Counseling
Benefit Specialist
Eligibility Determination
What if I came for caregiver support or to sign up for home delivered meals?

26. Try this instead… What services have you used at the Aging Unit?
Nutrition
Caregiver Services
Benefit Specialist
Healthy Aging Programs
Other
None
**List the programs you particularly want to track, but add “other” and “none” as a catch-all.

27. Another example… How did you hear about the Aging Unit?
Friend/relative
Agency newsletter
Local newspaper
Radio ad
Other _________________
**Again, choose the specific resources you want to track and add “other” for anything not listed.

28. Confusing Questions
Do you feel that from the time you contacted the Agency and when the Agency contacted you to arrange an appointment/referral was reasonable?
Yes No
CINDY:
What are they trying to ask???

29. Try this instead…
How would you rate the amount of time that passed until you discussed your situation with the staff person?
Very long
Long
Neutral/Don’t know/No opinion
Short
Very short
**Ask the question in a clear manner, with a non-biased rating scale for answer choices.

30. Leading Questions
Questions worded in a way that tell respondents exactly what you want to hear.
Respondents may feel compelled to “make you happy” or to avoid offending you.
Respondents may comply so they are not alienating themselves from additional services.

31. Examples of Leading Questions
Check “Yes” or “No” for each statement:
I liked the service I received.
___Yes ___No
I am comfortable visiting this agency.
I would recommend this agency to a friend.
___Yes ___No

32. Try this instead…
How satisfied or dissatisfied were you with the service you received?
Very satisfied
Satisfied
Neutral
Dissatisfied
Very dissatisfied

33. How comfortable or uncomfortable did you feel visiting this agency?
Very Comfortable d. Uncomfortable
Comfortable e. Very Uncomfortable
Neutral
How likely are you to recommend this agency to others?
Very Likely d. Unlikely
Likely e. Very Unlikely

34. Tips to Avoid Leading Questions
Keep questions as neutral as possible to elicit a true response.
Use a rating scale to allow for a wider range of choices. Their answer may not be a definitive Yes or No.
Include both the positive and negative when writing your question so you are not pushing a positive response.
CINDY:

35. Consequences of Bad Questions
No response – left it blank
i.e. they couldn’t figure out how to answer it
Text responses to check-box questions
i.e. they wrote in an answer because their honest answer wasn’t listed as an option
Ambiguous responses
i.e. answers do not make sense because question didn’t make sense
CINDY:

36. Consequences of Bad Questions
Response errors
i.e. checking more than one age category
Quitting the survey early
too frustrated to finish
too many questions that don’t make sense
too few questions may not give an opportunity to give a true picture about their whole experience

37. Consequences of Bad Questions
Misleading positive responses
i.e. they gave the answer they thought you wanted
Throwing the whole survey away!
If there are too many errors, ambiguous responses or incomplete surveys, participants may just give up on it.

38. In Summary
Work needs to be done to improve the questions used in surveys
Scrutinize your survey questions to see if they meet these criteria:
Clear and easy to understand
Open to full range of responses
All possible responses are accounted for
Allow for negative responses

39. Good questions lead to surveys that provide helpful information

40. Customer Satisfaction Tools
Caregiver Customer Satisfaction Survey
Home Delivered Meals Satisfaction Survey
Congregate Meals Satisfaction Survey
**More tools will be forthcoming
**Send specific requests to Jane Mahoney, GWAAR

41. Coming Soon!
Watch for the next presentation on Evaluating Program Success, where we will talk about more tools including:
Pre and Post Test
Evaluation Survey
Follow-up Interview
Scale Measurement Tools

42. Contact information:
Jane Mahoney, OAA Consultant (608)
Evaluate this webinar!