Presentation is loading. Please wait.

Presentation is loading. Please wait.

David Geddes, Ph.D. Senior Vice President & Partner Fleishman-Hillard Institute for Public Relations Measurement Commission 21 tips for conducting evaluation.

Similar presentations


Presentation on theme: "David Geddes, Ph.D. Senior Vice President & Partner Fleishman-Hillard Institute for Public Relations Measurement Commission 21 tips for conducting evaluation."— Presentation transcript:

1 David Geddes, Ph.D. Senior Vice President & Partner Fleishman-Hillard Institute for Public Relations Measurement Commission 21 tips for conducting evaluation research on a boot lace

2 2 Todays objectives Advice on doing it yourself Advice on being a good client

3 3 Where to get help University departments Social work, Sociology, Psychology, Political Science, Business, and others Contacts via board of directors American Marketing Association Market Research Association Market research firms LinkedIn (www.linkedin.com)www.linkedin.com National Opinion Research Center (University of Chicago)

4 4 mehow_02.asp Introduction to evaluation research Process evaluation examples Telephone survey with annotations Intercept and self- administered survey

5 5 Build a community Vision Lutheran Social Services Research Community Share best practices Methods Facebook or other collaboration tool

6 6 Set your objectives Program objectives Specific information needs 1

7 7 Do your background research 2

8 8 Think about different types of evaluation and measurement 1.Formative evaluation Helps planning 2.Process evaluation How well is the program operating? 3.Impact evaluation Building awareness and understanding? 4.Outcome evaluation Concrete program or business metrics 3

9 9 Dont assume you know audience attitudes and opinions You need formative research 4

10 10 Set up your process metrics 5

11 11 Measure impacts Awareness Knowledge Attitudes and perceptions Intended actions 6

12 12 Identify survey targets Known respondents Population = Sample frame Members Donors Clients Employees Civic organization leaders Defined population Sample frame derived from population General public Congregation members Potential donors Potential adoptive parents 7

13 13 Pick the right survey method to reach targets Telephone Mail Online Mall or store intercept 8

14 14 Telephone surveys Advantages Speed, in some cases Control Probe and clarify Cost if you do it yourself Disadvantages Cost if you go outside Hard to reach some socio-economic groups

15 15 Telephone surveys Case: General public, 10 minute survey of 500 adults in Saint Louis metro area. $8,000 = $16 cost per interview Review cost parameters Case: Survey of 200 adult children or guardians of seniors who receive services from LSS in St. Louis metro area. LSS provides names and phone list. $4,000= $20 cost per interview Review cost parameters

16 16 Telephone surveys Case: Survey of 200 married adults in the St. Louis metro area aged 18 to 45 who are at least somewhat open to the idea of adoption. 10 minute survey. 30% incidence based on other surveys. $7,000 = $35 cost per interview

17 17 Doing your own telephone surveys Sample list Staff Interns, volunteers, students, etc. Call management system Data recording system Discipline Persistence

18 18 Going outside for telephone surveys Smaller local firms Be flexible on timing

19 19 Mail surveys Advantages Lower cost Easy if you have a mailing list Easier to do yourself Disadvantages Slow Lower control Lower response rates Need a mailing list

20 20 Doing your own mail surveys Clear questionnaire layout Sample list Staff Interns, volunteers, students, etc. Lots of manual paper handling Discipline Patience

21 21 Online surveys Advantages Speed, in some cases Control Cost if you do it yourself (SurveyMonkey) Disadvantages Purchase a sample list Unrepresentative of some socioeconomic groups

22 22 Online surveys Case: General public, 10 minute survey of 500 adults in Saint Louis metro area using an online survey tool. $3,300 = $6.60 cost per interview Review cost parameters

23 23 Doing your own Online surveys SurveyMonkey programmer Sample list Staff Interns, volunteers, students, etc.

24 24 Mall intercept surveys Advantages Control Probe and clarify Cost if you do it yourself Disadvantages Slow Need mall or store cooperation Unrepresentative of some socioeconomic groups

25 25 Doing your own mall intercept surveys Cooperation of mall or Wal-Mart Staff Interns, volunteers, students, etc. Discipline Patience.. Figure on one interview/hour

26 26 Select an appropriate sample size What are possible sources of error? How far off could we be? What sample size do we need? 8a

27 27 Give advance notice If you know the target respondents Newsletter Postcard Benefit: improves response rate 9

28 28 Remember you are asking for their time This applies to… Pre-survey letters or cards Introduction to the survey And the questions themselves Dont ask meaningless questions 10

29 29 Write a good introduction to the questionnaire Introduce yourself Research not survey Survey topic: social services Not a sales call … not a fundraising call You want their opinion / input 11

30 30 Move from general to specific 12

31 31 Lead with an interesting question Engage the respondent Ask interesting questions and people will talk 13

32 32 Make judicious use of open-ended questions Example: Based on what you have read or heard, what are your personal perceptions of Lutheran Social Services? [PROBE] Anything else? 2 to 3 per questionnaire maximum Finish with a final thoughts question 14

33 33 Use good response scales Building a scale Clearly distinguish good and bad 1 Agree 2 Disagree 15

34 34 Use good response scales Building a scale Add a polite positive and negative 1 Strongly Agree 2 Agree 3 Disagree 4 Strongly disagree

35 35 Use good response scales Building a scale Add a top positive response 1 Very strongly agree 2 Strongly Agree 3 Agree 4 Disagree 5 Strongly disagree

36 36 Use good response scales Building a scale Add a dont know/no opinion 1 Very strongly agree 2 Strongly Agree 3 Agree 4 Disagree 5 Strongly disagree 6 Dont know / no opinion

37 37 Use good response scales Building a scale Add a mid-point if you really need it... and balance the scale 1 Very strongly agree 2 Strongly Agree 3 Agree 4 Neither agree nor disagree 5 Disagree 6 Strongly disagree 7 Very strongly disagree 8 Dont know / no opinion

38 38 Make sure you cover all possible responses Multiple choice questions 16

39 39 No opinion does not equal Dont know 17

40 40 Always include a refused option 18

41 41 Be careful with numeric responses About how much do you spend on gas each month $__________ Which of the following best describes the amount you spend on gas each month: Under $25 $26 to $50 $51 to $100 Over $100 19

42 42 Be careful about do you floss your teeth items Socially desirable responses Responses that may imply guilt 20

43 43 Test your questionnaire 21

44 44 Wrap-up Begin with your objectives and information needs Define the people you want to survey Select an appropriate survey method Write good questions Test Launch the survey Analyze the results

45 45 Exercise


Download ppt "David Geddes, Ph.D. Senior Vice President & Partner Fleishman-Hillard Institute for Public Relations Measurement Commission 21 tips for conducting evaluation."

Similar presentations


Ads by Google