1. ABCs of Satisfaction & Impact Surveys Elizabeth Harris, Ph.D. Evaluation, Management & Training Associates, Inc. May 21 st, 2008

2. ABCs of Satisfaction & Impact Surveys Objectives –Understand the importance and relevance of science- based surveying –Understand method of developing telephone and Internet surveys to measure inquirer satisfaction and key outcomes important to contact centers –Leave the workshop with the tools necessary to develop surveys tailored to the needs of each contact center Workshop exercise

3. Lessons Learned from surveying for: Centers for Disease Control & Prevention –Consumer Response Services Center (CDC-INFO) –Monitor calls & e-mail responses First 5 LA –Countywide Parent Helpline

4. We will discuss: Survey: –Design To Dos Common Pitfalls –Administration Options –Data Use

5. Satisfaction & Impact Satisfaction: –Immediate –Callers opinion about Overall Service Various Aspects of the Service (Drill Down) Impact: –What happened as a result of the call?

6. Satisfaction & Impact Surveys: Step 1 What is important to us when we consider the callers perspective and experience? What questions are our key stakeholders asking that we need to answer?

7. Satisfaction & Impact Surveys: Step 2 Determine method of administration –Interactive Voice Response –Live Surveyor –Web-Based Feasibility Balance between cost & knowledge gain

8. Satisfaction & Impact Surveys: Step 3 Develop questions Response options –Closed Yes/No Likert Scale –e.g. Very Satisfied to Very Dissatisfied, Press 1 for Very Satisfied –Allow a middle option? Neither Satisfied or Dissatisfied –Open Pro –Information Con –Labor consideration

9. Satisfaction & Impact Surveys: Step 3 Develop questions Handout example of closed and open- ended questions

10. Satisfaction & Impact Surveys: Step 4 Design Survey –Question wording Neutral –Response order effect Carefully consider which question is first and which is last

11. Satisfaction & Impact Surveys: Step 4 Design Survey –IVR Rules Brief and to the point No more than six response options –Consider length Longer = more respondents drop off –Language options

12. Satisfaction & Impact Surveys: Step 5 Determine Survey Protocol –Who is surveyed-Immediate Satisfaction Every caller –Works well if no automated transfer option TrainingBias Random selection –Works well with a robust system program algorithm that automatically chooses

13. Satisfaction & Impact Surveys: Step 5 Who is surveyedImpact (Call Back) –Again consider: Purpose of the survey Purpose of the survey –Caller preference –Part of service follow up effort How the information will be used –Other considerations: Labor allocation/resources –To survey –To interpret/make use of information

14. Satisfaction & Impact Surveys: Step 6 Pilot Testing –Works out the bugs –Improves introductory instructions –May revise questions and responses based on pilot test results

15. What to Expect: Response Rate Immediate Satisfaction Survey –22% to 33% Follow Up Survey –Will reach half of those you try to call If you try more than once Vary time/day called

16. Results: Over 90% give the First 5 LA Parent Helpline the highest satisfaction rating for overall service! Over half who received a referral successfully accessed services

17. Results: Over 75% give CDC-INFO the highest satisfaction rating for overall service Over 75% report learning something new from their call Over half report that what they learned makes them want to change a behavior

18. Questions?

19. Exercise: See handouts Select one question you would like to answer with a satisfaction survey

20. ExerciseBreak Out: Develop a survey question and responses: –Doable/feasible to implement in your setting

21. ExerciseDebriefing: Share examples from each group

22. Questions? Contact information: Elizabeth Harris, Ph.D. 818-990-8301 eharris@emt.org