ABCs of Satisfaction & Impact Surveys Elizabeth Harris, Ph.D. Evaluation, Management & Training Associates, Inc. May 21 st, 2008

ABCs of Satisfaction & Impact Surveys Objectives –Understand the importance and relevance of science- based surveying –Understand method of developing telephone and Internet surveys to measure inquirer satisfaction and key outcomes important to contact centers –Leave the workshop with the tools necessary to develop surveys tailored to the needs of each contact center Workshop exercise

Lessons Learned from surveying for: Centers for Disease Control & Prevention –Consumer Response Services Center (CDC-INFO) –Monitor calls & responses First 5 LA –Countywide Parent Helpline

We will discuss: Survey: –Design To Dos Common Pitfalls –Administration Options –Data Use

Satisfaction & Impact Satisfaction: –Immediate –Callers opinion about Overall Service Various Aspects of the Service (Drill Down) Impact: –What happened as a result of the call?

Satisfaction & Impact Surveys: Step 1 What is important to us when we consider the callers perspective and experience? What questions are our key stakeholders asking that we need to answer?

Satisfaction & Impact Surveys: Step 2 Determine method of administration –Interactive Voice Response –Live Surveyor –Web-Based Feasibility Balance between cost & knowledge gain

Satisfaction & Impact Surveys: Step 3 Develop questions Response options –Closed Yes/No Likert Scale –e.g. Very Satisfied to Very Dissatisfied, Press 1 for Very Satisfied –Allow a middle option? Neither Satisfied or Dissatisfied –Open Pro –Information Con –Labor consideration

Satisfaction & Impact Surveys: Step 3 Develop questions Handout example of closed and open- ended questions

Satisfaction & Impact Surveys: Step 4 Design Survey –Question wording Neutral –Response order effect Carefully consider which question is first and which is last

Satisfaction & Impact Surveys: Step 4 Design Survey –IVR Rules Brief and to the point No more than six response options –Consider length Longer = more respondents drop off –Language options

Satisfaction & Impact Surveys: Step 5 Determine Survey Protocol –Who is surveyed-Immediate Satisfaction Every caller –Works well if no automated transfer option TrainingBias Random selection –Works well with a robust system program algorithm that automatically chooses

Satisfaction & Impact Surveys: Step 5 Who is surveyedImpact (Call Back) –Again consider: Purpose of the survey Purpose of the survey –Caller preference –Part of service follow up effort How the information will be used –Other considerations: Labor allocation/resources –To survey –To interpret/make use of information

Satisfaction & Impact Surveys: Step 6 Pilot Testing –Works out the bugs –Improves introductory instructions –May revise questions and responses based on pilot test results

What to Expect: Response Rate Immediate Satisfaction Survey –22% to 33% Follow Up Survey –Will reach half of those you try to call If you try more than once Vary time/day called

Results: Over 90% give the First 5 LA Parent Helpline the highest satisfaction rating for overall service! Over half who received a referral successfully accessed services

Results: Over 75% give CDC-INFO the highest satisfaction rating for overall service Over 75% report learning something new from their call Over half report that what they learned makes them want to change a behavior

Questions?

Exercise: See handouts Select one question you would like to answer with a satisfaction survey

ExerciseBreak Out: Develop a survey question and responses: –Doable/feasible to implement in your setting

ExerciseDebriefing: Share examples from each group

Questions? Contact information: Elizabeth Harris, Ph.D