Improving the Effectiveness of Interviewer Administered Surveys though Refusal Avoidance Training Grace E. ONeill Presented by Anne Russell U.S. Census Bureau ICES-III - June 19, 2007
2 Outline of Presentation Background Respondent focus groups Refusal avoidance training Training content Discussion Future developments
3 Background
4 Respondent contact staff Interviewers: outgoing calls to gain cooperation and to gather data Clerks: incoming calls to provide basic information and resend forms Analysts: professional staff who place outgoing calls concerning data errors or nonresponse follow-up
5 Respondent contact training Interviewers: Centralized training on telephone skills and refusal avoidance and conversion Clerks: Shadow experienced clerks with informal discussion Analysts: Basic training on respondent contact techniques
6 Why clerks? Self-administered paper/ electronic forms Establishment respondents need less encouragement to participate Nature of telephone calls
7 However… Over time, the role of the clerk has evolved into the role of an interviewer Less cooperative respondents More data collected over the telephone More nonresponse follow-up telephone calls Yet, their training has not evolved
8 Respondent Focus Groups
9 Focus Groups Monthly Trade Surveys Wholesale and retail firms Conducted by outside firm Six focus groups, 44 participants Respondents views and impressions Ways to improve survey
10 Focus Groups Finding Inexperienced clerks needed skills to increase participation by respondents, especially on voluntary surveys Solution Refusal avoidance training
11 Refusal Avoidance Training
12 Refusal Avoidance Training Groves and McGonagle (2001) –Assemble respondent concerns –Develop responses –Train interviewers to classify concerns –Train interviewers to provide quick and appropriate responses
13 Refusal Avoidance Training Interactive Cooperative learning Flexible Survey specific Provides telephone skills, refusal avoidance techniques, and improves communication
14 Training specifics Monthly Trade Surveys (MTS) and Quarterly Services Survey (QSS) 43 MTS clerks, 14 QSS clerks Mix of tenure and survey experience Supervisor and survey manager involvement Eight hours over two days
15 Training Preparation Adapt household- based training to establishment-based survey Adapt training to the MTS or QSS Identify respondent concerns and develop solutions FAQ Job Aid
16 Training Content
17 Refusal avoidance training modules Module 1: Introduction Module 2: Survey Specific Module 3: Shared Experience Module 4: Preparing a Telephone Call Module 5: Telephone Skills Module 6: Identify, Analyzing, and Dealing with Reluctance and Refusal Module 7: Recovering from Negative Calls Module 8: Wrap-up and Evaluation
18 Module 1 Introduction Introduces training to clerks Introduces the trainer and clerks to each other Provides training schedule
19 Module 2 Survey Specific Introduces survey specific content Conducted by survey manager or survey staff
20 Module 3 Shared Experience Identify clerks biggest concerns and difficulties Develops solutions Concerns are used in a later module
21 Module 3 example Trainer: What are some of the things respondents say when you talk to them? Clerks: Why should I do this, Im not a wholesaler etc. Group: Identify appropriate responses Trainer: Make sure identified concerns are addressed and adds new concerns to list
22 Module 4 Preparing for the Telephone Call Asks clerks how they prepare for telephone calls Assess what tools clerks use to find information about company and what tools they might need
23 Module 5 Telephone Skills Assess clerks active listening skills Discuss tone Discuss mechanics of placing a telephone call
24 Module 6 Identifying, Analyzing and Dealing with Reluctance and Refusal Classify concerns identified in Module 3 as reluctance and/ or refusal Further discuss solutions Paired practice during class Reviewed updated FAQ Job Aid
25 Module 7 Recovering from Negative Calls Discusses recovering from refusals and other negative calls Helps clerks to evaluate negative experiences
26 Module 8 Wrap-up and Evaluation Review main training points Clarify any remaining concerns Evaluations by clerks
27 Results
28 Results Clerk evaluation Usefulness of workshop Usefulness of skills learned Increased confidence, preparation, and communication
29 Results Response rates Caveats: Not experimentally tested Confounded by other survey conditions Dont know how many potential refusals clerks prevented
30 Results: MTS Initially lower from previous response period Loss of clerks Misclassification of refusals
31 Results: MTS Over time Good response rates, decrease in wholesale refusals Continued improvement Clerks have more responsibilities
32 Results: QSS Decline in response rates Imputation remained stable More companies refused Larger companies continued to report
33 Results: QSS Communication has improved Clerks are open about sharing concerns with survey managers Bi-weekly telephone calls with call center Annual refresher training
34 Discussion
35 Benefits Centralized dissemination of skills and information Practice occurs in a test environment Provides training at regular intervals Proactive training instead of reactive training
36 Benefits Communication between clerks, supervisors, and survey managers Clerks feel invested in data collection process Survey managers gain direct insight into data collection process
37 Costs Staff time writing and delivering training Telephone coverage Monetary cost of training Difficult to provide conclusive evidence
38 Future Developments
39 Future Developments Formalized process for clerk training Data capture of call concerns Follow-up evaluation by clerks Analyst training
40 Thank You Please feel free to contact the author at: Grace E. ONeill Phone: