1. Increasing Survey Cooperation: Motivating Chronic Late Responders to an Annual Survey National Science Foundation Division of Science Resources Statistics Ronda Britt and Fran Featherston ICES III June 19, 2007 National Science Foundation Division of Science Resources Statistics www.nsf.gov/statistics

2. 1 NSF Academic R&D Survey National Science Foundation Division of Science Resources Statistics Voluntary survey of 700 research & development - performing universities and colleges Conducted annually since 1972 Web only since 2001 Requests expenditures for R&D performed during previous year Survey data based on respondents fiscal year

3. 2 Historical Response Pattern National Science Foundation Division of Science Resources Statistics Early November email launches survey & sets January 31 deadline 50-60% respond by deadline, despite multiple reminders Several months after deadline to reach response rate of ~95% Respondents were allowed to extend deadline, creating cycle of late response

4. 3 Goals of Experiments FY 2004 - FY 2005 National Science Foundation Division of Science Resources Statistics Increase response rate at survey deadline -Speed up first interaction with web survey -Increase number of reminders -Vary mode of reminders Decrease number of weeks to reach 95% response rate

5. 4 Experiment 1- FY 2004 Increase Response Rate at Survey Deadline, Part 1 National Science Foundation Division of Science Resources Statistics Hypothesis: Respondents who visit website soon after launch (acknowledge survey) are more likely to respond by the survey deadline 3 study groups received varying modes of reminders: Group 1 received one e-mail reminder in December (control group) Group 2 received two e-mail reminders in December Group 3 received 2 phone calls in December All groups received final e-mail in January requesting acknowledgement

6. 5 Experiment 1 Design National Science Foundation Division of Science Resources Statistics Key Dates Study Groups Group 1 (control) Group 2Group 3 November 23Survey Launch December 14 Acknowledgement e-mail Acknowledgement phone calls December 30 Acknowledgement e-mail Acknowledgement e-mail Acknowledgement phone calls January 11Acknowledgement e-mail January 31Survey Due Date

7. 6 Experiment 1 Results National Science Foundation Division of Science Resources Statistics You can lead a respondent to the website, but you cant encourage an earlier survey completion Multiple reminders to acknowledge succeeded in decreasing elapsed days until acknowledgement Multiple reminders to acknowledge had no effect on decreasing elapsed days to survey completion

8. 7 Experiment 1 Results National Science Foundation Division of Science Resources Statistics Control Group 1 E-mail reminders Group 2 Phone Reminders Group 3 All Groups Mean days from survey launch to acknowledgement 37 days (n=177) 29 days (n=179) 30 days (n=179) 32 days (n=535) Response rate at survey deadline 46.3%49.7%49.2%48.4%

9. 8 Experiment 2 – FY 2004 Increase Response Rate at Survey Deadline, Part 2 National Science Foundation Division of Science Resources Statistics Hypotheses: Respondents who receive more reminders are more likely to respond by the survey deadline Respondents who receive reminders that vary in mode are more likely to respond by the survey deadline During month prior to survey deadline: Group 1: One e-mail reminder (control group) Group 2: One e-mail, one mail reminder Group 3: Two e-mail reminders

10. 9 Experiment 2 Design National Science Foundation Division of Science Resources Statistics Key Dates Study Groups Group 1Group 2Group 3 January 18-20 (expected delivery) Reminder Mailing January 19 Reminder E-mail 1 Reminder E-mail 1 January 26Reminder E-mail 2 January 31Survey Due Date

11. 10 Experiment 2 Results National Science Foundation Division of Science Resources Statistics Variety and intensity will spice up the response rate. Group who received both mail and e-mail responded at a higher rate by deadline than group receiving only e-mail Experiment 4 (FY 2005) validated the effect of two contacts vs. one, but no significant difference between two e-mails vs. e- mail and mailing

12. 11 Experiment 2 Results – Single mode, 1 contact vs. mixed mode, 2 contacts National Science Foundation Division of Science Resources Statistics Group 1 1 e-mail Group 2 1 e-mail and 1 mailing Responded by deadline35.4%45.9% 5162 Total100% 144135

13. 12 Experiment 3 – FY 2004 Increase Timeliness after Survey Deadline National Science Foundation Division of Science Resources Statistics Hypothesis: Respondents will respond sooner after the deadline if follow-up messages are individualized and less frequent than weekly During 24 weeks after survey deadline: Group 1: Low tailoring – standard weekly messages (control group) Group 2: Medium tailoring – less frequent messages with different content each time Group 3: High tailoring –adjusted to individual respondent history (more frequent and tailored contacts for previous non- responders)

14. 13 Experiment 3 Design National Science Foundation Division of Science Resources Statistics Follow-up Period Group 1 Low Tailoring Group 2 Medium Tailoring Group 3 High Tailoring Likely non- responders All others 0-11 weeks 10 e-mail or phone contacts Same messages 3 e-mails Varied messages Dependent on past behavior Tailored messages 2 e-mails Same messages 12-24 weeks 12 contacts; alternating e-mail or phone weekly 4 contacts spaced 3 to 4 weeks; e- mail, phone, letter, and final phone call 6 contacts; alternating e- mail and phone every two or three weeks Same messagesVaried messages and senders Same messages

15. 14 Experiment 3 Results National Science Foundation Division of Science Resources Statistics It is more important to stay in contact than it is to use a particular contacting strategy Little variation of final response rate across 3 groups Weekly reminders were not found to perform better than tailored reminders sent every 3 to 4 weeks

16. 15 Experiment 3 Results National Science Foundation Division of Science Resources Statistics Group 1 Low Tailoring Group 2 Medium Tailoring Group 3 High Tailoring All Groups Number of respondents 10595102302 Response rate on January 31 43.9%49.2%45.5%46.2% Final response rate90.5%94.7%90.2%91.7% Response added between January 31 and August 10 50.8%48.1%49.2%49.3%

17. 16 Changing Procedures National Science Foundation Division of Science Resources Statistics Used experimental results as starting point for making changes Consulted with Dr. Don Dillman on message crafting and changing survey procedures Determined four areas for change: Acknowledgement process Mode/frequency of contacts Extension policy Survey close-out policy (letters to institution presidents)

18. 17 Acknowledgement Process National Science Foundation Division of Science Resources Statistics Simplified acknowledgement process in FY 2006 less burdensome for respondents Prior to FY 2006, asked respondents to log-on to survey website to acknowledge continuation as survey respondent Now, respondent replies to e-mail Prior to FY 2006, 23% acknowledged in first week of survey, 72.5% month before due date Now, increased to 63% and 98.7%, respectively Results: Earlier identification of changes in respondents

19. 18 Mode/frequency of Respondent contacts National Science Foundation Division of Science Resources Statistics Changed mode and frequency, varied content in FY 2006 Pre-due date reminders increased in frequency: Early December – additional e-mail reminder January - 2 reminders (1 mail and 1 e-mail) Post-due date follow-ups decreased in frequency: First 6 weeks - 3 e-mails (every 2 weeks) 7 th week - phone call 8 th week - letter to presidents office 9 th week – phone call 11 th week - final e-mail announcing survey closing date Results: Increased response rate at deadline from 56% in FY 2005 to 68% in FY 2006, shortened post-deadline follow-up period

20. 19 Extension Policy National Science Foundation Division of Science Resources Statistics Eliminated unlimited extensions Prior to FY 2005: Extension requests up to 36 weeks after survey due date FY 2005 and FY 2006: Requests for extensions restricted to shorter time frame (only granted for dates before Presidents letter) After presidents letter mail-out, respondents were told survey closes when target response rate reached and 2 week final warning would be given Results: Shortened post-deadline follow-up period

21. 20 Survey Close-out Policy National Science Foundation Division of Science Resources Statistics Presidents letter was added in FY 2004 as final call for response Goal set to mail Presidents letter to nonrespondents 1 month earlier each year FY 2004 - May 25 (87% response rate at mailing) FY 2005 - April 21 (88% response rate at mailing) FY 2006 - March 23 (92% response rate at mailing) Results: Earlier survey closeout on May 21, 2007 v. August 10, 2005

22. 21 Response Rate History, FY 1997- FY 2006 National Science Foundation Division of Science Resources Statistics Weeks to reach final response rate Weeks to reach 80% Weeks to reach 88%

23. 22 Final Response Rates, FY 1997- FY 2006 National Science Foundation Division of Science Resources Statistics

24. 23 Questions? For more information contact: Ronda Britt, Academic R&D Survey Manager 703-292-7765 rbritt@nsf.gov National Science Foundation Division of Science Resources Statistics