1. Association for Institutional Research Annual Forum May 2014 Amber D. Lambert, Ph.D. Angie L. Miller, Ph.D. Center for Postsecondary Research, School of Education, Indiana University Living with Smartphones: Does Completion Device Affect Survey Responses?

2. Literature Review In higher education, surveys are used frequently for collecting information to demonstrate effectiveness (Kuh & Ikenberry, 2009) Example purposes: curriculum improvement, internal evaluation, accreditation, outcomes assessment, strategic planning Student surveys are most prominent, but surveys are also used to gather information from other stakeholders, including faculty, staff, and alumni (Cabrera et al., 2005; Kuh & Ewell, 2010)

3. Literature Review Although online surveying is more efficient and convenient, survey response rates have actually been falling (Atrostic et al., 2001; Baruch, 1999; Porter, 2004) Transition to web-based surveys over the past decade has generated much research on the new mode of delivery (Dillman, 2007) Initial concerns over sampling bias and coverage (Couper, 2000)

4. Literature Review As internet access grows, now research focuses on how mode impacts responses themselves: Content of one’s response (Descombe, 2006) Humanizing aspects of interface (Tourangeau et al., 2003) Technology to make survey dynamic (rather than static) (Norman et al., 2001)

5. Literature Review Also research on web-based surveys and: Page breaks and scrolling (Peytchev, 2009) Effectiveness of progress bars on completion (Villar et al., 2013) Browser compatibility and response placement (Kaye & Johnson, 1999) Color contrast and placement of emphasis (Tourangeau, 2004)

6. Literature Review Research on web-based surveys now must shift away from laptops and desktops to smartphones and tablets Mobile devices offer internet access virtually anywhere, but touch screen functioning, truncated viewing area, and smaller keyboards can place additional burdens on survey respondents (Buskirk & Andrus, 2012; Peytchev & Hill, 2010)

7. Literature Review Recent research comparing survey patterns between PC, tablet, and smartphone users found: Young people more likely to use smartphones, young and employed people more likely to use tablets (de Bruijne & Wijnant, 2014) Mobile phone respondents have lower completion rates, shorter open-ended answers, and are younger; no gender or education differences (Mavletova, 2013)

8. Research Questions Goals of this study: A) explore patterns in responses to a multi-institution alumni survey, looking at how type of completion device is impacted by various demographic variables, including age, income, gender, and employment status B) examine relationships between type of device and other survey-taking characteristics, including breaking off, backing up, time duration, item nonresponse, and open-ended text box completion

9. Method: Participants Data from the 2012 and 2013 administrations of the Strategic National Arts Alumni Project (SNAAP) Participants were 58,768 alumni from 109 different arts high schools, arts colleges, or arts programs within larger universities Sample consisted of 2% high school level, 76% undergraduate level, and 22% graduate level alumni 41% male, 59% female,.2% transgender Majority (85%) reported ethnicity as Caucasian Average institutional response rate: 18%

10. What is SNAAP? On-line annual survey designed to assess and improve various aspects of arts-school education Investigates the educational experiences and career paths of arts graduates nationally Questionnaire topics include: Formal education and degrees Institutional experience and satisfaction Postgraduate resources for artists Career Arts engagement Income and debt Demographics

11. Method: Metadata Measures Completion device: tracked through data collection platform- PC (42%), Mac (43%), Smartphone (9%), and Tablet (5%) (with an “other”.4% not traceable) Breakoff: did respondents reach the end of the survey and hit the “submit” button? Backup status: did the respondent go back (using the browser) to previously completed pages?

12. Method: Metadata Measures Time duration: how long (in minutes) did respondents spend with the survey open in their browser? Item nonresponse: did respondents answer complex “matrix layout” items without leaving any missing? Open-ended text boxes (10 total throughout survey): did respondents write ANY response? If so, how long was their response?

13. Method: Metadata Measures Example of “matrix layout” question sets:

14. Method: Demographic Measures Demographic information collected for: Continuous variables of age (write-in number box) and income (used midpoints of response ranges) Categorical variable of gender Male, female, transgender Categorical variable of current employment status Full-time (35 hours or more per week) Part-time only Unemployed and looking for work In school full time Caring for family full time Retired Other

15. Analyses Series of 16 chi-squared analyses was done for each of the categorical metadata and demographic variables For gender, current employment status, completion status, backup status, item nonresponse status (for two sets of matrix layouts), and open-ended response status (for 10 open-ended questions) Series of ANOVAs and Mann-Whitney tests for continuous metadata and demographic variables For age, income, duration, and length of open-ended responses

16. Results: Demographic Variables Smartphone users are significantly younger, while tablet users have significantly higher income AgeIncome PCMean46.5955172.45 N2100517976 Std. Deviation15.8344474.32 MacMean43.0752632.77 N2137118397 Std. Deviation15.2044100.82 Smart PhoneMean37.3247452.60 N30632848 Std. Deviation12.4240824.37 TabletMean46.4560997.26 N25492186 Std. Deviation14.4348138.70 OtherMean41.3546791.91 N200173 Std. Deviation14.81043216.90 TotalMean44.4153791.37 N4818841580 Std. Deviation15.4944343.01 F 326.00537.690 Sign 0.000

17. Results: Demographic Variables Women are more likely to use tablets and smartphones; retired people less likely to use smartphones but more likely to use tablets PCMacSmart PhoneTabletTotal Count% % % % % Gender Male 898342.2%884540.9%102532.4%95836.8%1981140.7% Female 1226357.7%1276559.0%212267.2%163962.9%2878959.1% Transgender 24.1%35.2%13.4%7.3%790.2% Employment Status Full-time (35 hours or more per week) 1396065.5%1332961.5%203164.9%157761.0%3089763.5% Part-time only 268012.6%344115.9%48015.3%36314.0%696414.3% Unemployed and looking for work 6212.9%7383.4%1284.1%762.9%15633.2% In school full time 4792.2%7283.4%1344.3%642.5%14052.9% Caring for family full time 2601.2%2771.3%832.7%783.0%6981.4% Retired 19849.3%12966.0%902.9%26010.1%36307.5% Other 13216.2%18588.6%1825.8%1696.5%35307.3%

18. Results: Metadata Variables Smartphone users took a significantly longer amount time (selecting only for those who completed the survey) MedianN Std. Deviation PC 27.6221372244.61 Mac 27.6021706297.08 Smart Phone 31.82314199.38 Tablet 28.482593119.46 Other 29.2720476.80 Total 27.9849016258.10 F 5.907 Sign 0.000

19. Results: Metadata Variables Smartphone users were far more likely to break off, but all device users were equally likely to back up to previous pages PCMacSmart PhoneTabletTotal Count% % % % % Completion Status Complete 2137287.0%2170785.4%314157.6%259383.8%4881383.4% Partial complete 318312.9%371914.6%231042.4%50216.2%971416.6% Backup Status Respondent did not back up in survey 2344095.5%2428195.5%523396.0%294395.1%5589795.5% Respondent backed up in survey 11154.5%11454.5%2184.0%1524.9%26304.5%

20. Results: Metadata Variables Smartphone users were more likely to provide complete responses to complicated layout questions (i.e. lower item nonresponse) PCMacSmart PhoneTabletTotal Count% % % % % Response to Complicated Question 1 Did not respond to all items 218010.2%21529.9%1665.3%2409.3%47389.7% Did respond to all items 1919289.8%1955490.1%297594.7%235390.7%4407490.3% Response to Complicated Question 2 Did not respond to all items 214910.1%19839.2%1655.3%2198.5%45169.3% Did respond to all items 1918989.9%1968490.8%296794.7%237091.5%4421090.7%

21. Results: Metadata Variables Overall, smartphone and tablet users were less likely to write responses to open-ended questions PCMacSmart PhoneTabletTotal Count% % % % % Question 1 Respondent did NOT write something 476534.5%580742.4%74437.7%63236.5%1194838.3% Respondent did write something 903965.5%789857.6%123062.3%109863.5%1926561.7% Question 2 Respondent did NOT write something 722833.8%692831.9%120138.2%101539.1%1637233.5% Respondent did write something 1414466.2%1477868.1%194061.8%157860.9%3244066.5% Question 3 Respondent did NOT write something 1672478.3%1689977.9%257481.9%209780.9%3829478.5% Respondent did write something 464821.7%480722.1%56718.1%49619.1%1051821.5% Question 4 Respondent did NOT write something 578862.8%398360.3%94370.5%71165.6%1142562.6% Respondent did write something 342837.2%261939.7%39429.5%37334.4%681437.4% Question 5 Respondent did NOT write something 1944691.0%1964690.5%290692.5%240092.6%4439891.0% Respondent did write something 19269.0%20609.5%2357.5%1937.4%44149.0%

22. Results: Metadata Variables PCMacSmart PhoneTabletTotal Count% % % % % Question 6 Respondent did NOT write something 628632.0%677733.4%138347.7%90938.8%1535534.0% Respondent did write something 1338268.0%1350766.6%151652.3%143361.2%2983866.0% Question 7 Respondent did NOT write something 256013.0%278313.7%59120.4%37015.8%630413.9% Respondent did write something 1710887.0%1750186.3%230879.6%197284.2%3888986.1% Question 8 Respondent did NOT write something 1330362.2%1398364.4%234874.8%187172.2%3150564.5% Respondent did write something 806937.8%772335.6%79325.2%72227.8%1730735.5% Question 9 Respondent did NOT write something 1698979.5%1749680.6%275287.6%214682.8%3938380.7% Respondent did write something 438320.5%421019.4%38912.4%44717.2%942919.3% Question 10 Respondent did NOT write something 1936590.6%1964390.5%295994.2%241393.1%4438090.9% Respondent did write something 20079.4%20639.5%1825.8%1806.9%44329.1%

23. Results: Metadata Variables And of those who did write open-ended text responses, smartphone and tablet users overall wrote significantly shorter responses Question 1Question 2Question 3Question 4Question 5 PCMedian 352459210980 N 903914144464834281926 Std. Dev. 50.08426.84212.87213.18150.07 MacMedian 3326410910284 N 789814778480726192060 Std. Dev. 48.45458.49210.85188.51180.06 Smart PhoneMedian 2917973 46 N 12301940567394235 Std. Dev. 28.70305.00135.84111.1583.63 TabletMedian 30184758352 N 10981578496373193 Std. Dev. 51.40302.53148.12189.27138.32 OtherMedian 332029317274 N 79142463027 Std. Dev. 27.76367.66154.57313.9778.71 TotalMedian 332459710378 N 19344325821056468444441 Std. Dev. 48.41431.56206.28199.40162.13 F 16.51555.79316.90214.31810.893 Sign 0.000

24. Results: Metadata Variables Question 6Question 7Question 8Question 9Question 10 PCMedian 150119126278119 N 1338217108806943832007 Std. Dev. 194.52202.78164.34363.87286.35 MacMedian 147118127283125 N 1350717501772342102063 Std. Dev. 201.17206.43165.08399.88266.77 Smart PhoneMedian 103789218567 N 15162308793389182 Std. Dev. 124.94117.37120.12337.23133.72 TabletMedian 12291113228102 N 14331972722447180 Std. Dev. 161.72133.83145.75296.27188.84 OtherMedian 17610613132094 N 126157743928 Std. Dev. 206.36179.03176.38342.34111.47 TotalMedian 144114124274119 N 29964390461738194684460 Std. Dev. 193.99198.16162.61377.40269.10 F 54.96876.38720.97013.1515.917 Sign 0.000

25. Discussion Many patterns of results are consistent with previous literature Differences on type of device used based on age and employment status Potential generational effects? Unlike previous studies, also found differences for gender and income Tablets as “luxury” items? Are women more compliant to survey requests, regardless of device?

26. Discussion Types of devices do seem to affect (in some ways) respondents’ survey-taking behaviors Also mirrors previous literature that taking surveys on smartphones and tablets can increase respondent burden Smartphone users are far more likely to abandon the survey, and those who do finish require more time to complete it Smartphone and tablet users were less likely to answer open-ended questions, and when they did answer them their responses were much shorter

27. Discussion Interestingly, smartphone users were more likely to fully complete complex layout item sets Counterintuitive at first glance because these questions may require more scrolling (vertically and horizontally) on a truncated screen, so one would expect fewer complete responses Could be that those who persevered to these points in the survey on a smartphone (about 1/3 and ½ of the way through) are the more dedicated and conscientious survey takers

28. Conclusions Limitations of study: sample may not be completely representative of all survey takers (only arts alumni, lower response rates, and selective participation) When designing web-based surveys, need to take into account that respondents may use smartphones and tablets May need to rely less heavily on long layouts and open- ended questions Future research continuing to look at device type is necessary as technology rapidly becomes available to larger populations Convenience of “anytime, anywhere” internet access may have negative impact on data quality

29. Questions or Comments? Contact Information: Amber D. Lambert adlamber@indiana.eduadlamber@indiana.edu Angie L. Miller anglmill@indiana.eduanglmill@indiana.edu Strategic National Arts Alumni Project (SNAAP) www.snaap.indiana.edu (812) 856-5824 snaap@indiana.edu *Reference list available upon request or in full paper