1. Technical and methodological meta-information on current practices in online research: A full population study of online questionnaires from recruitment portals. Christoph Burger and Stefan Stieger Department of Basic Psychological Research and Research Methods, Faculty of Psychology, University of Vienna

2. Folie 2/13 2 Online research Still an ongoing trend in the Social Sciences Hits per year for the term "online" in the Social Science Citation Index database (Thomson Reuters, 2012)

3. Folie 3/13 3 Methodological literature Growing amount of methodological literature Disadvantageous features should be avoided –potential negative effects on participants‘ motivation, commitment, response rates and data quality Advantageous features should be implemented –Potential positive effects on participants‘ motivation, commitment, response rates and data quality

4. Folie 4/13 4 Disadvantageous features Pre-answered item options (Reips, 2002a, 2000b) Displaying all items on one page (Peytchev et al., 2006; Nosek et al., 2012) Plug-ins and non-standard technologies (Stieger et al., 2011) Forced response (Stieger et al., 2007) Too long online questionnaires (>20 min; Cape, 2010) Displaying commercial advertising (Thota et al., 2010) Using too long informed consent forms (<500 words; Varnhagen et al., 2005)

5. Folie 5/13 5 Advantageous features Providing progress information („how near till finish“; Conrad et al., 2010) Using professional survey services (Reips, 2007) Displaying a back button (Couper et al., 2011) Using the seriousness check technique (Reips, 2002b) Offering information on the expected duration Using short online questionnaires (<10 min; Crawford et al., 2001) Offering incentives (Gosling et al., 2004)

6. Folie 6/13 6 Research question Is this methodological knowledge reflected in current practices in online research? Are online questionnaires that are currently active avoiding disadvantageous features and/or implementing advantageous features? Azar (2000): “Researchers who collect data on the Web expose their methods and protocols to their colleagues long before they publish a completed research project. […] [T]he Web can also facilitate science at its best, opening up to scrutiny the entire process, from beginning to end.”

7. Folie 7/13 7 Sample of online questionnaires Full population study of active online questionnaires (data collection took place during the first half of 2012) 5 online recruitment portals –Web experiment/ survey list (Ulf-Dietrich Reips) –Psychological research on the net (John H. Krantz) –Online psychology research (Kathryn Gardner) –Online social psychology studies (Scott Plous) –Inquisitive Mind online research list (Michael van den Oudenalder) Online questionnaires –3036 raw data entries found in five online recruitment portals –720 non-redundant active online questionnaires

8. Folie 8/13 8 Geographical distribution of online questionnaires

9. Folie 9/13 9 Disadvantageous features Quality thresholds Using pre-answered item options 4.2% yes 95.8% no +++ Displaying all items on one page 8.9% yes 91.1% no +++ Using plug-ins and non-standard technologies 11.9% yes 88.1% no ++ Using forced response 52.2% yes 47.8% no failed Using too long online questionnaires (> 20 min) 31.4% yes 68.6% no + Displaying commercial advertising 1.5% yes 98.5% no +++ Using too long informed consent form (> 500 words) 31.1% yes 68.9% no + failed = no threshold achieved, + = <50%, ++ = <30%, +++ = <10% Results

10. Folie 10/13 10 failed = no threshold achieved, + = >50%, ++ = >70%, +++ = >90% Advantageous features Quality thresholds Using progress information ("how near to finish") 47.1% yes 52.9% no failed Using professional survey service or survey software packages 62.8% yes 37.2% no + Displaying a back button 41.9% yes 58.1% no failed Using the seriousness check technique 8.7% yes 91.3% no failed Offering information on the expected questionnaire duration 84.2% yes 15.8% no ++ Keeping expected questionnaire duration short (< 10 min) 14.7% yes 85.3% no failed Offering incentives 38.9% yes 61.1% no failed Results

11. Folie 11/13 11 Summary Good compliance with quality standards regarding the avoidance of disadvantageous features Lack of compliance with quality standards pertaining to the implementation of advantageous features Recommendations Online researchers should focus more on implementing advantageous features while maintaining their high quality levels regarding the avoidance of disadvantageous features

12. Folie 12/13 12 Thank you for your attention! xxx xxxxxxx o o o o o xxxxxxxxxxxx o o o o o xxxxxxxxxx o o o o o xx

13. Folie 13/13 13 References and further reading Azar, B. (2000). Online experiments: Ethically fair or foul? Monitor on Psychology, 31, 50–52. Cape, P. (2010, March). Questionnaire length, fatigue effects and response quality revisited. Paper presented at the ARF Re:think 2010, New York City, USA. Retrieved October 7, 2012 from http://www.surveysampling.com/ssi-media/Corporate/white_papers/SSI_QuestionLength_WP.image Conrad, F. G., Couper, M. P., Tourangeau, R., & Peytchev, A. (2010). The impact of progress indicators on task completion. Interacting with Computers, 22, 417–427. Couper, M. P., Baker, R. P., & Mechling, J. (2011). Placement and design of navigation buttons in Web surveys. Retrieved October 7, 2012 from http://surveypractice.wordpress.com/2011/02/14/navigation-buttons/ Crawford, S. D., Couper, M. P., & Lamias, M. J. (2001). Web Surveys: Perceptions of Burden. Social Science Computer Review, 19, 146–162. Gosling, S. D., Vazire, S., & Srivastava, S., & John, O. P. (2004). Should we trust web-based studies? A comparative analysis of six preconceptions about Internet questionnaires. American Psychologist, 59, 93–104. Nosek, B. A., Sriram, N., & Umansky, E. (2012). Presenting survey items one at a time compared to all at once decreases missing data without sacrificing validity in research with Internet volunteers. PLoS One, 7, e36771. Peytchev, A., Couper, M. P., McCabe, S. E., & Crawford, S. D. (2006). Web Survey Design: Paging versus scrolling. Public Opinion Quarterly, 70, 596–607. Reips, U.-D. (2002a). Internet-based psychological experimenting: Five dos and five don'ts. Social Science Computer Review, 20, 241–249. Reips, U.-D. (2002b). Standards for Internet-based experimenting. Experimental Psychology, 49, 243–256. Reips, U.-D. (2007). The methodology of Internet-based experiments. In A. Joinson, K. McKenna, T. Postmes, & U.-D. Reips (Eds.), The Oxford handbook of Internet psychology (pp. 373–390). Oxford, UK: Oxford University Press. Statowl. (2012). Web browser plugin market share: Web browser plugin market penetration and global usage. Retrieved October 7, 2012 from http://www.statowl.com/plugin_overview.php Stieger, S., Göritz, A. S., & Voracek, M. (2011). Handle with care: The impact of using java applets in web-based studies on dropout and sample composition. Cyberpsychology, Behavior, and Social Networking, 14, 327–330. Stieger, S., Reips, U.-D., & Voracek, M. (2007). Forced-response in online surveys: Bias from reactance and an increase in sex-specific dropout. Journal of the American Society for Information Science and Technology, 58, 1653–1660. Thomson Reuters. (2012). Social Sciences Citation Index. Retrieved October 7, 2012 from http://thomsonreuters.com/products_services/science/science_products/a-z/social_sciences_citation_index Thota, S. C., Song, J. H., & Larsen, V. (2010). Do animated banner ads hurt websites? The moderating roles of website loyalty and need for cognition. Academy of Marketing Studies Journal, 14, 91–116. Varnhagen, C. K., Gushta, M., Daniels, J., Peters, T. C., Parmar, N., Law, D., … Johnson, T. (2005). How informed is online informed consent? Ethics and Behavior, 15, 37–48.