1. Web-Based Research: Issues, Problems and an Example Technique John H. Krantz Hanover College

2. Outline Brief History Issues in Internet Research An Example Method Problems and Solutions

3. History Email for 2-3 Decades 1995 Krantz, Ballard, & Scher (1997) Within Subjects Reips (1997) Between Subjects First True Web Experiments July 1995 List of Online Psychological Research – sponsored by APS for awhile http://psych.hanover.edu/research/exponnet.html Almost 200 studies and study sites listed

4. Why Do Internet Research Musch & Reips (2000) Sample Size Statistical Power High Speed Ability to reach participants in other countries High external or ecological validity Low cost Ability to replicate a lab experiment with more power Special populations

5. Issues Is the Sample Representative Yes and No Important to remember base of comparison Ideal vs. Lab Krantz & Dalal (2000) Much more diverse that most lab samples Especially in age and education range Race is generally limited as is nationality

6. Sample Characteristics: Gender Are Internet Samples Male Dominated? Krantz & Dalal (2000)% Female GVU 1 st (1994) 5% Reips (1996) English: 43% German:18% Krantz, et al. (1997)44% Pasveer & Ellard (1998) 3 rd Study71% More Recent Caddell & Utt(2004)77% Meyerson & Tryon (2003)45%

7. Sample Characteristics: Age Are we still testing college sophomores? Krantz, et al. (1997)43% > 30 Smith & Leigh (1997)35% > 30 Pasveer & Ellard (1998)45% > 25 GVU 1 st 36% > 30 Caddell & Utt (2004)60% > 30 Pattison & Rouse(2003)16% > 30

8. Is the College Sophomore be Making a Comeback Plot of activity of Psychological Research on the Net Pattison & Rouse (2004) 76% 18-22

9. Sample Characteristics: Race How diverse are the samples? Unfortunately in general samples, diversity is still limited Krantz et al. (1997)89% White Smith & Leigh (1997)86% White GVU 10 th (1998b)87% White ONeil, Penrod, & Bornstein (2003)82% White Meyerson & Tryon (2003)93% White However, as will be discussed later, web can make it possible to access special populations

10. Sample Characteristics: Nationality Where to do the subjects come from? Largely North American, even US, even in some European studies Krantz, et al. (1997) 86% N. Am. Senior, et al (1999)>80% N. Am This study was conducted in England.

11. Is the Data Any Good Emphatically Yes

12. Data Quality: Direct Comparisons A number of studies, still, run both laboratory and internet samples. Krantz, et al. (1997) Regression of web means on laboratory samples (even though different types of samples): lab mean = 1*(Web mean) + 0 r 2 =.99

13. Compare to Established Data Compare web results to previously published data sets Myerson & Tryon (2003) Studied Sexual Boredom Scale of Watt & Ewing (1996) Matched sample characteristics Found same internal consistency Form of administration was not a significant factor

14. Direct Validity Comparison Use same techniques to validate results Pasveer & Ellard (1998) Developed new scale Internal consistency Psychomectric properties

15. Data Validity: Arguments Reips (2000) Statistical Power Limited Sample Population Limited External Validity Less than Optimal Voluntariness Motivational Confounding Experimenter Bias Nontransparency Limitations of what is feasible to research

16. Interesting Deviations Stern & Faber (1997) Milgrams lost letter technique Milgram, sent on Stern & Faber, returned to sender From doing favor to doing easy task Often effects are smaller

17. Getting Subjects: General http://psych.hanover.edu/research/exponnet. html http://psych.hanover.edu/research/exponnet. html Other Pages: Social Psychology: http://www.socialpsychology.org/expts.htm http://www.socialpsychology.org/expts.htm The Web Experiment List: http://genpsylab- wexlist.unizh.ch/http://genpsylab- wexlist.unizh.ch/

18. Getting Subjects: Special Populations Advertising Email Groups Careful, get permission Can be thought spam Netiquette Discussion Groups Same, permission

19. Sample Method for Doing Survey Birnbaum, M. H. (2001). Introduction to Behavioral Research on the Internet. Upper Saddle River, NJ: Prentice Hall. Lots of resources at http://ati.fullerton.edu/http://ati.fullerton.edu/ NSF sponsored classes at Fullerton on Social Psychology One more next January, look for the announcement at this site

20. First Step: Where to put it Setting up a server can be easy Apache web server http://www.apache.org Most popular server Freeware Any computer connected to web is now a server May have issues of firewalls & dynamic IP addresses

21. Second Step: SurveyWiz In Birnbaum (2001) Also linked to from http://ati.fullerton.edu/http://ati.fullerton.edu/ Under Birnbaums page Freely available We will build a very short survey to illustrate

22. Forms SurveyWiz uses web forms to collect data What are forms: Elements to allow web page viewers to input This data is then sent back to the server to be processed SurveyWiz What the page looks like My Take What the page looks likeMy Take Can edit it with any web page editor

23. Examples I have done this year Caddell & Utt Pattison & Reese

24. Third Step: Collecting the data First way, let Birnbaum do it for you SurveyWiz is set up this way FTP://guest:guest99@psych.fullerton.edu You can type in the address box of your browser. Note that this is FTP site. At the present time, password is guest99, as shown above, and permits download only.

25. Keeping the Data Local Second, get this perl script (found at http://psych.fullerton.edu/mbirnbaum/program s/PERL_script2.htm)perl script http://psych.fullerton.edu/mbirnbaum/program s/PERL_script2.htm The script and instructions are at this site Written by Billy Schmidt Also change code in form command At my school the line reads: At my school the line reads

26. What is CGI Common Gateway Interface Method for other programs to interact with web servers In this case, this perl program takes data from web forms and stores them in a file Stores data in Excel or SPSS readable files

27. Fourth Step: Example Data File Data File Data coding scheme: all variable names begin two numbers, see examplesee example The perl program sorts the variables by numerical order

28. Another Method of Survey Generation Schmits Survey Assistant He manages data More flexibility http://or.psychology.dal.ca/~wcs/hidden/home.htm l http://or.psychology.dal.ca/~wcs/hidden/home.htm l

29. What Do I Need? At NSF ATI site List of software resources

30. What Problems are important Musch & Reips (2000) – 5 point scale No control over participants behavior 3.6 No control over motivation3.4 Inability of participants to ask questions3.3 Nonrepresentative sample2.9 Manipulation and fraud2.4 Ethical problems1.5

31. Problems Eliminating Multiple Entries Dropout Security Data Integrity

32. Multiple Entries People submit too fast, while waiting for feedback People will fake being two people Usually have same IP address SurveyWiz and most other methods, sends the IP address of machine where survey is being taken Can eliminate more than one from same IP, e.g. Schmidt (1997)

33. Security & Data Integrity If on public server, others can access data and download Others might fake pages to send data (unlikely) Keep data in non-public directories so only researchers have access Have CGI check for origin of survey to make sure it is yours (Schmidt, 1997)

34. Dropout People will come but not finish Or data is incomplete Survey will sends a complete signal to help you track Can use: (ONeil, Penrod, & Bornstein, 2003) Short sweet Financial incentives. If multiple pages to survey, several pages warm- up so not drop-out during

35. Ethical Issues Should not think of Ethical Issues of Web Research Alone Compare to Ethics of doing Traditional Research Not let our comfort with what we have done blind us to those problems

36. Conclusion Many Benefits Easy Methods Easy Resources Not alternative to traditional methods A new tool still needs to be used thoughtfully