1. Pre-Conference Workshop ELI06 January 29, 2006 1 Student and Faculty Surveys Paul R. Hagner Associate Program Director EDUCAUSE Learning Initiative Copyright Paul R. Hagner, 2006. This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

2. Pre-Conference Workshop ELI06 January 29, 2006 2 Agenda Why Use Surveys? Types of Surveys Types of Samples Response Bias Question Design Quasi-Experimental Designs Ethics

3. Pre-Conference Workshop ELI06 January 29, 2006 3 What do you want to know?

4. Pre-Conference Workshop ELI06 January 29, 2006 4 Finding out what you need to know Unobtrusive Measures

5. Pre-Conference Workshop ELI06 January 29, 2006 5

6. 6

7. 7

8. 8 Finding out what you need to know Unobtrusive Measures Obtrusive Measures Error

9. Pre-Conference Workshop ELI06 January 29, 2006 9 Surveys Done Right…. Good snapshot Benchmarking Accreditation Good basis for policy making

10. Pre-Conference Workshop ELI06 January 29, 2006 10 Surveys Done Wrong Time well wasted Inaccurate results Easy to dismiss Dangerous to base policy decisions upon

11. Pre-Conference Workshop ELI06 January 29, 2006 11 Types of Survey Delivery

12. Pre-Conference Workshop ELI06 January 29, 2006 12 Paper Surveys

13. Pre-Conference Workshop ELI06 January 29, 2006 13 In-Person Surveys

14. Pre-Conference Workshop ELI06 January 29, 2006 14 Mail Surveys

15. Pre-Conference Workshop ELI06 January 29, 2006 15 Phone Surveys

16. Pre-Conference Workshop ELI06 January 29, 2006 16 Email Surveys

17. Pre-Conference Workshop ELI06 January 29, 2006 17 Web-Based Surveys

18. Pre-Conference Workshop ELI06 January 29, 2006 18 Types of Samples Non-Probability Probability Population

19. Pre-Conference Workshop ELI06 January 29, 2006 19 Non-Probability Available Subjects Purposive Snowball

20. Pre-Conference Workshop ELI06 January 29, 2006 20 Non-Probability Surveys Advantages Relatively easy Relatively cheap Disadvantages Not representative Not generalizable

21. Pre-Conference Workshop ELI06 January 29, 2006 21 Probability Samples

22. Pre-Conference Workshop ELI06 January 29, 2006 22 Probability Samples Advantages Can be representative Generalizable Allows targeted follow-up Allows for non- response analyses Disadvantages Lack of anonymity Time intensive Response rate

23. Pre-Conference Workshop ELI06 January 29, 2006 23 Population Web Surveys Advantages Can protect anonymity Inclusive Can be representative Not time intensive Non-Respondent analyses are possible Disadvantages No targeted follow-up Response rate Sample bias

24. Pre-Conference Workshop ELI06 January 29, 2006 24 Response Rate: Destroyer of Survey Results First thing to understand: Numbers don’t matter A randomly selected survey of 400 can have less sample bias than a non-random one of 10,000

25. Pre-Conference Workshop ELI06 January 29, 2006 25 Bias: Random v Systematic

26. Pre-Conference Workshop ELI06 January 29, 2006 26 Me & My Scales Scale 1  Day 1200  Day 2195  Day 3205  Day 4197  Day 5203  Average200 Scale 2  Day 1190  Day 2190  Day 3190  Day 4190  Day 5190  Average190

27. Pre-Conference Workshop ELI06 January 29, 2006 27 Sampling Error vs Sample Bias Sampling error is estimated based on the number of respondents in your sample –The bigger your sample the lower your sampling error

28. Pre-Conference Workshop ELI06 January 29, 2006 28

29. Pre-Conference Workshop ELI06 January 29, 2006 29 Sampling Error vs Sample Bias Sampling error is estimated based on the number of respondents in your sample –The bigger your sample the lower your sampling error –But!!! It assumes that the sample you have drawn is a random sample!!!

30. Pre-Conference Workshop ELI06 January 29, 2006 30 Sample Bias When there is a non-random reason why some are in your sample and some are not If this bias is significant, the size of the sample won’t solve the problem What matters is the response rate The higher the response rate, the less chance of having a high sample bias

31. Pre-Conference Workshop ELI06 January 29, 2006 31 Why this matters Sample generalizability If the reason people do not respond to the survey is related to the topic of the survey itself, you will get an unrepresentative inclusion or exclusion.

32. Pre-Conference Workshop ELI06 January 29, 2006 32 Exercise 2 Part 1 You are developing a student and faculty survey instruments that measure 1) What instructional technology students expect to be included in their coursework and 2) what instructional technology faculty think students want in their coursework. Q: For students and faculty, what would be some reasons why they would and would not reply to this survey?

33. Pre-Conference Workshop ELI06 January 29, 2006 33 Exercise 2 Part 2 Your student survey has a response rate of 60% Your faculty survey has a response rate of 50%. Q: What can you do to estimate how biased these samples are?

34. Pre-Conference Workshop ELI06 January 29, 2006 34 How High a Response Rate Do You Need? Anything under 50% challenges claims that the survey results represent your target population. National Center for Educational Statistics

35. Pre-Conference Workshop ELI06 January 29, 2006 35 Type of Survey Stage-Specific Design Response Rates ScreenerSchoolAll Other Universe Cross-sectional Longitudinal Assessment Random-Digit Dial> Household  70 90  85 70 80  95 85 90 85 90 85 STANDARD 2-2-3: NCES sample survey data collections must be designed to meet a target item response rate of at least 90 percent for each key item.

36. Pre-Conference Workshop ELI06 January 29, 2006 36 How High a Response Rate Do You Need? Anything under 50% challenges claims that the survey results represent your target population. National Center for Educational Statistics This is why random samples are better than population samples

37. Pre-Conference Workshop ELI06 January 29, 2006 37 How to Improve Your Response Rates Be brief! Make response opportunities easily understood Make the survey look professional Do a pre-notification: A survey is coming and it will demand very little of your time

38. Pre-Conference Workshop ELI06 January 29, 2006 38 How to Improve Your Response Rates Have the survey endorsed by a well- regarded (and highly placed) official Follow up: The first, no longer than a week after deployment Make them believe that this survey is important Assure them of confidentiality

39. Pre-Conference Workshop ELI06 January 29, 2006 39 Asking the Right Questions The return of random and systematic bias (Remember the bathroom scale?) When we’re talking about measurements, we are talking about measurement error

40. Pre-Conference Workshop ELI06 January 29, 2006 40

41. Pre-Conference Workshop ELI06 January 29, 2006 41 Reliability & Validity Reliability –How accurate is our measurement? Validity –Are we measuring what we think we are measuring?

42. Pre-Conference Workshop ELI06 January 29, 2006 42 Exercise 3 We’re trying to measure a student’s information literacy. Here’s the question: What is the frequency that you go on the Internet? __Often __From time to time __Seldom

43. Pre-Conference Workshop ELI06 January 29, 2006 43

44. Pre-Conference Workshop ELI06 January 29, 2006 44

45. Pre-Conference Workshop ELI06 January 29, 2006 45

46. Pre-Conference Workshop ELI06 January 29, 2006 46 Measurement Error Reliability is a quantifiable problem Validity is a theoretical problem

47. Pre-Conference Workshop ELI06 January 29, 2006 47 Reliability Fixes Use established measures Test/retest Use multiple indicators –A note about Response Set, however Split half techniques

48. Pre-Conference Workshop ELI06 January 29, 2006 48 Exercise 4 Suggest some multiple indicators for the concept: Information literacy

49. Pre-Conference Workshop ELI06 January 29, 2006 49 Validity Checks Face Validity Criterion Validity Content Validity

50. Pre-Conference Workshop ELI06 January 29, 2006 50 Exercise 5 Taking the multiple-indicator measure of information literacy developed in the last exercise, suggest ways that you can validate it.

51. Pre-Conference Workshop ELI06 January 29, 2006 51 A Word about Opinions Beliefs Attitudes Opinions The first two are internal Opinions are external Pseudo-opinions

52. Pre-Conference Workshop ELI06 January 29, 2006 52 Surveys: Summary Pre-test!! Publicize!! Randomize!! Put it on the Web! Follow-up!

53. Pre-Conference Workshop ELI06 January 29, 2006 53 The ELI Student & Faculty Survey http://survey.educause.edu/elitools/sf

54. Pre-Conference Workshop ELI06 January 29, 2006 54 Quasi-Experimental Designs What are they used for? Why ‘Quasi’ ? Experimental group Control group

55. Pre-Conference Workshop ELI06 January 29, 2006 55 Quasi-Experimental Designs

56. Pre-Conference Workshop ELI06 January 29, 2006 56 Why Use a Control Group? Maturation effects History Selection Bias Experimental mortality

57. Pre-Conference Workshop ELI06 January 29, 2006 57 What Should the Control Group Look Like? Ideally: The only difference should be that one gets the “treatment” and the other doesn’t Matching Problems for measuring the impact of new technologies on teaching and learning.

58. Pre-Conference Workshop ELI06 January 29, 2006 58 Ethical Issues in the Use of Surveys Babbie) Voluntary participation Harm to participants Anonymity Confidentiality Deception Institutional Review Boards (Human Subjects Committees)

59. Pre-Conference Workshop ELI06 January 29, 2006 59 Final Thoughts Make friends with someone at your institution who understands survey methodology Staff development opportunities Don’t do it, unless you can do it right “Having some data is better than having none at all” NOT!!!

60. Pre-Conference Workshop ELI06 January 29, 2006 60