Presentation is loading. Please wait.

Presentation is loading. Please wait.

Pre-Conference Workshop ELI06 January 29, 2006 1 Student and Faculty Surveys Paul R. Hagner Associate Program Director EDUCAUSE Learning Initiative Copyright.

Similar presentations


Presentation on theme: "Pre-Conference Workshop ELI06 January 29, 2006 1 Student and Faculty Surveys Paul R. Hagner Associate Program Director EDUCAUSE Learning Initiative Copyright."— Presentation transcript:

1 Pre-Conference Workshop ELI06 January 29, Student and Faculty Surveys Paul R. Hagner Associate Program Director EDUCAUSE Learning Initiative Copyright Paul R. Hagner, This work is the intellectual property of the author. Permission is granted for this material to be shared for non-commercial, educational purposes, provided that this copyright statement appears on the reproduced materials and notice is given that the copying is by permission of the author. To disseminate otherwise or to republish requires written permission from the author.

2 Pre-Conference Workshop ELI06 January 29, Agenda Why Use Surveys? Types of Surveys Types of Samples Response Bias Question Design Quasi-Experimental Designs Ethics

3 Pre-Conference Workshop ELI06 January 29, What do you want to know?

4 Pre-Conference Workshop ELI06 January 29, Finding out what you need to know Unobtrusive Measures

5 Pre-Conference Workshop ELI06 January 29,

6 6

7 7

8 8 Finding out what you need to know Unobtrusive Measures Obtrusive Measures Error

9 Pre-Conference Workshop ELI06 January 29, Surveys Done Right…. Good snapshot Benchmarking Accreditation Good basis for policy making

10 Pre-Conference Workshop ELI06 January 29, Surveys Done Wrong Time well wasted Inaccurate results Easy to dismiss Dangerous to base policy decisions upon

11 Pre-Conference Workshop ELI06 January 29, Types of Survey Delivery

12 Pre-Conference Workshop ELI06 January 29, Paper Surveys

13 Pre-Conference Workshop ELI06 January 29, In-Person Surveys

14 Pre-Conference Workshop ELI06 January 29, Mail Surveys

15 Pre-Conference Workshop ELI06 January 29, Phone Surveys

16 Pre-Conference Workshop ELI06 January 29, Surveys

17 Pre-Conference Workshop ELI06 January 29, Web-Based Surveys

18 Pre-Conference Workshop ELI06 January 29, Types of Samples Non-Probability Probability Population

19 Pre-Conference Workshop ELI06 January 29, Non-Probability Available Subjects Purposive Snowball

20 Pre-Conference Workshop ELI06 January 29, Non-Probability Surveys Advantages Relatively easy Relatively cheap Disadvantages Not representative Not generalizable

21 Pre-Conference Workshop ELI06 January 29, Probability Samples

22 Pre-Conference Workshop ELI06 January 29, Probability Samples Advantages Can be representative Generalizable Allows targeted follow-up Allows for non- response analyses Disadvantages Lack of anonymity Time intensive Response rate

23 Pre-Conference Workshop ELI06 January 29, Population Web Surveys Advantages Can protect anonymity Inclusive Can be representative Not time intensive Non-Respondent analyses are possible Disadvantages No targeted follow-up Response rate Sample bias

24 Pre-Conference Workshop ELI06 January 29, Response Rate: Destroyer of Survey Results First thing to understand: Numbers don’t matter A randomly selected survey of 400 can have less sample bias than a non-random one of 10,000

25 Pre-Conference Workshop ELI06 January 29, Bias: Random v Systematic

26 Pre-Conference Workshop ELI06 January 29, Me & My Scales Scale 1  Day 1200  Day 2195  Day 3205  Day 4197  Day 5203  Average200 Scale 2  Day 1190  Day 2190  Day 3190  Day 4190  Day 5190  Average190

27 Pre-Conference Workshop ELI06 January 29, Sampling Error vs Sample Bias Sampling error is estimated based on the number of respondents in your sample –The bigger your sample the lower your sampling error

28 Pre-Conference Workshop ELI06 January 29,

29 Pre-Conference Workshop ELI06 January 29, Sampling Error vs Sample Bias Sampling error is estimated based on the number of respondents in your sample –The bigger your sample the lower your sampling error –But!!! It assumes that the sample you have drawn is a random sample!!!

30 Pre-Conference Workshop ELI06 January 29, Sample Bias When there is a non-random reason why some are in your sample and some are not If this bias is significant, the size of the sample won’t solve the problem What matters is the response rate The higher the response rate, the less chance of having a high sample bias

31 Pre-Conference Workshop ELI06 January 29, Why this matters Sample generalizability If the reason people do not respond to the survey is related to the topic of the survey itself, you will get an unrepresentative inclusion or exclusion.

32 Pre-Conference Workshop ELI06 January 29, Exercise 2 Part 1 You are developing a student and faculty survey instruments that measure 1) What instructional technology students expect to be included in their coursework and 2) what instructional technology faculty think students want in their coursework. Q: For students and faculty, what would be some reasons why they would and would not reply to this survey?

33 Pre-Conference Workshop ELI06 January 29, Exercise 2 Part 2 Your student survey has a response rate of 60% Your faculty survey has a response rate of 50%. Q: What can you do to estimate how biased these samples are?

34 Pre-Conference Workshop ELI06 January 29, How High a Response Rate Do You Need? Anything under 50% challenges claims that the survey results represent your target population. National Center for Educational Statistics

35 Pre-Conference Workshop ELI06 January 29, Type of Survey Stage-Specific Design Response Rates ScreenerSchoolAll Other Universe Cross-sectional Longitudinal Assessment Random-Digit Dial> Household    STANDARD 2-2-3: NCES sample survey data collections must be designed to meet a target item response rate of at least 90 percent for each key item.

36 Pre-Conference Workshop ELI06 January 29, How High a Response Rate Do You Need? Anything under 50% challenges claims that the survey results represent your target population. National Center for Educational Statistics This is why random samples are better than population samples

37 Pre-Conference Workshop ELI06 January 29, How to Improve Your Response Rates Be brief! Make response opportunities easily understood Make the survey look professional Do a pre-notification: A survey is coming and it will demand very little of your time

38 Pre-Conference Workshop ELI06 January 29, How to Improve Your Response Rates Have the survey endorsed by a well- regarded (and highly placed) official Follow up: The first, no longer than a week after deployment Make them believe that this survey is important Assure them of confidentiality

39 Pre-Conference Workshop ELI06 January 29, Asking the Right Questions The return of random and systematic bias (Remember the bathroom scale?) When we’re talking about measurements, we are talking about measurement error

40 Pre-Conference Workshop ELI06 January 29,

41 Pre-Conference Workshop ELI06 January 29, Reliability & Validity Reliability –How accurate is our measurement? Validity –Are we measuring what we think we are measuring?

42 Pre-Conference Workshop ELI06 January 29, Exercise 3 We’re trying to measure a student’s information literacy. Here’s the question: What is the frequency that you go on the Internet? __Often __From time to time __Seldom

43 Pre-Conference Workshop ELI06 January 29,

44 Pre-Conference Workshop ELI06 January 29,

45 Pre-Conference Workshop ELI06 January 29,

46 Pre-Conference Workshop ELI06 January 29, Measurement Error Reliability is a quantifiable problem Validity is a theoretical problem

47 Pre-Conference Workshop ELI06 January 29, Reliability Fixes Use established measures Test/retest Use multiple indicators –A note about Response Set, however Split half techniques

48 Pre-Conference Workshop ELI06 January 29, Exercise 4 Suggest some multiple indicators for the concept: Information literacy

49 Pre-Conference Workshop ELI06 January 29, Validity Checks Face Validity Criterion Validity Content Validity

50 Pre-Conference Workshop ELI06 January 29, Exercise 5 Taking the multiple-indicator measure of information literacy developed in the last exercise, suggest ways that you can validate it.

51 Pre-Conference Workshop ELI06 January 29, A Word about Opinions Beliefs Attitudes Opinions The first two are internal Opinions are external Pseudo-opinions

52 Pre-Conference Workshop ELI06 January 29, Surveys: Summary Pre-test!! Publicize!! Randomize!! Put it on the Web! Follow-up!

53 Pre-Conference Workshop ELI06 January 29, The ELI Student & Faculty Survey

54 Pre-Conference Workshop ELI06 January 29, Quasi-Experimental Designs What are they used for? Why ‘Quasi’ ? Experimental group Control group

55 Pre-Conference Workshop ELI06 January 29, Quasi-Experimental Designs

56 Pre-Conference Workshop ELI06 January 29, Why Use a Control Group? Maturation effects History Selection Bias Experimental mortality

57 Pre-Conference Workshop ELI06 January 29, What Should the Control Group Look Like? Ideally: The only difference should be that one gets the “treatment” and the other doesn’t Matching Problems for measuring the impact of new technologies on teaching and learning.

58 Pre-Conference Workshop ELI06 January 29, Ethical Issues in the Use of Surveys Babbie) Voluntary participation Harm to participants Anonymity Confidentiality Deception Institutional Review Boards (Human Subjects Committees)

59 Pre-Conference Workshop ELI06 January 29, Final Thoughts Make friends with someone at your institution who understands survey methodology Staff development opportunities Don’t do it, unless you can do it right “Having some data is better than having none at all” NOT!!!

60 Pre-Conference Workshop ELI06 January 29,


Download ppt "Pre-Conference Workshop ELI06 January 29, 2006 1 Student and Faculty Surveys Paul R. Hagner Associate Program Director EDUCAUSE Learning Initiative Copyright."

Similar presentations


Ads by Google