Presentation on theme: "What are the differences between surveys, interviews, scales, and questionnaires? What makes surveys great?"— Presentation transcript:
What are the differences between surveys, interviews, scales, and questionnaires? What makes surveys great?
Determine content and purpose of question Choose the response format Figure out how to word it Figure out where to put it Pilot test! Ask for feedback from participants (at least have a comments box)
Is the question needed? At that level of detail? Is there a double-barreled question? Do you need to ask more than 1 question? Do p’s have the info needed to answer the question? Do you need to be more specific or more general? Are there biases in the question? Will people answer the question honestly? Hypothetical projective respondent
I feel completely secure in facing unknown new situations because I know that my partner will never let me down.
When would you want to use dichotomous vs. nominal, ordinal, or interval data? Advantages and disadvantages? In a Likert scale, how many options should you have? How should they be set up? Should there be a neutral point? How can the responses you offer affect results?
When would fill-in-the-blank be good or bad? When should you give multiple options? Do you have all the alternatives without going into too much detail? At what point do you quit adding categories? When are unstructured response formats good?
How often do you exercise? Infrequently 17% Occasionally 48% Often 35% In the last six months, how often have you engaged in at least 20 minutes of aerobic activity? Almost never 17% 3x/week 15% Less than 1x/week 13% 4x/week 15% 1x/week 12% >4x/week 13% 2x/week 15%
Will people misinterpret the question? Does the question include assumptions or need a time frame specified? Does the question fit the population you’re sampling? How personal is the wording? Is the wording too direct or not direct enough? Are there words people wouldn’t know? Are the alternatives clear? Is the wording unbiased?
in 2005 When I graduated from college When I was 18
In the past 30 days, were you able to climb the stairs with no difficulty? On days when you drink alcohol, how many drinks do you usually have? How many miles are you from the nearest hospital?
Survey wording Don't you agree that campus parking is a problem? There are many people who believe that campus parking is a problem. Are you one of them? Do you agree that campus parking is a problem and that the administration should be working diligently on a solution? What do you think about parking?
I oppose raising taxes.* (willing to pay)2.88 I would be willing to pay a few extra dollars in taxes to provide high-quality education to all children The primary task of the government should be to keep citizens safe from terrorism and crime The primary task of the government should be to preserve citizen’s rights and civil liberties n
I regularly perform routine maintenance on my car Sometimes I don’t change the oil in my car on time.* 3.27 I make it a practice to never lie.*2.65 Like all human beings, I occasionally tell a white lie Monogamy is important to me.5.02 Sexual freedom is important to me.4.43
People should wait to have sex until they are in a committed relationship Sex can strengthen a new relationship.2.40 My partner and I always use protection.4.08 Although I know it is important, sometimes I don’t practice safe sex.* 5.22
Do you approve of the government temporarily taking over major banks in danger of failing? 54% approved Do you approve of the government temporarily nationalizing major banks in danger of failing? 57% disapproved
Another polling example In any health care proposal, how important do you feel it is to give people a choice of both a public plan administered by the federal government and a private plan for their health insurance--extremely important, quite important, not that important, or not at all important? 77% extremely or quite important Would you favor or oppose creating a public health care plan administered by the federal government that would compete directly with private health insurance companies? 43% favor
opinioncenter.com 1. If a bottle of water costs you three times more, would you continue buying it? 2. If not, what would you do? 3. Are you aware of the steps use to process a generic bottle of mineral water? 4. Do you believe that bottled water can be more expensive than oil? 5. When travelling overseas, do you find it easy to find your preferred bottled water brands? 6. How much importance do you give to drinking water on daily basis? 7. How concerned are you that there may be water problems, including water shortages, around the world? 8. Please specify any global water issues or concerns you are aware of? 9. Have you taken any steps to help alleviate any water problems around the world? “The student makes the statement that Darwin killed religion”. “The student makes the statement that finding out whether religion is real is not a big deal in their life”. ‘The Store’ represents good value for the money.”
What should be early vs. late in the survey? What else affects placement? What about layout issues?
Thank them to start Keep it short Be alert to discomfort Thank them at the end Send a copy of the results if they want them
What is the role of the interviewer? How should they be trained? Should they be “blind”? Any examples of good/bad interviewers? What should an interviewer bring/do? How do you decide which person to interview?
What are good techniques for Getting entry Introduction Explanation of study Asking questions Probing for more information Recording the interview Ending the interview
Should you have one or multiple pages? Should you have a line telling people how far along they are? What other things may affect responses? Can you get representative samples?
30s-60s Developed methods 60s-90s Phone interviews, research on response rates, missing values 90s- Harder to get respondents Computer methods Future? Big data
What are advantages and disadvantages of surveys overall? How can the disadvantages be dealt with? How have cell phones affected surveys? How can you deal with unlisted numbers? What are issues with RDD? Mitofsky-Waksberg method List-assisted
Why are response rates decreasing? What are reasons for nonresponse and how can you deal with them? Call back rates? How can survey researchers increase rates? Are lower response rates a problem? When?
When is missing data likely to be a problem? What are the different types of missing data you can have? MCAR vs. MAR vs. MNAR In reality, a continuum between MAR and MNAR What problems do they cause? What are the “old” methods and when are they okay? Listwise deletion Pairwise deletion Mean substitution Missingness dummy variable Regression-based single imputation
Sidebar: What is an eigenvalue and where would you find one? EM algorithm (expectation maximization) Goes through values one at a time. If there is a value, it’s added to the model. If not, then the best guess based on predicting it in regression with all the other variables is put in. This continues until it becomes stable. Good for: mean, variance, covariance estimates, correlation matrices, coefficient alpha, exploratory FA No standard errors produced, so not good for hypo testing Use SAS, NORM, EMCOV, maybe SPSS
MI (multiple imputation): Better because it doesn’t assume that responses lie on the regression line—it adds in random error Use NORM, SAS, Splus PAN program—uses growth curves or clustered data FIML: Does it all in one step SEM software: Amos, LISREL, Mx
Is it okay to “make up data”? Inclusive variable inclusion strategies Include variables that are correlated, even if not in model Reduces bias Helps with power How to deal with missing scale items: If just part of scale, can use partial data if at least ½ variables and good alpha and all item-total correlations are about the same Otherwise, impute at the individual level
Issues with MI: Works even with n = 50, 18 predictors and 50% missing Works with nonnormal data Don’t round Use lots of imputations (e.g., 40 for 50% missing) Consider whether to impute separately for groups (e.g, gender) Can use with clustered data (dummy code) Can use with categorical data Can use with lots of variables
3 form design All get some questions, others just some get 2 method measurement Get data from everyone on cheap measure and from a sample on “expensive” measure Include good predictors of the missing values in your data set Measure p’s plans to drop out Follow up and try to get measures for some drop outs Why isn’t looking at difference between stayers and leavers a cure?
Longer history in sociology Sociograms, centrality Clustering, social influence, selection
Experimental design and mediators/moderators Make sure you understand the difference between them and how to test for them 2 chapters and 2 articles Class is on Tuesday! Outline for paper due (whole thing)