Presentation on theme: "Statistical Sampling Part IV – When the MOE Doesn’t Apply."— Presentation transcript:
Statistical Sampling Part IV – When the MOE Doesn’t Apply
This video is designed to accompany pages in Making Sense of Uncertainty Activities for Teaching Statistical Reasoning Van-Griner Publishing Company
Non-Sampling Errors The Margin of Error is a nice mathematical way of addressing sampling variability, also called “random sampling error.” There are lots of other “errors” that can affect data collection and the MOE simply doesn’t apply.
Working Definition Non-Sampling Error – An error or discrepancy caused by something other than the fact that a sample was selected instead of the entire population. These errors include, but are not limited to, data entry errors, nonresponse, biased questions in a questionnaire, question order, and false information provided by respondents.
Muddled Questions “Do you plan to leave your job and look for another one during the coming year?” “Have you often, sometimes, hardly ever, or never felt bad because you were unfaithful to your wife?” (Harris Poll: One percent said often, 14 percent said sometimes or hardly ever, and 85 percent said they never felt bad because of this.) “Should we increase taxes in order to get more housing and better schools, or should we keep them about the same?”
Sensitive Questions Roger Tourangeau and Ting Yan, Psychological Bulletin, 2007, Vol. 133, No. 5, 859–883 “Since 1971, the federal government has sponsored a series of recurring studies to estimate the prevalence of illicit drug use. Other surveys ask national samples of women whether they have ever had an abortion or ask samples of adults whether they voted in the most recent election. “ Item Percentage Nonresponse Total household income8.15 No. of lifetime male sexual partners3.05 Received public assistance2.22 No. of times had sex in past 4 weeks1.37 Age of first sexual intercourse0.87 Blood tested for HIV0.65 Age of first menstrual period0.39 Highest grade completed0.04 “An important question about such surveys is whether respondents answer the questions truthfully. Methodological research on the accuracy of reports in surveys about illicit drug use and other sensitive topics … is a major source of error, more specifically of bias, in the estimates derived from these surveys.”
How bad is it? No hard data to compare sampling errors to non-sampling errors, but it is generally thought that non-sampling errors might be the larger of the two. Harris Poll Disclaimer: All sample surveys and polls, whether or not they use probability sampling, are subject to multiple sources of error which are most often not possible to quantify or estimate, including sampling error, coverage error, error associated with nonresponse, error associated with question wording and response options, and post-survey weighting and adjustments. Therefore, Harris Interactive avoids the words “margin of error” as they are misleading.
What to do? Large body of literature on things to try and things that seem to work 1.Use of inducements for non-responders 2.Use of technology-assisted confidential interview techniques 3.Awareness of psychology of question order 4.Awareness of experiences with question wording 5.Use of internal consistency checks
One-Sentence Reflection Non-sampling errors can substantially detract from the accuracy of survey data, and are not addressed by the margin of error.