C1, L2, S1 Non-sampling Errors in Polls and Surveys L I – To identify different non-sampling errors and think of ways to reduce them.

C1, L2, S2 Methods of Data Collection Surveys and Polls Experimental Studies Observational Studies

C1, L2, S3 Polls and Surveys Poll  Few questions  Multi-choice type only Survey  Many questions  May have branches and skips  May have a “tick multiple boxes” option  May have open-ended, write a comment questions

C1, L2, S4 Non-sampling Errors Sampling Errors (random process) Sampling

C1, L2, S5 Response Errors Coverage Errors Selection bias Question effects Non-response bias Self selection bias Behavioural effects Interviewer effects Survey-format effects Non Sampling Errors Non Response Errors

Sampling frame (e.g. households with a landline phone) Not included in sampling frame Not eligible for survey Cannot be contacted Refuse to respond Incapable of responding SAMPLED POPULATION Selection Bias (Coverage error): Population sampled is not exactly the population of interest.

C1, L2, S7 Sources of Non-sampling Errors Self-selection bias (coverage error) People decide themselves whether to be surveyed or not. Dru Rose

C1, L2, S8 Self-selection bias: phone-in or internet polls

C1, L2, S9 Sources of Non-sampling Errors Non-response error When people who have been targeted to be surveyed do not respond: Potential bias if non-respondents are likely to behave differently to respondents with respect to the question being asked. e.g. Non-respondents in an employment survey are likely to be those who work long hours.

C1, L2, S10 Sources of Non-sampling Errors Question effects (response error) Subtle variations in wording can have an effect on responses. e.g. “Should euthanasia be legal?” vs. “Should voluntary euthanasia be legal?” People are more likely to favour “voluntary” euthanasia.

C1, L2, S11 18 August 1998 New York Times/CBS News Poll “Do you think there should be an amendment to the constitution prohibiting abortions?” Yes 29%No62% Later the same people were asked: “Do you think there should be an amendment to the constitution protecting the life of the unborn child?” Yes50%No39%

C1, L2, S12 Question Effects in the NZ Census 1986: “What is your ethnic origin? (Tick the box or boxes which apply to you.) 1991: “Which ethnic group do you belong to?” (Tick the box or boxes which apply to you.) 1996: “Tick as many circles as you need to show which ethnic group(s) you belong to.”

C1, L2, S13 Sources of Non-sampling Errors Behavioural considerations (response error) People tend to answer questions in a way they consider to be socially desirable. e.g. pregnant women being asked about their drinking habits may be reluctant to admit that they drink alcohol

C1, L2, S14 Sources of Non-sampling Errors Interviewer effects (response error) Different interviewers asking the same question can obtain different results. e.g. the sex, race, religion, manner of the interviewer may influence how people respond to a particular question. Dru Rose

C1, L2, S15 Interviewer Effects in Racial Questions In 1968, one year after a major racial disturbance in Detroit, a sample of black residents were asked: “Do you personally feel that you trust most white people, some white people or none at all?” White interviewer: 35% answered “most” Black interviewer: 7% answered “most”

C1, L2, S16 Sources of Non-sampling Errors Survey-format effects (response error) -question order e.g. “To what extent do you think teenagers are affected by peer pressure when drinking alcohol ?” followed by: “ Name the top 5 peer pressures you think teenagers face today.” -survey layout -interviewed by phone or in-person or mail.

C1, L2, S17 Non-sampling Errors can be much larger than sampling errors are always present can be virtually impossible to correct for after the completion of survey virtually impossible to determine how badly they will affect the result good surveys try to minimize them in the design of the survey (e.g. do a pilot survey first)

C1, L2, S18 Your Task Complete pages 23 and 24 of your Green workbook. There are 10 questions and you should have time to finish up to 35). Try to consider ALL possible non- sampling errors as many situations are affected by more than one.

C1, L2, S19 Surveys / Polls A report on a sample survey/poll should include: –Who carried it out and who funded it –target population (population of interest) –sample selection method –the sample size and the margin of error –the date of the survey –the exact question(s) being asked –the results –the claims (inferences) made