1. C1, L2, S1 Design Method of Data Collection Surveys and Polls Experimentation Observational Studies

2. C1, L2, S2 Nonsampling Errors Sampling/Chance/ Random Errors Selection bias Interviewer effects Non-response bias Behavioural considerations Self selection Transfer findings Question effects Survey-format effects Sampling

3. C1, L2, S3 Sources of Nonsampling Errors Selection bias Population sampled is not exactly the population of interest. e.g. KARE 11 poll, telephone interviews population sample

4. C1, L2, S4 Sources of Nonsampling Errors Non-response bias People who have been targeted to be surveyed do not respond. Non-respondents tend to behave differently to respondents with respect to the question being asked.

5. C1, L2, S5 1936 U.S. Election Country struggling to recover from the Great Depression 9 million unemployed 1929-1933 real income dropped by 1/3

6. C1, L2, S6 1936 U.S. Election Candidates: –Albert Landon (Republican) “The spenders must go!” – Franklin D Roosevelt (Democrat) Deficit financing - “Balance the budget of the people before balancing the budget of the Nation” This man has never been President.

7. C1, L2, S7 1936 U.S. Election Roosevelt’s percentage –Digest prediction of the election result –Gallup’s prediction of the Digest prediction –Gallup’s prediction of the election result –Actual election result 43% 44% 56% 62% Digest sent out 10 million questionnaires to people on club membership lists, telephone directories etc. – received 2.4 million responses Gallup Poll used another sample of 50,000 Gallup used a random sample of 3,000 from the Digest lists to predict Digest outcome

8. C1, L2, S8 Sources of Nonsampling Errors Self-selection bias People decide themselves whether to be surveyed or not. Much behavioral research can only use volunteers.

9. C1, L2, S9 Sources of Nonsampling Errors

10. C1, L2, S10 Sources of Nonsampling Errors This poll is not scientific and reflects the opinions of only those Internet users who have chosen to participate. The results cannot be assumed to represent the opinions of Internet users in general, nor the public as a whole. The QuickVote sponsor is not responsible for poll content, functionality or the opinions expressed therein.

11. C1, L2, S11 Sources of Nonsampling Errors

12. C1, L2, S12 Sources of Nonsampling Errors “The Center for the Study of Higher Education (University of Melbourne) has sought Macquarie's support to survey a representative sample of their academic staff. Interested members of staff should contact Professor John Loxton about obtaining copies of the survey questionnaire.” Staff Newsletter (Macquarie University) 13 November 1998

13. C1, L2, S13 Sources of Nonsampling Errors Question effects Subtle variations in wording can have an effect on responses. e.g. “Should euthanasia be legal?” vs. “Should voluntary euthanasia be legal?”

14. C1, L2, S14 18 August 1980 New York Times/CBS News Poll “Do you think there should be an amendment to the constitution prohibiting abortions?” Yes 29%No62% Later the same people were asked: “Do you think there should be an amendment to the constitution protecting the life of the unborn child?” Yes50%No39%

15. C1, L2, S15 Question Effects in the NZ Census 1986: “What is your ethnic origin? (Tick the box or boxes which apply to you.) 1991: “Which ethnic group do you belong to?” (Tick the box or boxes which apply to you.) 1996: “Tick as many circles as you need to show which ethnic group(s) you belong to.”

16. C1, L2, S16 Sources of Nonsampling Errors Interviewer effects Different interviewers asking the same question can obtain different results. e.g. the sex, race, religion of the interviewer

17. C1, L2, S17 Interviewer Effects in Racial Questions In 1968, one year after a major racial disturbance in Detroit, a sample of black residents were asked: “Do you personally feel that you trust most white people, some white people or none at all?” White interviewer: 35% answered “most” Black interviewer: 7% answered “most”

18. C1, L2, S18 Sources of Nonsampling Errors Behavioral considerations People tend to answer questions in a way they consider to be socially desirable. e.g. pregnant women being asked about their drinking habits

19. C1, L2, S19 Behavioral Considerations in Election Official vote counts show that 86.5 million people voted in the 1980 U.S. presidential elections. A census bureau survey of 64,000 households some weeks later estimated 93.1 million people voted.

20. C1, L2, S20 Sources of Nonsampling Errors Transferring findings Taking the data from one population and transferring the results to another. e.g. Minneapolis opinions may not be a good indication of Winonan opinions. Minneapolis sample Winona

21. C1, L2, S21 Sources of Nonsampling Errors Survey-format effects e.g. question order, survey layout, interviewed by phone or in- person or mail.

22. C1, L2, S22 Nonsampling Errors Sampling/Chance/ Random Errors Selection bias Interviewer effects Non-response bias Behavioural considerations Self selection Transfer findings Question effects Survey-format effects Sampling

23. C1, L2, S23 1999 New Zealand Newspaper Graphic

24. C1, L2, S24 Nonsampling Errors What nonsampling errors could be present if the results of the survey were used as representative of the views held by New Zealanders in general? Discuss in groups for 1 minute.

25. C1, L2, S25 Nonsampling Errors Selection bias –only those who read this paper –only those that can afford to pay for the call Question effects –how do you answer the question?? Self-selection bias

26. C1, L2, S26 Survey Errors Sampling/Chance/ Random Errors Nonsampling Errors

27. C1, L2, S27 Sampling / Chance / Random Errors errors caused by the act of taking a sample have the potential to be bigger in smaller samples than in larger ones possible to determine how large they can be unavoidable (price of sampling)

28. C1, L2, S28 Nonsampling Errors can be much larger than sampling errors are always present can be virtually impossible to correct for after the completion of survey virtually impossible to determine how badly they will affect the result must try to minimize in design of survey (use a pilot survey etc.)

29. C1, L2, S29 Surveys / Polls A pilot survey is a small survey that is carried out before the main survey and is often used to identify any problems with the survey design (such as potential sources of non-sampling errors).

30. C1, L2, S30 Surveys / Polls A report on a sample survey/poll should include: –target population (population of interest) –sample selection method –the sample size and the margin of error –the date of the survey –the exact question(s) being asked