1. Introduction & Errors
Dru Rose

2. Methods of Data Collection
Surveys
and Polls
Experimental
Studies
Observational
Studies
Dru Rose

3. Polls and Surveys Poll Survey Few questions Multi-choice type only
Many questions
May have branches and skips
May have a “tick multiple boxes” option
May have open-ended, write a comment questions
Dru Rose

4. Polls & Surveys should include:
Who carried it out
Who funded it
The target population (the population of interest)
Sample selection method
Sample size and margin of error (MOE – more later on this)
Date of survey / poll
Exact questions being asked
Results
The claims (inferences) being made

5. Experimental Study Type of experiment Experimental design features
Pose an investigative question
Randomly assign experimental units to groups
Groups – treatment & control
Treatment and response variables
Control variables (reducing sources of variation)
Replication
‘blind’ experiments
Comparing two independent groups
Paired comparisons – comparing the same group, comparing outcomes before and after a treatment

6. Observational study
This type of study draw inferences about a population from a sample whereby the independent variable is not under the control of the researcher.
The researchers only observe what is happening – they do manipulate any variables
The longest longtitudinal study in the world is the Dunedin Study.

7. 2 reasons for errors on reports
Non-sampling
Errors (human error)
Sampling Errors
(due to sampling)
Selection bias Non-response bias
Self selection bias Question effects
Behavioural considerations
Interviewer effects Survey-format effects
Transfer of findings

8. Sampling errors
Difference between the estimate & the actual population due to SAMPLING VARIATION.
The error is a result of taking a sample from the population (rather than using the whole population)
So they occur naturally
Eg a population parameter eg mean, median is likely to be different for different samples (of the same size) and each estimate is likely to be different from the population parameter.

9. Reducing sampling variability
Sampling variability can be reduced by increasing sample size but it CANNOT be removed.

10. Non-sampling errors (these create bias)
Bias: an inclination or prejudice for or against a person or group, in an unfair way.
This is the difference between the estimate and the actual population due to HUMAN ERROR from the people designing and carrying out the survey or due to the people being surveyed.
They are errors in the data collection process that result from factors OTHER than taking a sample. They can cause BIAS.
This can result in a survey over- or under- estimating the value of the pop. parameter

11. Not eligible for survey
Selection Bias: Population sampled is not exactly the population of interest.
Target population (e.g. adults in NZ)
Sampling frame (e.g. households with a landline phone)
Not included in sampling frame
Cannot be contacted
SAMPLED POPULATION
Not eligible for survey
Refuse to respond
Incapable of responding

12. Sources of Non-sampling Errors
Non-response bias
When people who have been targeted to be surveyed do not respond:
Potential bias if non-respondents are likely to behave differently to respondents with respect to the question being asked.
e.g. Non-respondents in an employment survey are likely to be those who work long hours.

13. Sources of Non-sampling Errors
Self-selection bias
People decide themselves whether to be surveyed or not.
Dru Rose

14. Self-selection bias: phone-in or internet polls

15. Sources of Non-sampling Errors
Question effects
Subtle variations in wording can have an effect on responses.
e.g. “Should euthanasia be legal?”
vs. “Should voluntary euthanasia be legal?”
People are more likely to favour “voluntary” euthanasia.
2009 smacking referendum (& on Moodle)

16. 18 August 1980 New York Times/CBS News Poll
“Do you think there should be an amendment to the constitution prohibiting abortions?”
Yes 29% No 62%
Later the same people were asked:
“Do you think there should be an amendment to the constitution protecting the life of the unborn child?”
Yes 50% No 39%

17. Question Effects in the NZ Census
1986: “What is your ethnic origin?
(Tick the box or boxes which apply to you.)
1991: “Which ethnic group do you belong to?” (Tick the box or boxes which apply to you.)
1996: “Tick as many circles as you need to show which ethnic group(s) you belong to.”
Changes in wording in the Question from 1986 to 1996 makes comparisons unreliable-apparent change in dual ethnicity may be due to wording.
Ethnicity
1986
1991
1996
Single
94.6
94.3
81.0
European
81.2
78.1
65.8
Maori
9.1
9.6
7.6
Two Ethnicities
4.0
4.5
11.2
European & Maori
2.9
2.7
4.7
Two European gps
0.0
0.6

18. Do you drive recklessly?
Thw word ‘recklessly’ would create a systematic error – why?

19. Leading questions. Think of an example and share
This is where the wording of a question is loaded to favour 1 response over another (eg: do you agree that…..prompts the respondent to agree)
A satisfaction survey asks you to indicate whether you are satisfied, dissatisfied or very dissatisfied with a service – where is the bias?

20. Sources of Non-sampling Errors
Behavioural considerations
People tend to answer questions in a way they consider to be socially desirable.
e.g. pregnant women being asked about their drinking habits may be reluctant to admit that they drink alcohol
Dru Rose

21. Sources of Non-sampling Errors
Interviewer effects
Different interviewers asking the same question can obtain different results.
e.g. the sex, race, religion , manner
of the interviewer
may influence how people respond to a particular question.
Dru Rose

22. Interviewer Effects in Racial Questions
In 1968, one year after a major racial disturbance in Detroit, a sample of black residents were asked:
“Do you personally feel that you trust most white people, some white people or none at all?”
White interviewer:
35% answered “most”
Black interviewer:
7% answered “most”

23. Sources of Non-sampling Errors
Survey-format effects
-question order e.g.
“To what extent do you think teenagers are affected by peer pressure when drinking alcohol ?”
followed by:
“ Name the top 5 peer pressures you think teenagers face today.”
-survey layout
-interviewed by phone or in-person
or mail.

24. Sources of Non-sampling Errors
Transferring findings
Taking the data from one population and transferring the results to another.
e.g. Auckland opinions may not be a good indication of New Zealand opinions.
sample
Auckland
New Zealand

25. Non-sampling Errors can be much larger than sampling errors
are always present
can be virtually impossible to correct for after the completion of survey
virtually impossible to determine how badly they will affect the result
good surveys try to minimize them in the design of the survey (e.g. do a pilot survey first)

26. Error & bias
Error – this is the difference between the values that are obtained and the true values of the targetted population
Errors describe by how much the results missed the mark (they take into account ALL flaws in a survey / poll).
Eg a study shows 20% of people’s favourite ice-cream is chocolate but in reality it is 25% - the difference could be due to a whole range of biases and errors but the total level of error is 5%

27. Error = all flaws in a study’s result
Bias = ONLY refers to error that are systematic ie due to the system (and related to human error in the design of the survey or how it is carried out).

28. Note:
Even if there were no non-sampling errors (therefore no bias), then estimates from different samples will still vary from sample to sample because………….

29. The margin of error (MOE)
The sampling error for a given sample is UNKNOWN but….
When sampling is random, the maximum size of the sampling error is called the margin of error.
The MOE indicates the amount of uncertainty due to SAMPLING ERROR