Dear Readers, If you had it to do all over again, would you have children? Ann Landers Ann Landers posed the question to the readers of her advice column. Almost 70% of the approximately responses said “NO!” When a carefully conducted (“scientific”) survey of 1373 parents was conducted, it was found that 91% answered “YES!” What went wrong? Minds on:

In 1936, the Literary Digest sent postcards to a sample of households taken from telephone directories and lists of automobile owners. About 57% of the respondents indicated they would vote for Republican Alf Landon. It was, however, Democratic candidate Franklin Roosevelt who was elected and he was elected with the greatest majority in history up to that time! What went wrong?

5.5 Avoiding Bias

What is Bias? An unintended influence on data-gathering method leads to high sampling variance and inaccurate results Different kinds: –Sampling bias –Non-response bias –Household/Undercoverage bias –Response bias

Sampling Bias When chosen sample does not accurately reflect population Due to non-random sampling –Not all outcomes have the same chance of occurring or –Some outcomes have no chance of occurring

Examples of Sampling Bias (not an exhaustive list) Quota sampling –A type of stratified sampling in which selection within strata is non-random E.g., you don’t use simple/systematic random sampling to choose within each strata, but (say) choose the first n students in each strata –Often used by market and opinion researchers –Pros: less costly, easier to administer –Cons: sample may be biased

Examples of Sampling Bias (not an exhaustive list) Convenience sampling –Items are only selected if they can be accessed easily and conveniently –Ex: the first 10 cookies out of the oven –Ex: people in the front row at a concert –Pros: ease of use –Cons: very, very biased

Examples of Sampling Bias (not an exhaustive list) Volunteer sampling –When the sample chooses you –Ex: phone-in sampling (TV/radio) or internet sampling to gauge public opinion on a issue, Canadian Idol, So You Think You Can Dance, YouTube “likes” –Often no limit to the number of times you can call –Sometimes it costs money to vote

Examples of Sampling Bias (not an exhaustive list) Volunteer sampling (continued) –Pros: cheap (time/money), easy to monitor and control –Cons: highly biased Only those with a phone/internet can vote Only those watching TV/listening to radio at that time aware of survey Some opinions over-represented (you can vote more than once) People not interested in calling not included

Non-Response Bias Occurs when particular groups under-represented because they choose not to participate (in a survey or a question) Biased because characteristics of non-respondents may differ from respondents Different from volunteer sampling, because original sample is chosen by you Often occurs with mail-in surveys To avoid: –Design and test survey carefully –Follow up with non-respondents

Household/Undercoverage Bias When one group is over-represented because groupings of different sizes polled equally Ex: stratify class into 4 ages (16, 17, 18, 19) and take 2 students from each age group 16, 18, 19 over-represented 17 very under-represented

Response Bias Factors in the survey method influence the result, such as –Questionnaires with leading/unclear questions –Interviewer bias (too friendly, aloof, prompting the respondent) –Social desirability Most people want to present themselves in a favourable light

Example of Response Bias 1986 Australian Census pilot test Wanted to gather info on ethnicity “What is your cultural background?” One reply was “none” HUH? People may interpret broad concepts (“cultural background”) differently “What is each person’s ancestry?” E.g. Greek, Armenian, English, … etc.

Find the bias!

Example 1 A survey asked students at a high-school football game whether a fund for extra-curricular activities should be used to buy new equipment for the football team or instruments for the school band. Sampling Bias (specifically convenience) Sample includes only football fans Not representative of entire student body A random sample selected from entire student body

Example 2 A science class asks every fifth student entering the cafeteria to answer a survey on environmental issues. Less than half agree to complete the questionnaire. The completed questionnaires show that a high proportion of respondents are concerned about the environment and well-informed about environmental issues. Non-response bias Students who don’t participate may be less interested in environmental issues Include questions that identify members of particular groups Frame questions so that people will answer them Offer incentives

Example 3a As part of a survey of the “Greatest Hits of All Time,” a radio station asks its listeners: Which was the best song by the Beatles? i) Help! ii) Nowhere Man iii) In My Life Response Bias (prompting) The survey could include listeners’ opinions

Example 3b A poll by a tabloid newspaper includes the question: “Do you favour the proposed bylaw in which the government will dictate whether you have the right to smoke in a restaurant?” Response Bias (leading) Distracting attention from real issue (smoking in restaurants) Question with straight-forward, neutral language will produce more accurate data E.g.: “Should smoking in restaurants be banned?”