Presentation on theme: "Www.andco.uk.com Oliver Mantell Myths and Magic Tricks."— Presentation transcript:
www.andco.uk.com Oliver Mantell Myths and Magic Tricks
What Im Going to Talk About The two basic myths about data: Myth 1: Statistics are irrelevant Myth 2: Statistics are hostile Types of magic tricks: What you can find out with no data at all Getting complexity from simple data Getting simplicity from complex data
What are the odds? If your coin is fair, that: It turns up heads on the next toss? It turns up heads more often than tails? It turns up tails, if it turned up heads last time?
What are the odds? If your audience are 50:50 male to female, that: The next random person you survey is male? You survey more women than men? The next randomly selected visitor is male, if the last was female?
What are the odds? If by knowing that the sample is 50:50 male to female, you can work out the odds of all 10 people you survey being male......you can also work out the probability that the sample is 50:50 male to female, if all 10 people you survey are male. You dont need to be able to work out the exact answer, just to know that it IS possible to work out.
MYTH 3: Randomness is difficult Randomness makes things easy: its when theyre not random it gets difficult.
MYTH 4: My Survey is Sampling My Visitors Sampling works on selecting items from a population that are equally likely, so that they can represent the whole. So youre probably sampling visits, not visitors. VS.
MYTH 5: 51% is bigger than 50% Just because your result is bigger than before, doesnt mean that thats what is happening in reality. You have to look at the margin of error. If its different, but not significant, it isnt different. Most newspaper stories about changes in opinion polls arent true.
MYTH 6: Significant changes matter Theres a difference between a change being real and it being big. You should only care about substantial AND significant changes.
MYTH 7: Significant changes are significant We only said we wanted to be 95% sure. If you look at enough examples, one in twenty wont be significant, although it will look it. LIAR? % % % % %
MYTH 8: A Bigger Sample is Always Better The fewer people you ask randomly, the less chance of a significant result. But asking more, non-randomly, doesnt help.
MYTH 9: Changes to outliers show changes to the odds If you praise people who do well, they do worse. If you shout at people who do badly, they do better. That doesnt mean that praise is bad or shouting is good.
MYTH 10: 67% of Our Focus Group Liked It Theres a reason qual and quant are kept separate. What would it take for that result to be totally different? Yes!No! Erm...?
Say It Aint So, Joe! Null hypotheses are a very powerful trick. A must be true, because B happens. Assume A isnt true: does B still happen?
Welcome to Monte Carlo! Its ok to completely make up the numbers (sometimes).
Slice and Dice What if: 50% of all visitors go to the cafe. and 50% of all visitors are vegetarian. A VeggieNot Veggie B VeggieNot Veggie Cafe 50%0% Cafe 25% Not Cafe 0%50% Not Cafe 25% OR
Judge by results: segmentation by response Response-based – based on likelihood of an answer. Uses same significance tests mentioned earlier. Automatic, shows you what has the biggest effect (and is significant).
Judge by results: segmentation by response Example:
Judge by results: segmentation by response Example:
Birds of a feather: segmentation by cluster Based on best grouping of clusters.
Birds of a feather: segmentation by cluster Shows natural groups within your audience Gets beyond using single categories to describe visitors Identifies similarities and differences between individuals based on patterns in whole audience.
Birds of a feather: segmentation by cluster Example: Used attitudinal and behavioural only Showed real differences that made sense There were real demographic differences between the groups.
Birds of a feather: segmentation by cluster ABCDEF Main reasonSocial / Passing Park / Work- shop Art / Kids / Day Out Local//VeryQuite// Repeat/Q LowHighLow// Satisfied?QuiteOKQuiteNot Very Very Dwell TimeLongQ ShortMediumShortMediumLong Gender2/3 F 1/2 MAll F2/3 F EthnicityAsianMixedAsianWhite AgeYoung /MiddleOlder Group TypeAdult SoloFamily Etc... % respondents12%14%6%2%37%30%
Summary Statistics matter, but they have to be used with care. Used well, however, that can provide real insight that help you to make decisions and do things better.
Just To Remind You... Myth 1: Statistics are irrelevant Myth 2: Statistics are hostile Myth 3: Randomness is difficult Myth 4: My survey is sampling my visitors Myth 5: 51% is bigger than 50% Myth 6: Significant changes matter Myth 7: Significant changes are significant Myth 8: A bigger sample is always better Myth 9: Changes to outliers show changes to the odds Myth 10: 67% of our focus group liked it.
Just To Remind You... Trick 1: Say it aint so! – Null hypotheses Myth 2: Welcome to Monte Carlo! – Using simulations Myth 3: Slice and dice – Cross-tabulation Myth 4: Judge by results – segmentation by response Myth 5: Birds of a feather –segmentation by clusters