Presentation on theme: "Bayesian inference PHILOSOPHY OF SCIENCE: Thomas Bayes"— Presentation transcript:
1Bayesian inference PHILOSOPHY OF SCIENCE: Thomas Bayes 1702-1761 Zoltán Dienes
2Subjective probability: Personal conviction in an opinion – to which a number is assigned that obeys the axioms of probability.Probabilities reside in the mind of the individual not the external world.Your subjective probability/conviction in a proposition is just up to you– but you must revise your probability in the light of data in ways consistent with the axioms of probability.
3Personal probabilities: Which of the following alternatives would you prefer? If your choice turns out to be true I will pay you 10 pounds:It will rain tomorrow in BrightonI have a bag with one blue chip and one red chip. I randomly pull out one chip and it is red.
4Personal probabilities: Which of the following alternatives would you prefer? If your choice turns out to be true I will pay you 10 pounds:It will rain tomorrow in BrightonI have a bag with one blue chip and one red chip. I randomly pull out one and it is red.If you chose 2, your p(‘it will rain’) is less than .5If you chose 1, your p(‘it will rain’) is greater than .5
5Assume you think it more likely than not that it will rain. Now which do you chose? I will give you 10 pounds if your chosen statement turns out correct.It will rain tomorrow in Brighton.I have a bag with 3 red chips and 1 blue. I randomly pick a chip and it is red.
6Assume you think it more likely than not that it will rain. Now which do you chose? I will give you 10 pounds if your chosen statement turns out correct.It will rain tomorrow in Brighton.I have a bag with 3 red chips and 1 blue. I randomly pick a chip and it is red.If you chose 2, your p(‘it will rain’) is less than .75 (but more than .5)If you chose 1, your p(‘it will rain’) is more than .75.
7By imagining a bag with differing numbers of red and blue chips, you can make your personal probability as precise as you like.
8This is a notion of probability that applies to the truth of theories (Remember objective probability does not apply to theories)So that means we can answer questions about p(H) – the probability of a hypothesis being true – and also p(H|D) – the probability of a hypothesis given data (which we cannot do on the Neyman-Pearson approach).
9Consider a theory you might be testing in your project. What is your personal probability that the theory is true?
10ODDS in favour of a theory = P(theory is true)/P(theory is false) For example, for the theory you may be testing ‘extroverts have high cortical arousal’Then you might think (it’s completely up to you)P(theory is true) = 0.5It follows you think thatP(theory is false) = 1 – P(theory is true) = 0.5So odds in favour of theory = 0.5/0.5 = 1Even odds
11If you thinkP(theory is true) = 0.8Then you must thinkP(theory is false) = 0.2So your odds in favour of the theory are 0.8/0.2 = 4 (4 to 1)What are your odds in favour of the theory you are testing in your project?
12Odds before you have collected your data are called your prior odds Experimental results tell you by how much to increase your odds (the Bayes Factor, B)Odds after collecting your data are called your posterior oddsPosterior odds = B * Prior oddsFor example, a Bayesian analysis might lead to a B of 5.What would your posterior odds for your project hypothesis be?
13You can convert back to probabilities with the formula: P (Theory is true) = odds/(odds + 1)So if your prior odds had been 1 and the Bayes factor 5Then posterior odds = 5Your posterior probability of your theory being true = 5/6 = .83Not a black and white decision like significance testing (conclude one thing if p = .048 and another if p = .052)
14While your prior odds are subjective ALL assumptions going into B are explicit, and can be argued about until - in principle - agreed on by everyoneThen B is to that degree objectiveContrast Neyman Pearson – e.g. nobody might know what the actual stopping rule was. Assumptions are not necessarily explicit. People can “cheat” in this way with Neyman Pearson – one can’t in Bayes.
15If B is greater than 1 then the data supported your experimental hypothesis over the null If B is less than 1, then the data supported the null hypothesis over the experimental oneIf B = about 1, experiment was not sensitive.(Automatically get a notion of sensitivity;contrast: just relying on p values in significance testing.)
16For each experiment you run, just keep updating your odds/personal probability by multiplying by each new Bayes factorNo need for p values at all!No need for power calculations!No need for critical values of t-tests!And as we will see:No need for post hoc tests!No need to plan number of subjects in advance!
17To know which theory data support need to know what the theories predict The null is normally the prediction of e.g. no differenceNeed to decide what difference or range of differences are consistent with one’s theoryDifficult - but forces one to think clearly about one’s theory.And this is what is required for power calculations, so psychologists should be doing it anyway. Bayes forces one to.
18Contrasts with Neyman-Pearson: A) A non-significant result can mean one should decrease one’s confidence in the null hypothesis.
19A theory predicts a difference between conditions, but a null result is obtained. Prediction of nullMean of dataSEDifference between conditionsShould one be less confident in the theory relative to the null hypothesis?
21What does the theory predict? Mean of dataPrediction of nullIf theory predicts a difference of 10 units, should slightly increase confidence in theory over nullSEDifference between conditions
22What does the theory predict? But if theory predicts a difference of 20 units, data should decrease confidence in theory over nullMean of dataPrediction of nullSEDifference between conditions
23NB It is simplistic to think the theory predicts only one value – but Bayes can deal with a range of values(it typically does)
24Contrasts with Neyman-Pearson: A) A non-significant result can mean one should decrease one’s confidence in the null.ConverselyB) A significant result (i.e. accepting the experimental hypothesis on Neyman Pearson) can mean one should increase one’s confidence in the null
25A theory predicts a difference between conditions, and one is obtained A theory predicts a difference between conditions, and one is obtained. Should one be more confident in the theory relative to the null?Data meanPrediction of nullSEDifference between conditions
27Data meanIf theory predicts e.g. 20 or more, then significant result is evidence against theoryPrediction of nullSEDifference between conditions
28More normally it is because the theory is vague that a significant result leads to less confidence in the theoryi.e. many of the possible predictions of the theory are far from the data
29i.e. because the theory makes no precise prediction at all! The fact the null makes a precise prediction makes it more falsifiable, and hence preferred – this CAN mean one should be more confident in nullBayesian inference would encourage scientists to make more falsifiable theories; significance testing (without power) encourages theories of low falsifiability
302. On Neyman Pearson, it matters whether you formulated your hypothesis before or after looking at the data.Post hoc vs planned comparisonsPredictions made in advance of rather than before looking at the data are treated differentlyBayesian inference: It does not matter what day of the week you thought of your theoryThe evidence for your theory is just as strong regardless of its timing
313. On Neyman Pearson standardly you should plan in advance how many subjects you will run. If you just miss out on a significant result you can’t just run 10 more subjects and test again.You cannot run until you get a significant result.Bayes: It does not matter when you decide to stop running subjects. You can always run more subjects if you think it will help.
324. On Neyman Pearson you must correct for how many tests you conduct in total. For example, if you ran 100 correlations and 4 were just significant, researchers would not try to interpret those significant results.On Bayes, it does not matter how many other statistical hypotheses you investigated. All that matters is the data relevant to each hypothesis under investigation.
33The strengths of Bayesian analyses are also its weaknesses: 1) P-values do not require specifying a predicted effect size so significance testing is simple. Bayes requires saying what the theory predicts – there may be argument over what effect size a theory predicts. Using p-values side steps this argument.
34The strengths of Bayesian analyses are also its weaknesses: 1) P-values do not require specifying a predicted effect size so is simple. Bayes requires saying what the theory predicts – there may be argument over what effect size a theory predicts.BUT isn’t this just the sort of argument the field needs?If someone disagrees they can argue their case – in Bayes all assumptions must be explicit.In Neyman Pearson some important aspects are hidden.Neyman Pearson done properly requires specifying effect size.
35The strengths of Bayesian analyses are also its weaknesses: 2) Bayesian procedures – because they are not concerned with long term frequencies - are not guaranteed to control error probabilities (Type I, type II).
36The strengths of Bayesian analyses are also its weaknesses: 2) Bayesian procedures – because they are not concerned with long term frequencies - are not guaranteed to control error probabilities (Type I, type II).Which is more important to you –to use a procedure with known long term error rates or to know the degree of support for your theory (the amount by which you should change your conviction in a theory)?
37To calculate a Bayes factor must decide what range of differences is consistent with the theory Uniform distributionHalf normalNormal
38Example: The theory predicts a difference will be in one direction. Minimally informative prior, other than difference is positive:Probability density51020Difference between conditionsMaximum difference allowed
39But how to scale the rate of drop? Seems more plausible to think the larger effects are less likely than the smaller ones:Probability densityDifference between conditionsBut how to scale the rate of drop?
40Similar sorts of effects as those predicted in the past have been on the order of a 5% difference between conditions in classification accuracy.Probability density5Difference between conditionsImplies: Smaller effects more likely than bigger ones; effects bigger than 10% very unlikely
41Probability density510Difference between conditions
42To calculate Bayes factor in a t-test situation Need same information from the data as for a t-test:Mean difference, MdiffSE of difference, SEdiff
43To calculate Bayes factor in a t-test situation Need same information from the data as for a t-test:Mean difference, MdiffSE of difference, SEdiffNote: t = Mdiff / SEdiffSo if a paper gives Mdiff and tFor any of between- or within-subjects or single-sample t-test you knowSEdiff = Mdiff / tNB: F(1,x) = t2(x)
44To calculate a Bayes factor: 1) Google “Zoltan Dienes”2) First site to come up is the right one:3) Click on link to book4) Click on link to Chapter Four5) Scroll down and click on “Click here to calculate your Bayes factor!”
47difference = control condition – memory load condition Example: On a task where people make binary classifications about 65% correct (i.e. 15% above chance) we load working memory to see if it interferesdifference = control condition – memory load conditionProbability density51015Difference between conditionsMaximum difference expected
51Similar sorts of effects as those predicted in the past have been on the order of a 5% difference between conditions in classification accuracy.Probability density5Difference between conditionsImplies: Smaller effects more likely than bigger ones; effects bigger than 10% very unlikely
57Assignment:12) What was the mean difference obtained in the study?13) What was the standard error of this difference?14) Extending your answer in (7), specify a probability distribution for the difference expected by the theory and justify it15) What is the Bayes factor in favour of the theory over the null hypothesis?16) What does this Bayes factor tell you that the t-test does not?