1. Introduction to Bayesian Model Comparison
P548: Bayesian Stats with Psych Applications Instructor: John Miyamoto 2/13/2017: Lecture 07-1
Note: This Powerpoint presentation may contain macros that I wrote to help me create the slides. The macros aren’t needed to view the slides. You can disable or delete the macros without any change to the presentation.

2. Model Estimation versus Model Comparison
Only look at one class of models
Look at posterior distribution of this class of models
Look at “residuals” (posterior predictive distributions) to check systematic failures of the model.
Model Comparison
Look at two or more models.
Compute P(Model m | data) for every model m. Usually assume that all models in the comparison have equal prior probability.
Decide which model is favored by the data.
Psych 548, Miyamoto, Win '17
How to Perform Model Comparisons

3. How to Perform a Model Comparison?
Assume that we have two models, Model 1 and Model 2.
Both models predict the data.
Given the data, which model has the higher posterior probability? I.e., how can we compare
P(Model m = 1 | data) to P(Model m = 2 | data)?
Psych 548, Miyamoto, Win '17
Odds Form of Bayes Rule and Bayes Factor

4. Odds Form of Bayes Rule and the Bayes Factor
Odds Form of Bayes Rule: Typically, in a model comparison we assume that all models have equal prior probability, i.e., Under this assumption,
Bayes Factor
Psych 548, Miyamoto, Win '17
Same Slide Without Bayes Factor Label

5. Odds Form of Bayes Rule and the Bayes Factor
Odds Form of Bayes Rule: Typically, in a model comparison we assume that all models have equal prior probability, i.e., Under this assumption,
Psych 548, Miyamoto, Win '17
Two Factory Problem

6. Two Factory Problem from Kruschke Ch 10.2
We know that Acme Magic Company has two factories that produce biased coins.
Factory 1 produces tail-biased coins and Factory 2 produces head-biased coins.
Let θ represent the probability of "heads" from a particular coin.
SUMMARY:
Psych 548, Miyamoto, Win '17
Four Methods for Computing P(D|Model m)

7. Four Methods for Computing P(data | Model m)
Psych 548, Miyamoto, Win '17
Point out that JM Has Modified the genMCMC Function

8. Look at the Model File in Row 10 - 17 of Table 3
Note that ‘model.choice’ argument was added to the genMCMC function (Row 1 of Table 3).
Note that string variables are defined at Rows 7 – 9 of Table 3.
The model file is defined in terms of a ‘paste’ operation (See Row 10 of Table 3).
Psych 548, Miyamoto, Win '17
Explain Method 3

9. Method 3: Compute P(Model m = i | data) separately for each i
Method 3 depends on the approximation: (pp of Kruschke’s textbook)
We can use the values of i that JAGS samples from the posterior of  to evaluate the summation in Eq. (4).
Psych 548, Miyamoto, Win '17 #

10. Run Table 6 (Method 3)
Psych 548, Miyamoto, Win '17

11. Psych 548, Miyamoto, Aut ‘16