Bayesian inference of normal distribution Problem statement Objective is to estimate unknown two parameters q={m,s2} of normal distribution based on observations y = {y1, y2, …}. Prior Joint pdf of non-informative prior Joint posterior distribution This is function of two variables, which can be plotted as a surface or contour. In the todays lecture, the same thing is addressed but with different problem. In this case, it is normal distribution. And the obejctive is … remember that the normal … In this case, we have two unknwon parameters. So at first we estimate mean only while the other is assumed as given a priori. Then we move to the other case, the two parameters are switched. I mean, estiamte variance while the mean is assumed as known. Finally, which is the most practical case, estimate both parameters as unknowns.

Joint posterior distribution There is no inherent pdf function by matlab in this case. This is function of two variables, which can be plotted as a surface or contour. Let’s consider a case with n=20; ȳ =2.9; s=0.2; Remark Analysis of posterior pdf: mean, median & confidence bounds. Marginal distribution Once we have the marginal pdf, we can evaluate its mean and confidence bounds. Posterior prediction: predictive distribution of new y based on observed y. We need some basic understanding of this function within the matlab environment. So lets start the matlab program. Consider the parameters being 100 & 10. First, we can draw the shape of the function. We can compute a pdf value at a certain x. like 90. This can also be obtained using the original expression the same value obtained. The probability less than an x which is the definition of the cdf is also obtained at x=90. Or we can draw the cdf over a range of x too. See here the value at x=90 represents the cdf value which is the probability that the value is less than 90. We can also get the inverse of the cdf. Which means that when a probability is given, then the inverse is where the probability that x is less than that value. We can further generate virtual data using random sampling. In case of 50 #, we can make data using ***. Histogram can be drawn at the interval of 5 from 50 to 150. Mean, mode and median of the data are obtained. Repeat the random sampling, then you will find out the different results are obtained, which is no doubt at all. During the explanation, I would advise the students may follow the same thign that I do. So that you may understand at your hand. Because we have made virtual data frm the true parameters mew and sig, theoretically we expect the mean and stdev may be close to the true values. If we try smaller #, however, the results are more distant from the true values. This is likelihood of new y This is posterior pdf of m & s2

Marginal distributions Analytical solutions Marginal mean Marginal variance How to evaluate marginal distributions Before doing this, note that in normal dist., variable y can be transformed to standard normal variable z. Marginal mean & variance can be evaluated in the same manner. One can evaluate various characteristics using the matlab functions. Unfortunately, unlike the normal case, matlab does not provide t dist of original variable.

Marginal distributions Supplementary information for normal pdf The original pdf fY(y) and normalized pdf fZ(z) have following relation. Therefore, If we want use f(z) instead of f(y), use this relation. Supplementary information for chi2 pdf

Posterior prediction Predictive distribution of new y based on observed y Analytical solution One can evaluate characteristics of this using the matlab function. Compare with the marginal mean. Marginal mean: t-distribution with location ȳ and variance s^2/n with n-1 dof. Predictive distribution: t-distribution with location ȳ and variance s^2*(1+1/n) with n-1 dof. Likelihood of new y posterior pdf of m & s2

Simulation of joint posterior distribution Why ? Even when analytic solutions available, some are not easy to evaluate. Once exercised, may find it more convenient and more general. Practice simulation & validate with analytic solution. Factorization approach Review: joint probability of A & B Likewise, joint posterior pdf of m & s2 The two at the right are marginal pdfs. Simulation by drawing random samples We can evaluate characteristics of joint posterior pdf using simulation techniques.

Simulation of joint posterior distribution Approach 1: use marginal variance. Conditional pdf of m on s2 is already derived, which is the pdf of mean with known variance. (page 10 of Lec. #4) Marginal pdf of s2 is given in this lecture. In order to sample the posterior pdf of p(m,s2|y) Draw s2 from the marginal pdf Draw m from the conditional pdf Conditional pdf of m on s2 Marginal pdf of s2

Simulation of joint posterior distribution Approach 1: use marginal variance. Once you have obtained samples of joint pdf, compare (validate) the results with the analytic solution. Compare the samples of (M,S2) with the analytic joint pdf. In terms of scattered pot & contour. In terms of 3-D histogram & surface (mesh). Compare the samples of M with the marginal pdf, which is t distribution. Compare the samples of S2 with the marginal pdf which is inv-chi2 distribution. Extract features of the samples M and compare with analytic solution. Extract features of the samples S2 and compare with analytic solution.

Simulation of joint posterior distribution Approach 2: use marginal mean. Conditional pdf of s2 on m is already derived, which is the pdf of variance with known mean. (page 13 of Lec. #4) Marginal pdf of m is given in this lecture. In order to sample the posterior pdf of p(m,s2|y) Draw m from the marginal pdf Draw s2 from the conditional pdf Compare (validate) the resulting samples with the analytic solution. Conditional pdf of s2 on m Marginal pdf of m

Simulation of joint posterior distribution Approach 2: use marginal mean. Once you have obtained samples of joint pdf, compare (validate) the results with the analytic solution. Details are omitted for limited time.

Posterior prediction by simulation Analytic approach Result of analytic solution by double integral over infinity to get Simulation In practice, posterior predictive distribution is obtained by random draws. Once we have posterior distribution for m & s2 in the form of samples, the predictive new y are easily obtained by drawing each one from conditional on each individual m & s2. Mean & conf. intervals of posterior prediction can be obtained easily.

Homework

Homework

Homework Problems 1 Use the approach 1 to obtain samples of joint pdf (M,S2). Compare scattered dots with contour of analytic solution. Compare 3-D histogram with mesh shape of analytic solution. Compare the samples of M with the marginal pdf, which is t distribution. Compare the samples of S2 with the marginal pdf which is inv-chi2 dist. Obtain mean, 95% conf. interval of M, & compare with analytic solution. Obtain mean, 95% conf. interval of S2, & compare with analytic solution. Obtain samples of posterior prediction, and obtain mean, 95% conf. interval of ynew & compare with analytic solution.