Presentation on theme: "580.691 Learning Theory Reza Shadmehr Bayesian learning 1: Bayes rule, priors and maximum a posteriori."— Presentation transcript:
Learning Theory Reza Shadmehr Bayesian learning 1: Bayes rule, priors and maximum a posteriori
Frequentist vs. Bayesian Statistics Frequentist Thinking True parameter: Estimate of this parameter: Bayesian Thinking Does not have the concept of a true parameter. Rather, at every given time we have knowledge about w (the prior), gain new data, and then update our knowledge using Bayes rule (the posterior). Many different ways in which we can come up with estimates (e.g. Maximum Likelihood estimate), and we can evaluate them. Posterior distr. Prior Distr. Conditional Distr. Given Bayes rule, there is only ONE correct way of learning.
Binomial distribution and discrete random variables Suppose a random variable can only take one of two variables (e.g., 0 and 1, success and failure, etc.). Such trials are termed Bernoulli trials. Probability distribution of a specific sequence of successes and failures Probability density or distribution
Poor performance of ML estimators with small data samples Suppose we have a coin and wish to estimate the outcome (head or tail) from observing a series of coin tosses. = probability of tossing a head. After observing n coin tosses, we note that: out of which h trials are head. To estimate whether the next toss will be head or tail, we form an ML estimator: After one toss, if it comes up tail, our ML estimate predicts zero probability of seeing heads. If first n tosses are tails, the ML continues to predict zero prob. of seeing heads. Probability of observing a particular sequence of heads and tails in D
The numerator is just the joint distribution of and D, evaluated at a particular D. The denominator is the marginal distribution of the data D, that is, it is just a number that makes the Numerator integrate to one. Posterior distr. Prior Distr.Conditional Distr. Including prior knowledge into the estimation process Even though the ML estimator might say, we know that the coin can come up both heads and tails, i.e.: Starting point for our consideration is that is not only a number, but we will give a full probability distribution function Suppose we know that the coin is either fair ( =0.5) with prob. or in favor of tails ( =0.4) with probability 1-. We want to combine this prior knowledge with new data D (i.e. number of heads in n throws) to arrive at a posterior distribution for. We will apply Bayes rule:
Bayesian estimation for a potentially biased coin Suppose that we believe that the coin is either fair, or that it is biased toward tails: = probability of tossing a head. After observing n coin tosses, we note that: out of which h trials are head. Now we can accurately calculate the probability that we have a fair coin, given some data D. In contrast to the ML estimate, which only gave us one number ML, we have here a full probability distribution, that is we know also how certain we are that we have a fair or unfair coin. In some situation we would like a single number, that represents our best guess of. One possibility for this best guess is the maximum a-posteriori estimate (MAP).
MAP estimator: Maximum a-posteriori estimate We define the MAP estimate as the maximum (i.e. mode) of the posterior distribution. The latter version makes the comparison to the maximum likelihood estimate easy: We see that ML and MAP are identical, if p( ) is a constant that does not depend on. Thus our prior would be a uniform distribution over the domain of. We call such a prior for obvious reasons a flat or uniformed prior.
Formulating a continuous prior for the coin toss problem In the last example the probability of tossing a head, represented by, could only be either 0.5 or p=0.4. How should we choose a prior distribution if can be between 0 and 1? Suppose we observed n tosses. The probability density that exactly h of those tosses were heads is: = probability of tossing a head Binomial distribution
Formulating a continuous prior for the coin toss problem represents the probability of a head. We want a continuous distribution that is defined between =0 and =1, and is 0 for =0 and =1. = probability of tossing a head Beta distribution normalizing constant
Formulating a continuous prior for the coin toss problem In general, lets assume our knowledge comes in the form of a beta distribution: When we apply Bayes rule to integrate some old knowledge (the prior) in the form of a beta-distribution with parameters and, with some new knowledge h and n (coming from a binomial distribution), then we find that the posterior distribution also has the form of a beta distribution with parameters +h and +n-h. Beta and binomial distribution are therefore called conjugate distributions.
MAP estimator for the coin toss problem Let us look at the MAP estimator if we start with a prior of =1, n=2, i.e. we have a slight belief in the fact that the coin is fair. Our posterior is then: Note that after one toss, if we get a tail, our probability of tossing a head is 0.33, not zero as in the ML case. Lets calculate the MAP-estimate so that we can compare it to the ML estimate
Assume you only know the height of a person, but not their gender. Can height tell you something about gender? Assume y=height and x=gender (0=male or 1=female). Height is normally distributed in the population of men and in the population of women, with different means, and similar variances. Let x be an indicator variable for being a female. Then the conditional distribution of y (the height becomes): What we have: densities What we want: probability Classification with a continuous conditional distribution
Let us further assume that we start with a prior distribution, such that x is 1 with probability. The posterior is a logistic function of a linear function of the data and parameters (remember this result the section on classification!). The maximum-likelihood argument would just have decided under which model the data would have been more likely. The posterior distribution gives us the full probability that we have a male or female. We can also include prior knowledge in our scheme. Classification with a continuous conditional distribution
Computing the probability that the subject is female, given that we observed height y. Our prior probability Posterior probability: Classification with a continuous conditional distribution
Summary Bayesian estimation involves the application of Bayes rule to combine a prior density and a conditional density to arrive at a posterior density. Maximum a posteriori (MAP) estimation: If we need a best guess from our posterior distribution, often the maximum of the posterior distribution is used. The MAP and ML estimate are identical, when our prior is uniformly distributed on, i.e. is flat or uniformed. With a two-way classification problem and data that is Gaussian given the category membership, the posterior is a logistic function, linear in the data.