Bayesian Learning, Cont’d

Administrivia Various homework bugs: Due: Oct 12 (Tues) not 9 (Sat) Problem 3 should read: (duh) (some) info on naive Bayes in Sec. 4.3 of text

Administrivia Another bug in last time’s lecture: Multivariate Gaussian should look like:

5 minutes of math... Joint probabilities Given d different random vars, The “joint” probability of them taking on the simultaneous values given by Or, for shorthand, Closely related to the “joint PDF”

5 minutes of math... Independence: Two random variables are statistically independent iff: Or, equivalently (usually for discrete RVs): For multivariate RVs:

Exercise Suppose you’re given the PDF: Where z is a normalizing constant. What must z be to make this a legitimate PDF? Are and independent? Why or why not? What about the PDF:

Parameterizing PDFs Given training data, [X, Y], w/ discrete labels Y Break data out into sets, etc. Want to come up with models,, Suppose the individual f() s are Gaussian, need the params μ and σ How do you get the params? Now, what if the f()s are something really funky you’ve never seen before in your life, with parameters, etc.

Maximum likelihood Principle of maximum likelihood: Pick the parameters that make the data as probable (or, in general “likely”) as possible Regard the probability function as a func of two variables: data and parameters: Function L is the “likelihood function” Want to pick the that maximizes L

Example Consider the exponential PDF: Can think of this as either a function of x or τ

Exponential as fn of x

Exponential as a fn of τ

Max likelihood params So, for a fixed set of data, X, want the parameter that maximizes L Hold X constant, optimize How? More important: f() is usually a function of a single data point (possibly vector), but L is a func. of a set of data How do you extend f() to set of data?

IID Samples In supervised learning, we usually assume that data points are sampled independently and from the same distribution IID assumption: data are independent and identically distributed

IID Samples In supervised learning, we usually assume that data points are sampled independently and from the same distribution IID assumption: data are independent and identically distributed ⇒ joint PDF can be written as product of individual (marginal) PDFs:

The max likelihood recipe Start with IID data Assume model for individual data point, f(X; Θ ) Construct joint likelihood function (PDF): Find the params Θ that maximize L (If you’re lucky): Differentiate L w.r.t. Θ, set =0 and solve Repeat for each class

Exercise Find the maximum likelihood estimator of μ for the univariate Gaussian: Find the maximum likelihood estimator of β for the degenerate gamma distribution: Hint: consider the log of the likelihood fns in both cases

Putting the parts together [X,Y][X,Y] complete training data

5 minutes of math... Marginal probabilities If you have a joint PDF:... and want to know about the probability of just one RV (regardless of what happens to the others) Marginal PDF of or :

5 minutes of math... Conditional probabilities Suppose you have a joint PDF, f(H,W) Now you get to see one of the values, e.g., H=“183cm” What’s your probability estimate of A, given this new knowledge?

5 minutes of math... Conditional probabilities Suppose you have a joint PDF, f(H,W) Now you get to see one of the values, e.g., H=“183cm” What’s your probability estimate of A, given this new knowledge?

Everything’s random... Basic Bayesian viewpoint: Treat (almost) everything as a random variable Data/independent var: X vector Class/dependent var: Y Parameters: Θ E.g., mean, variance, correlations, multinomial params, etc. Use Bayes’ Rule to assess probabilities of classes