# INTRODUCTION TO MACHINE LEARNING Bayesian Estimation.

## Presentation on theme: "INTRODUCTION TO MACHINE LEARNING Bayesian Estimation."— Presentation transcript:

INTRODUCTION TO MACHINE LEARNING Bayesian Estimation

Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2  Estimating parameters of a model from the data  Regression  Classification  Have some prior knowledge on possible parameter range  Before looking at the data  Distribution of the parameter

Generative Model Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 3

Bayes Rule Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 4

Multinomial variable Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 5  Sample of multinomial data taking one of K state  Sample Likelihood  Good way to specify prior distribution on state probabilities q

Dirichlet Distribution Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 6  Probability of each combination of state probabilities  Parameters: approximate proportions of data in state q i

Posteriori Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 7  Likelihood  Posteriori

Conjugate Prior Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 8  Posteriori and prior have the same form  Sequential learning  Instance by instance  Calculate posteriori for the current item  Make it prior for the next item

Continuous Variable Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 9  Instances are Gaussian Distributed with unknown parameters  Conjugate prior

Continuous Variable Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 10 Posteriori Mean is weighted combination of sample mean and prior mean More samples, estimate is closer to m Little prior uncertainty=>closer to prior mean

Precision/Variance Prior Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 11  More convenient to work with precision  Conjugate prior is a Gamma Distribution

Precision Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 12  Posteriori is a weighted sum of prior and sample statistics

Parameter Estimation Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 13  Used prior to refine distribution parameter estimates  User prior to refine parameter of some function of the input  Regression  Classification discriminant

Regression Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 14

Regression Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 15  Maximum Likelihood  Prediction  Gaussian Prior

Prior on weights Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 16

Examples 17