Machine Learning 5. Parametric Methods

Parametric Methods Need a probabilities to make decisions (prior, evidence, likelihood) Probability is a function of input (observables) Represent function by Selecting its general form (model) with several unknown parameters Find(estimate) parameters from data that optimize certain criteria (e.g. minimize generalization error) Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Parametric Estimation Assume sample comes from a distribution known up to its parameters Sufficient statistics : parameters that completely define distribution (e.g. mean and variance) X = { xt }t where xt ~ p (x) Parametric estimation: Assume a form for p (x | θ) and estimate θ, its sufficient statistics, using X e.g., N ( μ, σ2) where θ = { μ, σ2} Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Estimation Assume form of distribution Estimate its sufficient parameters from a sample Use distribution in classification or regressions Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Maximum Likelihood Estimation X consists of independent and identically distributed (iid) samples Likelihood of θ given the sample X l (θ|X) = p (X |θ) = ∏t p (xt|θ) Log likelihood L(θ|X) = log l (θ|X) = ∑t log p (xt|θ) Maximum likelihood estimator (MLE) θ* = argmaxθ L(θ|X) Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Why to use log likelihood Log is increasing function Increased input->increased output Maximizing log of a function is equivalent to maximizing function itself Log convert products to sum log (abc)=log(a)+log(b)+logc Makes analysis/computation simpler Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Examples: Bernoulli/Multinomial Bernoulli: Two states, failure/success, x in {0,1} P (x) = pox (1 – po ) (1 – x) Solving L(p|X)/dp=0 => Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Examples: Multinomial Generalization of Bernoulli : multinomial Outcome is mutually exclusive K states Probability of occurring pi P (x1,x2,...,xK) = ∏i pixi L(p1,p2,...,pK|X) = log ∏t ∏i pixit MLE: pi = ∑t xit / N Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Gaussian (Normal) Distribution p(x) = N ( μ, σ2) Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Bias and Variance of an estimator Actual value of en estimator depends on data Get different N samples from a true distribution , results in different value of an estimator Value of an estimator is a random variables Can ask about its mean and variance Difference between the true value of a parameter and mean of estimator is bias of the estimator Usually looking for formula resulting in unbiased estimator with small variance Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Bias and Variance Unknown parameter θ Estimator di = d (Xi) on sample Xi Bias: bθ(d) = E [d] – θ Variance: E [(d–E [d])2] Mean square error: r (d,θ) = E [(d–θ)2] = (E [d] – θ)2 + E [(d–E [d])2] = Bias2 + Variance Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Mean Estimator Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Variance Estimator Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Optimal Estimators Estimators are random variables as depend on random sample from the distribution We look for estimators (functions of samples) that have minimum expected square error Expected square error can be decomposed into bias (drift) and variance Sometimes accept larger errors but require unbiased estimators. Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Bayes’ Estimator Treat θ as a random var with prior p (θ) Bayes’ rule: p (θ|X) = p(X|θ) p(θ) / p(X) Full: p(x|X) = ∫ p(x|θ) p(θ|X) dθ Maximum a Posteriori (MAP): θMAP = argmaxθ p(θ|X) Maximum Likelihood (ML): θML = argmaxθ p(X|θ) Bayes’: θBayes’ = E[θ|X] = ∫ θ p(θ|X) dθ Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Bayes estimator Sometimes have some prior information on the possible value range that parameter may take Can’t have exact value but can provide a probability for each value (density function) Probability of a parameter before looking at data Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Bayes Estimator Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Bayes Estimator Assume this function have a peak around true value of the parameter Maximum a posteriori estimate will minimize error Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Bayes’ Estimator: Example xt ~ N (θ, σo2) and θ ~ N ( μ, σ2) θML = m θMAP = Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Parametric Classification Bayes Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Assumptions Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Example Input: customer income Output: which car out of K models he will prefer Assume that for each model there is a group of customers with certain income Income in a single class distributed normal Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Example: True distributions Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Example: Estimate parameters Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Example: Discriminant Evaluate on a new input Select class with largest discriminant Assume: priors and variances are equal: Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Boundary between classes Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Equal variances Single boundary at halfway between means Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Variances are different Two boundaries Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Two approaches Likelihood-based approach (till now) Calculate probabilities using Bayes rule Compute discriminant function Discriminant function approach(later) Directly estimate discriminant Bypass probabilities estimation Boundary between classes? Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Regression x is independent variable, r is dependant variable Unknown f, want to approximate to predict future values Parametric approach: assume model with small number of parameters y= Find best parameters from data Also have to make assumption on noise Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Regressions Have a training data (x,r) Find parameters to maximize likelihood In other words, what parameters makes data most probable Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Regressions Ignore the last term,(does not depend on parameters Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Regression Minimize last term Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Least Square Estimate Minimize this Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Linear Regression Assume linear model Need to minimize Set derivatives to zero 2 linear equations in 2 unknowns Can solve easily Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Polynomial Regression Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Bias and Variance noise squared error bias variance Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Bias and Variance Given single sample (x,r), what is the expected error Variations are due to noise and training sample First term is due to noise Does not depend on the estimate Can’t be removed Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Variance Second term Deviation of estimator from regression function Depends on estimator and training set Average over all possible training samples Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Example: polynomial regression As we increase degree of the polynomial Bias decreases as allow better fit to points Variance increases as small deviation in training sample might result in large deviation in model parameters Bias/variance dilemma true for any machine learning systems Need a way to find optimal model complexity to balance between bias and variance Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Bias/Variance Dilemma Example: gi(x)=2 has no variance and high bias gi(x)= ∑t rti/N has lower bias with variance As we increase complexity, bias decreases (a better fit to data) and variance increases (fit varies more with data) Bias/Variance dilemma: (Geman et al., 1992) Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Bias/Variance Dilemma gi Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Polynomial Regression Best fit “min error” Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Model Selection How to select right model complexity? Different from estimating model parameters There are several procedures Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Cross-Validation Can’t calculate bias and variance as don’t know true model But can estimate total generalization error Set aside portion of data (validation set) Increase model complexity, find parameters Calculate error on validation set Stop when error cease to decrease or even start increasing Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Cross-Validation Best fit, “elbow” Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)

Regularization Introduce penalty for model complexity into an error function Find optimal model complexity (e.g. degree of polynomial) and optimal parameters (coefficients) which minimize this function Lambda is penalty for model complexity If lambda is too large only very simple models will be admitted Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1)