CSE 446 Logistic Regression Perceptron Learning Winter 2012 Dan Weld Some slides from Carlos Guestrin, Luke Zettlemoyer

Machine Learning 2 Supervised Learning Labeled Examples  Function Reinforcement Learning Experience + Rewards  Policy Unsupervised Learning Unlabeled Examples  Clusters

Machine Learning 3 Supervised Learning Labeled Examples  Function Non-parametric Parametric

Machine Learning 4 Supervised Learning Labeled Examples  Function Y DiscreteY Continuous Gaussians Learned in closed form Linear Functions 1. Learned in closed form 2. Using gradient descent Parametric

Machine Learning 5 Supervised Learning Labeled Examples  Function Y DiscreteY Continuous Gaussians Learned in closed form Linear Functions 1. Learned in closed form 2. Using gradient descent Decision Trees Greedy search; pruning Probability of Class | Features Parametric

Probabilistic Inference (Making Predictions) After learning – (And also during learning) We need to – Determine probabilities of hypotheses – Use these hypotheses to make predictions MLE MAP Bayesian Inference

Which Coin is Dan Using? P(H|C 2 ) = 0.5 P(H|C 1 ) = 0.1 C1C1 C2C2 P(H|C 3 ) = 0.9 C3C3

Prior Probabilities P(H|C 2 ) = 0.5 P(H|C 1 ) = 0.1 C1C1 C2C2 P(H|C 3 ) = 0.9 C3C3 P(C 1 ) = 0.05P(C 2 ) = 0.25P(C 3 ) = 0.70

Forms of Inference P(H|C 2 ) = 0.5 P(H|C 1 ) = 0.1 C1C1 C2C2 P(H|C 3 ) = 0.9 C3C3 P(C 1 ) = 0.05P(C 2 ) = 0.25P(C 3 ) = 0.70 Evidence: Heads; Tails P(C 1 |HT) =0.035P(C 2 |HT)=0.481P(C 3 |HT)=0.485 Predict Next Toss?? Bayesian Inference 68% heads MAP 90% heads

Making predictions after (and during) learning from a distribution of hypotheses PriorHypothesis Maximum Likelihood Estimate Maximum A Posteriori Estimate Bayesian Estimate UniformThe most likely AnyThe most likely Any Weighted combination

Themes: Learning as function approximation – What’s a good approximation? Learning as optimization – Minimize loss over training data (test data) – Loss(h,data) = error(h, data) + complexity(h) 11

Machine Learning 12 Supervised Learning Labeled Examples  Function Y DiscreteY Continuous Gaussians Learned in closed form Linear Functions 1. Learned in closed form 2. Using gradient descent Decision Trees Greedy search; pruning Probability of class | features 1.Learn P(Y), P(X|Y) in closed form Then apply Bayes rule 2.Learn P(Y|X) w/ gradient descent Parametric

Generative vs. Discriminative Classifiers Want to Learn: h:X  Y – X – features – Y – target classes Generative classifier, e.g., Naïve Bayes: – Assume some functional form for P(X|Y), P(Y) – Estimate parameters of P(X|Y), P(Y) directly from training data – Use Bayes rule to calculate P(Y|X= x) – This is a ‘generative’ model Indirect computation of P(Y|X) through Bayes rule As a result, can also generate a sample of the data, P(X) =  y P(y) P(X|y) Discriminative classifiers, e.g., Logistic Regression: – Assume some functional form for P(Y|X) – Estimate parameters of P(Y|X) directly from training data – This is the ‘discriminative’ model Directly learn P(Y|X) But cannot obtain a sample of the data, because P(X) is not available 13 P(Y | X)  P(X | Y) P(Y) Are decision trees generative or discriminative?

Discriminative Classification What Functional Form should we use for P(Y|X)? 14 P(edible|X)=1 P(edible|X)=0 Decision Boundary Want something smooth…

Logistic Function in n Dimensions 15 Sigmoid applied to a linear function of the data: Features can be discrete or continuous!

Understanding Sigmoids w 0 =0, w 1 =-1 w 0 =-2, w 1 =-1 w 0 =0, w 1 = -0.5

Very convenient! implies linear classification rule! 17 ©Carlos Guestrin

Logistic regression more generally Logistic regression in more general case, where Y  {y 1,…,y R } for k<R for k=R (normalization, so no weights for this class) Features can be discrete or continuous! 18©Carlos Guestrin

Generative (Naïve Bayes) Loss function: Data likelihood Discriminative (Logistic Regr.) Loss funct: Conditional Data Likelihood Discriminative models can’t compute P(x j |w)! Or, … “They don’t waste effort learning P(X)” Focus only on P(Y|X) - all that matters for classification Loss Functions (“Error” Term): Likelihood vs. Conditional Likelihood 19 ©Carlos Guestrin

Expressing Conditional Log Likelihood 20©Carlos Guestrin } Probability of predicting 1 Probability of predicting 0 } 1 when correct answer is 01 when correct answer is 1 ln(P(Y=0|X,w) = -ln(1+exp(w 0 +  i w i X i ) ln(P(Y=0|X,w) = w 0 +  i w i X i - ln(1+exp(w 0 +  i w i X i )

Expressing Conditional Log Likelihood ln(P(Y=0|X,w) = -ln(1+exp(w 0 +  i w i X i ) ln(P(Y=0|X,w) = w 0 +  i w i X i - ln(1+exp(w 0 +  i w i X i )

Maximizing Conditional Log Likelihood Bad news: no closed-form solution to maximize l(w) Good news: l(w) is concave function of w! No local minima Concave functions easy to optimize 22©Carlos Guestrin

Optimizing concave function – Gradient ascent Conditional likelihood for Logistic Regression is concave ! Gradient ascent is simplest of optimization approaches – e.g., Conjugate gradient ascent much better (see reading) Gradient: Update rule: Learning rate,  >0

Maximize Conditional Log Likelihood: Gradient ascent

Gradient Descent for LR Gradient ascent algorithm: (learning rate η > 0) do: For i=1 to n: (iterate over weights) until “change” <  Loop over training examples!

Large parameters… Maximum likelihood solution: prefers higher weights – higher likelihood of (properly classified) examples close to decision boundary – larger influence of corresponding features on decision – can cause overfitting!!! Regularization: penalize high weights a=1 a=5 a=10

That’s all M C LE. How about M C AP? One common approach is to define priors on w – Normal distribution, zero mean, identity covariance – “Pushes” parameters towards zero Often called Regularization – Helps avoid very large weights and overfitting MAP estimate:

M C AP as Regularization Penalizes high weights, like we did in linear regression Add log p(w) to objective: – Quadratic penalty: drives weights towards zero – Adds a negative linear term to the gradients

MLE vs. MAP Maximum conditional likelihood estimate Maximum conditional a posteriori estimate

Remember Gaussian Naïve Bayes? P(X i =x| Y=y k ) = N(  ik,  ik ) P(Y | X)  P(X | Y) P(Y) N(  ik,  ik ) = Sometimes Assume Variance – is independent of Y (i.e.,  i ), – or independent of X i (i.e.,  k ) – or both (i.e.,  )

Logistic Regression vs. Naïve Bayes Consider learning f: X  Y, where – X is a vector of real-valued features, – Y is boolean Could use a Gaussian Naïve Bayes classifier – assume all X i are conditionally independent given Y – model P(X i | Y = y k ) as Gaussian N(  ik,  i ) – model P(Y) as Bernoulli( ,1-  ) What does that imply about the form of P(Y|X)? Cool!!!!

Derive form for P(Y|X) for continuous X i only for Naïve Bayes modelsup to now, all arithmetic Can we solve for w i ? Yes, but only in Gaussian case Looks like a setting for w 0 ?

Ratio of class-conditional probabilities … Linear function! Coefficents expressed with original Gaussian parameters!

Derive form for P(Y|X) for continuous X i

Gaussian Naïve Bayes vs. Logistic Regression Representation equivalence – But only in a special case!!! (GNB with class-independent variances) But what’s the difference??? LR makes no assumptions about P(X|Y) in learning!!! Loss function!!! – Optimize different functions ! Obtain different solutions Set of Gaussian Naïve Bayes parameters (feature variance independent of class label) Set of Logistic Regression parameters Can go both ways, we only did one way

Naïve Bayes vs. Logistic Regression Consider Y boolean, X= continuous Number of parameters: Naïve Bayes: 4n +1 Logistic Regression: n+1 Estimation method: Naïve Bayes parameter estimates are uncoupled Logistic Regression parameter estimates are coupled

Naïve Bayes vs. Logistic Regression Generative vs. Discriminative classifiers Asymptotic comparison (# training examples  infinity) – when model correct GNB (with class independent variances) and LR produce identical classifiers – when model incorrect LR is less biased – does not assume conditional independence – therefore LR expected to outperform GNB [Ng & Jordan, 2002]

Naïve Bayes vs. Logistic Regression Generative vs. Discriminative classifiers Non-asymptotic analysis – convergence rate of parameter estimates, (n = # of attributes in X) Size of training data to get close to infinite data solution Naïve Bayes needs O(log n) samples Logistic Regression needs O(n) samples – GNB converges more quickly to its (perhaps less helpful) asymptotic estimates [Ng & Jordan, 2002]

Some experiments from UCI data sets ©Carlos Guestrin Naïve bayes Logistic Regression

What you should know about Logistic Regression (LR) Gaussian Naïve Bayes with class-independent variances representationally equivalent to LR – Solution differs because of objective (loss) function In general, NB and LR make different assumptions – NB: Features independent given class ! assumption on P(X|Y) – LR: Functional form of P(Y|X), no assumption on P(X|Y) LR is a linear classifier – decision rule is a hyperplane LR optimized by conditional likelihood – no closed-form solution – concave ! global optimum with gradient ascent – Maximum conditional a posteriori corresponds to regularization Convergence rates – GNB (usually) needs less data – LR (usually) gets to better solutions in the limit

41

Perceptrons 42

Who needs probabilities? Previously: model data with distributions Joint: P(X,Y) – e.g. Naïve Bayes Conditional: P(Y|X) – e.g. Logistic Regression But wait, why probabilities? Lets try to be error- driven!

Generative vs. Discriminative Generative classifiers: – E.g. naïve Bayes – A joint probability model with evidence variables – Query model for causes given evidence Discriminative classifiers: – No generative model, no Bayes rule, often no probabilities at all! – Try to predict the label Y directly from X – Robust, accurate with varied features – Loosely: mistake driven rather than model driven

Linear Classifiers Inputs are feature values Each feature has a weight Sum is the activation If the activation is: – Positive, output class 1 – Negative, output class 2  f1f1 f2f2 f3f3 w1w1 w2w2 w3w3 >0?

Example: Spam Imagine 3 features (spam is “positive” class): – free (number of occurrences of “free”) – money (occurrences of “money”) – BIAS (intercept, always has value 1) BIAS : -3 free : 4 money : 2... BIAS : 1 free : 1 money : 1... “free money” w.f(x) > 0  SPAM!!!

Binary Decision Rule In the space of feature vectors – Examples are points – Any weight vector is a hyperplane – One side corresponds to Y=+1 – Other corresponds to Y=-1 BIAS : -3 free : 4 money : free money +1 = SPAM -1 = HAM

Binary Perceptron Algorithm Start with zero weights For each training instance (x,y*): – Classify with current weights – If correct (i.e., y=y*), no change! – If wrong: update

Examples: Perceptron Separable Case

Examples: Perceptron Inseparable Case

From Logistic Regression to the Perceptron: 2 easy steps! Logistic Regression: (in vector notation): y is {0,1} Perceptron: y is {0,1}, y(x;w) is prediction given w Differences? Drop the Σ j over training examples: online vs. batch learning Drop the dist’n: probabilistic vs. error driven learning

Multiclass Decision Rule If we have more than two classes: – Have a weight vector for each class: – Calculate an activation for each class – Highest activation wins

Example BIAS : win : game : vote : the :... BIAS : win : game : vote : the :... BIAS : win : game : vote : the :... “win the vote” “win the election” “win the game”

Example BIAS : -2 win : 4 game : 4 vote : 0 the : 0... BIAS : 1 win : 2 game : 0 vote : 4 the : 0... BIAS : 2 win : 0 game : 2 vote : 0 the : 0... “win the vote” BIAS : 1 win : 1 game : 0 vote : 1 the : 1...

The Multi-class Perceptron Alg. Start with zero weights Iterate training examples – Classify with current weights – If correct, no change!  If wrong: lower score of wrong answer, raise score of right answer

Properties of Perceptrons Separability: some parameters get the training set perfectly correct Convergence: if the training is separable, perceptron will eventually converge (binary case) Mistake Bound: the maximum number of mistakes (binary case) related to the margin or degree of separability Separable Non-Separable

Problems with the Perceptron Noise: if the data isn’t separable, weights might thrash – Averaging weight vectors over time can help (averaged perceptron) Mediocre generalization: finds a “barely” separating solution Overtraining: test / held-out accuracy usually rises, then falls – Overtraining is a kind of overfitting

Linear Separators  Which of these linear separators is optimal?

Support Vector Machines  Maximizing the margin: good according to intuition, theory, practice  Support vector machines (SVMs) find the separator with max margin SVM

Three Views of Classification (more to come later in course!) Naïve Bayes: – Parameters from data statistics – Parameters: probabilistic interpretation – Training: one pass through the data Logistic Regression: – Parameters from gradient ascent – Parameters: linear, probabilistic model, and discriminative – Training: one pass through the data per gradient step, use validation to stop The perceptron: – Parameters from reactions to mistakes – Parameters: discriminative interpretation – Training: go through the data until held- out accuracy maxes out Training Data Held-Out Data Test Data