Logistic Regression [Many of the slides were originally created by Prof. Dan Jurafsky from Stanford.]

Slides:

Advertisements

Similar presentations

Notes Sample vs distribution “m” vs “µ” and “s” vs “σ” Bias/Variance Bias: Measures how much the learnt model is wrong disregarding noise Variance: Measures.

Advertisements

INTRODUCTION TO MACHINE LEARNING Bayesian Estimation.

PROBABILISTIC MODELS David Kauchak CS451 – Fall 2013.

Middle Term Exam 03/01 (Thursday), take home, turn in at noon time of 03/02 (Friday)

Naïve Bayes Classifier

What is Statistical Modeling

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Assuming normally distributed data! Naïve Bayes Classifier.

Bayes Rule How is this rule derived? Using Bayes rule for probabilistic inference: –P(Cause | Evidence): diagnostic probability –P(Evidence | Cause): causal.

Classification and risk prediction

Logistic Regression Rong Jin. Logistic Regression Model  In Gaussian generative model:  Generalize the ratio to a linear model Parameters: w and c.

Logistic Regression Rong Jin. Logistic Regression Model  In Gaussian generative model:  Generalize the ratio to a linear model Parameters: w and c.

Predictive Automatic Relevance Determination by Expectation Propagation Yuan (Alan) Qi Thomas P. Minka Rosalind W. Picard Zoubin Ghahramani.

Announcements  Project proposal is due on 03/11  Three seminars this Friday (EB 3105) Dealing with Indefinite Representations in Pattern Recognition.

Today Linear Regression Logistic Regression Bayesians v. Frequentists

Review. 2 Statistical modeling  “Opposite” of 1R: use all the attributes  Two assumptions: Attributes are  equally important  statistically independent.

Generative Models Rong Jin. Statistical Inference Training ExamplesLearning a Statistical Model  Prediction p(x;  ) Female: Gaussian distribution N(

Kernel Methods Part 2 Bing Han June 26, Local Likelihood Logistic Regression.

Naïve Bayes Classification Debapriyo Majumdar Data Mining – Fall 2014 Indian Statistical Institute Kolkata August 14, 2014.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

CHAPTER 4: Parametric Methods. Lecture Notes for E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2 Parametric Estimation X = {

MACHINE LEARNING 6. Multivariate Methods 1. Based on E Alpaydın 2004 Introduction to Machine Learning © The MIT Press (V1.1) 2 Motivating Example  Loan.

Simple Bayesian Supervised Models Saskia Klein & Steffen Bollmann 1.

INTRODUCTION TO Machine Learning ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Bayes Classifier, Linear Regression 10701/15781 Recitation January 29, 2008 Parts of the slides are from previous years’ recitation and lecture notes,

Review Rong Jin. Comparison of Different Classification Models  The goal of all classifiers Predicating class label y for an input x Estimate p(y|x)

Crash Course on Machine Learning

CSE 446 Gaussian Naïve Bayes & Logistic Regression Winter 2012

ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –Classification: Naïve Bayes Readings: Barber

1 Logistic Regression Adapted from: Tom Mitchell’s Machine Learning Book Evan Wei Xiang and Qiang Yang.

INTRODUCTION TO MACHINE LEARNING 3RD EDITION ETHEM ALPAYDIN © The MIT Press, Lecture.

Text Classification, Active/Interactive learning.

1 CS 391L: Machine Learning: Bayesian Learning: Beyond Naïve Bayes Raymond J. Mooney University of Texas at Austin.

Empirical Research Methods in Computer Science Lecture 7 November 30, 2005 Noah Smith.

CSE 446 Logistic Regression Winter 2012 Dan Weld Some slides from Carlos Guestrin, Luke Zettlemoyer.

1 Generative and Discriminative Models Jie Tang Department of Computer Science & Technology Tsinghua University 2012.

Machine Learning CUNY Graduate Center Lecture 4: Logistic Regression.

INTRODUCTION TO MACHINE LEARNING 3RD EDITION ETHEM ALPAYDIN Modified by Prof. Carolina Ruiz © The MIT Press, 2014 for CS539 Machine Learning at WPI

Artificial Intelligence 8. Supervised and unsupervised learning Japan Advanced Institute of Science and Technology (JAIST) Yoshimasa Tsuruoka.

ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –Classification: Logistic Regression –NB & LR connections Readings: Barber.

29 August 2013 Venkat Naïve Bayesian on CDF Pair Scores.

Over-fitting and Regularization Chapter 4 textbook Lectures 11 and 12 on amlbook.com.

Regress-itation Feb. 5, Outline Linear regression – Regression: predicting a continuous value Logistic regression – Classification: predicting a.

Introduction to Machine Learning Multivariate Methods 姓名 : 李政軒.

Lecture 3: MLE, Bayes Learning, and Maximum Entropy

Linear Models (II) Rong Jin. Recap  Classification problems Inputs x  output y y is from a discrete set Example: height 1.8m  male/female?  Statistical.

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Naïve Bayes Classification Recitation, 1/25/07 Jonathan Huang.

Naive Bayes (Generative Classifier) vs. Logistic Regression (Discriminative Classifier) Minkyoung Kim.

Text Classification and Naïve Bayes Formalizing the Naïve Bayes Classifier.

ECE 5984: Introduction to Machine Learning Dhruv Batra Virginia Tech Topics: –(Finish) Model selection –Error decomposition –Bias-Variance Tradeoff –Classification:

Bayesian Learning Reading: Tom Mitchell, “Generative and discriminative classifiers: Naive Bayes and logistic regression”, Sections 1-2. (Linked from.

Matt Gormley Lecture 3 September 7, 2016

Quadratic Classifiers (QC)

Oliver Schulte Machine Learning 726

Ch8: Nonparametric Methods

Discriminative and Generative Classifiers

ECE 5424: Introduction to Machine Learning

Tackling the Poor Assumptions of Naive Bayes Text Classifiers Pubished by: Jason D.M.Rennie, Lawrence Shih, Jamime Teevan, David R.Karger Liang Lan 11/19/2007.

ECE 5424: Introduction to Machine Learning

with observed random variables

Mathematical Foundations of BME Reza Shadmehr

Unsupervised Learning II: Soft Clustering with Gaussian Mixture Models

Logistic Regression.

LECTURE 23: INFORMATION THEORY REVIEW

Multivariate Methods Berlin Chen

Multivariate Methods Berlin Chen, 2005 References:

Naïve Bayes Classifier

Presentation transcript:

Logistic Regression [Many of the slides were originally created by Prof. Dan Jurafsky from Stanford.]

Logistic regression classifier Naive Bayes classifier Likelihood P(x|y) Prior P(y) Logistic regression classifier No likelihood No prior Use differing way to estimate probabilities.

Features and weights Movie review N = 4 features 2 classes: +, − Negative weight for enjoy. It is evidence against the class negative.

Posterior probability x = ( ‘great’, ‘second-rate’, ‘no’, ‘enjoy’ ). A feature function, fi, that takes on only the values 0 and 1 is called an indicator function. f1(+,x)=1, f2(+,x)=0, f3(+,x)=0, f3(+,x)=0 f1(−,x)=0, f2(−,x)=1, f3(−,x)=1, f3(−,x)=1 P

Classification

Classify example P(+|x) ∝ 1.9 P(−|x) ∝ .9+.7−.8 = .8

Training logistic regression How are the parameters of the model, the weights w, learned? Logistic regression is trained with conditional maximum likelihood estimation. w Euclidean norm alpha is a constant Bayes probabilities are estimated from relative frequencies by counting. Regression probabilities are estimated from Gaussian means and variance, and Euclidean norm.

Multinomial logistic regression Logistic regression has two possible classes. Multinomial logistic regression has more than two classes. Multinomial logistic regression is also called maximum entropy modeling, MaxEnt for short.

Bayes vs regression Naive Bayes assumes conditional independence. Regression does not. When there are many correlated features, logistic regression will assign a more accurate probability than naive Bayes. Naive Bayes works well on small datasets or short documents. Naive Bayes is easy to implement and fast to train.