1. Prof. Navneet Goyal CS & IS BITS, Pilani
Logistic Regression
Prof. Navneet Goyal
CS & IS
BITS, Pilani

2. Perceptron

3. Logistic Regression
In Linear regression, the dependent variable is continuous
What if the dependent is dichotomous or binary?
A person will vote for Reagan (1) or Carter (0)?
A woman will give birth to a low weight baby(1) or not (0)?
Does the person have a disease? Yes (1) or No (0)
Outcome of a baseball game? Win (1) or loss (0)
A linear regression model will not be able solve it with acceptable error
Moreover, values above 1 and below 0 do not make any sense
Also, we are more interested in probabilites (or odds ratio) than in 0 or 1 output
What can be done?

4. Logistic Regression
Actuary Example: Model P(death|X), the probability that a person X will die within the next 10 years
X = {x1=age,x2=M/F,x3= cholestrol level}
∑wixi = wTX is not a probability!!
Introduce logistic function: σ (a) = 1/(1+e-a)

5. Probit Regression 2-class Multi-class Parameters using
Logistic Regression
2-class
Multi-class
Parameters using
Maximum Likelihood
Iterative Reweighted Least Squares (IRLS)
Probit Regression

6. The Logistic Function
Sigmoid fn is S-shaped and is also called the squashing fn because it maps the whole real line into a finite interval
(maps real a ε (-∞, +∞) to finite (0,1) interval)
Plays an important role in many classification algorithms
Symmetry property:
Inverse of the logistic sigmoid fn is given by
& is called the logit or log odds function because it represents the log of the ratio of the probabilities

7. Probabilistic Discriminative Models
Logistic Regression Example Data Set
Response Variable –> Admission to Grad School (Admit)
0 if admitted, 1 if not admitted
Predictor Variables
GRE Score (gre)
Continuous
University Prestige (topnotch)
1 if prestigious, 0 otherwise
Grade Point Average (gpa)

8. Probabilistic Discriminative Models
First 10 Observations of the Data Set
ADMIT GRE TOPNOTCH GPA

9. Dot-plot: Data from Table 2

10. Logistic regression (2)
Table 3 Prevalence (%) of signs of CD according to age group

11. Dot-plot: Data from Table 3
Diseased %
Age (years)

12. Logistic Regression Consider the linear probability model
Issue: π(Xi) can take on values less than 0 or greater than 1
Issue: Predicted probability for some subjects fall outside of the [0,1] range. 12

13. Logistic Regression Consider the logistic regression model
GLM with binomial random component and identity link g(μ) = logit(μ)
Range of values for π(Xi) is 0 to 1 13

14. Logistic Regression Consider the logistic regression model
And the linear probability model
Then the graph of the predicted probabilities for different grade point averages: 14

15. What is Logistic Regression?
In a nutshell:
A statistical method used to model dichotomous or binary outcomes (but not limited to) using predictor variables.
Used when the research method is focused on whether or not an event occurred, rather than when it occurred (time course information is not used).

16. What is Logistic Regression?
What is the “Logistic” component?
Instead of modeling the outcome, Y, directly, the method models the log odds(Y) using the logistic function.

17. Logistic Regression
Simple logistic regression = logistic regression with 1 predictor variable
Multiple logistic regression = logistic regression with multiple predictor variables
Multiple logistic regression =
Multivariable logistic regression =
Multivariate logistic regression

18. Logistic Regression predictor variables dichotomous outcome
is the log(odds) of the outcome.

19. Logistic Regression is the log(odds) of the outcome. intercept
model coefficients
is the log(odds) of the outcome.

20. Odds & Probability

21. Maximum Likelihood Flipped a fair coin 10 times:
T, H, H, T, T, H, H, T, H, H
What is the Pr(Heads) given the data?
1/100? 1/5? /2? 6/10?

22. Maximum Likelihood T, H, H, T, T, H, H, T, H, H
What is the Pr(Heads) given the data?
Most reasonable data-based estimate would be 6/10.
In fact,
is the ML estimator of p.

23. Maximum Likelihood: Example
Discrete distribution, finite parameter space
How biased an unfair coin is?
Call the probability of tossing a HEAD p. Determine p.
Toss the coin 80 times
Outcome is 49 HEADS and 31 TAILS,
Suppose the coin was taken from a box containing three coins:
one which gives HEADS with probability p = 1/3,
one which gives HEADS with probability p = 1/2 and
another which gives HEADS with probability p = 2/3.
NO labels on these coins
Using maximum likelihood estimation the coin that has the largest likelihood can be found, given the data that were observed.
By using the probability mass function of the binomial distribution with sample size equal to 80, number successes equal to 49 but different values of p (the "probability of success"), the likelihood function (defined below) takes one of three values:

24. Maximum Likelihood: Example
Discrete distribution, finite parameter space
The likelihood is maximized when p = 2/3, and so this the maximum likelihood estimate for p.

25. Maximum Likelihood: Example
Discrete distribution, continuous parameter space
Now suppose that there was only one coin but its p could have been any value 0 ≤ p ≤ 1.
The likelihood function to be maximized is:
and the maximization is over all possible values 0 ≤ p ≤ 1.
differentiating with respect to p
(solutions p = 0, p = 1, and p = 49/80)
The solution which maximizes the likelihood is clearly p = 49/80
Thus the maximum likelihood estimator for p is 49/80.

26. Maximum Likelihood: Example
Continuous distribution, continuous parameter space
Do it for Gaussian Distribution yourself!
Two parameters, μ & σ
Its expectation value is equal to the parameter μ of the given distribution,
which means that the maximum-likelihood estimator μ is unbiased.
This means that the estimator is biased. However, is consistent.
In this case it could be solved individually for each parameter. In general, it may not be the case.

27. Maximum Likelihood
The method of maximum likelihood estimation chooses values for parameter estimates (regression coefficients) which make the observed data “maximally likely.”
Standard errors are obtained as a by-product of the maximization process

28. The Logistic Regression Model
intercept
model coefficients
is the log(odds) of the outcome.

29. Maximum Likelihood
We want to choose β’s that maximizes the probability of observing the data we have:
Assumption: independent y’s

30. Linear probability model
Obvious possibility is to use traditional linear regression model
But this has problems
Distribution of dependent variable hardly normal
Predicted probabilities cannot be less than 0, greater than 1

31. Linear probability model predictions

32. Logistic regression model
Instead, use logistic transformation (logit) of probability, log of the odds

33. Logistic regression model predictions

34. Estimation of logistic regression model
Least-squares no longer best way of estimating parameters of logistic regression model
Instead use maximum likelihood estimation
Finds values of parameters that have greatest probability, given data

35. Space shuttle data
Data on 24 space shuttle launches prior to Challenger
Dependent variable, whether shuttle flight experienced thermal distress incident
Independent variables
Date – whether shuttle changes or age has effect
Temperature – whether joint temperature on booster has effect

36. First model—date as single independent variable
Any thermal distress on launch
Independent variable
Date (days since 1/1/60)
SPSS procedure
Regression, Binary logistic

37. Predicted probability of thermal distress using date

38. Exponential of B as change in odds

39. What does “odds” mean?
Odds is the ratio of probability of success to probability of failure
Like odds on horse races
Even odds, odds = 1, implies probability equals 0.5
Odds = 2 means 2 to 1 in favor of success, implies probability of 0.667
Odds = 0.5 means 1 to 2 in favor (or 2 to 1 against) success, implies probability of 0.333

40. Multiple logistic regression
Logistic regression can be extended to use multiple independent variables exactly like linear regression

41. Adding joint temperature to the logistic regression model
Dependent variable
Any thermal distress on launch
Independent variables
Date (days since 1/1/60)
Joint temperature, degrees F

42. Probabilistic Discriminative Models
Find posterior class probabilities directly
Use functional form of generalized linear model and determine its parameters directly by using maximum likelihood principle
Iterative Reweighted Least Squares (IRLS)
Maximize a likelihood fn defined through the conditional distribution p(Ck|x), which represents a form of discriminative training
Advantage of discriminative approach – fewer no. of adaptive parameters to be determined (linear in M)
Improved predictive performance particularly when the class-conditional density assumptions give a poor approximation of the true distributions

43. Probabilistic Discriminative Models
Classification methods work directly with the original input vector
All such algorithms are still applicable if we first make a fixed NL transformation of the inputs using a vector of basis fns ϕ(x)
Decision boundaries linear in the feature space ϕ
In linear models of regression, one of the basis fn is typically set to a constant say , so that the corresponding parameter plays the role of bias.
A fixed basis fn transformation ϕ(x) will be used in

44. Probabilistic Discriminative Models
Original Input Space (x1,x2)
Feature Space (φ1,φ2)
Although we use linear classification models
Linear-separability in feature space does not imply
linear-separability in input space