1. Logistic regression

2. Programme 2:05 pm Talk 3:15 pm Coffee break 3:45 pm A short exercise
4:30 pm Finish

3. Revision Correlation and regression

4. A study
Does watching screened violence promote violent behaviour in children?
In a study of the effects of media violence, some children were measured on their Actual violence and on their Exposure to screened violence.
Here is a scatterplot of Actual violence against Exposure.

5. The scatterplot Each point in the plot represents one child.
The coordinates of the point are the child’s scores on Exposure to and Actual violence.
A strong statistical ASSOCIATION between Exposure to and Actual violence is evident from the shape of the cloud of points.

6. A linear association
An elliptical scatterplot such as this indicates a LINEAR association between the variables.

7. The Pearson correlation
Where the association is linear, its strength is measured by the Pearson correlation, the formula for which is:

8. Sums of squares and products

9. Alternative formula for the Pearson correlation

10. Regression
Regression is a set of statistical techniques enabling the researcher to exploit an association among variables to PREDICT the values of one variable from those of others.
From regression, you can also ascertain the extent to which the variance of a target variable can be EXPLAINED or accounted for in terms of the other variables.

11. Some key terms
The variable we are trying to predict or account for is known variously as the DEPENDENT VARIABLE (DV), the CRITERION, or the TARGET VARIABLE.
The predictors are known as the INDEPENDENT VARIABLES (IVs) or REGRESSORS.
In our current example, the DV is Actual violence and the IV is Exposure to screen violence.

12. Simple regression and multiple regression
In SIMPLE REGRESSION, there is just ONE IV or regressor.
In MULTIPLE REGRESSION, there are TWO OR MORE IVs or regressors.

13. The regression line
In simple regression, a line called the REGRESSION LINE is drawn through the points.
The regression line is the line that fits the points most closely, according to the LEAST SQUARES CRITERION.
The traditional approach to regression is therefore referred to as ORDINARY LEAST SQUARES (OLS) REGRESSION.

14. A linear equation

15. The equation of a straight line

16. The regression line

17. The regression equation

18. On the graph …

19. Interpretation of the slope or regression coefficient
The slope or REGRESSION COEFFICIENT is the average number of units of change in the DV that result from a change of one unit on the IV.
In our example, slope = .74 .
So, an increase of one unit in Exposure produces, on average, an increase of .74 (rather less than one unit) in Actual violence.

20. The residual (e)

21. Joe’s residual score Joe scored 8 on Exposure and 9 on Actual.
Joe’s predicted score from regression Y/ is 8.
Joe’s residual score is : (9 – 8) = 1.

22. Least squares criterion for goodness-of-fit
The values of the slope and intercept of the regression line are such that the sum of the squares of the residuals (SSerror) is a minimum.

23. A unique solution
The values of b and c needed to achieve the least squares criterion are given by the formulae below.
Clearly, the regression coefficient b is closely related to the Pearson correlation.

24. Relation between the regression coefficient and the correlation coefficient
The value of the regression coefficient is directly proportional to the value of the correlation coefficient.

25. Regression and correlation
Regression and correlation are two sides of the same associative coin.
The higher the correlation, the narrower the elliptical cloud of points in the scatterplot.
For fixed values of the variances of X and Y, the higher the correlation, the greater will be the value of the regression coefficient.

26. The violence data

27. A negative correlation
So far, I have considered only positive correlations. Here’s a negative one.
Does the number of complaints made against GPs very inversely with the average length of their appointments?
The following scatterplot supports this hypothesis.

28. A strong negative correlation

29. The signs of b and r
The regression coefficient and the correlation always have the same sign.
For the violence data, both are positive.
For the data on GPs’ appointments, both are negative.
Note that a negative correlation of – 0.89 indicates an association as strong as one for which the correlation is +.89.

30. Complete independence
I take two random samples, each of size 10,000 from a normal population with mean 100 and SD 25. (The Syntax for doing this is shown in the appendix.)
Since there should be no association between the two samples, the correlation between them should be zero.
The scatterplot will be CIRCULAR.
The regression line will be HORIZONTAL, that is, with zero slope.

31. No association

32. The case of independence

33. Intercept-only regression
The regression line is horizontal and passes through the value 100 on the y-axis. This is the MEAN VALUE OF THE DISTRIBUTION OF THE DEPENDENT VARIABLE.
Here the intercept of the regression line is equal to the mean value of Y and its slope is zero.
When X and Y are independent, you can only predict the mean value of Y, WHATEVER THE VALUE OF X.
This is known as INTERCEPT- ONLY REGRESSION.

34. Model-building
When testing the goodness-of-fit of regression models to the data, a useful baseline is provided by the INDEPENDENCE MODEL, which makes intercept-only predictions of the dependent variable by predicting the mean value of the DV whatever the value of the IV.
In several computing procedures, this is labelled as STEP 0 in the analysis.
A good regression model should be a big improvement upon the independence model.

35. The coefficient of determination (r2)
The square of the Pearson correlation is known as the COEFFICIENT OF DETERMINATION.
It is so-called because r2 is the proportion of the variance of Y that is accounted for by regression upon X.

36. Coefficient of determination

37. Cohen’s (1988) guidelines

38. Multiple regression model
We could try to expand our regression model to include additional variables, such as level of parental violence and their levels of formal education.
We should then have to determine the relative importance of the various IVs and whether we needed to include all of them in the regression model.
These are problems in MULTIPLE REGRESSION.

39. Equations for simple and multiple regression
In the multiple regression equation, b0 is the CONSTANT and b1, b2, …,bp are the PARTIAL REGRESSION COEFFICIENTS.

40. Partial regression coefficients
In multiple regression, a PARTIAL REGRESSION COEFFICIENT is the estimated average change in the DV resulting from an increase of one unit in one particular IV with ALL THE OTHER IVs HELD CONSTANT.

41. The multiple correlation coefficient R
The MULTIPLE CORRELATION COEFFICIENT (R) is the correlation between the target variable Y and the corresponding predictions Y/ of Y from regression upon the IVs.

42. Notation
When it is necessary to specify which variables are involved in a multiple regression, a subscript notation is used.
The multiple correlation between Y and X1, X2, …, Xp is

43. Properties of R
R can never take a negative value, because the sign of the slope of the regression line is always the same as that of the correlation.
Recall that the Pearson correlation varies within the range from –1 to +1, inclusive.
The multiple correlation R, however, can only take values between zero and +1, inclusive.

44. The case of one IV
The multiple correlation coefficient is defined even in simple regression, where there is only one IV.
Here, remembering that R can never be negative, it takes the ABSOLUTE VALUE of the Pearson correlation (r ) between X and Y, even when r has a negative value.
So in SPSS, R is included in the output for simple regression as well as in the output for multiple regression.

45. The coefficient of multiple determination R2
In multiple regression, THE COEFFICIENT OF MULTIPLE DETERMINATION R2 is the proportion of variance of the dependent variable Y that is accounted for by regression upon the IVs.

46. A spatial representation of the coefficient of multiple determination

47. What if the DV is a set of categories?
Simple and multiple OLS regression assume that all the variables are CONTINUOUS, that is, measures on an independent scale with units.
But suppose we want to predict whether a person will suffer from a heart attack or contract a certain illness with known risk factors.
Here, we are predicting not a VALUE, but CATEGORY MEMBERSHIP.

48. Regression with a categorical DV
The two most commonly used techniques are:
Logistic regression
Discriminant analysis

49. Discriminant analysis
If all (or most) IVs are continuous, you might consider using DISCRIMINANT ANALYSIS (DA).
But the DA model makes assumptions about the distributions of the IVs (such as multivariate normality) which research data often fail to satisfy.
Moreover, DA doesn’t like qualitative IVs, such as sex or nationality.
For these reasons, logistic regression is increasingly preferred to DA when the DV is categorical.

50. Categorical IVs
Unlike DA, logistic regression is happy with qualitative IVs; in fact, logistic regression is happy even if ALL the IVs are qualitative.

51. A research question
It is suspected that smoking and drinking are risk factors in the incidence of a pre-morbid blood condition.
Records of the incidence of the condition in 100 patients are available, together with estimates of the amounts that they smoke and drink.

52. The data

53. How many of the patients have the antibody?

54. Use Frequencies

55. Frequencies dialog

56. Forty-four of the hundred patients have the antibody

57. The odds
In an EXPERIMENT OF CHANCE (tossing a coin, rolling a die) the ODDS in favour of an event is the number of ways in which the event could occur, divided by the number of ways in which it could fail to occur.
If a die is rolled, there is one way of getting a six and there are five ways of not getting a six.
The odds in favour of a six are therefore 1/5.

58. Odds in favour of having the antibody
We know that out of 100 patients, 44 have the antibody. We select a person at random from this group.
There are 44 ways of selecting a person with the antibody; and there are 56 ways of selecting someone without it.
The odds in favour of the person having the antibody are 44/56 = .79.

59. Probability
A probability is a measure of likelihood ranging from 0 (an impossibility) to 1 (a certainty).
The CLASSICAL DEFINITION of probability, like that of the odds, also arises in the context of an experiment of chance.
The probability p of an event is the number of ways it can happen divided by the TOTAL number of possible outcomes.
When a die is rolled, there are six possible outcomes.
There is one way of getting a six.
The probability of a six when a die is rolled is therefore 1/6.

60. Relationship between probability and the odds
Probability and the odds are both measures of likelihood and have been defined in the same context – an experiment of chance.
They are related according to the equation on the left.

61. Logarithms
In a logarithmic system, numbers are expressed as powers (logarithms) of a constant called the BASE of the system.
In COMMON LOGS, the base is 10.
In NATURAL LOGS, the base is the mathematical constant e, where e is approximately

62. Definition of a logarithm
The logarithm of a number is the POWER to which the base must be raised to give the number itself.
So the log of 100 to the base ten is 2 because 102 = 100.
The log of 1000 to the base ten is 3, because 103 = 1000.

63. Notation

64. Logs and antilogs
Before the IT revolution, calculations involving large numbers were done by converting the numbers to logs, working with the logs (which are much smaller numbers), then reversing the log function with the ANTILOG FUNCTION to get back to the original number scale.

65. Logs of sums and products
The log of the PRODUCT of two numbers is the SUM of their logs.
The log of the QUOTIENT of two numbers is the DIFFERENCE between their logs.

66. Two advantages of logs You are working with SMALLER numbers.
You are replacing the operations of MULTIPLICATION and DIVISION with the easier ones of ADDITION and SUBTRACTION.

67. Antilogs
The ANTILOG function reverses the process of obtaining a logarithm.

68. The common antilog

69. The natural antilog

70. Antilogs (base 10)

71. Antilogs (base e)

72. The antilog of a sum
The antilog of the SUM of two logs is the PRODUCT of the numbers themselves.

73. An asymmetrical measure
The ODDS suffers from ASYMMETRY OF RANGE.
Extremely unlikely events have odds confined between 0 and 1; whereas very likely events can have huge odds running into millions.
Two very likely events could be separated by millions in terms of odds; two very unlikely events will be separated by minute fractions.

74. The log odds or logit
The LOG ODDS (LOGIT) is the natural logarithm (log to the base e) of the odds.

75. Even Steven Suppose the odds were 50 to 50 (50/50 =1).
The natural log of 1 is zero (e0 = 1).
So for raw odds of 50 to 50, the logit (log odds) is zero.

76. Range of the logit
The logit has a symmetrical range: a positive sign means the odds are in favour; a negative sign means the odds are against.
Unlike the odds, which has a lower limit of zero, the logit has neither an upper nor a lower finite limit.

77. Example
In our current example, the odds in favour of a case having the antibody are 44/56 = 11/14 = .79
logit = ln(.79) = –.24
The event is less likely than not, hence the negative sign.
If the odds in favour had been 56/44, the logit would have been ln(56/44) = ln(1.27) = +.24.
Notice the symmetry of the scale of magnitude around the neutral point at 0.

78. Odds as antilogs
A number such as the odds can be written as an ANTILOG, that is, the base e to the power of the natural log of the odds (the logit):

79. Probability and the logit
We can therefore express the probability in terms of the logit, rather than the odds.

80. The logistic regression function
We have arrived at the LOGISTIC REGRESSION FUNCTION:

81. Assumptions of logistic regression
Either you have the antibody or you don’t.
As smoking and alcohol increase, however, the probability of having the antibody is assumed to increase CONTINUOUSLY as a function of the IVs.
In logistic regression, we estimate the probability of having the antibody with the LOGISTIC REGRESSION FUNCTION
If the estimated probability exceeds a cut-off (usually set at 0.5), the case is classified by the program as a Yes, rather than a No.

82. A logistic regression function

83. Logistic regression and logit functions
We have seen that the logistic regression function is non-linear.
The logit function, however, is assumed to be linear.

84. The logit equation
The logit function looks like an OLS linear regression equation, with a constant and partial regression coefficients.

85. Typical graph of the logit function

86. The decision rule

87. Interpretation of a logistic regression coefficient
The partial regression coefficient is the increase in the LOG ODDS or LOGIT arising from an increase of one unit in the independent variable.
If one unit is added, the logit becomes:

88. New odds
To find the new odds you must MULTIPLY the old odds by exp(b).

89. Example
In terms of the ODDS, an increase of one unit in the IV MULTIPLIES the original odds by the ANTILOG of b, that is, by eb, or exp(b).
If b = 1.1, exp(1.1) = 3.0
So an increase of one smoking unit results in the odds being MULTIPLIED by 3, that is, the antibody is THREE times as likely to be present in the blood of those who smoke a unit more.

90. The regression problem
In the logit equation, we must find values of the constant and partial regression coefficients such that correct assignment to categories by the logistic regression function is maximised.

91. No mathematical solution
In logistic regression, there is no equivalent of the formulae for the intercept and coefficients in OLS regression.
A ‘brute force’ computing algorithm is used whereby, starting at arbitrary values of the coefficients, the values are progressively adjusted to try to arrive at a set which maximises the likelihood of obtaining the observed frequencies.

92. Iteration and ‘convergence’
In a process known as ITERATION, estimates of the parameters are calculated again and again in the hope that they will ‘converge’ to stable values.
IT DOESN’T ALWAYS HAPPEN!
We must therefore check that this ‘convergence’ really has been achieved by examining the ITERATION HISTORY in the SPSS output.

93. Potential difficulties
The algorithm will not run successfully if the IVs are too highly correlated. This is the familiar MULTICOLLINEARITY PROBLEM sometimes encountered in OLS regression.

94. Centring
As with OLS multiple regression, it is a good idea to CENTRE variables, by subtracting the mean from each score, so that the mean of the transformed scores is zero.
CENTRING leaves the correlations among the variables unchanged.
But centring makes the algorithm less likely to crash when there are substantial correlations among the variables.

95. Finding binary logistic regression

96. Covariates
In SPSS logistic regression dialogs, IVs that are scale or continuous variables are known as COVARIATES.

97. Always ask for the ITERATION HISTORY, so that you can check whether the algorithm was able to arrive at a stable estimate.

98. Dire warning!
Should the iteration history show failure to converge, the results of the analysis can be ridiculous!
The effects of failure to converge are not limited to the IV concerned: they can mess up the whole analysis!

99. The logistic regression dialog

100. Fitting a model
The goodness-of-fit of a model is measured by a log likelihood chi-square statistic.
The SMALLER the value of chi-square, the BETTER the fit.
The LARGER the p -value the better.

101. Step 0 in logistic regression
We know that 44/100 people have the condition.
Armed only with this fact, and with no knowledge of any associations there might be among the variables, we shall maximise our hit rate if we predict ABSENCE of the condition for ANY person selected at random.
This is the equivalent, in logistic regression, of INTERCEPT-ONLY (no-regression) prediction in OLS regression: there you just guess MY, whatever the value of X; here you always guess that the condition is absent.

102. Here is the logistic regression output for Step 0

103. Iteration history at Step 0

104. Classification table at Step 0

105. Iteration history after applying regression model

106. Classification table at Step 1 (after the regression model has been applied)

107. The Nagelkerke R2 statistic
The NAGELKERKE statistic is the counterpart of the coefficient of determination R2 in OLS multiple regression.
It is a measure of the PROPORTION OF THE TOTAL VARIANCE in incidence of the antibody accounted for by regression.

108. The Nagelkerke R2 statistic

109. Hosmer and Lemeshow contingency table

110. Goodness-of-fit test

111. The Wald statistic
The WALD STATISTIC tests a regression coefficient for significance.
The null hypothesis is that, in the population, the coefficient is zero.
The Wald statistic is distributed approximately as chi-square on one degree of freedom.

112. Some regression statistics
The Wald statistic confirms that Smoking has an effect (p-value is very small) but Alcohol does not (the p-value is large).

113. The regression coefficient

114. The logit equation

115. The logistic regression equation

116. Graph of accuracy of prediction

117. Conclusion
The incidence of the blood condition is indeed predictable from regression and raises the hit rate from 54% to 85%.
Smoking contributes significantly to the model.
Alcohol does not contribute significantly to the model.

118. The next step
This session has been merely an introduction to the technique of logistic regression.
The next step is to do some further reading.

119. Getting started
There’s an elementary section on logistic regression in
Kinnear, P., & Gray, C. (2011). IBM SPSS Statistics 18 made simple. Hove: Psychology Press. Chapter 14.
This is mainly a practical, get-started guide; but there is an outline of the rationale of the technique as well.

120. The next step
Dugard, P., Todman, J., & Staines, H. (2010) Approaching multivariate analysis: a practical introduction. (2nd ed.) London & New York: Routledge.

121. Sage paperbacks
Menard, S. (2002). Applied logistic regression analysis (2nd ed.). London: Sage.
Jaccard, J. (2001). Interaction effects in logistic regression. London: Sage.

122. Tabachnik, B. G. , & Fidell, L. S. (2007)
Tabachnik, B. G., & Fidell, L. S. (2007). Using multivariate statistics (5th ed.). Boston: Allyn & Bacon. Chapter 10.
Field, A. (2009). Discovering statistics using SPSS for Windows: Advanced techniques for the beginner (3rd ed.). London: Sage. Chapter 6.

123. Appendix Using syntax to draw random samples from specified populations

124. Drawing two samples from a normal distribution with mean 100 and SD 15