4A studyDoes watching screened violence promote violent behaviour in children?In a study of the effects of media violence, some children were measured on their Actual violence and on their Exposure to screened violence.Here is a scatterplot of Actual violence against Exposure.
5The scatterplot Each point in the plot represents one child. The coordinates of the point are the child’s scores on Exposure to and Actual violence.A strong statistical ASSOCIATION between Exposure to and Actual violence is evident from the shape of the cloud of points.
6A linear associationAn elliptical scatterplot such as this indicates a LINEAR association between the variables.
7The Pearson correlation Where the association is linear, its strength is measured by the Pearson correlation, the formula for which is:
10RegressionRegression is a set of statistical techniques enabling the researcher to exploit an association among variables to PREDICT the values of one variable from those of others.From regression, you can also ascertain the extent to which the variance of a target variable can be EXPLAINED or accounted for in terms of the other variables.
11Some key termsThe variable we are trying to predict or account for is known variously as the DEPENDENT VARIABLE (DV), the CRITERION, or the TARGET VARIABLE.The predictors are known as the INDEPENDENT VARIABLES (IVs) or REGRESSORS.In our current example, the DV is Actual violence and the IV is Exposure to screen violence.
12Simple regression and multiple regression In SIMPLE REGRESSION, there is just ONE IV or regressor.In MULTIPLE REGRESSION, there are TWO OR MORE IVs or regressors.
13The regression lineIn simple regression, a line called the REGRESSION LINE is drawn through the points.The regression line is the line that fits the points most closely, according to the LEAST SQUARES CRITERION.The traditional approach to regression is therefore referred to as ORDINARY LEAST SQUARES (OLS) REGRESSION.
19Interpretation of the slope or regression coefficient The slope or REGRESSION COEFFICIENT is the average number of units of change in the DV that result from a change of one unit on the IV.In our example, slope = .74 .So, an increase of one unit in Exposure produces, on average, an increase of .74 (rather less than one unit) in Actual violence.
21Joe’s residual score Joe scored 8 on Exposure and 9 on Actual. Joe’s predicted score from regression Y/ is 8.Joe’s residual score is : (9 – 8) = 1.
22Least squares criterion for goodness-of-fit The values of the slope and intercept of the regression line are such that the sum of the squares of the residuals (SSerror) is a minimum.
23A unique solutionThe values of b and c needed to achieve the least squares criterion are given by the formulae below.Clearly, the regression coefficient b is closely related to the Pearson correlation.
24Relation between the regression coefficient and the correlation coefficient The value of the regression coefficient is directly proportional to the value of the correlation coefficient.
25Regression and correlation Regression and correlation are two sides of the same associative coin.The higher the correlation, the narrower the elliptical cloud of points in the scatterplot.For fixed values of the variances of X and Y, the higher the correlation, the greater will be the value of the regression coefficient.
27A negative correlation So far, I have considered only positive correlations. Here’s a negative one.Does the number of complaints made against GPs very inversely with the average length of their appointments?The following scatterplot supports this hypothesis.
29The signs of b and rThe regression coefficient and the correlation always have the same sign.For the violence data, both are positive.For the data on GPs’ appointments, both are negative.Note that a negative correlation of – 0.89 indicates an association as strong as one for which the correlation is +.89.
30Complete independence I take two random samples, each of size 10,000 from a normal population with mean 100 and SD 25. (The Syntax for doing this is shown in the appendix.)Since there should be no association between the two samples, the correlation between them should be zero.The scatterplot will be CIRCULAR.The regression line will be HORIZONTAL, that is, with zero slope.
33Intercept-only regression The regression line is horizontal and passes through the value 100 on the y-axis. This is the MEAN VALUE OF THE DISTRIBUTION OF THE DEPENDENT VARIABLE.Here the intercept of the regression line is equal to the mean value of Y and its slope is zero.When X and Y are independent, you can only predict the mean value of Y, WHATEVER THE VALUE OF X.This is known as INTERCEPT- ONLY REGRESSION.
34Model-buildingWhen testing the goodness-of-fit of regression models to the data, a useful baseline is provided by the INDEPENDENCE MODEL, which makes intercept-only predictions of the dependent variable by predicting the mean value of the DV whatever the value of the IV.In several computing procedures, this is labelled as STEP 0 in the analysis.A good regression model should be a big improvement upon the independence model.
35The coefficient of determination (r2) The square of the Pearson correlation is known as the COEFFICIENT OF DETERMINATION.It is so-called because r2 is the proportion of the variance of Y that is accounted for by regression upon X.
38Multiple regression model We could try to expand our regression model to include additional variables, such as level of parental violence and their levels of formal education.We should then have to determine the relative importance of the various IVs and whether we needed to include all of them in the regression model.These are problems in MULTIPLE REGRESSION.
39Equations for simple and multiple regression In the multiple regression equation, b0 is the CONSTANT and b1, b2, …,bp are the PARTIAL REGRESSION COEFFICIENTS.
40Partial regression coefficients In multiple regression, a PARTIAL REGRESSION COEFFICIENT is the estimated average change in the DV resulting from an increase of one unit in one particular IV with ALL THE OTHER IVs HELD CONSTANT.
41The multiple correlation coefficient R The MULTIPLE CORRELATION COEFFICIENT (R) is the correlation between the target variable Y and the corresponding predictions Y/ of Y from regression upon the IVs.
42NotationWhen it is necessary to specify which variables are involved in a multiple regression, a subscript notation is used.The multiple correlation between Y and X1, X2, …, Xp is
43Properties of RR can never take a negative value, because the sign of the slope of the regression line is always the same as that of the correlation.Recall that the Pearson correlation varies within the range from –1 to +1, inclusive.The multiple correlation R, however, can only take values between zero and +1, inclusive.
44The case of one IVThe multiple correlation coefficient is defined even in simple regression, where there is only one IV.Here, remembering that R can never be negative, it takes the ABSOLUTE VALUE of the Pearson correlation (r ) between X and Y, even when r has a negative value.So in SPSS, R is included in the output for simple regression as well as in the output for multiple regression.
45The coefficient of multiple determination R2 In multiple regression, THE COEFFICIENT OF MULTIPLE DETERMINATION R2 is the proportion of variance of the dependent variable Y that is accounted for by regression upon the IVs.
46A spatial representation of the coefficient of multiple determination
47What if the DV is a set of categories? Simple and multiple OLS regression assume that all the variables are CONTINUOUS, that is, measures on an independent scale with units.But suppose we want to predict whether a person will suffer from a heart attack or contract a certain illness with known risk factors.Here, we are predicting not a VALUE, but CATEGORY MEMBERSHIP.
48Regression with a categorical DV The two most commonly used techniques are:Logistic regressionDiscriminant analysis
49Discriminant analysis If all (or most) IVs are continuous, you might consider using DISCRIMINANT ANALYSIS (DA).But the DA model makes assumptions about the distributions of the IVs (such as multivariate normality) which research data often fail to satisfy.Moreover, DA doesn’t like qualitative IVs, such as sex or nationality.For these reasons, logistic regression is increasingly preferred to DA when the DV is categorical.
50Categorical IVsUnlike DA, logistic regression is happy with qualitative IVs; in fact, logistic regression is happy even if ALL the IVs are qualitative.
51A research questionIt is suspected that smoking and drinking are risk factors in the incidence of a pre-morbid blood condition.Records of the incidence of the condition in 100 patients are available, together with estimates of the amounts that they smoke and drink.
56Forty-four of the hundred patients have the antibody
57The oddsIn an EXPERIMENT OF CHANCE (tossing a coin, rolling a die) the ODDS in favour of an event is the number of ways in which the event could occur, divided by the number of ways in which it could fail to occur.If a die is rolled, there is one way of getting a six and there are five ways of not getting a six.The odds in favour of a six are therefore 1/5.
58Odds in favour of having the antibody We know that out of 100 patients, 44 have the antibody. We select a person at random from this group.There are 44 ways of selecting a person with the antibody; and there are 56 ways of selecting someone without it.The odds in favour of the person having the antibody are 44/56 = .79.
59ProbabilityA probability is a measure of likelihood ranging from 0 (an impossibility) to 1 (a certainty).The CLASSICAL DEFINITION of probability, like that of the odds, also arises in the context of an experiment of chance.The probability p of an event is the number of ways it can happen divided by the TOTAL number of possible outcomes.When a die is rolled, there are six possible outcomes.There is one way of getting a six.The probability of a six when a die is rolled is therefore 1/6.
60Relationship between probability and the odds Probability and the odds are both measures of likelihood and have been defined in the same context – an experiment of chance.They are related according to the equation on the left.
61LogarithmsIn a logarithmic system, numbers are expressed as powers (logarithms) of a constant called the BASE of the system.In COMMON LOGS, the base is 10.In NATURAL LOGS, the base is the mathematical constant e, where e is approximately
62Definition of a logarithm The logarithm of a number is the POWER to which the base must be raised to give the number itself.So the log of 100 to the base ten is 2 because 102 = 100.The log of 1000 to the base ten is 3, because 103 = 1000.
64Logs and antilogsBefore the IT revolution, calculations involving large numbers were done by converting the numbers to logs, working with the logs (which are much smaller numbers), then reversing the log function with the ANTILOG FUNCTION to get back to the original number scale.
65Logs of sums and products The log of the PRODUCT of two numbers is the SUM of their logs.The log of the QUOTIENT of two numbers is the DIFFERENCE between their logs.
66Two advantages of logs You are working with SMALLER numbers. You are replacing the operations of MULTIPLICATION and DIVISION with the easier ones of ADDITION and SUBTRACTION.
67AntilogsThe ANTILOG function reverses the process of obtaining a logarithm.
72The antilog of a sumThe antilog of the SUM of two logs is the PRODUCT of the numbers themselves.
73An asymmetrical measure The ODDS suffers from ASYMMETRY OF RANGE.Extremely unlikely events have odds confined between 0 and 1; whereas very likely events can have huge odds running into millions.Two very likely events could be separated by millions in terms of odds; two very unlikely events will be separated by minute fractions.
74The log odds or logitThe LOG ODDS (LOGIT) is the natural logarithm (log to the base e) of the odds.
75Even Steven Suppose the odds were 50 to 50 (50/50 =1). The natural log of 1 is zero (e0 = 1).So for raw odds of 50 to 50, the logit (log odds) is zero.
76Range of the logitThe logit has a symmetrical range: a positive sign means the odds are in favour; a negative sign means the odds are against.Unlike the odds, which has a lower limit of zero, the logit has neither an upper nor a lower finite limit.
77ExampleIn our current example, the odds in favour of a case having the antibody are 44/56 = 11/14 = .79logit = ln(.79) = –.24The event is less likely than not, hence the negative sign.If the odds in favour had been 56/44, the logit would have been ln(56/44) = ln(1.27) = +.24.Notice the symmetry of the scale of magnitude around the neutral point at 0.
78Odds as antilogsA number such as the odds can be written as an ANTILOG, that is, the base e to the power of the natural log of the odds (the logit):
79Probability and the logit We can therefore express the probability in terms of the logit, rather than the odds.
80The logistic regression function We have arrived at the LOGISTIC REGRESSION FUNCTION:
81Assumptions of logistic regression Either you have the antibody or you don’t.As smoking and alcohol increase, however, the probability of having the antibody is assumed to increase CONTINUOUSLY as a function of the IVs.In logistic regression, we estimate the probability of having the antibody with the LOGISTIC REGRESSION FUNCTIONIf the estimated probability exceeds a cut-off (usually set at 0.5), the case is classified by the program as a Yes, rather than a No.
87Interpretation of a logistic regression coefficient The partial regression coefficient is the increase in the LOG ODDS or LOGIT arising from an increase of one unit in the independent variable.If one unit is added, the logit becomes:
88New oddsTo find the new odds you must MULTIPLY the old odds by exp(b).
89ExampleIn terms of the ODDS, an increase of one unit in the IV MULTIPLIES the original odds by the ANTILOG of b, that is, by eb, or exp(b).If b = 1.1, exp(1.1) = 3.0So an increase of one smoking unit results in the odds being MULTIPLIED by 3, that is, the antibody is THREE times as likely to be present in the blood of those who smoke a unit more.
90The regression problem In the logit equation, we must find values of the constant and partial regression coefficients such that correct assignment to categories by the logistic regression function is maximised.
91No mathematical solution In logistic regression, there is no equivalent of the formulae for the intercept and coefficients in OLS regression.A ‘brute force’ computing algorithm is used whereby, starting at arbitrary values of the coefficients, the values are progressively adjusted to try to arrive at a set which maximises the likelihood of obtaining the observed frequencies.
92Iteration and ‘convergence’ In a process known as ITERATION, estimates of the parameters are calculated again and again in the hope that they will ‘converge’ to stable values.IT DOESN’T ALWAYS HAPPEN!We must therefore check that this ‘convergence’ really has been achieved by examining the ITERATION HISTORY in the SPSS output.
93Potential difficulties The algorithm will not run successfully if the IVs are too highly correlated. This is the familiar MULTICOLLINEARITY PROBLEM sometimes encountered in OLS regression.
94CentringAs with OLS multiple regression, it is a good idea to CENTRE variables, by subtracting the mean from each score, so that the mean of the transformed scores is zero.CENTRING leaves the correlations among the variables unchanged.But centring makes the algorithm less likely to crash when there are substantial correlations among the variables.
96CovariatesIn SPSS logistic regression dialogs, IVs that are scale or continuous variables are known as COVARIATES.
97Always ask for the ITERATION HISTORY, so that you can check whether the algorithm was able to arrive at a stable estimate.
98Dire warning!Should the iteration history show failure to converge, the results of the analysis can be ridiculous!The effects of failure to converge are not limited to the IV concerned: they can mess up the whole analysis!
100Fitting a modelThe goodness-of-fit of a model is measured by a log likelihood chi-square statistic.The SMALLER the value of chi-square, the BETTER the fit.The LARGER the p -value the better.
101Step 0 in logistic regression We know that 44/100 people have the condition.Armed only with this fact, and with no knowledge of any associations there might be among the variables, we shall maximise our hit rate if we predict ABSENCE of the condition for ANY person selected at random.This is the equivalent, in logistic regression, of INTERCEPT-ONLY (no-regression) prediction in OLS regression: there you just guess MY, whatever the value of X; here you always guess that the condition is absent.
102Here is the logistic regression output for Step 0
105Iteration history after applying regression model
106Classification table at Step 1 (after the regression model has been applied)
107The Nagelkerke R2 statistic The NAGELKERKE statistic is the counterpart of the coefficient of determination R2 in OLS multiple regression.It is a measure of the PROPORTION OF THE TOTAL VARIANCE in incidence of the antibody accounted for by regression.
111The Wald statisticThe WALD STATISTIC tests a regression coefficient for significance.The null hypothesis is that, in the population, the coefficient is zero.The Wald statistic is distributed approximately as chi-square on one degree of freedom.
112Some regression statistics The Wald statistic confirms that Smoking has an effect (p-value is very small) but Alcohol does not (the p-value is large).
117ConclusionThe incidence of the blood condition is indeed predictable from regression and raises the hit rate from 54% to 85%.Smoking contributes significantly to the model.Alcohol does not contribute significantly to the model.
118The next stepThis session has been merely an introduction to the technique of logistic regression.The next step is to do some further reading.
119Getting startedThere’s an elementary section on logistic regression inKinnear, P., & Gray, C. (2011). IBM SPSS Statistics 18 made simple. Hove: Psychology Press. Chapter 14.This is mainly a practical, get-started guide; but there is an outline of the rationale of the technique as well.
120The next stepDugard, P., Todman, J., & Staines, H. (2010) Approaching multivariate analysis: a practical introduction. (2nd ed.) London & New York: Routledge.
121Sage paperbacksMenard, S. (2002). Applied logistic regression analysis (2nd ed.). London: Sage.Jaccard, J. (2001). Interaction effects in logistic regression. London: Sage.
122Tabachnik, B. G. , & Fidell, L. S. (2007) Tabachnik, B. G., & Fidell, L. S. (2007). Using multivariate statistics (5th ed.). Boston: Allyn & Bacon. Chapter 10.Field, A. (2009). Discovering statistics using SPSS for Windows: Advanced techniques for the beginner (3rd ed.). London: Sage. Chapter 6.
123Appendix Using syntax to draw random samples from specified populations
124Drawing two samples from a normal distribution with mean 100 and SD 15