1. 1 Lecture 8 Regression: Relationships between continuous variables Slides available from Statistics & SPSS page of www.gpryce.com www.gpryce.com Social Science Statistics Module I Gwilym Pryce

2. Notices: Register Revision lecture next week: –Worked examples on: Confidence Intervals? Hypothesis Tests? Regression? –Email me any particular issues Learning & Support strategy:

3. 1.Independent learning: –this is a PG course and a degree of independent learning is assumed. –do the reading, attend the labs, review the lectures, make use of the computer labs and online help in your own time. 2.Lab Overview & Feedback: –Please feedback to the tutors & Class Reps how you think that is going, how it could be improved. –Tutors and Class Reps will then report back to me how things are going each week. 3.Talk to tutors if you are struggling –Let the tutors know if you are struggling (assuming you have done the reading, attended labs etc.) –Tutors cannot guarantee extra support, but it might be possible to arrange extra tutorials etc. 4.Support from Maths Advisor Shazia Ahmed, University’s Maths Adviser: –If you have gone through steps 1 to 3, Shazia has agreed to run one-on-one sessions with students that are struggling with particular mathematical or statistical concepts (though she has made it clear that she cannot advise on SPSS problems, nor will she do the assignment for you). –Students who have particular problems in this regard can contact her directly: Shazia Ahmed, Maths Adviser, Student Learning Service McMillan Reading Room, Tel: 330 5631 Fax: 330 8063 5.Departmental Support: –Struggling students should enquire whether their own dept has support to offer. –All the grad school courses are only intended to constitute a generic training component; –Individual depts & supervisors should supplement with additional training and support as necessary. 6.Tutor of Last Resort: –Students who have gone through steps 1 to 5 above, and who still feel they are not receiving enough support, can email me directly –I will try to arrange individual or small group meetings for people who have tried all other avenues. You will need to demonstrate that you have gone through steps 1 to 5.

4. Plan: 1. Linear & Non-linear Relationships 2. Fitting a line using OLS 3. Inference in Regression 4. Omitted Variables & R 2 5. Categorical Explanatory Variables 6. Summary

5. 1. Linear & Non-linear relationships between variables Often of greatest interest in social science is investigation into relationships between variables: –is social class related to political perspective? –is income related to education? –is worker alienation related to job monotony? We are also interested in the direction of causation, but this is more difficult to prove empirically: –our empirical models are usually structured assuming a particular theory of causation

6. Relationships between scale variables The most straight forward way to investigate evidence for relationship is to look at scatter plots: –traditional to: put the dependent variable (I.e. the “effect”) on the vertical axis –or “y axis” put the explanatory variable (I.e. the “cause”) on the horizontal axis –or “x axis”

7. Scatter plot of IQ and Income:

8. We would like to find the line of best fit: Predicted values (i.e. values of y lying on the line of best fit) are given by:

9. What does the output mean?

10. Sometimes the relationship appears non-linear:

11. … straight line of best fit is not always very satisfactory:

12. Could try a quadratic line of best fit:

13. We can simulate a non-linear relationship by first transforming one of the variables:

14. e.g. squaring IQ and taking the natural log of IQ:

15. … or a cubic line of best fit: ( over-fitted?)

16. Or could try two linear lines: “structural break”

17. 2. Fitting a line using OLS The most popular algorithm for drawing the line of best fit is one that minimises the sum of squared deviations from the line to each observation: Where: y i = observed value of y = predicted value of y i = the value on the line of best fit corresponding to x i

18. y = school performance; x = ave. HH income of pupils (£000s) 1.Write this model output as an equation. 2.When x i = 41 what is the value of y i ? 3.When x i = 41 what is the value of y_hat? 4.What is the difference between y i and y_hat when x i = 41, and what does this difference mean? 5.Where does the line of best fit cut the vertical axis? 6.What is the value of school performance when average HH income of pupils is zero? 7.How sensitive is school performance to the economic status of its intake? 8.How is this sensitivity calculated? Example: School Performance in 8 Schools etc

19. 1.y_hat = 6 + 2*x i –y i = 6 + 2*x i + e i 2.From the table of observations we can see that, when x i = 41, y i = 91.7. NB if there was another school with xi = 41, the observed value of y might not be the same due to random variation. 3.When x i = 41 what is the value of y_hat? y_hat = 6 + 2*41 = 88 4.The difference between y i and y_hat when x i = 41, is 91.7 – 88.0 = 3.7. This difference is the “error” or “residual”. –i.e. our model predicts that school performance will equal 88 when x = 41, but for this particular school, the actual performance is 91.7, so the model underpredicts performance by 3.7. 5.The line of best fit (our model) cuts the vertical axis where x = 0. y_hat = 6 + 2*x i = 6 + 2*0 = 6 6.The value of school performance = 6 when average HH income of pupils, x, is zero. 7.The regression slope, also called b, also called the slop coefficient is a measure of how sensitive the dependent variable is to change in the explanatory variables. SPSS has estimated that the slope in this case = 2. –i.e. for every unit increase in the explanatory variable (average income of parents measured in £000s) school performance rises by two units. –i.e. for every extra £1,000 average income, school performance goes up by one unit. 8.How is this sensitivity calculated? Good question! It is the slope of the line of best fit, calculated using the OLS formula which minimises the sum squared residuals…

20. Regression estimates of a, b using Ordinary Least Squares (OLS): Solving the min[error sum of squares] problem yields estimates of the slope b and y-intercept a of the straight line: = 2 = 6 y_hat = 6 + 2*x i

21. A Second random sample of 8 schools: = 2.1= 7.6 Now consider what would happen if we collected another sample and calculated the line of best fit for this new sample…

22. A Third Random Sample of 8 Schools: = 1.9= 15.2

23. A Fourth Random Sample of 8 Schools: = 2.0= 14.5

24. A Fifth Random Sample of 8 Schools: = 1.9= 14.0

25. Further random samples… Sample 6 Sample 7 Sample 8 Sample 9

26. Notice that, in the second, third etc samples we have found schools with exactly the same values of x as in the first sample. Despite this, we find random variation in the performance of the school for a given value of x. This means that the slope coefficient will also vary from sample to sample. Sample 1: b = 2.0 Sample 2: b = 2.1 Sample 3: b = 1.9 Sample 4: b = 2.0 Sample 5: b = 1.9 Sample 6: b = 1.7 Sample 7: b = 1.8 Sample 8: b = 2.5 Sample 9: b = 2.2 Average b from 9 samples = 2.0 Standard deviation of b from 9 samples = 0.2 i.e. average deviation of b from sample to sample = 0.2 = Standard Error of the slope

27. Q1/ What would the sampling distribution of b look like if the sample size was large? Q2/ What will the average of all sample slopes by and what symbol do we use to denote this value? Q3/ What section of that distribution are we usually most interested in?

28. If n is large… A1/ sample slope b is normally distributed if n is large. A2/ average of all sample slopes = population slope  A3/ we are usually most interested in the central 95% of the distribution of b –We want to be 95% sure that the population value of the slope lies between some lower bound and some upper bound.  || Average b b

29. Q/ Why is it useful that b is normally distributed?

30. A/ If b is normally distributed, it means that we can use the standard normal curve to help us work out the lower and upper bounds of the central 95% of the sampling distribution of b

31. Convert to z value z a b c where s b is the SE of b

32. Because the sampling distribution of the regression slope from large samples is normal (i.e. has a bell-shaped histogram), we can use the standard normal curve (z distribution) to work out confidence intervals and hypothesis tests on b. –i.e we can use the known probabilities for areas under the standard normal curve to work out: The lower and upper bounds for the central 95% of b The probability of observing a sample like our own with a value of b at least as far away from the H 0 assumed value of 

33. Small samples If the sample is small, b will have a t- distribution. Since the t-distribution is asymptotically normal (i.e. tends towards the z distribution as n increases) we tend to use the t- distribution whether the sample is large or small.

34. Convert to t value t a b c where s b is the SE of b

35. 3. Hypothesis tests on the slope coefficient Regressions are usually run on samples, but usually we want to say something about the population value of the relationship between x and y. Repeated samples would yield a range of values for estimates of b ~ N( , s b ) I.e. b is normally distributed with mean =  = population mean = value of b if regression run on population If there is no relationship in the population between x and y, then  = 0 H 0 :  = 0, H 1 :   0 is the hypothesis test which SPSS runs automatically on every regression you run and produces the output in two columns headed “t” and “Sig.” in the Coefficients table. –i.e. every SPSS output table of coefficients includes the results of a hypothesis test on whether there is any relationship at all between x and y.

36. Some examples…

37. Q1/ what is the estimate of slope in this sample and what does it tell us? Q2/ what is the standard error and what does it mean? Q3/ what is the value of the intercept term and what does it mean? Q4/ how would we test the hypothesis that  = 0, and what does this hypothesis mean? Returning to our IQ example:

38. A1/ the estimate of slope in this sample is 260. This tells us that for every unit increase in IQ, income typically rises by around £260. A2/ the standard error tells us how much the estimate of the slope typically varies from sample to sample. We do not know the SE of b for sure, but SPSS estimates it at £11 –i.e. the slope estimate is likely to vary by around £11 from sample to sample. A3/ the value of the intercept term is estimated to be - 8,237. The intercept term tells us the value of the dependenet variable when the explanatory variables are all zero. –i.e. where the line of best fit cuts the vertical axis –So we estimate that for someone with zero IQ, their income will typically be -£8,237.

39. A4/ we would test the hypothesis that  = 0 by calculating the probability of observing a sample with an estimated slope of £260 when the value of the population slope is zero. –We would calculate this probability (=sig. = probability of falsely rejecting H 0 :  = 0 ) by calculating the associated value on the t-distribution and use this to work out the areas in the tails. t c = (258.5 – 0)/11.01 = 23.5 where tc is the value of t you have calculated. You then want to work out what proportion of t lies above t c and below –t c. –We would then look up this value for t in the t tables for the degrees of freedom associated with out regression = sample size -(1 + the number of explanatory variables).

40. Hypothesis test on  : (1) H 0 :  = 0 (I.e. slope coefficient, if regression run on population, would = 0) H 1 :  (2)  = 0.05 or 0.01 etc. (3) Reject H 0 iff P <  (N.B. Rule of thumb if n fairly large: P < 0.05 if t c  2) (4) Calculate P and conclude.

41. Floor Area Example: You run a regression of house price on floor area which yields the following output. Use this output to answer the following questions: Q/ What is the “Constant”? What does it’s value mean here? Q/ What is the slope coefficient and what does it tell you here? Q/ What is the estimated value of an extra square metre? Q/ How would you test for the existence of a relationship between purchase price and floor area? Q/ How much is a 200m 2 house worth? Q/ How much is a 100m 2 house worth? Q/ On average, how much is the slope coefficient likely to vary from sample to sample? NB Write down your answers – you’ll need them later!

42. Floor area example: (1) H 0 : no relationship between house price and floor area. H 1 : there is a relationship (2), (3), (4): P = 1- CDF.T(24.469,554) = 0.000000 Reject H 0

43. 4. Omitted Variables, Goodness of Fit and R 2 Q/ is floor area the only factor? Q/ How much of the variation in Price does it explain?

44. R-square R-square tells you how much of the variation in y is explained by your model 0 < R 2 < 1 (NB: you want R 2 to be near 1). If your have more than one explanatory variable, use Adjusted R 2 which takes into account the distortion caused by adding extra variables.

45. House Price Example cont’d: Two explanatory variables Q/ How has the estimated value of an extra square metre changed? Q/ Do a hypothesis test for the existence of a relationship between price and number of bathrooms. Q/ How much will an extra bathroom typically add to the value of a house? Q/ What is the value of a 200m 2 house with one bathroom? Compare your estimate with that from the previous model. Q/ What is the value of a 100m 2 house with one bathroom? Compare your estimate with that from the previous model. Q/ What is the value of a 100m 2 house with two bathrooms? Compare your estimate with that from the previous model. Q/ On average, how much is the slope coefficient on floor area likely to vary from sample to sample? Now add number of bathrooms as an extra explanatory variable…

46. Scatter plot (with floor spikes)

47. 3D Surface Plots: Construction, Price & Unemployment during a boom Q = - 246 + 27 P - 0.2 P 2 - 73 U + 3 U 2 Non-linear effects can also be modelled when you have more than one explanatory variable

48. Construction Equation in a Slump Q = 315 + 4 P - 73 U + 5 U 2

49. 5. Categorical Explanatory Variables Sometimes certain observations display consistently higher y values for a particular subgroup in the sample. –i.e. for a particular category of observations. If you assume the slope will have the same value, and that only the intercept is shifting, you can model the effect of categorical variables by including “dummy” variables –A dummy variable is simply a binary variable –e.g. male = 1 or 0

50. To model the effect of a categorical explanatory variable in this way you need to: 1.Decide on a “baseline” category. This is usually an arbitrary decision, so just choose the largest or most familiar category. E.g. if the category is UK Region, choose London as the baseline 2.Create dummies (binary variables) for all remaining categories E.g. Compute yorksh_dum = 0. if (Region = “Yorkshire”) yorksh_dum = 1. Execute. 3.Include in your regression the dummies for all categories except your baseline category. E.g. suppose you only have two regions in your sample, London and Yorkshire, –you would do a regression of house price on floorarea and yorksh_dum

51. By including dummy variables you are saying that the difference between categories can be modelled as a parallel shift of the regression line above or below the baseline category –The value of the coefficient on the dummy variable tells you how much higher the value of the dependent variable would be observations in that category E.g. if the regression output were as follows: price = -2000 + 500*floorarea - 27500*yorksh_dum then the results tell us that a house of a given size is £27,500 cheaper in Yorkshire compared with London. –i.e. the coefficient tells you the size of the intercept shift associated with that category of observations…

52. Coefficient on Dummy Variable = size of Intercept Shift: Floorarea House price £27,500 London Yorkshire £27,500 Slope = £500; same for both areas

53. Summary 1. Linear & Non-linear Relationships 2. Fitting a line using OLS 3. Inference in Regression 4. Omitted Variables & R 2 5. Categorical Explanatory variables Revision lecture next week: –Worked examples on: Confidence Intervals? Hypothesis Tests? Regression?

54. Reading: Regression Analysis: –*Pryce chapter on relationships. –*Field, A. chapters on regression. –*Moore and McCabe Chapters on regression. –Kennedy, P. ‘A Guide to Econometrics’ –Bryman, Alan, and Cramer, Duncan (1999) “Quantitative Data Analysis with SPSS for Windows: A Guide for Social Scientists”, Chapters 9 and 10. –Achen, Christopher H. Interpreting and Using Regression (London: Sage, 1982).