Christopher Dougherty EC220 - Introduction to econometrics (chapter 4) Slideshow: nonlinear regression Original citation: Dougherty, C. (2012) EC220 - Introduction to econometrics (chapter 4). [Teaching Resource] © 2012 The Author This version available at: Available in LSE Learning Resources Online: May 2012 This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License. This license allows the user to remix, tweak, and build upon the work even for commercial purposes, as long as the user credits the author and licenses their new creations under the identical terms

1 NONLINEAR REGRESSION Suppose you believe that a variable Y depends on a variable X according to the relationship shown and you wish to obtain estimates of  1,  2, and  3 given data on Y and X.

NONLINEAR REGRESSION There is no way of transforming the relationship to obtain a linear relationship, and so it is not possible to apply the usual regression procedure. 2

NONLINEAR REGRESSION Nevertheless, one can still use the principle of minimizing the sum of the squares of the residuals to obtain estimates of the parameters. We will describe a simple nonlinear regression algorithm that uses the principle. It consists of a series of repeated steps. 3

NONLINEAR REGRESSION Nevertheless, one can still use the principle of minimizing the sum of the squares of the residuals to obtain estimates of the parameters. We will describe a simple nonlinear regression algorithm that uses the principle. It consists of a series of repeated steps. 4

Nonlinear regression algorithm 1.You start by guessing plausible values for the parameters. 2.You calculate the predicted values of Y from the data on X, using these values of the parameters. 3.You calculate the residual for each observation in the sample, and hence RSS, the sum of the squares of the residuals. 4.You then make small changes in one or more of your estimates of the parameters. 5.You calculate the new predicted values of Y, residuals, and RSS. 6.If RSS is smaller than before, your new estimates of the parameters are better than the old ones and you take them as your new starting point. 7.You repeat steps 4, 5 and 6 again and again until you are unable to make any changes in the estimates of the parameters that would reduce RSS. 8.You conclude that you have minimized RSS, and you can describe the final estimates of the parameters as the least squares estimates. NONLINEAR REGRESSION 5

Nonlinear regression algorithm 1.You start by guessing plausible values for the parameters. 2.You calculate the predicted values of Y from the data on X, using these values of the parameters. 3.You calculate the residual for each observation in the sample, and hence RSS, the sum of the squares of the residuals. 4.You then make small changes in one or more of your estimates of the parameters. 5.You calculate the new predicted values of Y, residuals, and RSS. 6.If RSS is smaller than before, your new estimates of the parameters are better than the old ones and you take them as your new starting point. 7.You repeat steps 4, 5 and 6 again and again until you are unable to make any changes in the estimates of the parameters that would reduce RSS. 8.You conclude that you have minimized RSS, and you can describe the final estimates of the parameters as the least squares estimates. NONLINEAR REGRESSION 6

Nonlinear regression algorithm 1.You start by guessing plausible values for the parameters. 2.You calculate the predicted values of Y from the data on X, using these values of the parameters. 3.You calculate the residual for each observation in the sample, and hence RSS, the sum of the squares of the residuals. 4.You then make small changes in one or more of your estimates of the parameters. 5.You calculate the new predicted values of Y, residuals, and RSS. 6.If RSS is smaller than before, your new estimates of the parameters are better than the old ones and you take them as your new starting point. 7.You repeat steps 4, 5 and 6 again and again until you are unable to make any changes in the estimates of the parameters that would reduce RSS. 8.You conclude that you have minimized RSS, and you can describe the final estimates of the parameters as the least squares estimates. NONLINEAR REGRESSION 7

Nonlinear regression algorithm 1.You start by guessing plausible values for the parameters. 2.You calculate the predicted values of Y from the data on X, using these values of the parameters. 3.You calculate the residual for each observation in the sample, and hence RSS, the sum of the squares of the residuals. 4.You then make small changes in one or more of your estimates of the parameters. 5.You calculate the new predicted values of Y, residuals, and RSS. 6.If RSS is smaller than before, your new estimates of the parameters are better than the old ones and you take them as your new starting point. 7.You repeat steps 4, 5 and 6 again and again until you are unable to make any changes in the estimates of the parameters that would reduce RSS. 8.You conclude that you have minimized RSS, and you can describe the final estimates of the parameters as the least squares estimates. NONLINEAR REGRESSION 8

Nonlinear regression algorithm 1.You start by guessing plausible values for the parameters. 2.You calculate the predicted values of Y from the data on X, using these values of the parameters. 3.You calculate the residual for each observation in the sample, and hence RSS, the sum of the squares of the residuals. 4.You then make small changes in one or more of your estimates of the parameters. 5.You calculate the new predicted values of Y, residuals, and RSS. 6.If RSS is smaller than before, your new estimates of the parameters are better than the old ones and you take them as your new starting point. 7.You repeat steps 4, 5 and 6 again and again until you are unable to make any changes in the estimates of the parameters that would reduce RSS. 8.You conclude that you have minimized RSS, and you can describe the final estimates of the parameters as the least squares estimates. NONLINEAR REGRESSION 9

Nonlinear regression algorithm 1.You start by guessing plausible values for the parameters. 2.You calculate the predicted values of Y from the data on X, using these values of the parameters. 3.You calculate the residual for each observation in the sample, and hence RSS, the sum of the squares of the residuals. 4.You then make small changes in one or more of your estimates of the parameters. 5.You calculate the new predicted values of Y, residuals, and RSS. 6.If RSS is smaller than before, your new estimates of the parameters are better than the old ones and you take them as your new starting point. 7.You repeat steps 4, 5 and 6 again and again until you are unable to make any changes in the estimates of the parameters that would reduce RSS. 8.You conclude that you have minimized RSS, and you can describe the final estimates of the parameters as the least squares estimates. NONLINEAR REGRESSION 10

Nonlinear regression algorithm 1.You start by guessing plausible values for the parameters. 2.You calculate the predicted values of Y from the data on X, using these values of the parameters. 3.You calculate the residual for each observation in the sample, and hence RSS, the sum of the squares of the residuals. 4.You then make small changes in one or more of your estimates of the parameters. 5.You calculate the new predicted values of Y, residuals, and RSS. 6.If RSS is smaller than before, your new estimates of the parameters are better than the old ones and you take them as your new starting point. 7.You repeat steps 4, 5 and 6 again and again until you are unable to make any changes in the estimates of the parameters that would reduce RSS. 8.You conclude that you have minimized RSS, and you can describe the final estimates of the parameters as the least squares estimates. NONLINEAR REGRESSION 11

Nonlinear regression algorithm 1.You start by guessing plausible values for the parameters. 2.You calculate the predicted values of Y from the data on X, using these values of the parameters. 3.You calculate the residual for each observation in the sample, and hence RSS, the sum of the squares of the residuals. 4.You then make small changes in one or more of your estimates of the parameters. 5.You calculate the new predicted values of Y, residuals, and RSS. 6.If RSS is smaller than before, your new estimates of the parameters are better than the old ones and you take them as your new starting point. 7.You repeat steps 4, 5 and 6 again and again until you are unable to make any changes in the estimates of the parameters that would reduce RSS. 8.You conclude that you have minimized RSS, and you can describe the final estimates of the parameters as the least squares estimates. NONLINEAR REGRESSION It should be emphasized that, long ago, mathematicians devised sophisticated techniques to minimize the number of steps required by algorithms of this type. 12

NONLINEAR REGRESSION We will return to the relationship between employment growth rate, e, and GDP growth rate, g, in the first slideshow for this chapter. e and g are hypothesized to be related as shown. 13

NONLINEAR REGRESSION According to this specification, as g becomes large, e will tend to a limit of  1. Looking at the figure, we see that the maximum value of e is about 2. So we will take this as our initial value for  1. We then hunt for the optimal value of  2, conditional on this guess for  1. 14

NONLINEAR REGRESSION The figure shows RSS plotted as a function of b 2, conditional on b 1 = 2. From this we see that the optimal value of b 2, conditional on b 1 = 2, is –2.87. RSS as a function of b 2, conditional on b 1 = 2 15

NONLINEAR REGRESSION Next, holding b 2 at –2.87, we look to improve on our guess for b 1. The figure shows RSS as a function of b 1, conditional on b 2 = –2.87. We see that the optimal value of b 1 is RSS as a function of b 1, conditional on b 2 = –

NONLINEAR REGRESSION We continue to do this both parameter estimates cease to change. The figure shows the first 11 iterations. 17

NONLINEAR REGRESSION Eventually, the estimates will reach limits. We will then have reached the values that yield minimum RSS. 18

NONLINEAR REGRESSION The limits are shown in the diagram as the horizontal lines. Convergence to them is painfully slow because this is a very inefficient algorithm. More sophisticated ones usually converge remarkably quickly. 19

NONLINEAR REGRESSION The limits must be the values from the transformed linear regression shown in the first slideshow for this chapter: b 1 = 2.60 and b 2 = –4.05. They have been determined by the same criterion, the minimization of RSS. All that we have done is to use a different method. 20

. nl (e = {beta1} + {beta2}/g) (obs = 25) Iteration 0: residual SS = Iteration 1: residual SS = Source | SS df MS Number of obs = 25 Model | R-squared = Residual | Adj R-squared = Root MSE = Total | Res. dev. = e | Coef. Std. Err. t P>|t| [95% Conf. Interval] /beta1 | /beta2 | NONLINEAR REGRESSION The table shows such output for the present hyperbolic regression of e on g. It is, as usual, Stata output, but output from other regression applications will look similar. 21

NONLINEAR REGRESSION The Stata command for a nonlinear regression is ‘nl’. This is followed by the hypothesized mathematical relationship within parentheses. The parameters must be given names placed within braces. Here b 1 is {beta1} and b 2 is {beta2}. The output is effectively the same as the linear regression output in the first slideshow for this chapter.. nl (e = {beta1} + {beta2}/g) (obs = 25) Iteration 0: residual SS = Iteration 1: residual SS = Source | SS df MS Number of obs = 25 Model | R-squared = Residual | Adj R-squared = Root MSE = Total | Res. dev. = e | Coef. Std. Err. t P>|t| [95% Conf. Interval] /beta1 | /beta2 |

NONLINEAR REGRESSION The hyperbolic function imposes the constraint that the function plunges to minus infinity for positive g as g approaches zero. 23

NONLINEAR REGRESSION This feature can be relaxed by using the variation shown. Unlike the previous function, this cannot be linearized by any kind of transformation. Here, nonlinear regression must be used. 24

. nl (e = {beta1} + {beta2}/({beta3} + g)) (obs = 25) Iteration 0: residual SS = Iteration 1: residual SS = Iteration 15: residual SS = Source | SS df MS Number of obs = 25 Model | R-squared = Residual | Adj R-squared = Root MSE = Total | Res. dev. = e | Coef. Std. Err. t P>|t| [95% Conf. Interval] /beta1 | /beta2 | /beta3 | Parameter beta1 taken as constant term in model & ANOVA table NONLINEAR REGRESSION The output for this specification is shown, with most of the iteration messages deleted. 25

NONLINEAR REGRESSION The figure compares the original (black) and new (red) hyperbolic functions. The fit is a considerable improvement, reflected in a higher R 2, 0.64 instead of

Copyright Christopher Dougherty These slideshows may be downloaded by anyone, anywhere for personal use. Subject to respect for copyright and, where appropriate, attribution, they may be used as a resource for teaching an econometrics course. There is no need to refer to the author. The content of this slideshow comes from Section 4.4 of C. Dougherty, Introduction to Econometrics, fourth edition 2011, Oxford University Press. Additional (free) resources for both students and instructors may be downloaded from the OUP Online Resource Centre Individuals studying econometrics on their own and who feel that they might benefit from participation in a formal course should consider the London School of Economics summer school course EC212 Introduction to Econometrics or the University of London International Programmes distance learning course 20 Elements of Econometrics