1. Chengyuan Yin School of Mathematics
Econometrics
Chengyuan Yin
School of Mathematics

2. 14. Nonlinear Regression and Nonlinear Least Squares
Econometrics
14. Nonlinear Regression and
Nonlinear Least Squares

3. Nonlinear Regression What makes a regression model “nonlinear?”
Nonlinear functional form?
Regression model: yi = f( xi ,  ) + i
Not necessarily: yi = exp(α) + β2*xi + i
β1 = exp(α)
yi = exp(α)xiβexp(i) is “loglinear”
Models can be nonlinear in the functional form of the relationship between y and x, and not be nonlinear for purposes here.
We will redefine “nonlinear” shortly, as we proceed.

4. Nonlinear Least Squares
Least squares: Minimize wrt  ½ i [yi - f(xi,)]2
= ½ i ei2
First order conditions:
½i[yi- f(xi,)]2 ]/
= ½i(-2)[yi- f(xi,)] f(xi,)/
= -i ei xi0 = 0 (familiar?)
There is no explicit solution, b = f(data) like LS.
(Nonlinearity of the FOC defines nonlinear model)

5. Example: NIST How to solve this kind of set of equations: Example,
yi = 0 + 1xi2 + i.
[ ½i ei2]/0 = i (-1) (yi - 0 - 1xi2) = 0
[ ½i ei2]/1 = i (-1) (yi - 0 - 1 xi2) xi = 0
[ ½i ei2]/2 = i (-1) (yi - 0 - 1 xi2) 1 xi2lnxi = 0
Nonlinear equations require a nonlinear solution. We’ll return to that problem shortly.
This defines a nonlinear regression model. I.e., when the first order conditions are not linear in .
(!!!) Check your understanding. What does this produce if f( xi ,  ) = xi? (I.e., a linear model)

6. The Linearized Regression Model
Linear Taylor series: y = f(xi,) + .
Expand the regression around some point, 0.
f(xi,)  f(xi,0) + k[f(xi,0)/k0]( k - k0)
= f(xi,0) + k xi0 ( k - k0)
= [f(xi,0) - k xi0k0] + k xi0k
= f0 + k xi0k which looks linear.
The ‘pseudo-regressors’ are the derivative functions in the linearized model.

7. Estimating Asy.Var[b]
Computing the asymptotic covariance matrix for the nonlinear least squares estimator using the pseudo regressors and the sum of squares.

8. Gauss-Marquardt Algorithm
Given a coefficient vector at step m, find the vector for step m+1 by b(m),
b(m+1) = b(m) + [X0(m)X0(m)]-1X0(m)e0(m)
Columns of X0(m) are the derivatives,
f(xi,b(m))/b(m)
e0 = vector of residuals, y - f[x,b(m)]
“Update” vector is the slopes in the regression of the residuals on the pseudo-regressors. Update is zero when they are orthogonal. (Just like LS)

9. A NIST Application
Y X
y = 0 + 1x2 + .
xi0 = [1, x2, 1x2logx ]

10. Iterations NLSQ;LHS=Y ;FCN=b0+B1*X^B2 ;LABELS=b0,B1,B2
;MAXIT=500;TLF;TLB;OUTPUT=1;DFC ;START=0,1,5 $
Begin NLSQ iterations. Linearized regression.
Iteration= 1; Sum of squares= ; Gradient=
Iteration= 2; Sum of squares= ; Gradient=
Iteration= 3; Sum of squares= E-01; Gradient= E-01
Iteration= 4; Sum of squares= E-01; Gradient= E-01
Iteration= 5; Sum of squares= E-02; Gradient= E-08
Iteration= 6; Sum of squares= E-02; Gradient= E-15
Iteration= 7; Sum of squares= E-02; Gradient= E-20
Convergence achieved

11. Stata – Version 9
Most maximum likelihood estimators now test for convergence using the Hessian-scaled gradient, g*inv(H)*g'. This criterion ensures that the gradient is close to zero when scaled by the Hessian (the curvature of the likelihood or pseudolikelihood surface at the optimum) and provides greater assurance of convergence for models whose likelihoods tend to be difficult to optimize, such as those for arch, asmprobit, and scobit. See [R] maximize.

12. Results
| User Defined Optimization |
| Nonlinear least squares regression Weighting variable = none |
| Number of iterations completed = |
| Dep. var. = Y Mean= , S.D.= |
| Model size: Observations = , Parameters = 3, Deg.Fr.= |
| Residuals: Sum of squares= D-02, Std.Dev.= |
| Fit: R-squared= , Adjusted R-squared = |
| (Note: Not using OLS. R-squared is not bounded in [0,1] |
| Model test: F[ 2, ] = , Prob value = |
| Diagnostic: Log-L = , Restricted(b=0) Log-L = |
| LogAmemiyaPrCrt.= , Akaike Info. Crt.= |
|Variable | Coefficient | Standard Error |b/St.Er.|P[|Z|>z] | Mean of X| B B B

13. NLS Solution The pseudo regressors and residuals at the solution are:
x 1x2 lnx e0
X0e0 = D-13
D-12
D-10

14. Application: Doctor Visits
German Individual Health Care data: N=27,236
Model for number of visits to the doctor

15. Conditional Mean and Projection
This area is outside the range of the data
Most of the data are in here
Notice the problem with the linear approach. Negative predictions.

16. Nonlinear Model Specification
Nonlinear Regression Model y=exp(β’x) + ε
X =one,age,health_status, married,
education, household_income, nkids

17. NLS Iterations
--> nlsq;lhs=docvis;start=0,0,0,0,0,0,0;labels=k_b;fcn=exp(b1'x);maxit=25;out...
Begin NLSQ iterations. Linearized regression.
Iteration= 1; Sum of squares= ; Gradient=
Iteration= 2; Sum of squares= E+11; Gradient= E+11
Iteration= 3; Sum of squares= E+10; Gradient= E+10
Iteration= 4; Sum of squares= ; Gradient=
Iteration= 5; Sum of squares= ; Gradient=
Iteration= 6; Sum of squares= ; Gradient=
Iteration= 7; Sum of squares= ; Gradient=
Iteration= 8; Sum of squares= ; Gradient=
Iteration= 9; Sum of squares= ; Gradient=
Iteration= 10; Sum of squares= ; Gradient=
Iteration= 11; Sum of squares= ; Gradient= E-03
Iteration= 12; Sum of squares= ; Gradient= E-05
Iteration= 13; Sum of squares= ; Gradient= E-07
Iteration= 14; Sum of squares= ; Gradient= E-08
Iteration= 15; Sum of squares= ; Gradient= E-10

18. Nonlinear Regression Results
| Nonlinear least squares regression |
| LHS=DOCVIS Mean = |
| Standard deviation = |
| WTS=none Number of observs. = |
| Model size Parameters = |
| Degrees of freedom = |
| Residuals Sum of squares = |
| Standard error of e = |
| Fit R-squared = |
| Adjusted R-squared = |
| Info criter. LogAmemiya Prd. Crt. = |
| Akaike Info. Criter. = |
| Not using OLS or no constant. Rsqd & F may be < 0. |
|Variable | Coefficient | Standard Error |b/St.Er.|P[|Z|>z] | B B B B B B B

19. Partial Effects in the Nonlinear Model

20. Asymptotic Variance of the Slope Estimator

21. Computing the Slopes calc;k=col(x)$
nlsq;lhs=docvis;start=0,0,0,0,0,0,0
;labels=k_b;fcn=exp(b1'x);
matr;xbar=mean(x)$
calc;mean=exp(xbar'b)$
matr;me=b*mean$
matr;g=mean*iden(k)+mean*b*xbar'$
matr;vme=g*varb*g'$
matr;stat(me,vme)$

22. Partial Effects at the Means of X
|Number of observations in current sample = |
|Number of parameters computed here = |
|Number of degrees of freedom = |
|Variable | Coefficient | Standard Error |b/St.Er.|P[|Z|>z] |
ME_
ME_
ME_
ME_
ME_
ME_
ME_

23. What About Just Using LS?
|Variable | Coefficient | Standard Error |b/St.Er.|P[|Z|>z] | Mean of X|
Least Squares Coefficient Estimates
Constant
AGE
NEWHSAT
MARRIED
EDUC
HHNINC
HHKIDS
Estimated Partial Effects
ME_ Constant term, marginal effect not computed
ME_
ME_
ME_
ME_
ME_
ME_