1. Basis Expansion and Regularization
Prof. Liqing Zhang
Dept. Computer Science & Engineering,
Shanghai Jiaotong University

2. Basis Expansion and Regularization
Outline
Piece-wise Polynomials and Splines
Wavelet Smoothing
Smoothing Splines
Automatic Selection of the Smoothing Parameters
Nonparametric Logistic Regression
Multidimensional Splines
Regularization and Reproducing Kernel Hilbert Spaces
2017/4/15
Basis Expansion and Regularization

3. Piece-wise Polynomials and Splines
Linear basis expansion
Some basis functions that are widely used
2017/4/15
Basis Expansion and Regularization

4. Basis Expansion and Regularization
Three approaches for controlling the complexity of the model.
Restriction
Selection
Regularization:
2017/4/15
Basis Expansion and Regularization

5. Piecewise Polynomials and Splines
2017/4/15
Basis Expansion and Regularization

6. Piecewise Cubic Polynomials
Increasing orders of continuity at the knots.
A cubic spline with knots at and :
Cubic spline truncated power basis
2017/4/15
Basis Expansion and Regularization

7. Piecewise Cubic Polynomials
An order-M spline with knots , j=1,…,K is a piecewise-polynomial of order M, and has continuous derivatives up to order M-2.
A cubic spline has M=4.
Truncated power basis set:
2017/4/15
Basis Expansion and Regularization

8. Natural cubic spline
Natural cubic spline adds additional constraints, namely that the function is linear beyond the boundary knots.
Linear
Cubic
Natural boundary constraints
2017/4/15
Basis Expansion and Regularization

9. Basis Expansion and Regularization
B-spline
The augmented knot sequenceτ:
Bi,m(x), the i-th B-spline basis function of order m for the knot-sequenceτ, m≤M.
2017/4/15
Basis Expansion and Regularization

10. Basis Expansion and Regularization
B-spline
The sequence of B-spline up to order 4 with ten knots evenly spaced from
0 to 1
The B-spline have local support; they are nonzero on an interval spanned by M+1 knots.
2017/4/15
Basis Expansion and Regularization

11. Basis Expansion and Regularization
Smoothing Splines
Base on the spline basis method:
So , is the noise.
Minimize the penalized residual sum of squares
is a fixed smoothing parameter
f can be any function that interpolates the data
the simple least squares line fit
2017/4/15
Basis Expansion and Regularization

12. Basis Expansion and Regularization
Smoothing Splines
The solution is a natural spline:
Then the criterion reduces to:
where
So the solution:
The fitted smoothing spline:
2017/4/15
Basis Expansion and Regularization

13. Basis Expansion and Regularization
Smoothing Splines
Function of age, that response the relative change in bone mineral density measured at the spline in adolescents
Separate smoothing splines fit the males and females,
12 degrees of freedom
脊骨BMD—骨质密度
2017/4/15
Basis Expansion and Regularization

14. Basis Expansion and Regularization
Smoothing Matrix
the N-vector of fitted values
The finite linear operator — the smoother matrix
Compare with the linear operator in the LS-fitting:
M cubic-spline basis functions, knot sequenceξ
Similarities and differences:
Both are symmetric, positive semidefinite matrices
idempotent（幂等的) ; shrinking
rank:
2017/4/15
Basis Expansion and Regularization

15. Basis Expansion and Regularization
Smoothing Matrix
Effective degrees of freedom of a smoothing spline
in the Reinsch form:
Since , solution:
is symmetric and has a real eigen-decomposition
is the corresponding eigenvalue of K
2017/4/15
Basis Expansion and Regularization

16. Basis Expansion and Regularization
Smoothing spline fit of ozone(臭氧) concentration versus Daggot pressure gradient.
Smoothing parameter df=5 and df=10.
The 3rd to 6th eigenvectors of the spline smoothing matrices
2017/4/15
Basis Expansion and Regularization

17. Basis Expansion and Regularization
The smoother matrix for a smoothing spline is nearly banded, indicating an equivalent kernel with local support.
2017/4/15
Basis Expansion and Regularization

18. Bias-Variance Tradeoff
Example:
For
The diagonal contains the pointwise variances at the training
Bias is given by
is the (unknown) vector of evaluations of the true f
2017/4/15
Basis Expansion and Regularization

19. Bias-Variance Tradeoff
df=5, bias high, standard error band narrow
df=9, bias slight, variance not increased appreciably
df=15, over learning, standard error widen
2017/4/15
Basis Expansion and Regularization

20. Bias-Variance Tradeoff
The EPE and CV curves have the a similar shape.
And, overall the CV curve is approximately unbiased as an estimate of the EPE curve
The integrated squared prediction error (EPE) combines both bias and variance in a single summary:
N fold (leave one) cross-validation:
2017/4/15
Basis Expansion and Regularization

21. Basis Expansion and Regularization
Logistic Regression
Logistic regression with a single quantitative input X
The penalized log-likelihood criterion
2017/4/15
Basis Expansion and Regularization

22. Multidimensional Splines
Tensor product basis
The M1×M2 dimensional tensor product basis
, basis function for coordinate X1
, basis function for coordinate X2
2017/4/15
Basis Expansion and Regularization

23. Tenor product basis of B-splines, some selected pairs
2017/4/15
Basis Expansion and Regularization

24. Multidimensional Splines
High dimension smoothing Splines
J is an appropriate penalty function
a smooth two-dimensional surface, a thin-plate spline.
The solution has the form
2017/4/15
Basis Expansion and Regularization

25. Multidimensional Splines
The decision boundary of an additive logistic regression model. Using natural splines in each of two coordinates.
df = 1 +(4-1) + (4-1) = 7
2017/4/15
Basis Expansion and Regularization

26. Multidimensional Splines
The results of using a tensor product of natural spline basis in each coordinate.
df = 4 x 4 = 16
2017/4/15
Basis Expansion and Regularization

27. Multidimensional Splines
A thin-plate spline fit to the heart disease data.
The data points are indicated, as well as the lattice of points used as knots.
2017/4/15
Basis Expansion and Regularization

28. Reproducing Kernel Hilbert space
A regularization problems has the form:
L(y,f(x)) is a loss-function.
J(f) is a penalty functional, and H is a space of functions on which J(f) is defined.
The solution
span the null space of the penalty functional J
2017/4/15
Basis Expansion and Regularization

29. Spaces of Functions Generated by Kernel
Important subclass are generated by the positive kernel K(x,y).
The corresponding space of functions Hk is called reproducing kernel Hilbert space.
Suppose that K has an eigen-expansion
Elements of H have an expansion
2017/4/15
Basis Expansion and Regularization

30. Spaces of Functions Generated by Kernel
The regularization problem become
The finite-dimension solution(Wahba,1990)
Reproducing properties of kernel function
2017/4/15
Basis Expansion and Regularization

31. Spaces of Functions Generated by Kernel
The penalty functional
The regularization function reduces to a finite-dimensional criterion
K is NxN matrix, the ij-th entry K(xi, xj)
2017/4/15
Basis Expansion and Regularization

32. Basis Expansion and Regularization
RKHS
Penalized least squares
The solution of :
The fitted values:
The vector of N fitted value is given by
2017/4/15
Basis Expansion and Regularization

33. Basis Expansion and Regularization
Example of RKHS
Polynomial regression
Suppose M huge
Given
Loss function:
The penalty polynomial regression:
2017/4/15
Basis Expansion and Regularization

34. Penalized Polynomial Regression
Kernel: has eigen-functions
E.g. d=2, p=2:
The penalty polynomial regression:
2017/4/15
Basis Expansion and Regularization

35. Basis Expansion and Regularization
RBF kernel & SVM kernel
Gaussian Radial Basis Functions
Support Vector Machines
2017/4/15
Basis Expansion and Regularization

36. Basis Expansion and Regularization
Wavelet smoothing
Another type of bases——Wavelet bases
Wavelet bases are generated by translations and dilations of a single scaling function
If ,then generates an orthonormal basis for functions with jumps at the integers.
form a space called reference space V0
The dilations form an orthonormal basis for a space V V0
Generally, we have
2017/4/15
Basis Expansion and Regularization

37. Basis Expansion and Regularization
2017/4/15
Basis Expansion and Regularization

38. Basis Expansion and Regularization
2017/4/15
Basis Expansion and Regularization

39. Basis Expansion and Regularization
Wavelet smoothing
The L2 space dividing
Mother wavelet generate function form an orthonormal basis for W0. Likewise form a basis for Wj.
2017/4/15
Basis Expansion and Regularization

40. Basis Expansion and Regularization
Wavelet smoothing
Wavelet basis on W0 :
Wavelet basis on Wj :
The symmlet-p wavelet:
A support of 2p-1 consecutive intervals.
p vanishing moments:
2017/4/15
Basis Expansion and Regularization

41. Basis Expansion and Regularization
2017/4/15
Basis Expansion and Regularization

42. Basis Expansion and Regularization
2017/4/15
Basis Expansion and Regularization

43. Basis Expansion and Regularization
2017/4/15
Basis Expansion and Regularization

44. Adaptive Wavelet Filtering
Wavelet transform:
y: response vector, W: NxN orthonormal wavelet basis matrix
Stein Unbiased Risk Estimation (SURE)
The solution:
Fitted function is given by inverse wavelet transform: , LS coefficients truncated to 0
Simple choice for
2017/4/15
Basis Expansion and Regularization

45. Basis Expansion and Regularization
2017/4/15
Basis Expansion and Regularization