1. (CH. 5 Part 2) Speaker: Brian Quanz (bquanz@ittc.ku.edu) 7/3/2008
Basis Expansions and Regularization Continued From Elements of Statistical Learning
(CH. 5 Part 2)
Speaker: Brian Quanz
7/3/2008

2. Overview Nonparametric Logistic Regression Multidimensional Splines
Regularization and Reproducing Kernel Hilbert Spaces
Wavelet Smoothing

3. Review: Logistic Regression
Uses logistic (a.k.a. sigmoid) function:
f(x)
Goal: Fit logistic curve to the data using iterative procedure to calculate maximum likelihood parameters x
P(Class = 1|X=x)
P(Class = 2|X=x)
Can be used to associate probabilities with a discriminative classifier (i.e. P(Class=1|X=x)).
For example, sigmoid fit is used with Support Vector Machines (SVM), where x is distance from separating hyperplane, to assign probabilities to a classification.
H.-T. Lin, C.-J. Lin, and R. C. Weng. A note on Platt's probabilistic outputs for support vector machines. Technical report, Department of Computer Science, National Taiwan University, URL
*Original sigmoid image taken from wikipedia.org

4. Review: Logistic Regression - Fitting
Maximum Likelihood (2 class case):
Sample data consists of data samples x1, x2,…,xn with labels y1, y2,…,yn, where xi has dimension p and yi are 0 or 1.
Maximize Prob(Parameters and Data) = P(B;X;Y) = P(Y|B;X)P(B;X)
L(B) = P(Y|B;X) is called the likelihood function
Then, assuming IID and taking log to simplify gives log-likelihood function:
Goal: find B that maximizes ℓ(B), take derivative to obtain score functions:
Text uses Newton-Raphson algorithm to find zeros

5. Logistic Regression – Newton-Rapshon Method
To find zeros of arbitrary function
Approximate function at starting point with tangent; find x-intercept to attain new starting point; repeat
(f(xn)-0)/(xn-xn+1) = f’(xn)
=> xn+1 = xn – f(xn)/f’(xn)
Likely to converge since log-likelihood is concave
Update rule for logistic regression:
*Image taken from wikipedia.org

6. Nonparametric Logistic Regression
No longer fix the log odds as linear, allow a smoother fit:
Fit f(x) smoothly to allow smoother conditional probability function
As with smoothing spline, penalize curvature:
As with smoothing splines, optimal f is finite dimensional natural spline with knots at unique x. We can define:

7. Nonparametric Logistic Regression
Which implies:
Where p is N-vector with elements pi, and as defined previously:
And W is diagonal matrix with entries P(Y = 1|X = xi)(1-P(Y = 1|X = xi))
Using Newton-Raphson update as before:

8. Multidimensional Splines
Many options for how to do multi-dimensional splines
The most basic, not even defined by text is additive
Simply add together the spline bases for different dimensions
The tensor product basis combines the bases from different dimensions through all possible multiplications with one basis from each. Example:
Tensor product basis:

9. Multidimensional Splines – Tensor Product
Simply expressed as new basis, so same fitting applies as before, e.g. least-squares
With increasing dimension, the resulting basis dimension grows exponentially
Selecting only important basis functions to solve problem, discussed in ch. 9
*Image taken from the book

10. Example Comparison: Additive and Tensor Natural Splines
left
right
*Image taken from the book

11. Smoothing Splines of Higher Dimension
Same problem as before, except x now has d dimensions:
J is an appropriate penalty. Text gives example of a two-dimensional penalty extending the one-dimensional penalty previously presented:
This optimization results in a thin-plate spline, which shares many properties with previously presented smoothing splines
Thin-plate splines can be generalized to higher dimensions by using the appropriate penalty J

12. Thin-Plate Splines Properties similar to 1-D smoothing spline:
λ -> 0 solution approaches interpolating function
λ -> ∞ solution approaches least-squares linear
For intermediate λ solution expressed as linear expansion of basis functions, coefficients from generalized ridge regression
hj are in fact radial basis functions, as discussed in previous presentation
Computational complexity O(N3) can be reduced by choosing subset of knots K < N resulting in O(NK2 + K3)

13. Thin-Plate Spline Example:
*Image taken from the book

14. Additional Multidimensional Splines
In general, there are many possibilities for multi-dimensional splines; we can use any suitably large basis expansion of different basis types and use a suitable regularizer
E.g. Tensor products of B-splines
Additive splines are just one class that come from additive penalty (f are univariate splines):
This can be extended to bases of functions with higer-order interactions:
Many choices: maximum order, which terms to include, basis type, etc. Automatic selection may be preferred (ch. 9 and 10).

15. Overview Nonparametric Logistic Regression Multidimensional Splines
Regularization and Reproducing Kernel Hilbert Spaces
Wavelet Smoothing

16. Motivation: a truly general penalty
Regularization and Reproducing Kernel Hilbert Spaces “This section is quite technical and can be skipped by the disinterested or intimidated reader”
Idea is to generalize the fitting/regression problem as much as possible
Motivation: a truly general penalty
Start by considering abstract vector spaces, where vectors can represent any number of objects, points in Euclidean space, functions, graphs, etc., as long as certain conditions are met, the same rules apply to all
Space of functions on which J(f) is defined
Penalty function
Loss function

17. A General Penalty
One set of general penalties proposed is of the form:
denotes the fourier transform of f, and is positive and approaches 0 for large s, so that high frequency components are more heavily penalized. Under additional assumptions this has solution:
span the null space of J (the null space of J is the set of all functions it maps to the same value)

18. Hilbert Spaces: Introduction
Example to introduce Hilbert spaces which is also closely related to wavelets
Recall Fourier series: all continuous functions f(x) defined on an interval of length L, 0 < x < L, (let D denote this set of functions) can be expanded as a sum of a sine series:
This defines a vector space since, given f,g in D and real constants a,b, h=af + bg can be defined as: which defines a continuous function (an element in D), fulfilling axioms of vector space.
This Fourier series representation could also be expressed as:
*Ideas of introduction taken from course notes by Professor Edwin Langmann, “Notes on Hilbert space theory”

19. Hilbert Spaces: Introduction
Thus un represent a set of special functions in D and from Fourier series theory, every element in D can be written as linear combination of these special functions
Immediate analogy with RN:
Also we can compute components with a scalar product:
A component has formula:
This can be easily shown:

20. Hilbert Spaces: Introduction
This is in fact the same as the expression for Fourier series components and can be shown in the same way. We can define scalar product of two functions in D as:
And un are orthogonal in the same sense:

21. Hilbert Spaces: Introduction
Question of completeness: can every function in D be represented as a combination of un? In general orthogonality of a system of functions vn shows best approximation off :
We can define any system is complete if error goes to 0 as M goes to infinity
Thus the un form an orthonormal basis of D that is of infinite dimension.
This is an infinite dimensional vector space. Both this space and the Euclidean space RN are special cases of the theory of Hilbert spaces.
The Hilbert space allows a generalized way of considering many different types of vector spaces

22. Hilbert Spaces: Definition
A Hilbert space is an inner product space that is complete under the norm defined by the inner product < ∙, ∙ > by
“Complete” means that if a sequence of vectors approaches a limit in the space, than that limit is in the space as well.
For example real numbers are complete, rational numbers are not, since some sequences approach irrational numbers like sqrt(2)
An inner product space is a vector space of arbitrary dimension, with an inner product, which associates a scalar quantity with each pair of vectors
A vector space is a collection of objects having operations of vector addition and scalar multiplication and satisfying 8 axioms, such as operations being associative, commutative, distributive, containing an identity element, etc.

23. Reproducing Kernel Hilbert Space
*Taken from slides by Dr. Christian Igel:

24. Reproducing Kernel Hilbert Space
*Taken from slides by Dr. Christian Igel:

25. Reproducing Kernel Hilbert Space
Moore-Aroszajn Theorem: For every positive definite function K(∙, ∙) on X x X, there exists a unique RKHS and vice versa.
This allows us to apply the ideas from Euclidean geometry to non-geometric problems, so long as we can define a suitable Kernel K(∙, ∙)
*Taken from slides by Dr. Christian Igel:

26. Results Presented in the Text about RKHS
Text considers an important subclass of problems of:
for which H is the space of functions defined by a positive definite kernel K(x,y), HK, a RKHS
Suppose K has eigen-expansion:
Elements of HK expanded as:

27. Results Presented in the Text about RKHS
We can define the penalty as: which can be interpreted as generalized ridge penalty, where functions with large eigenvalues are penalized less
It can be shown solution has finite-dimensional form:
Consists of basis function known as representer of evaluation at xi

28. Results Presented in the Text about RKHS
Then by reproducing property:
And the objective function reduces to the following finite-dimensional problem, known as the kernel property in support vector machine literature:
We can have the penalty apply to only a subspace of the functions in H by penalizing the projection of functions onto the subspace
Solution then has form:
(First term represents expansion of H0)

29. RKHS Examples Squared-error loss:
Generalized ridge regression, solution obtained as:

30. RKHS Examples Penalized Polynomial Regression Example:
Objective Function:
By substitution can be expressed as the squared-error loss problem

31. RKHS Examples Gaussian Radial Basis Functions
This is an expansion in radial basis functions. As we saw earlier, a thin-plate spline was also an expansion in radial basis functions

32. RKHS Examples
Support Vector Classifiers (ch. 12)

33. Wavelet Smoothing
Similar idea to the Fourier series representation, except wavelets are localized in both time and frequency
We have a complete dictionary of orthonormal basis functions to represent functions
Sparse representation is obtained by shrinking and selecting the coefficients of the basis functions, as we’ve seen before

34. Wavelet example
Fits basis coefficients by least-squares, thresholds smaller coefficients, like the Lasso

35. Wavelet Derivation
We define father and mother wavelets. The rest are then created from them, by increasing the frequency, as with Fourier series (translations and dilations):

36. Wavelet Derivation For example, for Haar wavelet Father wavelet
Build orthogonal mother wavelet:
All these basis functions are orthonormal
The father wavelets form the basis for the rough components of a function, and the orthogonal mother wavelets build the detail:
Haar wavelet is often too course; many other wavelets have been invented that are smoother, such as the symlet

37. Wavelet Smoothing Example
FIGURE The top panel shows a NMR signal, with the wavelet-shrunk
version superimposed in green. The lower left panel represents the wavelet transform
of the original signal, down to V4, using the symmlet-8 basis. Each coefficient
is represented by the height (positive or negative) of the vertical bar. The
lower right panel represents the wavelet coefficients after being shrunken using
the waveshrink function in S-PLUS, which implements the SureShrink method
of wavelet adaptation of Donoho and Johnstone.

38. Adaptive Wavelet Filtering
Lattice of N points, y is response vector and W is NxN orthonormal wavelet basis evaluated at the N uniformly spaced observations. The following is called wavelet transform of y:
Popular method for adaptive wavelet fitting is known as SURE shrinkage (Stein Unbiased Risk Estimation)
Same as previously seen Lasso criterion
Because W is orthogonal, simple solution:
Fitted function obtained from inverse wavelet transform:

39. Wavelets: Key Idea
In general, any basis could be used, such as the smoothing splines we’ve seen before.
The key difference is that wavelets allow localization in time as well as frequency (roughness), and along with the L1 penalty allow sparse solutions
Smoothing splines compress by imposing smoothness; Wavelets compress by imposing sparsity

40. Wavelet Compared to Smoothing Spline

41. Wavelet Compared to Smoothing Spline

42. The End
Questions?