Jim Ramsay McGill University Basis Basics

Overview  What are basis functions?  What properties should they have?  How are they usually constructed?  Fourier bases.  B-spline bases: Lots of detail here.  Wavelet bases.  Advice, wrap-up, and beyond.

What are basis functions?  We need flexible method for constructing a function f(t) that can track local curvature.  We pick a system of K basis functions φ k (t), and call this the basis for f(t).  We express f(t) as a weighted sum of these basis functions: f(t) = a 1 φ 1 (t) + a 2 φ 2 (t) + … + a K φ K (t) f(t) = a 1 φ 1 (t) + a 2 φ 2 (t) + … + a K φ K (t) The coefficients a 1, …, a K determine the shape of the function. The coefficients a 1, …, a K determine the shape of the function.

What do we want from basis functions?  Fast computation of individual basis functions.  Flexible: can exhibit the required curvature where needed, but also be nearly linear when appropriate.  Fast computation of coefficients a k : possible if matrices of values are diagonal, banded or sparse.  Differentiable as required: We make lots of use of derivatives in functional data analysis.  Constrained as required, such as periodicity, positivity, monotonicity, asymptotes and etc.

What are some commonly used basis functions?  Powers: 1, t, t 2, and so on. They are the basis functions for polynomials. These are not very flexible, and are used only for simple problems.  Fourier series: 1, sin(ωt), cos(ωt), sin(2ωt), cos(2ωt), and so on for a fixed known frequency ω. These are used for periodic functions.  B-spline functions: These have now more or less replaced polynomials for non-periodic problems. More explanation follows.

How do we construct basis systems?  We start with a prototype basis function φ(t). Let’s call it the mother function.  We apply three operations to it:  Lateral shift φ(t+a) to focus fitting power near a position a.  Scale change φ(bt), to increase resolving power, or the capacity for local curvature.  Smoothing, to increase differentiability and smoothness.

Can I see an example? Consider the Fourier series :  Mother functions: φ(t) = sin(ωt), and φ(t) = 1  Lateral Shift: sin[ω(t+π/2)] = cos(ωt)  Scale change: sin(kωt), cos(kωt), k=1,2,…  Smoothing: None; these are already infinitely differentiable functions.  Strengths: Orthogonality makes computing coefficients very fast  Limitations: Fourier series are only natural if f(t) is periodic with period ω.

Five Fourier Basis Functions  φ 1 (t) = 1  A sine/cosine pair with period 1: Captures low frequency variation  A sine/cosine pair with period ½: Captures higher frequency variation

What about polynomials?  The mother function is φ(t) = 1.  Smoothing: Multiply by t to increase both differentiability and curvature.  No shift or scaling: Polynomials are invariant with respect to shift and scaling.  Strengths: Okay for very simple curves such as linear or slight curvature.  Limitations: Just not flexible enough. We can’t focus fitting power on a specific location.

What basis should I use for non- periodic functions? Spline bases have nearly everything:  Infinite and independent control over shift, scale, and smoothness.  Potential to model sharp changes in f(t) as well as its smooth variation.  As fast as polynomials to evaluate.  Band-structured matrix of values.

How do B-splines work?  Start with a discrete mesh of values, called knots: ξ 0, ξ 1, …, ξ L.  Space out the knots where you don’t need much resolution, make them dense where you need a lot of resolution.

 Mother function is the box function over [ξ 0, ξ 1 ]: φ(t) = 1, ξ 0 ≤ t < ξ 1, and 0 otherwise.  Shift: just shift to another interval.  Scale: determined by spacing between knots.

What about smoothness?  Smoothness control is the heart of spline technology.  It is defined by a recursion relation.  The recursion relation uses spline basis functions at a given smoothness level to make new ones at the next higher smoothness level.

B-spline notation The spline basis system we use is the B-spline system.  To keep notation simple, let the knots be the natural numbers j = 0, 1, …  We need two indices for a B-spline basis function B jm (t): j to indicate the knot where it starts, and m to indicate its order or degree of smoothness.  For our box functions, m = 1.

 B-splines of order 2 are tent functions, starting at a knot, rising linearly to 1 at the next knot, and decaying linearly to 0 two knots over.  They are continuous. Order 2 implies a continuous derivative of order 0.  Order 2 knots are piecewise linear.  What about order 3 B- splines?

The recursion for integer knots For the B-spline function of order 2 beginning at 0, the recursion is

In the first term, the mother box function B 01 (t) is multiplied by t, and is therefore t itself. In the second, the mother box is shifted to the right and multiplied by 2-t.In the second, the mother box is shifted to the right and multiplied by 2-t. The second term amounts to a shift and reflection of the first term.The second term amounts to a shift and reflection of the first term.

Constructing a Tent B 02 (t) from Two Boxes B 01 (t) and B 11 (t)

Constructing Order 3 B 03 (t) from Two Tent Functions

 Multiplying by (t-j) increases the polynomial degree of the B-spline in the first term by one.  Multiplying (m+j-t) does the same thing for the reflected and shifted B-spline in the second term.  The result is a new B-spline of one degree or order higher, spread over an additional interval, and having an one more level of continuous derivatives.  Order 3 B-splines are piece-wise quadratic, spread over 3 intervals, and have a continuous first derivative.

How do we get the extra smoothness?  tB 01 (t) and (2-t)B 11 (t) have the same values at knot 1. The sum is continuous.  tB 02 (t) and (3-t)B 12 (t) have the same derivative values at knots 1 and 2. The sum has a continuous first derivative.  Each time we multiply by t, and then shift and reflect the result, we add another order of smooth derivative.

Suppose I don’t want so much smoothness at a fixed point?  You can have multiple knots at a point.  For each additional knot, the spline function will have one less derivative at that knot.  An order m Bspline with m knots at a point can be discontinuous at that point.  Most spline software places m knots at each of the boundaries.

How many B-spline basis functions am I using?  When all interior knots are unique, the total number of B-spline basis functions is equal to the number of interior knots plus the order.  When there are no interior knots, splines become polynomials: The order equals the number of basis functions.

Wavelets  Wavelets take this shift/scale/smooth trio of operations even further.  They define basis functions organized into orthogonal levels of resolution.  The resulting multi-resolution analysis is one of the most exciting developments in mathematics since Fourier’s series.

Some advice  If you have periodic data, you’ll probably use Fourier series.  If you don’t, you’ll probably use B-splines.  Especially if you need to look at or work with derivatives.  If you want a derivative of order m, fit the data with a spline of order at least m+2, although order 2m has some advantages.

Where have we been?  Linear combinations of basis functions are a convenient way to model functions.  The more basis functions we use, the more complex the fitted function can be.  Most basis function systems involve turning three knobs, marked shift, scale, and smooth, to modify a mother function.  The best systems permit independent control of these features.  Like splines and wavelets.

Where do we go from here?  Although we can control the resolution of a spline fit by the number of spline basis functions and the spacing of knots, we often need easier more continuous control over smoothness.  Roughness penalties can give us the additional control that we need.