3 Review: Logistic Regression Uses logistic (a.k.a. sigmoid) function:f(x)Goal: Fit logistic curve to the data using iterative procedure to calculate maximum likelihood parametersxP(Class = 1|X=x)P(Class = 2|X=x)Can be used to associate probabilities with a discriminative classifier (i.e. P(Class=1|X=x)).For example, sigmoid fit is used with Support Vector Machines (SVM), where x is distance from separating hyperplane, to assign probabilities to a classification.H.-T. Lin, C.-J. Lin, and R. C. Weng. A note on Platt's probabilistic outputs for support vector machines. Technical report, Department of Computer Science, National Taiwan University, URL*Original sigmoid image taken from wikipedia.org
4 Review: Logistic Regression - Fitting Maximum Likelihood (2 class case):Sample data consists of data samples x1, x2,…,xn with labels y1, y2,…,yn, where xi has dimension p and yi are 0 or 1.Maximize Prob(Parameters and Data) = P(B;X;Y) = P(Y|B;X)P(B;X)L(B) = P(Y|B;X) is called the likelihood functionThen, assuming IID and taking log to simplify gives log-likelihood function:Goal: find B that maximizes ℓ(B), take derivative to obtain score functions:Text uses Newton-Raphson algorithm to find zeros
5 Logistic Regression – Newton-Rapshon Method To find zeros of arbitrary functionApproximate function at starting point with tangent; find x-intercept to attain new starting point; repeat(f(xn)-0)/(xn-xn+1) = f’(xn)=> xn+1 = xn – f(xn)/f’(xn)Likely to converge since log-likelihood is concaveUpdate rule for logistic regression:*Image taken from wikipedia.org
6 Nonparametric Logistic Regression No longer fix the log odds as linear, allow a smoother fit:Fit f(x) smoothly to allow smoother conditional probability functionAs with smoothing spline, penalize curvature:As with smoothing splines, optimal f is finite dimensional natural spline with knots at unique x. We can define:
7 Nonparametric Logistic Regression Which implies:Where p is N-vector with elements pi, and as defined previously:And W is diagonal matrix with entries P(Y = 1|X = xi)(1-P(Y = 1|X = xi))Using Newton-Raphson update as before:
8 Multidimensional Splines Many options for how to do multi-dimensional splinesThe most basic, not even defined by text is additiveSimply add together the spline bases for different dimensionsThe tensor product basis combines the bases from different dimensions through all possible multiplications with one basis from each. Example:Tensor product basis:
9 Multidimensional Splines – Tensor Product Simply expressed as new basis, so same fitting applies as before, e.g. least-squaresWith increasing dimension, the resulting basis dimension grows exponentiallySelecting only important basis functions to solve problem, discussed in ch. 9*Image taken from the book
10 Example Comparison: Additive and Tensor Natural Splines leftright*Image taken from the book
11 Smoothing Splines of Higher Dimension Same problem as before, except x now has d dimensions:J is an appropriate penalty. Text gives example of a two-dimensional penalty extending the one-dimensional penalty previously presented:This optimization results in a thin-plate spline, which shares many properties with previously presented smoothing splinesThin-plate splines can be generalized to higher dimensions by using the appropriate penalty J
12 Thin-Plate Splines Properties similar to 1-D smoothing spline: λ -> 0 solution approaches interpolating functionλ -> ∞ solution approaches least-squares linearFor intermediate λ solution expressed as linear expansion of basis functions, coefficients from generalized ridge regressionhj are in fact radial basis functions, as discussed in previous presentationComputational complexity O(N3) can be reduced by choosing subset of knots K < N resulting in O(NK2 + K3)
13 Thin-Plate Spline Example: *Image taken from the book
14 Additional Multidimensional Splines In general, there are many possibilities for multi-dimensional splines; we can use any suitably large basis expansion of different basis types and use a suitable regularizerE.g. Tensor products of B-splinesAdditive splines are just one class that come from additive penalty (f are univariate splines):This can be extended to bases of functions with higer-order interactions:Many choices: maximum order, which terms to include, basis type, etc. Automatic selection may be preferred (ch. 9 and 10).
16 Motivation: a truly general penalty Regularization and Reproducing Kernel Hilbert Spaces “This section is quite technical and can be skipped by the disinterested or intimidated reader”Idea is to generalize the fitting/regression problem as much as possibleMotivation: a truly general penaltyStart by considering abstract vector spaces, where vectors can represent any number of objects, points in Euclidean space, functions, graphs, etc., as long as certain conditions are met, the same rules apply to allSpace of functions on which J(f) is definedPenalty functionLoss function
17 A General PenaltyOne set of general penalties proposed is of the form:denotes the fourier transform of f, and is positive and approaches 0 for large s, so that high frequency components are more heavily penalized. Under additional assumptions this has solution:span the null space of J (the null space of J is the set of all functions it maps to the same value)
18 Hilbert Spaces: Introduction Example to introduce Hilbert spaces which is also closely related to waveletsRecall Fourier series: all continuous functions f(x) defined on an interval of length L, 0 < x < L, (let D denote this set of functions) can be expanded as a sum of a sine series:This defines a vector space since, given f,g in D and real constants a,b, h=af + bg can be defined as: which defines a continuous function (an element in D), fulfilling axioms of vector space.This Fourier series representation could also be expressed as:*Ideas of introduction taken from course notes by Professor Edwin Langmann, “Notes on Hilbert space theory”
19 Hilbert Spaces: Introduction Thus un represent a set of special functions in D and from Fourier series theory, every element in D can be written as linear combination of these special functionsImmediate analogy with RN:Also we can compute components with a scalar product:A component has formula:This can be easily shown:
20 Hilbert Spaces: Introduction This is in fact the same as the expression for Fourier series components and can be shown in the same way. We can define scalar product of two functions in D as:And un are orthogonal in the same sense:
21 Hilbert Spaces: Introduction Question of completeness: can every function in D be represented as a combination of un? In general orthogonality of a system of functions vn shows best approximation off :We can define any system is complete if error goes to 0 as M goes to infinityThus the un form an orthonormal basis of D that is of infinite dimension.This is an infinite dimensional vector space. Both this space and the Euclidean space RN are special cases of the theory of Hilbert spaces.The Hilbert space allows a generalized way of considering many different types of vector spaces
22 Hilbert Spaces: Definition A Hilbert space is an inner product space that is complete under the norm defined by the inner product < ∙, ∙ > by“Complete” means that if a sequence of vectors approaches a limit in the space, than that limit is in the space as well.For example real numbers are complete, rational numbers are not, since some sequences approach irrational numbers like sqrt(2)An inner product space is a vector space of arbitrary dimension, with an inner product, which associates a scalar quantity with each pair of vectorsA vector space is a collection of objects having operations of vector addition and scalar multiplication and satisfying 8 axioms, such as operations being associative, commutative, distributive, containing an identity element, etc.
23 Reproducing Kernel Hilbert Space *Taken from slides by Dr. Christian Igel:
24 Reproducing Kernel Hilbert Space *Taken from slides by Dr. Christian Igel:
25 Reproducing Kernel Hilbert Space Moore-Aroszajn Theorem: For every positive definite function K(∙, ∙) on X x X, there exists a unique RKHS and vice versa.This allows us to apply the ideas from Euclidean geometry to non-geometric problems, so long as we can define a suitable Kernel K(∙, ∙)*Taken from slides by Dr. Christian Igel:
26 Results Presented in the Text about RKHS Text considers an important subclass of problems of:for which H is the space of functions defined by a positive definite kernel K(x,y), HK, a RKHSSuppose K has eigen-expansion:Elements of HK expanded as:
27 Results Presented in the Text about RKHS We can define the penalty as: which can be interpreted as generalized ridge penalty, where functions with large eigenvalues are penalized lessIt can be shown solution has finite-dimensional form:Consists of basis function known as representer of evaluation at xi
28 Results Presented in the Text about RKHS Then by reproducing property:And the objective function reduces to the following finite-dimensional problem, known as the kernel property in support vector machine literature:We can have the penalty apply to only a subspace of the functions in H by penalizing the projection of functions onto the subspaceSolution then has form:(First term represents expansion of H0)
33 Wavelet SmoothingSimilar idea to the Fourier series representation, except wavelets are localized in both time and frequencyWe have a complete dictionary of orthonormal basis functions to represent functionsSparse representation is obtained by shrinking and selecting the coefficients of the basis functions, as we’ve seen before
34 Wavelet exampleFits basis coefficients by least-squares, thresholds smaller coefficients, like the Lasso
35 Wavelet DerivationWe define father and mother wavelets. The rest are then created from them, by increasing the frequency, as with Fourier series (translations and dilations):
36 Wavelet Derivation For example, for Haar wavelet Father wavelet Build orthogonal mother wavelet:All these basis functions are orthonormalThe father wavelets form the basis for the rough components of a function, and the orthogonal mother wavelets build the detail:Haar wavelet is often too course; many other wavelets have been invented that are smoother, such as the symlet
37 Wavelet Smoothing Example FIGURE The top panel shows a NMR signal, with the wavelet-shrunkversion superimposed in green. The lower left panel represents the wavelet transformof the original signal, down to V4, using the symmlet-8 basis. Each coefficientis represented by the height (positive or negative) of the vertical bar. Thelower right panel represents the wavelet coefficients after being shrunken usingthe waveshrink function in S-PLUS, which implements the SureShrink methodof wavelet adaptation of Donoho and Johnstone.
38 Adaptive Wavelet Filtering Lattice of N points, y is response vector and W is NxN orthonormal wavelet basis evaluated at the N uniformly spaced observations. The following is called wavelet transform of y:Popular method for adaptive wavelet fitting is known as SURE shrinkage (Stein Unbiased Risk Estimation)Same as previously seen Lasso criterionBecause W is orthogonal, simple solution:Fitted function obtained from inverse wavelet transform:
39 Wavelets: Key IdeaIn general, any basis could be used, such as the smoothing splines we’ve seen before.The key difference is that wavelets allow localization in time as well as frequency (roughness), and along with the L1 penalty allow sparse solutionsSmoothing splines compress by imposing smoothness; Wavelets compress by imposing sparsity
Your consent to our cookies if you continue to use this website.