Dimensionality Reduction Part 2: Nonlinear Methods Comp 790-090 Spring 2007

Why Dimensionality Reduction Two approaches to reduce number of features Feature selection: select the salient features by some criteria Feature extraction: obtain a reduced set of features by a transformation of all features Data visualization and exploratory data analysis also need to reduce dimension Usually reduce to 2D or 3D

Deficiencies of Linear Methods Data may not be best summarized by linear combination of features Example: PCA cannot discover 1D structure of a helix

Intuition: how does your brain store these pictures?

Brain Representation

Brain Representation Every pixel? Or perceptually meaningful structure? Up-down pose Left-right pose Lighting direction So, your brain successfully reduced the high-dimensional inputs to an intrinsically 3-dimensional manifold!

Manifold Learning Y X latent observed Discover low dimensional representations (smooth manifold) for data in high dimension. Linear approaches(PCA, MDS) Non-linear approaches (ISOMAP, LLE, others)

Linear Approach- PCA PCA Finds subspace linear projections of input data.

Linear Method Linear Methods for Dimensionality Reduction PCA: rotate data so that principal axes lie in direction of maximum variance MDS: find coordinates that best preserve pairwise distances PCA

Motivation Linear Dimensionality Reduction doesn’t always work Data violates underlying “linear” assumptions Data is not accurately modeled by “affine” combinations of measurements Structure of data, while apparent, is not simple In the end, linear methods do nothing more than “globally transform” (rate, translate, and scale) all of the data, sometime what’s needed is to “unwrap” the data first

Stopgap Remedies Local PCA Neural Networks Compute PCA models for small overlapping item neighborhoods Requires a clustering preprocess Fast and simple, but results in no global parameterization Neural Networks Assumes a solution of a given dimension Uses relaxation methods to deform given solution to find a better fit Relaxation step is modeled as “layers” in a network where properties of future iterations are computed based on information from the current structure Many successes, but a bit of an art

Why Linear Modeling Fails Suppose that your sample data lies on some low-dimensional surface embedded within the high-dimensional measurement space. Linear models allow ALL affine combinations Often, certain combinations are atypical of the actual data Recognizing this is harder as dimensionality increases

What does PCA Really Model? Principle Component Analysis assumptions Mean-centered distribution What if the mean, itself is atypical? Eigenvectors of Covariance Basis vectors aligned with successive directions of greatest variance Classic 1st Order statistical model Distribution is characterized by its mean and variance (Gaussian Hyperspheres)

Non-Linear Dimensionality Reduction Non-linear Manifold Learning Instead of preserving global pairwise distances, non-linear dimensionality reduction tries to preserve only the geometric properties of local neighborhoods Discover a lower-dimensional “embedding” manifold Find a parameterization over that manifold Linear parameter space Projection mapping from original M-D space to d-D embedding space “reprojection, elevating, or lifting” “projection” Linear Embedding Space

Nonlinear DimRedux Steps Discover a low-dimensional embedding manifold Find a parameterization over the manifold Project data into parameter space Analyze, interpolate, and compress in embedding space Orient (by linear transformation) the parameter space to align axes with salient features Linear (affine) combinations are valid here In the case of interpolation and compression use “lifting” to estimate M-D original data

Nonlinear Methods Local Linear Embeddings [Roweis 2000] Isomaps [Tenenbaum 2000] These two papers ignited the field Principled approach (Asymptotically, as the amount of data goes to infinity they have been proven to find the “real” manifold) Widely applied Hotly contested

Nonlinear Approaches- Isomap Josh. Tenenbaum, Vin de Silva, John langford 2000 Constructing neighbourhood graph G For each pair of points in G, Computing shortest path distances ---- geodesic distances. Use Classical MDS with geodesic distances. Euclidean distance Geodesic distance

Sample points with Swiss Roll Altogether there are 20,000 points in the “Swiss roll” data set. We sample 1000 out of 20,000.

Construct neighborhood graph G K- nearest neighborhood (K=7) DG is 1000 by 1000 (Euclidean) distance matrix of two neighbors (figure A)

Compute all-points shortest path in G Now DG is 1000 by 1000 geodesic distance matrix of two arbitrary points along the manifold (figure B)

Use MDS to embed graph in Rd Find a d-dimensional Euclidean space Y (Figure c) to preserve the pariwise diatances.

Isomap Small Euclidean distance Key Observation: On a manifold distances are measured using geodesic distances rather than Euclidean distances Large geodesic distance

Problem: How to Get Geodesics Without knowledge of the manifold it is difficult to compute the geodesic distance between points It is even difficult if you know the manifold Solution Use a discrete geodesic approximation Apply a graph algorithm to approximate the geodesic distances

Dijkstra’s Algorithm Efficient Solution to all-points-shortest path problem Greedy breath-first algorithm

Dijkstra’s Algorithm Efficient Solution to all-points-shortest path problem Greedy breath-first algorithm

Dijkstra’s Algorithm Efficient Solution to all-points-shortest path problem Greedy breath-first algorithm

Dijkstra’s Algorithm Efficient Solution to all-points-shortest path problem Greedy breath-first algorithm

Isomap algorithm Compute fully-connected neighborhood of points for each item Can be k nearest neighbors or ε-ball Neighborhoods must be symmetric Test that resulting graph is fully-connected, if not increase either K or  Calculate pairwise Euclidean distances within each neighborhood Use Dijkstra’s Algorithm to compute shortest path from each point to non-neighboring points Run MDS on resulting distance matrix

Isomap Results Find a 2D embedding of the 3D S-curve (also shown for LLE) Isomap does a good job of preserving metric structure (not surprising) The affine structure is also well preserved

Residual Fitting Error

Neighborhood Graph

More Isomap Results

More Isomap Results

Isomap Failures Isomap also has problems on closed manifolds of arbitrary topology

Non-Linear Example A Data-Driven Reflectance Model (Matusik et al, Siggraph2003) Bidirectional Reflectance Distribution Functions(BRDF) Define ratio of the reflected radiance in a particular direction to the incident irradiance from direction. Isotropic BRDF

Measurement Modeling Bidirectional Reflectance Distribution Functions (BRDFs)

Measurement A “fast” BRDF measurement device inspired by Marshner[1998]

Measurement 20-80 million reflectance measurements per material Each tabulated BRDF entails 90x90x180x3=4,374,000 measurement bins

Measurement 20-80 million reflectance measurements per material Each tabulated BRDF entails 90x90x180x3=4,374,000 measurement bins

Rendering from Tabulated BRDFs Even without further analysis, our BRDFs are immediately useful Renderings made with Henrik Wann Jensen’s Dali renderer Nickel Hematite Gold Paint Pink Felt

BRDFs as Vectors in High-Dimensional Space Each tabulated BRDF is a vector in 90x90x180x3 =4,374,000 dimensional space 180 Unroll 90 90 4,374,000

Linear Analysis (PCA) Find optimal “linear basis” for our data set Eigenvalue magnitude Find optimal “linear basis” for our data set 45 components needed to reduce residue to under measurement error 20 40 60 80 100 120 Dimension mean 5 10 20 30 45 60 all

Problems with Linear Subspace Modeling Large number of basis vectors (45) Some linear combinations yield invalid or unlikely BRDFs (outside convex hull)

Problems with Linear Subspace Modeling Large number of basis vectors (45) Some linear combinations yield invalid or unlikely BRDFs (inside convex hull)

Results of Non-Linear Manifold Learning At 15 dimensions reconstruction error is less than 1% Parameter count similar to analytical models Error 5 10 15 Dimensionality

Non-Linear Advantages 15-dimensional parameter space More robust than linear model More extrapolations are plausible Linear Model Extrapolation Non-linear Model Extrapolation

Non-Linear Model Results

Non-Linear Model Results

Non-Linear Model Results

Representing Physical Processes Steel Oxidation

Local Linear Embeddings First Insight Locally, at a fine enough scale, everything looks linear

Local Linear Embeddings First Insight Find an affine combination the “neighborhood” about a point that best approximates it

Finding a Good Neighborhood This is the remaining “Art” aspect of nonlinear methods Common choices -ball: find all items that lie within an epsilon ball of the target item as measured under some metric Best if density of items is high and every point has a sufficient number of neighbors K-nearest neighbors: find the k-closest neighbors to a point under some metric Guarantees all items are similarly represented, limits dimension to K-1

Characterictics of a Manifold x1 x2 R2 Rn M x1 x2 R2 Rn z x x: coordinate for z Locally it is a linear patch Key: how to combine all local patches together?

LLE: Intuition Assumption: manifold is approximately “linear” when viewed locally, that is, in a small neighborhood Approximation error, e(W), can be made small Local neighborhood is effected by the constraint Wij=0 if zi is not a neighbor of zj A good projection should preserve this local geometric property as much as possible

LLE: Intuition We expect each data point and its neighbors to lie on or close to a locally linear patch of the manifold. Each point can be written as a linear combination of its neighbors. The weights chosen to minimize the reconstruction Error.

LLE: Intuition The weights that minimize the reconstruction errors are invariant to rotation, rescaling and translation of the data points. Invariance to translation is enforced by adding the constraint that the weights sum to one. The weights characterize the intrinsic geometric properties of each neighborhood. The same weights that reconstruct the data points in D dimensions should reconstruct it in the manifold in d dimensions. Local geometry is preserved

LLE: Intuition Use the same weights from the original space Low-dimensional embedding the i-th row of W Use the same weights from the original space

Local Linear Embedding (LLE) Assumption: manifold is approximately “linear” when viewed locally, that is, in a small neighborhood Approximation error, e(W), can be made small Meaning of W: a linear representation of every data point by its neighbors This is an intrinsic geometrical property of the manifold A good projection should preserve this geometric property as much as possible

Constrained Least Square problem Compute the optimal weight for each point individually: Neightbors of x Zero for all non-neighbors of x

Finding a Map to a Lower Dimensional Space Yi in Rk: projected vector for Xi The geometrical property is best preserved if the error below is small Y is given by the eigenvectors of the lowest d non-zero eigenvalues of the matrix Use the same weights computed above

Find Weights Rewriting as a matrix for all x Reorganizing Want to find W that minimizes , and satisfies “sum-to-one” constraint Ends up as constrained “least-squares” problem “Unknown W matrix” N N N M M N

Find Linear Embedding Space Now that we have the weight matrix W, find the linear vector that satisfies the following where W is N x N and X is M x N This can be found by finding the null space of Classic problem: run SVD on and find the orthogonal vector associated with the smallest d singular values (the smallest singular value will be zero and represent the system’s invariance to translation)

Numerical Issues Numerical problems can arise in computing LLEs The least-squared covariance matrix that arises in the computation of the weighting matrix, W, solution can be ill-conditioned Regularization (rescale the measurements by adding a small multiple of the Identity to covariance matrix) Finding small singular (eigen) values is not as well conditioned as finding large ones. The small ones are subject to numerical precision errors, and to get mixed Good (but slow) solvers exist, you have to use them

Results The resulting parameter vector, yi, gives the coordinates associated with the item xi The dth embedding coordinate is formed from the orthogonal vector associated with the dst singular value of A.

Reprojection Often, for data analysis, a parameterization is enough For interpolation and compression we might want to map points from the parameter space back to the “original” space No perfect solution, but a few approximations Delauney triangulate the points in the embedding space, find the triangle that the desired parameter setting falls into, and compute the baricenric coordinates of it, and use them as weights Interpolate by using a radially symmetric kernel centered about the desired parameter setting Works, but mappings might not be one-to-one

LLE Example 3-D S-Curve manifold with points color-coded Compute a 2-D embedding The local affine structure is well maintained The metric structure is okay locally, but can drift slowly over the domain (this causes the manifold to taper)

More LLE Examples

More LLE Examples

LLE Failures Does not work on to closed manifolds Cannot recognize Topology

Summary Non-Linear Dimensionality Reduction Methods Comparisons Issues These methods are considerably more powerful and temperamental than linear method Applications of these methods are a hot area of research Comparisons LLE is generally faster, but more brittle than Isomaps Isomaps tends to work better on smaller data sets (i.e. less dense sampling) Isomaps tends to be less sensitive to noise (perturbation of the input vectors) Issues Neither method handles closed manifolds and topological variations well