Nonlinear Dimension Reduction: Semi-Definite Embedding vs. Local Linear Embedding Li Zhang and Lin Liao.

Slides:

Advertisements

Similar presentations

Text mining Gergely Kótyuk Laboratory of Cryptography and System Security (CrySyS) Budapest University of Technology and Economics

Advertisements

Principal Component Analysis Based on L1-Norm Maximization Nojun Kwak IEEE Transactions on Pattern Analysis and Machine Intelligence, 2008.

Image classification Given the bag-of-features representations of images from different classes, how do we learn a model for distinguishing them?

Local Discriminative Distance Metrics and Their Real World Applications Local Discriminative Distance Metrics and Their Real World Applications Yang Mu,

A Geometric Perspective on Machine Learning 何晓飞浙江大学计算机学院 1.

Input Space versus Feature Space in Kernel- Based Methods Scholkopf, Mika, Burges, Knirsch, Muller, Ratsch, Smola presented by: Joe Drish Department of.

Graph Embedding and Extensions: A General Framework for Dimensionality Reduction Keywords: Dimensionality reduction, manifold learning, subspace learning,

Manifold Learning Dimensionality Reduction. Outline Introduction Dim. Reduction Manifold Isomap Overall procedure Approximating geodesic dist. Dijkstra’s.

Presented by: Mingyuan Zhou Duke University, ECE April 3, 2009

1 A Survey on Distance Metric Learning (Part 1) Gerry Tesauro IBM T.J.Watson Research Center.

Graph Laplacian Regularization for Large-Scale Semidefinite Programming Kilian Weinberger et al. NIPS 2006 presented by Aggeliki Tsoli.

A novel supervised feature extraction and classification framework for land cover recognition of the off-land scenario Yan Cui

Non-linear Dimensionality Reduction CMPUT 466/551 Nilanjan Ray Prepared on materials from the book Non-linear dimensionality reduction By Lee and Verleysen,

University of Joensuu Dept. of Computer Science P.O. Box 111 FIN Joensuu Tel fax Isomap Algorithm.

Principal Component Analysis CMPUT 466/551 Nilanjan Ray.

“Random Projections on Smooth Manifolds” -A short summary

ETHEM ALPAYDIN © The MIT Press, Lecture Slides for.

Dimensionality reduction. Outline From distances to points : – MultiDimensional Scaling (MDS) – FastMap Dimensionality Reductions or data projections.

LLE and ISOMAP Analysis of Robot Images Rong Xu. Background Intuition of Dimensionality Reduction Linear Approach –PCA(Principal Component Analysis) Nonlinear.

The Terms that You Have to Know! Basis, Linear independent, Orthogonal Column space, Row space, Rank Linear combination Linear transformation Inner product.

Three Algorithms for Nonlinear Dimensionality Reduction Haixuan Yang Group Meeting Jan. 011, 2005.

Lecture outline Support vector machines. Support Vector Machines Find a linear hyperplane (decision boundary) that will separate the data.

Distance Metric Learning: A Comprehensive Survey

Proceedings of the 2007 SIAM International Conference on Data Mining.

A Global Geometric Framework for Nonlinear Dimensionality Reduction Joshua B. Tenenbaum, Vin de Silva, John C. Langford Presented by Napat Triroj.

Continuous Latent Variables --Bishop

Atul Singh Junior Undergraduate CSE, IIT Kanpur.  Dimension reduction is a technique which is used to represent a high dimensional data in a more compact.

Dimensionality Reduction

Lightseminar: Learned Representation in AI An Introduction to Locally Linear Embedding Lawrence K. Saul Sam T. Roweis presented by Chan-Su Lee.

Dimensionality Reduction. Multimedia DBs Many multimedia applications require efficient indexing in high-dimensions (time-series, images and videos, etc)

Nonlinear Dimensionality Reduction by Locally Linear Embedding Sam T. Roweis and Lawrence K. Saul Reference: "Nonlinear dimensionality reduction by locally.

Nonlinear Dimensionality Reduction Approaches. Dimensionality Reduction The goal: The meaningful low-dimensional structures hidden in their high-dimensional.

Manifold learning: Locally Linear Embedding Jieping Ye Department of Computer Science and Engineering Arizona State University

Summarized by Soo-Jin Kim

Structure Preserving Embedding Blake Shaw, Tony Jebara ICML 2009 (Best Student Paper nominee) Presented by Feng Chen.

1 Graph Embedding (GE) & Marginal Fisher Analysis (MFA) 吳沛勳劉冠成韓仁智

Feature extraction 1.Introduction 2.T-test 3.Signal Noise Ratio (SNR) 4.Linear Correlation Coefficient (LCC) 5.Principle component analysis (PCA) 6.Linear.

Graph Embedding: A General Framework for Dimensionality Reduction Dong XU School of Computer Engineering Nanyang Technological University

1 Recognition by Appearance Appearance-based recognition is a competing paradigm to features and alignment. No features are extracted! Images are represented.

IEEE TRANSSACTIONS ON PATTERN ANALYSIS AND MACHINE INTELLIGENCE

Learning a Kernel Matrix for Nonlinear Dimensionality Reduction By K. Weinberger, F. Sha, and L. Saul Presented by Michael Barnathan.

Computer Vision Lab. SNU Young Ki Baik Nonlinear Dimensionality Reduction Approach (ISOMAP, LLE)

ISOMAP TRACKING WITH PARTICLE FILTER Presented by Nikhil Rane.

GRASP Learning a Kernel Matrix for Nonlinear Dimensionality Reduction Kilian Q. Weinberger, Fei Sha and Lawrence K. Saul ICML’04 Department of Computer.

Spoken Language Group Chinese Information Processing Lab. Institute of Information Science Academia Sinica, Taipei, Taiwan

Manifold learning: MDS and Isomap

Nonlinear Dimensionality Reduction Approach (ISOMAP)

Jan Kamenický.  Many features ⇒ many dimensions  Dimensionality reduction ◦ Feature extraction (useful representation) ◦ Classification ◦ Visualization.

Non-Linear Dimensionality Reduction

EE4-62 MLCV Lecture Face Recognition – Subspace/Manifold Learning Tae-Kyun Kim 1 EE4-62 MLCV.

Tony Jebara, Columbia University Advanced Machine Learning & Perception Instructor: Tony Jebara.

Supervisor: Nakhmani Arie Semester: Winter 2007 Target Recognition Harmatz Isca.

Math 285 Project Diffusion Maps Xiaoyan Chong Department of Mathematics and Statistics San Jose State University.

CSC321: Lecture 25: Non-linear dimensionality reduction Geoffrey Hinton.

CSC321: Extra Lecture (not on the exam) Non-linear dimensionality reduction Geoffrey Hinton.

Out of sample extension of PCA, Kernel PCA, and MDS WILSON A. FLORERO-SALINAS DAN LI MATH 285, FALL

Multi-index Evaluation Algorithm Based on Locally Linear Embedding for the Node importance in Complex Networks Fang Hu

Machine Learning Supervised Learning Classification and Regression K-Nearest Neighbor Classification Fisher’s Criteria & Linear Discriminant Analysis Perceptron:

Spectral Methods for Dimensionality

Nonlinear Dimensionality Reduction

Dimensionality Reduction

Dimensionality Reduction

Outline Nonlinear Dimension Reduction Brief introduction Isomap LLE

Singular Value Decomposition

Object Modeling with Layers

CS 2750: Machine Learning Support Vector Machines

Principal Component Analysis

Introduction PCA (Principal Component Analysis) Characteristics:

Video Analysis via Nonlinear Dimensionality Reduction Technique

Nonlinear Dimension Reduction:

Presentation transcript:

Nonlinear Dimension Reduction: Semi-Definite Embedding vs. Local Linear Embedding Li Zhang and Lin Liao

Outline Nonlinear Dimension Reduction Semi-Definite Embedding Local Linear Embedding Experiments

Dimension Reduction To understand images in terms of their basic modes of variability. Unsupervised learning problem: Given N high dimensional input X i R D, find a faithful one-to-one mapping to N low dimensional output Y i R d and d<D. Methods: Linear methods (PCA, MDS): subspace Nonlinear methods (SDE, LLE): manifold

Semi-Definite Embedding Given input X=(X 1,...,X N ) and k Find the k nearest neighbors for each input X i Formulate and solve a corresponding semi-definite programming problem; find optimal Gram matrix of output K=Y T Y Extract approximately a low dimensional embedding Y from the eigenvectors and eigenvalues of Gram matrix K

Semi-Definite Programming Maximize C·X Subject to AX=b matrix(X) is positive semi-definite where X is a vector with size n 2, and matrix(X) is a n by n matrix reshaped from X

Semi-Definite Programming Constraints: Maintain the distance between neighbors |Y i -Y j | 2= |X i -X j | 2 for each pair of neighbor (i,j)  K ii +K jj -K ij -K ji = G ii +G jj -G ij -G ji where K=Y T Y,G=X T X Constrain the output centered on the origin ΣY i =0  ΣK ij =0 K is positive semidefinite

Semi-Definite Programming Objective function Maximize the sum of pairwise squared distance between outputs Σ ij |Y i -Y j | 2  Tr(K)

Semi-Definite Programming Solve the best K using any SDP solver CSDP (fast, stable) SeDuMi (stable, slow) SDPT3 (new, fastest, not well tested)

Locally Linear Embedding

Swiss Roll N=800 SDE, k=4 LLE, k=18

LLE on Swiss Roll, varying K K=5K=6 K=8K=10

LLE on Swiss Roll, varying K K=12K=14 K=16K=18

LLE on Swiss Roll, varying K K=20K=30 K=40K=60

Twos N=638 SDE, k=4 LLE, k=18

Teapots N=400 SDE, k=4 LLE, k=12

LLE on Teapot, varying N N=400N=200 N=100N=50

Faces N=1900 SDE, failed LLE, k=12

SDE versus LLE Similar idea First, compute neighborhoods in the input space Second, construct a square matrix to characterize local relationship between input data. Finally, compute low-dimension embedding using the eigenvectors of the matrix

SDE versus LLE Different performance SDE: good quality, more robust to sparse samples, but optimization is slow and hard to scale to large data set LLE: fast, scalable to large data set, but low quality when samples are sparse, due to locally linear assumption