Enhancing Tensor Subspace Learning by Element Rearrangement Dong XU School of Computer Engineering Nanyang Technological University http://www.ntu.edu.sg/home/dongxu dongxu@ntu.edu.sg

Outline Summary of our recent works on Tensor (or Bilinear) Subspace Learning Element Rearrangement for Tensor (or Bilinear) Subspace Learning

What is Tensor? Tensors are arrays of numbers which transform in certain ways under coordinate transformations. Vector Matrix 3rd-order Tensor Bilinear (or 2D) Subspace Learning: each image is represented as a 2nd-order tensor (i.e., a matrix) Tensor Subspace Learning (more general case): each image is represented as a higher order tensor

Definition of Mode-k Product Original Tensor Projection: high-dimensional space -> low-dimensional space Reconstruction: low-dimensional space -> high-dimensional space Product for two Matrices Projection Matrix 2 = New Tensor Notation: Original Matrix Projection Matrix New Matrix

Definition of Mode-k Flattening Matrix Tensor Potential Assumption in Previous Tensor-based Subspace Learning: Intra-tensor correlations: Correlations along column vectors of mode-k flattened matrices.

Data Representation in Dimensionality Reduction Vector Matrix 3rd-order Tensor Gray-level Image Filtered Image Video Sequence High Dimension Low Dimension PCA, LDA Rank-1 Decomposition, 2001 Shashua and A. Levin, Our Work Xu et al., 2005 Yan et al., 2005 … Examples Tensorface, 2002 M. Vasilescu and D. Terzopoulos,

What is Gabor Features? … Gabor features can improve recognition performance in comparison to grayscale features. Chengjun Liu T-IP, 2002 Five Scales … Input: Grayscale Image Eight Orientations Output: 40 Gabor-filtered Images Gabor Wavelet Kernels

Why Represent Image Objects as Tensors instead of Vectors? Natural Representation Gray-level Images (2D structure) Videos (3D structure) Gabor-filtered Images (3D structure) Enhance Learnability in Real Application Curse of Dimensionality (Gabor-filtered image: 100*100*40 -> Vector: 400,000) Small sample size problem (less than 5,000 images in common face databases) Reduce Computation Cost

Concurrent Subspace Analysis (CSA) as an Example (Criterion: Optimal Reconstruction) Dimensionality Reduction Reconstruction Input sample Sample in Low- dimensional space The reconstructed sample Objective Function: Projection Matrices? D. Xu, S. Yan, Lei Zhang, H. Zhang et al., CVPR 2005 and T-CSVT 2008

Tensorization - New Research Direction: Other Related Works Discriminant Analysis with Tensor Representation (DATER): CVPR 2005 and T-IP 2007 Coupled Kernel-based Subspace Learning (CKDA): CVPR 2005 Rank-one Projections with Adaptive Margins (RPAM): CVPR 2006 and T-SMC-B 2007 Enhancing Tensor Subspace Learning by Element Rearrangement: CVPR 2007 and T-PAMI 2009 Discriminant Locally Linear Embedding with High Order Tensor Data (DLLE/T): T-SMC-B 2008 Convergent 2D Subspace Learning with Null Space Analysis (NS2DLDA): T-CSVT 2008 Semi-supervised Bilinear Subspace Learning: T-IP 2009 Applications in Human Gait Recognition CSA+DATER: T-CSVT 2006 Tensor Marginal Fisher Analysis (TMFA): T-IP 2007 Note: Other researchers also published several papers along this direction!!!

Tensorization - New Research Direction Tensorization in Graph Embedding Framework Google Citation: 174 (until 15-Sep-2009) Direct Graph Embedding Linearization Kernelization Original PCA & LDA, ISOMAP, LLE, Laplacian Eigenmap PCA, LDA, LPP, MFA KPCA, KDA, KMFA Tensorization Type Formulation CSA, DATER, TMFA Example S. Yan, D. Xu, H. Zhang et al., CVPR, 2005 and T-PAMI,2007

Element Rearrangement: Motivations The success of tensor-based subspace learning relies on the redundancy among the unfolded vector However, such correlation/redundancy is usually not strong for real data Our Solution: Element rearrangement is employed as a preprocessing step to increase the intra-tensor correlations for existing tensor subspace learning methods

Motivations-Continued Intra-tensor correlations: Correlations among the features within certain tensor dimensions, such as rows, columns and Gabor features… Sets of highly correlated pixels Columns of highly Element Rearrangement Low correlation High correlation

Problem Definition The task of enhancing correlation/redundancy among 2nd–order tensor is to search for a pixel rearrangement operator R, such that 1. is the rearranged matrix from sample 2. The column numbers of U and V are predefined After the pixel rearrangement, we can use the rearranged tensors as input for tensor subspace learning algorithms!

Solution to Pixel Rearrangement Problem Initialize U0, V0 Compute reconstructed matrices n=n+1 Optimize U and V Optimize operator R

Step for Optimizing R It is Integer Programming problem Sender Original matrix q Receiver Reconstructed matrix Linear programming problem in Earth Mover’s Distance (EMD) has integer solution. We constrain the rearrangement within spatially local neighborhood or feature-based neighborhood for speedup.

Convergence Speed

Rearrangement Results

Reconstruction Visualization

Reconstruction Visualization

Classification Accuracy

Summary . Our papers published in CVPR 2005 are the first works to address dimensionality reduction with the image objects represented as high-order tensors of arbitrary order. Our papers published in CVPR 2005 opens a new research direction. We also published a series of works along this direction. Element arrangement can further improve data compression performance and classification accuracy.