Download presentation

Presentation is loading. Please wait.

Published byQuinton Wyke Modified over 2 years ago

1
Richard G. Baraniuk Chinmay Hegde Manifold Learning in the Wild A New Manifold Modeling and Learning Framework for Image Ensembles Aswin C. Sankaranarayanan Rice University

2
Sensor Data Deluge

3
Internet Scale Databases Tremendous size of corpus of available data –Google Image Search of “Notre Dame Cathedral” yields 3m results 3Tb of data

4
Concise Models Efficient processing / compression requires concise representation Our interest in this talk: Collections of images

5
Concise Models Our interest in this talk: Collections of image parameterized by \in –translations of an object : x-offset and y-offset –rotations of a 3D object pitch, roll, yaw –wedgelets : orientation and offset

6
Concise Models Our interest in this talk: Collections of image parameterized by \in –translations of an object : x-offset and y-offset –rotations of a 3D object pitch, roll, yaw –wedgelets : orientation and offset Image articulation manifold

7
Image Articulation Manifold N-pixel images: K-dimensional articulation space Then is a K-dimensional manifold in the ambient space Very concise model –Can be learnt using Non-linear dim. reduction articulation parameter space

8
Ex: Manifold Learning LLE ISOMAP LE HE Diff. Geo … K=1 rotation

9
Ex: Manifold Learning K=2 rotation and scale

10
Smooth IAMs N-pixel images: Local isometry image distance parameter space distance Linear tangent spaces are close approximation locally Low dimensional articulation space articulation parameter space

11
Smooth IAMs articulation parameter space N-pixel images: Local isometry image distance parameter space distance Linear tangent spaces are close approximation locally Low dimensional articulation space

12
Smooth IAMs articulation parameter space N-pixel images: Local isometry image distance parameter space distance Linear tangent spaces are close approximation locally Low dimensional articulation space

13
Theory/Practice Disconnect Smoothness Practical image manifolds are not smooth! If images have sharp edges, then manifold is everywhere non-differentiable [Donoho and Grimes] Tangent approximations ?

14
Theory/Practice Disconnect Smoothness Practical image manifolds are not smooth! If images have sharp edges, then manifold is everywhere non-differentiable [Donoho and Grimes] Tangent approximations ?

15
Failure of Tangent Plane Approx. Ex: cross-fading when synthesizing / interpolating images that should lie on manifold Input Image Geodesic Linear path

16
Ex:translation manifold all blue images are equidistant from the red image Local isometry –satisfied only when sampling is dense Theory/Practice Disconnect Isometry

17
Theory/Practice Disconnect Nuisance articulations Unsupervised data, invariably, has additional undesired articulations –Illumination –Background clutter, occlusions, … Image ensemble is no longer low-dimensional

18
Image representations Conventional representation for an image –A vector of pixels –Inadequate! pixel image

19
Image representations Replace vector of pixels with an abstract bag of features –Ex: SIFT (Scale Invariant Feature Transform) selects keypoint locations in an image and computes keypoint descriptors for each keypoint –Very popular in many many vision problems

20
Features (including SIFT) ubiquitous in fusion and processing apps (15k+ cites for 2 SIFT papers) SIFT Features building 3D models part-based object recognition organizing internet-scale databases image stitching Figures courtesy Rob Fergus (NYU), Phototourism website, Antonio Torralba (MIT), and Wei Lu

21
Image representations Replace vector of pixels with an abstract bag of features –Ex: SIFT (Scale Invariant Feature Transform) selects keypoint locations in an image and computes keypoint descriptors for each keypoint –Keypoint descriptors are local; it is very easy to make them robust to nuisance imaging parameters

22
Loss of Geometrical Info Bag of features representations hide potentially useful image geometry Goal: make salient image geometrical info more explicit for exploitation Image space Keypoint space

23
Key idea Keypoint space can be endowed with a rich low-dimensional structure in many situations

24
Key idea Keypoint space can be endowed with a rich low-dimensional structure in many situations Mechanism: define kernels, between keypoint locations, keypoint descriptors

25
Keypoint Kernel Keypoint space can be endowed with a rich low-dimensional structure in many situations Mechanism: define kernels, between keypoint locations, keypoint descriptors Joint keypoint kernel between two images is given by

26
Many Possible Kernels Euclidean kernel Gaussian kernel Polynomial kernel Pyramid match kernel [Grauman et al. ’07] Many others

27
Keypoint Kernel Joint keypoint kernel between two images is given by Using Euclidean/Gaussian (E/G) combination yields

28
From Kernel to Metric Lemma: The E/G keypoint kernel is a Mercer kernel –enables algorithms such as SVM Lemma: The E/G keypoint kernel induces a metric on the space of images –alternative to conventional L 2 distance between images –keypoint metric robust to nuisance imaging parameters, occlusion, clutter, etc.

29
Keypoint Geometry Theorem: Under the metric induced by the kernel certain ensembles of articulating images form smooth, isometric manifolds Keypoint representation compact, efficient, and … Robust to illumination variations, non-stationary backgrounds, clutter, occlusions

30
Keypoint Geometry Theorem: Under the metric induced by the kernel certain ensembles of articulating images form smooth, isometric manifolds In contrast: conventional approach to image fusion via image articulation manifolds (IAMs) fraught with non-differentiability (due to sharp image edges) –not smooth –not isometric

31
Application: Manifold Learning 2D Translation

32
Application: Manifold Learning 2D Translation IAM KAM

33
Manifold Learning in the Wild Rice University’s Duncan Hall Lobby –158 images –360° panorama using handheld camera –Varying brightness, clutter

34
Duncan Hall Lobby Ground truth using state of the art structure-from-motion software Manifold Learning in the Wild Ground truthIAMKAM

35
Manifold Learning in the Wild Viewing angle – 179 images IAM KAM

36
Manifold Learning in the Wild Rice University’s Brochstein Pavilion –400 outdoor images of a building –occlusions, movement in foreground, varying background

37
Manifold Learning in the Wild Brochstein Pavilion –400 outdoor images of a building –occlusions, movement in foreground, background IAMKAM

38
Internet scale imagery Notre-dame cathedral –738 images –Collected from Flickr –Large variations in illumination (night/day/saturations), clutter (people, decorations), camera parameters (focal length, fov, …) –Non-uniform sampling of the space

39
Organization k-nearest neighbors

40
Organization “geodesics’ 3D rotation “Walk-closer” “zoom-out”

41
Summary Challenges for manifold learning in the wild are both theoretical and practical Need for novel image representations –Sparse features Robustness to outliers, nuisance articulations, etc. Learning in the wild: unsupervised imagery Promise lies in fast methods that exploit only neighborhood properties –No complex optimization required

Similar presentations

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google