1. Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Richard G. Baraniuk Rice University

2. Aswin Sankaranarayanan Chinmay Hegde Sriram Nagaraj Go With The Flow A New Manifold Modeling and Learning Framework for Image Ensembles Richard G. Baraniuk Rice University

3. Sensor Data Deluge

6. Concise Models Efficient processing / compression requires concise representation Sparsity of an individual image pixels large wavelet coefficients (blue = 0)

7. Concise Models Efficient processing / compression requires concise representation Our interest in this talk: Collections of images

8. Concise Models Our interest in this talk:Collections of images parameterized by  2  2  –translations of an object : x-offset and y-offset –rotations of a 3D object pitch, roll, yaw –wedgelets : orientation and offset

9. Concise Models Our interest in this talk:Collections of images parameterized by  2  2  –translations of an object : x-offset and y-offset –rotations of a 3D object pitch, roll, yaw –wedgelets : orientation and offset Image articulation manifold

10. Image Articulation Manifold In practice: N-pixel images: In theory: K-dimensional articulation space Then is a K-dimensional manifold in the ambient space Very concise model articulation parameter space

11. Smooth IAMs In practice: N-pixel images: In theory: If images are smooth then so is Local isometry: image distance parameter space distance Linear tangent spaces are close approximation locally articulation parameter space

12. Smooth IAMs In practice: N-pixel images: In theory: If images are smooth then so is Local isometry: image distance parameter space distance Linear tangent spaces are close approximation locally articulation parameter space

13. Ex: Manifold Learning LLE ISOMAP HE … K=1 rotation

14. Ex: Manifold Learning K=2 rotation and scale

15. Theory/Practice Disconnect: Smoothness Practical image manifolds are not smooth! If images have sharp edges, then manifold is everywhere non-differentiable [Donoho, Grimes] Local isometry Local tangent approximations

16. Theory/Practice Disconnect: Smoothness Practical image manifolds are not smooth! If images have sharp edges, then manifold is everywhere non-differentiable [Donoho, Grimes] Local isometry Local tangent approximations

17. Failure of Local Isometry Ex:translation manifold Local isometry all blue images are equidistant from the red image

18. Failure of Tangent Plane Approx. Ex: cross-fading when synthesizing / interpolating images that should lie on manifold Input Image Geodesic Linear path

19. Image Articulation Manifold Linear tangent space at is K-dimensional –provides a mechanism to transport along manifold –problem: since manifold is non-differentiable, tangent approximation is poor Our goal: replace tangent space with new transport operator that respects the nonlinearity of the imaging process Tangent space at Articulations IAM

20. Optical Flow Transport Brightness constancy: Given two images I 1 and I 2, we seek a displacement vector field f(x, y) = [u(x, y), v(x, y)] such that Ex: Linearized brightness constancy (LBC)

21. History of Optical Flow Dark ages (<1985): special cases solved –LBC an under-determined set of linear equations Horn and Schunk (1985): –Regularization term: smoothness prior on the flow Brox et al (2005) –shows that linearization of brightness constancy is a bad assumption –develops optimization framework to handle BC directly Brox et al (2010), Black et al (2010), Liu et al (2010) –practical systems with reliable code

22. Optical Flow Manifold IAM OFM at Articulations Consider a reference image and a K-dimensional articulation Collect optical flows from to all images reachable by a K-dimensional articulation Provides a mechanism to transport along manifold

23. Optical Flow Manifold IAM OFM at Articulations Consider a reference image and a K-dimensional articulation Collect optical flows from to all images reachable by a K-dimensional articulation Provides a mechanism to transport along manifold Theorem: Collection of OFs is a smooth, K-dimensional manifold (even if IAM is not smooth) [N,S,H,B,2010]

24. Flow Metric Replace intensity-based image distance –root cause of IAM non-differentiability … with distance along the OF field from I 1 to I 2 Enables differential geometric tools –ex: vector fields, curvature, parallel transport, … curve c IAM Flow Field V t

25. OFM is Smooth (Translation) Euclidean distance between image intensities Flow metric between images (globally linear)

26. OFM is Smooth (Rotation) Articulation θ in [ ⁰ ] Intensity I(θ) Op. flow v(θ) Euclidean distance between image intensities Flow metric between images (nearly linear)

27. The Story So Far… IAM OFM at Articulations flow metric Euclidean metric Tangent space at Articulations IAM

28. Input Image Geodesic Linear path IAM OFM

29. OFM Synthesis

30. ISOMAP embedding error for OFM and IAM 2D rotations Reference image Manifold Learning

31. Embedding of OFM 2D rotations Reference image

32. Data 196 images of two bears moving linearly and independently IAM OFM Task Find low-dimensional embedding OFM Manifold Learning

33. IAM OFM Data 196 images of a cup moving on a plane Task 1 Find low-dimensional embedding Task 2 Parameter estimation for new images (tracing an “R”) OFM ML + Parameter Estimation

34. Point on the manifold such that the sum of geodesic distances to every other point is minimized Important concept in nonlinear data modeling, compression, shape analysis [Srivastava et al] Karcher Mean 10 images from an IAM ground truth KM OFM KM linear KM

35. Goal: build a generative model for an entire IAM/OFM based on a small number of base images El Cheapo TM algorithm: –choose a reference image randomly –find all images that can be generated from this image by OF –compute Karcher (geodesic) mean of these images –compute OF from Karcher mean image to other images –repeat on the remaining images until no images remain Exact representation when no occlusions Manifold Charting

36. IAM Goal: build a generative model for an entire IAM/OFM based on a small number of base images Ex:cube rotating about axis All cube images can be representing using 4 reference images + OFMs Many applications –selection of target templates for classification –“next-view” selection for adaptive sensing applications

37. Summary IAMs a useful concise model for many image processing problems involving image collections and multiple sensors/viewpoints But practical IAMs are non-differentiable –IAM-based algorithms have not lived up to their promise Optical flow manifolds (OFMs) –smooth even when IAM is not –OFM ~ nonlinear tangent space –support accurate image synthesis, learning, charting, … Barely discussed here: OF enables the safe extension of differential geometry concepts –Log/Exp maps, Karcher mean, parallel transport, …

38. Related Work Analytic transport operators –transport operator has group structure [Xiao and Rao 07][Culpepper and Olshausen 09] [Miller and Younes 01] [Tuzel et al 08] –non-linear analytics [Dollar et al 06] –spatio-temporal manifolds [Li and Chellappa 10] –shape manifolds [Klassen et al 04] Analytic approach limited to a small class of standard image transformations (ex: affine transformations, Lie groups) In contrast, OFM approach works reliably with real-world image samples (point clouds) and broader class of transformations

39. Open Questions Our treatment is specific to image manifolds What are the natural transport operators for other data manifolds? dsp.rice.edu

41. Open Questions Theorem: random measurements stably embed a K-dim manifold whp [B, Wakin, FOCM ’08] Q: Is there an analogous result for OFMs?