1. Matthew Brown University of British Columbia (prev.) Microsoft Research [ Collaborators: † Simon Winder, *Gang Hua, † Rick Szeliski † =MS Research, *=MS Live Labs]

2.  Panoramic Stitching Digital Image Pro, Windows Live Photogallery, Expression, HDView  3D Modelling Photosynth  Virtual Earth Location Recognition  Image Search Lincoln [ yellow = product, white = technology preview, grey = research ]

3. [ http://labs.live.com/photosynth ]

4. [ Slide credit: Noah Snavely] Scene reconstruction Photo Explorer Input photographs Relative camera positions and orientations Point cloud Sparse correspondence [ http://photour.cs.washington.edu ]  Photosynth is based on Photo Tourism [Snavely, Seitz, Szeliski SIGGRAPH 2006 ]  Photo Tourism uses SIFT for correspondence

5. [ Seitz et al CVPR 2006, Goesele et al ICCV 2007 ]

6. [ Photo Tourism – Snavely, Seitz, Szeliski - SIGGRAPH 2006 ] 3D Point Cloud

7. [ Photo Tourism – Snavely, Seitz, Szeliski - SIGGRAPH 2006 ] 3D Point Cloud

8. 3D Point Cloud [ Photo Tourism – Snavely, Seitz, Szeliski - SIGGRAPH 2006 ]

9. 3D Point Cloud [ Photo Tourism – Snavely, Seitz, Szeliski - SIGGRAPH 2006 ]

11.  † = for simplicity + efficiency  * = measured by ROC curve  Q: Form of the descriptor function f(.)? Find a function of a local image patch descriptor = f ( ) s.t. a nearest neighbour classifier † is optimal*

12. Algorithm Normalized Image Patch Descriptor Vector Gradients Quantized to k Orientations Normalize Summation [ SIFT – Lowe ICCV 1999 ]

13. Algorithm Normalized Image Patch Descriptor Vector Gradients Quantized to k Orientations Normalize (plus PCA) Summation [ GLOH – Mikolajzcyk Schmid PAMI 2005 ]

14. Algorithm Normalized Image Patch Descriptor Vector Create Edge Map Normalize Summation [ Shape Context – Belongie Malik Puzicha NIPS 2000 ]

15. Algorithm Normalized Image Patch Descriptor Vector Feature Detector Normalize Summation TSN [ Geometric Blur – Berg Malik CVPR 2001 ]

16. Normalized Image Patch Descriptor Vector TSN Parameters  Propose a framework for descriptor algorithms  Learn parameters to find best performance  Train on a ground truth data set based on accurate 3D matches

17. Normalized Image Patch (w x h) Descriptor Vector TSN  Transformation block Local gradients Steerable filters Isotropic filters  Haar wavelets  Local classifier  Quantized intensities (w x h x k)  Output: one length k vector per source pixel

18. Normalized Image Patch (w x h) Descriptor Vector SNT (w x h x k) (m x k)  Spatial summation block with m regions  Output: m length k vectors S1 S2 S3 S4

19. Normalized Image Patch (w x h) Descriptor Vector SNT (w x h x k) (m x k)  Normalization Block Unit normalization SIFT normalization with clipping

20. STN

21. S2T1aN2 Parameters Training Pairs Incorrect Match % Correct Match % Update Parameters (Powell) Descriptor Distances

22. S2T1aN2 Parameters Test Pairs Incorrect Match % Correct Match % Final Error Rate Descriptor Distances 95%

24.  Polar lattice S2 always has lower error rate than rectangular S1  Gradient and DOG with S2 beat our SIFT reference (4% vs 6% error)

26.  Steerable filters produce great results if phase information is kept

28.  SIFT normalization is important  Best result: 4 th order steerable filters with phase information combined with polar S4-25 Gaussian summation block (2% error vs SIFT at 6%)  Very large numbers of dimensions

29. w PCA

30. w LDA

33.  LDA on pixels ≈ SIFT (6%)  PCA gave small improvement  Normalised patches Gradient patches

34. Effect of # of Training Pairs  LDA on pixels ≈ SIFT (6%)  PCA gave small improvement  Need ~100,000 training examples

35.  LDA on T1-T3 < 4.5%  Optimal #dimensions ~20-30  Post-normalisation important T1 T3

36. LDA using T blocks T1–T4  LDA on T1-T3 < 4.5%  Optimal #dimensions ~20-30  Post-normalisation important

37. LDA using CVPR 07 descriptors  Overall best results  #dimensions reduced from 100’s to 10’s  Need more challenging dataset!

38. Algorithm Normalized Image Patch Descriptor Vector Feature Detector Normalize Summation TSN “complex” “simple”

39.  Used learning to obtain good descriptors  Achieved error rates 1/3 of SIFT  Produced useful ground truth data set  Future Work  Use multi-view stereo ground truth  Multi-level simple-complex architecture  + non-parametric T blocks  Learn interest point detectors [ refs: 1) Winder, Brown CVPR 2007 2) Hua, Brown, Winder ICCV 2007 ] mbrown@cs.ubc.ca

41. [http://research.microsoft.com/ivm/hdview.htm ]