1. A New Correspondence Algorithm Jitendra Malik Computer Science Division University of California, Berkeley Joint work with Serge Belongie, Jan Puzicha, Alex Berg

2. Key contributions: Years 1-4 The FAÇADE system for semi- automated modeling of architectural scenes The FAÇADE system for semi- automated modeling of architectural scenes High dynamic range image acquisition High dynamic range image acquisition Image based lighting Image based lighting Inverse global illumination for recovering reflectance and lighting properties Inverse global illumination for recovering reflectance and lighting properties Segmented objects from range images Segmented objects from range images

3. Contributors Paul Debevec, now at ICT Paul Debevec, now at ICT George Borshukov, recipient of Technical Achievement Award 2001 with colleagues at Manex visual effects George Borshukov, recipient of Technical Achievement Award 2001 with colleagues at Manex visual effects Yizhou Yu, Asst. Prof., UIUC Yizhou Yu, Asst. Prof., UIUC

4. What remains? High quality automated correspondence is essential High quality automated correspondence is essential 3D Structure recovery algorithms need to scale up 3D Structure recovery algorithms need to scale up Geometric and reflectance properties need to be modeled for a much larger range of scenes than previously considered Geometric and reflectance properties need to be modeled for a much larger range of scenes than previously considered

5. Towards better correspondence Humans use contextual information much more effectively than current algorithms. Humans use contextual information much more effectively than current algorithms. Features are not robust to changes in viewpoint. Features are not robust to changes in viewpoint.

6. How big a window?

7. The solution to the dilemma. Large windows capture more context but suffer from increased distortion. Large windows capture more context but suffer from increased distortion. Goal: Design a similarity measure which can tolerate affine distortion. Goal: Design a similarity measure which can tolerate affine distortion. Similarity should decrease linearly with the amount of distortion. Similarity should decrease linearly with the amount of distortion. Cross correlation does not have this property Cross correlation does not have this property

8. An example Solution is to blur the signals, but how exactly?

9. Blurring the right way

10. Affine Robustness Condition The similarity function s(f, f T) should be close to a linear function L of the amount of distortion  (T). We can obtain an s that satisfies this condition: Where B is a bounded distortion blur…

11. Affine Robust Feature The bounded distortion blur of a signal f is the Affine Robust Feature B(f). Constructively B is a linear mapping with: And we take 012 012

12. In 2d Six oriented filters, half-wave rectified to provide12 channels Six oriented filters, half-wave rectified to provide12 channels Bounded distortion blur applied to each channel Bounded distortion blur applied to each channel Similarity is the sum of similarities in each channel computed separately Similarity is the sum of similarities in each channel computed separately

13. Bounded Distortion Blur in 2D

14. Comparing three techniques

15. Another example… Given points in one image, find corresponding points.

16. Find correspondences between points on shape Find correspondences between points on shape Estimate transformation Estimate transformation Measure similarity Measure similarity modeltarget... Another application: Matching shapes

17. Shape Context Count the number of points inside each bin, e.g.: Count = 4 Count = 10... FCompact representation of distribution of points relative to each point

18. Hand-written Digit Recognition MNIST 60 000: MNIST 60 000: linear: 12.0% linear: 12.0% 40 PCA+ quad: 3.3% 40 PCA+ quad: 3.3% 1000 RBF +linear: 3.6% 1000 RBF +linear: 3.6% K-NN: 5% K-NN: 5% K-NN (deskewed) : 2.4% K-NN (deskewed) : 2.4% K-NN (tangent dist.) : 1.1% K-NN (tangent dist.) : 1.1% SVM: 1.1% SVM: 1.1% LeNet 5: 0.95% LeNet 5: 0.95% MNIST 600 000 (distortions): MNIST 600 000 (distortions): LeNet 5: 0.8% LeNet 5: 0.8% SVM: 0.8% SVM: 0.8% Boosted LeNet 4: 0.7% Boosted LeNet 4: 0.7% MNIST 20 000 MNIST 20 000 K-NN, Shape context matching: 0.63 % K-NN, Shape context matching: 0.63 %

19. Conclusion A new image descriptor which is robust to affine image deformations A new image descriptor which is robust to affine image deformations Preliminary results suggest that this could result in a considerable improvement in quality of correspondence for long baseline multiple view analysis. Preliminary results suggest that this could result in a considerable improvement in quality of correspondence for long baseline multiple view analysis.

20. Plans for next 6 months Combine the use of the affine robust window features with the use of epipolar constraints and probabilistic matching. Combine the use of the affine robust window features with the use of epipolar constraints and probabilistic matching. Test technique on stereo and motion imagery. Test technique on stereo and motion imagery. Explore this in the context of an end to end system for scene reconstruction. Explore this in the context of an end to end system for scene reconstruction.