1. Recognising Panoramas
M. Brown and D. Lowe,
University of British Columbia

2. Introduction Are you getting the whole picture?
Compact Camera FOV = 50 x 35°

3. Introduction Are you getting the whole picture?
Compact Camera FOV = 50 x 35°
Human FOV = 200 x 135°

4. Introduction Are you getting the whole picture?
Compact Camera FOV = 50 x 35°
Human FOV = 200 x 135°
Panoramic Mosaic = 360 x 180°

5. Why “Recognising Panoramas”?

6. Why “Recognising Panoramas”?
1D Rotations ()
Ordering  matching images

7. Why “Recognising Panoramas”?
1D Rotations ()
Ordering  matching images

8. Why “Recognising Panoramas”?
1D Rotations ()
Ordering  matching images

9. Why “Recognising Panoramas”?
1D Rotations ()
Ordering  matching images
2D Rotations (, )
Ordering  matching images

10. Why “Recognising Panoramas”?
1D Rotations ()
Ordering  matching images
2D Rotations (, )
Ordering  matching images

11. Why “Recognising Panoramas”?
1D Rotations ()
Ordering  matching images
2D Rotations (, )
Ordering  matching images

12. Why “Recognising Panoramas”?

13. Overview Feature Matching Image Matching Bundle Adjustment
Multi-band Blending
Results
Conclusions

14. Bundle Adjustment and Multi-band Blending
Outline
Bundle Adjustment and Multi-band Blending
Image Matching
Feature Extraction
Input Images

15. Overview Feature Matching Image Matching Bundle Adjustment
Multi-band Blending
Results
Conclusions

16. Overview Feature Matching Image Matching Bundle Adjustment
SIFT Features
Nearest Neighbour Matching
Image Matching
Bundle Adjustment
Multi-band Blending
Results
Conclusions

17. Overview Feature Matching Image Matching Bundle Adjustment
SIFT Features
Nearest Neighbour Matching
Image Matching
Bundle Adjustment
Multi-band Blending
Results
Conclusions

18. Invariant Features
Schmid & Mohr 1997, Lowe 1999, Baumberg 2000, Tuytelaars & Van Gool 2000, Mikolajczyk & Schmid 2001, Brown & Lowe 2002, Matas et. al. 2002, Schaffalitzky & Zisserman 2002

19. SIFT Features – Location
Extrema of DoG
Discard Low Contrast
Discard Edge Points
Output: x,y,σ
Picture Credit:

20. SIFT Features – Orientation
Difference of Gaussian Function:
G(x, y, kσ) = Gaussian Kernel with SD = kσ
L(x, y, kσ) = I(x, y) * G(x, y, kσ)
D(x, y, σ) = L(x, y, kiσ) – L(x, y, kjσ)
Gradient Magnitude:
m(x,y) = √{ (L(x + 1, y) – L(x - 1, y))2 + (L(x, y + 1) – L(x, y - 1))2 }
Θ(x, y) = tan-1{ (L(x, y + 1) – L(x, y - 1)) / (L(x + 1, y) – L(x - 1, y)) }

21. SIFT Features – Orientation
Local Gradients
Adjusted Gradients
Gaussian Weighting
Orientation Histogram – 36 bins with 10 degrees each

22. SIFT Features – Orientation
Local Gradients
Adjusted Gradients
Gaussian Weighting
Orientation Histogram – 36 bins with 10 degrees each θ

23. SIFT Features – Descriptor Vector
4 x 4 region around point oriented according to θ
Take 8 versions of the gradient image corresponding to 8 different directions.
Each version has only gradients that fall in corresponding direction range
Concatenate the 8 versions (each 4 x 4) together to get 128 dimensional descriptor vector

24. SIFT Features Invariant Features SIFT features are…
Establish invariant frame
Maxima/minima of scale-space DOG  x, y, s
Maximum of distribution of local gradients  
Form descriptor vector
Histogram of smoothed local gradients
128 dimensions
SIFT features are…
Geometrically invariant to similarity transforms,
some robustness to affine change
Photometrically invariant to affine changes in intensity

25. Overview Feature Matching Image Matching Bundle Adjustment
SIFT Features
Nearest Neighbour Matching
Image Matching
Bundle Adjustment
Multi-band Blending
Results
Conclusions

26. Nearest Neighbour Matching
Find k-NN for each feature
k  number of overlapping images (we use k = 4)
Use k-d tree
k-d tree recursively bi-partitions data at mean in the dimension of maximum variance
Approximate nearest neighbours found in O(nlogn)

27. Overview Feature Matching Image Matching Bundle Adjustment
SIFT Features
Nearest Neighbour Matching
Image Matching
Bundle Adjustment
Multi-band Blending
Results
Conclusions

28. Overview Feature Matching Image Matching Bundle Adjustment
Multi-band Blending
Results
Conclusions

29. Overview Feature Matching Image Matching Bundle Adjustment
Multi-band Blending
Results
Conclusions

30. Overview Feature Matching Image Matching Bundle Adjustment
RANSAC for Homography
Probabilistic model for verification
Bundle Adjustment
Multi-band Blending
Results
Conclusions

31. Overview Feature Matching Image Matching Bundle Adjustment
RANSAC for Homography
Probabilistic model for verification
Bundle Adjustment
Multi-band Blending
Results
Conclusions

32. Homography Defn – Homography:
A one-to-one mapping between two images. In computer vision, it is typically used to describe the correspondence between two images taken of the same scene from different camera angles.
For Images:
Homography has four parameters, θ1 θ2 θ3 f, corresponding to the three angles of camera rotation and the focal length.
Picture Credit:

33. RANSAC (RANdom SAmple Consensus)
bestH = 0% success
While i < max iterations:
- Select a random sample of the data
- currentH = solve for the homography using sample data
- for each pair of points, test to see if currentH is a suitable solution
- if more points work with currentH than with bestH
set bestH = currentH

34. RANSAC for Homography

35. RANSAC for Homography

36. RANSAC for Homography

37. Overview Feature Matching Image Matching Bundle Adjustment
RANSAC for Homography
Probabilistic model for verification
Bundle Adjustment
Multi-band Blending
Results
Conclusions

38. Probabilistic model for verification

39. Probabilistic model for verification
Want to solve for the probability that the image match is correct given the set of RANSAC inliers and outliers.
p( match | inliers )
Using Bayes’ Rule:
p( match | inliers ) = p( inliers | match ) * p( match ) / p( inliers )
This can be simplified if we make some assumptions
p( inliers | match ) = B( ni; nf, p1)
p( inliers | ~match ) = B( ni; nf, p0)
(B is a binomial distribution function)

40. Probabilistic model for verification
Compare probability that this set of RANSAC inliers/outliers was generated by a correct/false image match
ni = #inliers, nf = #features
p1 = p(inlier | match), p0 = p(inlier | ~match)
pmin = acceptance probability
Choosing values for p1, p0 and pmin

41. Finding the panoramas

42. Finding the panoramas

43. Finding the panoramas

44. Finding the panoramas

45. Overview Feature Matching Image Matching Bundle Adjustment
RANSAC for Homography
Probabilistic model for verification
Bundle Adjustment
Multi-band Blending
Results
Conclusions

46. Overview Feature Matching Image Matching Bundle Adjustment
Multi-band Blending
Results
Conclusions

47. Overview Feature Matching Image Matching Bundle Adjustment
Multi-band Blending
Results
Conclusions

48. Bundle Adjustment Overlay the images together one at a time.
For each image, solve for the camera parameters θ1 θ2 θ3 f to align it with the current compilation using Levenberg-Marquardt to minimize an error function.

49. Overview Feature Matching Image Matching Bundle Adjustment
Error function
Multi-band Blending
Results
Conclusions

50. Overview Feature Matching Image Matching Bundle Adjustment
Error function
Multi-band Blending
Results
Conclusions

51. Error function Sum of squared projection errors Robust error function
n = #images
I(i) = set of image matches to image i
F(i, j) = set of feature matches between images i,j
rijk = residual of kth feature match between images i,j
Robust error function

52. Homography for Rotation
Parameterise each camera by rotation and focal length
This gives pairwise homographies

53. Bundle Adjustment
New images initialised with rotation, focal length of best matching image

54. Bundle Adjustment
New images initialised with rotation, focal length of best matching image

55. Overview Feature Matching Image Matching Bundle Adjustment
Error function
Multi-band Blending
Results
Conclusions

56. Overview Feature Matching Image Matching Bundle Adjustment
Multi-band Blending
Results
Conclusions

57. Overview Feature Matching Image Matching Bundle Adjustment
Multi-band Blending
Results
Conclusions

58. Multi-band Blending Burt & Adelson 1983
Blend frequency bands over range  

59. 2-band Blending Low frequency ( > 2 pixels)
High frequency ( < 2 pixels)

60. Linear Blending

61. 2-band Blending

64. Overview Feature Matching Image Matching Bundle Adjustment
Multi-band Blending
Results
Conclusions

65. Overview Feature Matching Image Matching Bundle Adjustment
Multi-band Blending
Results
Conclusions

66. Results

67. Results

68. Overview Feature Matching Image Matching Bundle Adjustment
Multi-band Blending
Results
Conclusions

69. Overview Feature Matching Image Matching Bundle Adjustment
Multi-band Blending
Results
Conclusions

70. Conclusions Fully automatic panoramas Invariant feature based method
A recognition problem…
Invariant feature based method
SIFT features, RANSAC, Bundle Adjustment, Multi- band Blending
O(nlogn)
Future Work
Advanced camera modelling
radial distortion, camera motion, scene motion, vignetting, exposure, high dynamic range, flash …
Full 3D case – recognising 3D objects/scenes in unordered datasets

71. Questions?

72. Analytical computation of derivatives

73. Levenberg-Marquardt
Iteration step of form