1. Image Stitching Computer Vision CS 678
Some slides from Richard Szeliski

2. Panoramic Image Mosaics
Full screen panoramas (cubic):
Mars:
2003 New Years Eve:
Richard Szeliski
Image Stitching

3. Gigapixel panoramas & images
Mapping / Tourism / WWT
Medical Imaging
Richard Szeliski
Image Stitching

4. Image Mosaics
… + =
Goal: Stitch together several images into a seamless composite
Richard Szeliski
Image Stitching

5. Today’s lecture Image alignment and stitching motion models
image warping
point-based alignment
complete mosaics (global alignment)
Richard Szeliski
Image Stitching

6. Motion models

7. Motion models
What happens when we take two images with a camera and try to align them?
translation?
rotation?
scale?
affine?
Perspective?
Richard Szeliski
Image Stitching

8. Image Warping

9. Image Warping image filtering: change range of image g(x) = h(f(x))
image warping: change domain of image
g(x) = f(h(x)) f x f x h f x f x h
Richard Szeliski
Image Stitching

10. Image Warping image filtering: change range of image g(x) = h(f(x))
image warping: change domain of image
g(x) = f(h(x)) f g h f g h
Richard Szeliski
Image Stitching

11. Parametric (global) warping
Examples of parametric warps:
aspect
translation
rotation
perspective
affine
cylindrical
Richard Szeliski
Image Stitching

12. 2D coordinate transformations
translation: x’ = x + t x = (x,y)
rotation: x’ = R x + t
similarity: x’ = s R x + t
affine: x’ = A x + t
perspective: x’  H x x = (x,y,1) (x is a homogeneous coordinate)
These all form a nested group (closed w/ inv.)
Richard Szeliski
Image Stitching

13. Image Warping
Given a coordinate transform x’ = h(x) and a source image f(x), how do we compute a transformed image g(x’) = f(h(x))?
h(x) x x’
f(x)
g(x’)
Richard Szeliski
Image Stitching

14. Forward Warping
Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’)
What if pixel lands “between” two pixels?
h(x) x x’
f(x)
g(x’)
Richard Szeliski
Image Stitching

15. Forward Warping
Send each pixel f(x) to its corresponding location x’ = h(x) in g(x’)
What if pixel lands “between” two pixels?
Answer: add “contribution” to several pixels, normalize later (splatting)
h(x) x x’
f(x)
g(x’)
Richard Szeliski
Image Stitching

16. Inverse Warping
Get each pixel g(x’) from its corresponding location x’ = h(x) in f(x)
What if pixel comes from “between” two pixels?
h(x) x x’
f(x)
g(x’)
Richard Szeliski
Image Stitching

17. Inverse Warping
Get each pixel g(x’) from its corresponding location x’ = h(x) in f(x)
What if pixel comes from “between” two pixels?
Answer: resample color value from interpolated (prefiltered) source image x x’
f(x)
g(x’)
Richard Szeliski
Image Stitching

18. Interpolation Possible interpolation filters:
nearest neighbor
bilinear
bicubic (interpolating)
sinc / FIR
Needed to prevent “jaggies” and “texture crawl”
Richard Szeliski
Image Stitching

19. Motion models (reprise)

20. Motion models Translation 2 unknowns Affine 6 unknowns Perspective
3D rotation
3 unknowns
Richard Szeliski
Image Stitching

21. Plane perspective mosaics
8-parameter generalization of affine motion
works for pure rotation or planar surfaces
Limitations:
local minima
slow convergence
difficult to control interactively
Richard Szeliski
Image Stitching

22. Image warping with homographies
image plane in front
image plane below
Richard Szeliski
Image Stitching

23. Image Mosaics (Stitching)
[Szeliski & Shum, SIGGRAPH’97]
[Szeliski, FnT CVCG, 2006]

24. Image Mosaics (stitching)
Blend together several overlapping images into one seamless mosaic (composite) … + =
Richard Szeliski
Image Stitching

25. Mosaics for Video Coding
Convert masked images into a background sprite for content-based coding =
Richard Szeliski
Image Stitching

26. Establishing correspondences
Direct method:
Use generalization of affine motion model [Szeliski & Shum ’97]
Feature-based method
Extract features, match, find consisten inliers [Lowe ICCV’99; Schmid ICCV’98, Brown&Lowe ICCV’2003]
Compute R from correspondences (absolute orientation)
Richard Szeliski
Image Stitching

27. Panoramas What if you want a 360 field of view?
mosaic Projection Cylinder
Richard Szeliski
Image Stitching

28. Cylindrical panoramas
Steps
Reproject each image onto a cylinder
Blend
Output the resulting mosaic
Richard Szeliski
Image Stitching

29. Cylindrical Panoramas
Map image to cylindrical or spherical coordinates
need known focal length
Image 384x300
f = 180 (pixels)
f = 280
f = 380
Therefore, it is desirable if the global motion model we need to recover is translation, which has only two parameters. It turns out that for a pure panning motion, if we convert two images to their cylindrical maps with known focal length, the relationship between them can be represented by a translation.
Here is an example of how cylindrical maps are constructed with differently with f’s. Note how straight lines are curved.
Similarly, we can map an image to its longitude/latitude spherical coordinates as well if f is given.
Richard Szeliski
Image Stitching

30. Image Stitching Align the images over each other
camera pan ↔ translation on cylinder
Blend the images together
Richard Szeliski
Image Stitching

31. image stitching Take pictures on a tripod (or handheld)
Warp images to spherical coordinates
Extract features
Align neighboring pairs using RANSAC
Write out list of neighboring translations
Correct for drift
Read in warped images and blend them
Crop the result and import into a viewer
Richard Szeliski
Image Stitching

32. What do we do about the “bad” matches?
Matching features
What do we do about the “bad” matches?
Richard Szeliski
Image Stitching

33. RAndom SAmple Consensus
Select one match, count inliers
Richard Szeliski
Image Stitching

34. RAndom SAmple Consensus
Select one match, count inliers
Richard Szeliski
Image Stitching

35. Find “average” translation vector
Least squares fit
Find “average” translation vector
Richard Szeliski
Image Stitching

36. Assembling the panorama
Stitch pairs together, blend, then crop
Richard Szeliski
Image Stitching

37. Problem: Drift Error accumulation
small (vertical) errors accumulate over time
apply correction so that sum = 0 (for 360° pan.)
Richard Szeliski
Image Stitching

38. Full-view (360° spherical) panoramas

39. Full-view Panorama + + + +
Richard Szeliski
Image Stitching

40. Texture Mapped Model
Richard Szeliski
Image Stitching

41. Global alignment Register all pairwise overlapping images
Use a 3D rotation model (one R per image)
Use direct alignment (patch centers) or feature based
Infer overlaps based on previous matches (incremental)
Optionally discover which images overlap other images using feature selection (RANSAC)
Richard Szeliski
Image Stitching

42. Image Blending

43. Image feathering
Weight each image proportional to its distance from the edge (distance map [Danielsson, CVGIP 1980]
1. Generate weight map for each image
2. Sum up all of the weights and divide by sum: weights sum up to 1: wi’ = wi / ( ∑i wi)
Richard Szeliski
Image Stitching

44. Image Feathering
Richard Szeliski
Image Stitching

45. Feathering 1 + =
Richard Szeliski
Image Stitching

46. Effect of window size 1
left 1
right
Richard Szeliski
Image Stitching

47. Effect of window size 1 1
Richard Szeliski
Image Stitching

48. Good window size “Optimal” window: smooth but not ghosted1
“Optimal” window: smooth but not ghosted
Doesn’t always work...
Richard Szeliski
Image Stitching

49. Pyramid Blending
Burt, P. J. and Adelson, E. H., A multiresolution spline with applications to image mosaics, ACM Transactions on Graphics, 42(4), October 1983,
Richard Szeliski
Image Stitching

50. Laplacian level 4 Laplacian level 2 Laplacian level left pyramid
Richard Szeliski
Image Stitching
left pyramid
right pyramid
blended pyramid

51. Laplacian image blend Compute Laplacian pyramid
Compute Gaussian pyramid on weight image (can put this in A channel)
Blend Laplacians using Gaussian blurred weights
Reconstruct the final image
Q: How do we compute the original weights?
A: For horizontal panorama, use mid-lines
Q: How about for a general “3D” panorama?
Richard Szeliski
Image Stitching

52. Poisson Image Editing
Blend the gradients of the two images, then integrate
For more info: Perez et al, SIGGRAPH 2003
Richard Szeliski
Image Stitching