1. Computational Photography Image Warping/Morphing Jinxiang Chai

2. Review Image Processing - Median filtering - Bilateral filtering - Edge detection - Corner detection

3. Review: Basic Edge Detection INPUT IMAGE 1) Noise Smoothing EDGE IMAGE 2) Edge Enhancement Horizontal [-1 0 1] Vertical [-1 0 1] T “GRADIENT” IMAGE 3)Threshold

4. Review: Corner Detection 1. Compute image gradients 2. Construct the matrix from it and its neighborhood values 3. Determine the 2 eigenvalues λ(i.j)= [λ 1, λ 2 ]. 4. If both λ 1 and λ 2 are big, we have a corner

5. Image Gradients Closeup of image orientation at each pixel

6. The Orientation Field Corners are detected where both λ1 and λ2 are big

7. Outline Image warping Image morphing

8. Required Readings Section 3.6 (Szeliski book) Image morphing paperImage morphing

9. Image Warping Image filtering: change range of image g(x) = h(f(x)) f x h f x f x h f x Image warping: change domain of image g(x) = f(h(x))

10. Image Warping Image filtering: change range of image g(x) = h(f(x)) Image warping: change domain of image g(x) = f(h(x)) h fg h f g

11. Image Warping Why? - texture mapping - image processing (rotation, zoom in/out, etc) - image morphing/blending - image editing - image based-modeling & rendering

12. Parametric (global) Warping Examples of image warps: translation rotation aspect affineperspective cylindrical

13. Transformation Function f, g Transform the geometry of an image to a desired geometry

14. Definition: Image Warping Source Image: Image to be used as the reference. The geometry of this image is no changed Target Image: this image is obtained by transforming the reference image. (x,y): coordinates of points in the reference image (u,v): coordinates of points in the target image f,g or F,G: x and y components of a transformation function

15. Definition: Image Warping Control points: Unique points in the reference and target images. The coordinates of corresponding control points in images are used to determine a transformation function. Source ImageTarget Image

16. A Transformation Function Used to compute the corresponding points Source Image S(x,y)Target Image T(u,v) u = f(x,y) v = g(x.y) x = F(u,v) y = G(x.v)

17. Warping Types Simple mappings: - Similarity - Affine mapping - Projective mapping These can be applied globally over a subdivision of the plane: - Piecewise affine over triangulation - Piecewise projective over quadrilaterization - Piecewise bilinear over a rectangular grid Or other, arbitrary functions can be used, e.g. - Bieer-neely warp (popular for morphs) - Store u(x,y) and v(x,y) in large arrays

18. Similarity Transform A combination of 2-D scale, rotation, and translation transformations. Allows a square to be transformed into any rotated rectangle. Angle between lines is preserved 5 degrees of freedom (s x,s y,θ,t x,t y ) Inverse is of same form (is also similarity). Given by inverse of 3X3 matrix above Have the form: In matrix notation:

19. Affine Transform A combination of 2-D scale, rotation, shear, and translation transformations. Allows a square to be distorted into any parallelogram. 6 degrees of freedom (a,b,c,d,e,f) Inverse is of same form (is also affine). Given by inverse of 3X3 matrix above Good when controlling a warp with triangles, since 3 points in 2D determined the 6 degrees of freedom Have the form: In matrix notation:

20. Projective Transform (a.k.a “perspective”) Linear numerator & denominator If g=h=0, then you get affine as a special case Allow a square to be distorted into any quadrilateral 8 degrees of freedom (a-h). We can choose i=1, arbitrarily Inverse is of same form (is also projective). Good when controlling a warp with quadrilaterals, since 4 points in 2D determine the 8 degrees of freedom Have the form: In matrix notation:

21. Image Warping x y u v Given a coordinate transform function f,g or F,G and source image S(x,y), how do we compute a transformed image T(u,v)? S(x,y) T(u,v)

22. Forward Warping x y u v S(x,y) T(u,v) Forward warping algorithm: for y = y min to y max for x = x min to x max u = f(x,y); v = g(x,y) copy pixel at source S(x,y) to T(u,v)

23. Forward Warping x y u v S(x,y) T(u,v) Forward warping algorithm: for y = y min to y max for x = x min to x max u = f(x,y); v = g(x,y) copy pixel at source S(x,y) to T(u,v) - Any problems for forward warping?

24. Forward Warping x y u v S(x,y) T(u,v) Q: What if the transformed pixel located between pixels?

25. Forward Warping x y u v S(x,y) T(u,v) Q: What if the transformed pixel located between pixels? A: Distribute color among neighboring pixels - known as “splatting”

26. Forward Warping Iterate over source, sending pixels to destination Some source pixels maps to the same dest. pixel Some dest. pixels may have no corresponding source Holes in reconstruction Must splat etc. for y = y min to y max for x = x min to x max u = f(x,y); v = g(x,y) copy pixel at source S(x,y) to T(u,v) x y u v

27. Forward Warping Iterate over source, sending pixels to destination Some source pixels map to the same dest. pixel Some dest. pixels may have no corresponding source Holes in reconstruction Must splat etc. for y = y min to y max for x = x min to x max u = f(x,y); v = g(x,y) copy pixel at source S(x,y) to T(u,v) x y u v - How to remove the holes?

28. Forward Warping Iterate over source, sending pixels to destination Some source pixels map to the same dest. pixel Some dest. pixels may have no corresponding source Holes in reconstruction Must splat etc. for y = y min to y max for x = x min to x max u = f(x,y); v = g(x,y) copy pixel at source S(x,y) to T(u,v) x y u v - How to remove the holes?

29. Inverse Warping x y u v S(x,y) T(u,v) Inverse warping algorithm: for v = v min to v max for u = u min to u max x = F(u,v); y = G(u,v) copy pixel at source S(x,y) to T(u,v)

30. Inverse Warping x y u v S(x,y)T(u,v) Q: What if pixel comes from “between” two pixels? A: Interpolate color values from neighboring pixels

31. Inverse Warping Iterate over dest., finding pixels from source Non-integer evaluation source image, resample May oversample source But no holes Simpler, better than forward mapping for v = v min to v max for u = u min to u max x = F(u,v); y = G(u,v) copy pixel at source S(x,y) to T(u,v) x y u v

32. Resampling Filter

33. Resampling x y u v This is a 2D signal reconstruction problem!

34. Review: Signal Reconstruction in Freq. Domain T2T … -2T-T … 0 x f s (x) f(x) x F s (u) u -f max f max F(u) u -f max f max Inverse Fourier transform Fourier transform

35. Review: Signal Reconstruction in Spatial Domain T2T … -2T-T … 0 x x f s (x) sinc(x)

36. Resampling Compute weighted sum of pixel neighborhood - Weights are normalized values of kernel function - Equivalent to convolution at samples with kernel - Find good filters using ideas of previous lectures x y u v

37. Point Sampling Nearest neighbor x y u v - Copies the color of the pixel with the closest integer coordinate - A fast and efficient way to process textures if the size of the target is similar to the size of the reference - Otherwise, the result can be a chunky, aliased, or blurred image.

38. Bilinear Filter Weighted sum of four neighboring pixels x y u v

39. Bilinear Filter Sampling at S(x,y): (i+1,j) (i,j) (i,j+1) (i+1,j+1) S(x,y) = (1-a)*(1-b)*S(i,j) + a*(1-b)*S(i+1,j) + (1-a)*b*S(i,j+1) + a*b*S(i+1,j+1) u v y x

40. Bilinear Filter Sampling at S(x,y): (i+1,j) (i,j) (i,j+1) (i+1,j+1) S(x,y) = a*b*S(i,j) + a*(1-b)*S(i+1,j) + (1-a)*b*S(i,j+1) + (1-a)*(1-b)*S(i+1,j+1) S i = S(i,j) + a*(S(i,j+1)-S(i)) S j = S(i+1,j) + a*(S(i+1,j+1)-S(i+1,j)) S(x,y) = S i +b*(S j -S i) To optimize the above, do the following u v y x

41. Bilinear Filter (i+1,j) (i,j) (i,j+1) (i+1,j+1) y x

42. Inverse Warping and Resampling Inverse warping algorithm: for v = v min to v max for u = u min to u max float x = F(u,v); float y = G(u,v); T(u,v) = resample_souce(x,y,w); x y u v (u,v) (x,y)

43. Outline Image warping Image morphing

44. Morphing = Object Averaging The aim is to find “an average” between two objects –Not an average of two images of objects… –…but an image of the average object! –How can we make a smooth transition in time? Do a “weighted average” over time t How do we know what the average object looks like? –We haven’t a clue! –But we can often fake something reasonable Usually required user/artist input

45. Averaging Points P Q v = Q - P P + 0.5v = P + 0.5(Q – P) = 0.5P + 0.5 Q P + 1.5v = P + 1.5(Q – P) = -0.5P + 1.5 Q (extrapolation) Linear Interpolation (Affine Combination): New point aP + bQ, defined only when a+b = 1 So aP+bQ = aP+(1-a)Q P and Q can be anything : –points on a plane (2D) or in space (3D) –Colors in RGB or HSV (3D) –Whole images (m-by-n D)… etc. What’s the average of P and Q?

46. Idea #1: Cross-Dissolve Interpolate whole images: - Imagehalfway = (1-t)*Image1 + t*image2 This is called cross-dissolve in film industry But what if the images are not aligned?

47. Dog Averaging What to do? –Cross-dissolve doesn’t work –Any ideas?

48. Dog Averaging What to do? –Cross-dissolve doesn’t work –Any ideas? Feature matching! –Nose to nose, tail to tail, etc. This is a local (non-parametric) warp

49. Idea #2: Local Warping Morphing procedure: for every t, 1. Find the average shape (the “mean dog” ) - local warping 2. Find the average color - Cross-dissolve the warped images

50. Local (non-parametric) Image Warping Need to specify a more detailed warp function –Global warps were functions of a few (2,4,8) parameters –Non-parametric warps u(x,y) and v(x,y) can be defined independently for every single location x,y! –Once we know vector field u,v we can easily warp each pixel (use backward warping with interpolation) –Will it work for these dogs? –Probably not… Need user control.

51. Warp Specification How can we specify the warp? –Specify corresponding points interpolate to a complete warping function How do we do it?

52. Finding Transformation Function Generally this is a 2D scattered data interpolation problem - radial basis function, etc

53. Finding Transformation Function Generally this is a 2D scattered data interpolation problem - radial basis function, etc Piecewise affine transformation

54. Finding Transformation Function Generally this is a 2D scattered data interpolation problem - radial basis function, etc Piecewise affine transformation

55. Finding Transformation Function Generally this is a 2D scattered data interpolation problem - radial basis function, etc Piecewise affine transformation

56. Triangular Mesh Input correspondences at key feature points Define a triangular mesh over the points (Delaunay triangulation) –Same mesh in both images! –Now we have triangle-to-triangle correspondences Warp each triangle separately from source to destination –How do we warp a triangle? –3 points = affine warp! –Just like texture mapping

57. Warp Specification How can we specify the warp? –Specify corresponding vectors interpolate to a complete warping function The Beier & Neely Algorithm

58. Beier&Neely (SIGGRAPH 1992) Single line-pair PQ to P’Q’:

59. Algorithm (single line-pair) For each X in the destination image: –Find the corresponding u,v –Find X’ in the source image for that u,v –destinationImage(X) = sourceImage(X’) Examples:

60. Multiple Lines Length: length of the line segment, Dist: distance to line segment a, p, b: constants specified by the user

61. Resulting Warp

62. Full Algorithm

63. Results

64. Morph Sequences

65. Image Morphing We know how to warp one image into the other, but how do we create a morphing sequence? –Create an intermediate warping field (by interpolation) –Warp both images towards it –Cross-dissolve the colors in the newly warped images

66. Morphing Video Click here!here

67. Summary Image warping Image morphing

68. Final Project Work on your own project or implement a research project based on - Siggraph papers (click here)here - Eurographics, SCA, SGP, etc, (click here)here Approved by the instructor Submit one page project proposal - deadline: 3/8 - what, why, how? Have a meeting with me before you submit the project proposal

69. Final Project 10% Originality 20% Reports (10% each) - 4/15: Midterm project report - 5/6: Final project report 10% Final Oral Presentation 60% Quality of Work

70. Next Lecture Image registration Image Mosaicing