1. View Morphing by Steven M. SeitzCharles R. Dyer Irwin Chiu Hau Computer Science McGill University Winter 2004 Comp 767: Advanced Topics in Graphics: Image-Based Rendering

2. Overview What is view morphing? How to do view morphing? Results Conclusion Comp 767: Advanced Topics in Graphics: Image-Based Rendering Mona Lisa view morphs Source: http://www.cs.washingto n.edu/homes/seitz/vmorp h/vmorph.htm

3. View Morphing What is view morphing? –What is it? –Why do we care about it? –Where do we see them? –Image Morphing vs View interpolation vs View Morphing Comp 767: Advanced Topics in Graphics: Image-Based Rendering Virtual Cameras Source: View Morphing; Steven M. Seitz, Charles R. Dyer

4. View Morphing: Key Idea Beier-Neely morph is NOT shape-preserving! –distortions –un-natural Comp 767: Advanced Topics in Graphics: Image-Based Rendering A Shape-Distorting Morph Source: Steven M. Seitz, Charles R. Dyer

5. View Morphing: Key Idea View morphing uses 3D shape preserving morph! –no distortions –natural A morph is 3D shape preserving if the results of two different views represent new views of the same object Comp 767: Advanced Topics in Graphics: Image-Based Rendering A Shape-Distorting Morph Source: Steven M. Seitz, Charles R. Dyer View morphing fixes these intermediate steps!

6. Why do we care? View morphing is efficient Produces new views without –3D modelling –Taking additional photos View morphing creates impressive effects –Camera motion –Image morphing Comp 767: Advanced Topics in Graphics: Image-Based Rendering trueSpace Source: www.caligari.com

7. Image Morphing vs View interpolation vs View Morphing View Morphing is an extension to Image Morphing (Beier and Neely, 1992) –Produces physically plausible new views of a scene View Morphing (Seitz and Dyer, 1996) is an improvement over View Interpolation (Chen and Williams, 1993) –Addresses non-rigid transformations problems –Does not require depth values –Creates realistic image transitions Comp 767: Advanced Topics in Graphics: Image-Based Rendering

8. Overview What is view morphing? How to do view morphing? Results Conclusion Comp 767: Advanced Topics in Graphics: Image-Based Rendering Mona Lisa view morphs Source: http://www.cs.washingto n.edu/homes/seitz/vmorp h/vmorph.htm

9. How to do View Morphing? View morphing in three steps 1.Prewarp two images 2.Compute a morph between the prewarped images 3.Postwarp each in between images produced by the morph Comp 767: Advanced Topics in Graphics: Image-Based Rendering View Morphing Procedure Source: Steven M. Seitz, Charles R. Dyer

10. How to do View Morphing? View morphing in three steps 1.Prewarp two images 2.Compute a morph between the prewarped images 3.Postwarp each in between images produced by the morph Comp 767: Advanced Topics in Graphics: Image-Based Rendering View morphing in 1 steps: Assume parallel views! 1.Compute a morph between the parallel images

11. Parallel Views Basic Theory Comp 767: Advanced Topics in Graphics: Image-Based Rendering image point p 0 = (x 0,y 0 ) scene point P = (X,Y,Z) p 0 = Π 0 P where Π 0 is a projection matrix Morphing Parallel Views Source: Steven M. Seitz, Charles R. Dyer image point p 1 = (x 1,y 1 ) scene point P = (X,Y,Z) p 1 = Π 1 P where Π 1 is a projection matrix

12. Parallel Views Basic Theory Comp 767: Advanced Topics in Graphics: Image-Based Rendering image point p = (x,y) scene point P = (X,Y,Z) p = Π P where Π is a homogenous projection matrix Π = [ H | - HC] H: position and orientation of image plane C: euclidean position of the camera Morphing Parallel Views Source: Steven M. Seitz, Charles R. Dyer

13. Parallel Views Mathematics for View Interpolation Comp 767: Advanced Topics in Graphics: Image-Based Rendering Πs: Linear interpolation of Π 0 and Π 1 Πs = (1 – s) Π 0 + s Π 1 s = [0,1] Cs = (sCx, sCy, 0) fs = (1 – f) f 0 + s f 1 f : focal lengh C : center of a camera

14. View Interpolation Recap View Interpolation vs View Morphing Comp 767: Advanced Topics in Graphics: Image-Based Rendering Morphing Parallel Views Source: Steven M. Seitz, Charles R. Dyer View Interpolation Source: 3D Games by Alan Watt and Fabio Policarpo

15. How to do View Morphing? View morphing in three steps 1.Prewarp two images 2.Compute a morph between the prewarped images 3.Postwarp each in between images produced by the morph Comp 767: Advanced Topics in Graphics: Image-Based Rendering View morphing in 1 steps: Assume parallel views! 1.Compute a morph between the parallel images

16. Non-Parallel Views The General Case Comp 767: Advanced Topics in Graphics: Image-Based Rendering This is where the 3-Step Algorithm comes into play 1.Prewarping: I 0 to Î 0 and I 1 to Î 1 2.Morphing: Î 0 and Î 1 into Î s 3.Postwarping: Î s into I s Morphing in Three Steps Source: Steven M. Seitz, Charles R. Dyer

17. Non-Parallel Views Mathematics for Image Reprojection Comp 767: Advanced Topics in Graphics: Image-Based Rendering ĤH -1 p = p H and Ĥ are 3x3 matrices that represent the position and the orientation of their image planes The resulting 3x3 matrix, ĤH -1 is a projective transformation that reprojects the image plane I into Î Morphing in Three Steps Source: Steven M. Seitz, Charles R. Dyer ^

18. Limitations Singular Views cannot be reprojected to form parallel views Singular configurations are settings where one of the camera resides in the field of view of another camera Still works, just kind of conceptually hacky Comp 767: Advanced Topics in Graphics: Image-Based Rendering Parallel view Source: Steven M. Seitz, Charles R. Dyer Singular view Source: Steven M. Seitz, Charles R. Dyer

19. Traditional Problems Change in visibility creates 1.Folds –Occurs when a visible area becomes occluded 2.Holes –Occurs when an occluded area becomes visible Comp 767: Advanced Topics in Graphics: Image-Based Rendering Penumbra, umbra and hole regions Source: Chen and Williams Fold Source: Irwin Chiu Hau Area Point

20. Producing The Morph We have talked about theory behind algorithm 1.How to project/unproject images to parallel planes 2.How to warp between parallel planes 3.Theoretical problems Now, let’s do an actual View Dependent Morph! Comp 767: Advanced Topics in Graphics: Image-Based Rendering

21. Producing The Morph We need: 1.Two images I 0 and I 1 2.Two perspective projection matrices Π 0 and Π 1 3.Correspondence between pixels Note that a sequence of projection matrices Π s is required to control the entire animation, but Π s can be computed automatically if we know H s. Comp 767: Advanced Topics in Graphics: Image-Based Rendering

22. Controlling The Morph H s can be obtained indirectly by establishing constraints –Recall H s is position and orientation of image plane –One way is to specify four control points Note: Control points implicitly determine the postwarping transformation Comp 767: Advanced Topics in Graphics: Image-Based Rendering View Morphing Procedure Source: Steven M. Seitz, Charles R. Dyer Four control points form the red bounding box to determine the postwarping stage Yellow lines are set of features to determine the prewarping stage

23. View Morphing sans Prewarping Prewarping is not necessary for: –Objects that aren’t closely related Prewarping is less effective Computation is unstable –Images that are approx. orthographic (eg. telephoto) However, postwarping should not be left out to: –Reduce image plane distorsions Comp 767: Advanced Topics in Graphics: Image-Based Rendering

24. Overview What is view morphing? How to do view morphing? Results Conclusion Comp 767: Advanced Topics in Graphics: Image-Based Rendering Mona Lisa view morphs Source: http://www.cs.washingto n.edu/homes/seitz/vmorp h/vmorph.htm

25. Results Comp 767: Advanced Topics in Graphics: Image-Based Rendering Facial view morphs Source: Steven M. Seitz, Charles R. Dyer

26. Results Comp 767: Advanced Topics in Graphics: Image-Based Rendering Facial view morphs Source: http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm

27. Results Comp 767: Advanced Topics in Graphics: Image-Based Rendering Mona Lisa view morphs Source: http://www.cs.washington.edu/homes/seitz/vmorph/vmorph.htm

28. Results Image Morphing vs View Morphing Comp 767: Advanced Topics in Graphics: Image-Based Rendering Image Morphing vs View Morphing Source: Steven M. Seitz, Charles R. Dyer

29. Conclusions Things to remember View Morphing Powerful extension to image morphing Produces new views of a scene 3D shape preserving Comp 767: Advanced Topics in Graphics: Image-Based Rendering

30. References View Morphing - Seitz and Dyer, 1996 View Interpolation - Chen and Williams, 1993 Image Morphing - Beier and Neely, 1992 3D Games: Realtime rendering and Sofware Technology - Alan Watt and Fabio Policarpo Comp 767: Advanced Topics in Graphics: Image-Based Rendering

31. Questions? Comp 767: Advanced Topics in Graphics: Image-Based Rendering