1. Visualization of Urban Scenes from Video Footages Siyuan Fang Supervised by Dr. Neill Campbell 29/10/2007

2. Overview Generate panorama (or image mosaicing) from a video of street scenes Potential applications: Virtual environment Robot navigation On-line tourist guide Combine different source frames (or images) together

3. Outline Background Research Proposal Achievements Future Plan

4. Mosaicing form Single-Viewpoint Images Source images are captured using a camera with pure rotation movement around a fixed viewpoint, (the intrinsic parameter can also change) Align (or register) source images and stitch them together, e.g., the general alignment (registration) model: homography for a pair of images. The homography can be estimated by points (pixels or features) correspondence, robust estimations (e.g., RANSAC) are used to remove the outliers of the point correspondence Further refinements: pixel value blending bundle adjustment The limitation of the single-viewpoint image mosaicing --cannot put viewpoint far away enough

5. Mosaicing form Multi-Viewpoint Images : Perspective Mosaicing The difficulty: there isnt a single uniform alignment --only planar scenes have the same homography if viewpoint is changing Traditional approaches: view synthesis and image based rendering e.g., [GORT96], [SHUM99] --picture surface alignment + residual pixel movement [KUMA95] User-interactive design: picture surface alignment + user refinement for the off-plane scenes [AGAR06] Limitations of these approaches

6. Strip Panorama: Parallel Projection Mosaicing A simulation of the push-broom camera. A common setup: sideway translating camcorder and vertical trip scanning. [ZHEN03] Parallel projection along the horizontal direction, hence, remain the perspective along the vertical direction, cause the aspect ratio distortion: Further scenes get expanded Closer scenes get squashed

7. Strip Panorama: Parallel Projection Mosaicing Cont. Two kinds of distortion, the same reason: Aspect ratio distortion Blurred-truncated distortion

8. A Better Strip Panorama: Crossed-Slit Mosaicing The Crossed-Slit Cameral Model, and its special case: POX-slit model. Strict sideway translation camcorder can be used to simulate POX-slit model: the cameral path is presumably considered as the horizontal slit and the vertical slit is determined by the camera path and the width of the image plane [ZOME03]

9. The Crossed-Slit Mosaicing cont.

10. Comparison between the Crossed-Slit Mosaicing and the Parallel projection Mosaicing

11. The Limitations of the Crossed-Slit Mosaicing (Distortions)

12. Aims and Objectives Aims and basic requirements:. Use moving pinhole camcorder (normal) Capture real long street scenes Scalable Least distortion Automaticity

13. The Proposed Research Approach (provisional) The proposed approach, 3 main tasks: I. Synthesis novel view based on the crossed-slit mosaic, but fully perspective remove two kinds of distortion based on geometry information II. Handle extremely long scenes, III. * Handle moving objects

14. Summary of Achievements The literature survey on image mosaicing Single-viewpoint mosaicing View synthesis and imaged based rendering Approaches of strip mosaicng A (provisional) research proposal Some design and implementation work A single-viewpoint mosaic tool A part of the task 1

15. Future Plan Further refine (modify) the research proposal Complete the task 1 A formal math model and an approximation algorithm Curved camera path and corner effect Writing up Start the design work of task 2 The main problem User interactive may be needed The result of task 1 can be served as the building block

16. Reference [AGAR06] A.Agarwala, M.Agrawala, M.Cohen and D.Salesin, R.Szeliski, Photographing long scenes with multi-viewpoint panoramas, ACM Transactions on Graphics, 25(3), 853 - 861, 2006. [GORT96] S.J.Gortler, R.Grzeszczuk, R. Szeliski and M.F. Cohen, The Lumigraph, The Journal of Computer Graphics, 30, 43--54, 1996. [KUMA95] R.Kumar, P.Anandan, M.Irani, J.Bergen and K.Hanna, Representation of scenes from collections of images, Proceedings of IEEE Workshop on Representation of Visual Scenes, 10-17,1995 [PELE00] S.Peleg, B.Rousso, A.Rav-Acha and A.Zomet, Mosaicing on Adaptive Manifold, IEEE Transactions on Pattern Analysis and Machine Intelligence, 22(10), 1144-1154, 2000. [SHUM99] H.Y.Shum and L.W.He, Rendering with concentric mosaics, The Journal of Computer Graphics, 33, 299--306, 1999. [ZHEN03] J.Y.Zheng, Digital route panoramas, IEEE Multimedia, 10(3), 57- 67, 2003. [ZOME03] A.Zomet, D.Feldman, S.Peleg and D.Weinshall, Mosaicing New Views: The Crossed-Slits Projection, IEEE Transactions on Pattern Analysis and Machine Intelligence, 741- 754, 2003.