Image Based Rendering And Modeling Techniques And Their Applications Jiao-ying Shi State Key laboratory of Computer Aided Design and Graphics Zhejiang University, Hangzhou, China

Image Based Rendering (IBR) PART I

Traditional Computer Graphics Use geometry and lighting model to simulate the imaging process and generate realistic scene –No guarantees for the rightness of the models –A lot of computation time needed

Use of Images in Computer Graphics Texture mapping Environment map How about more images?

Image Based Rendering IBR: To Synthesize a new scene with novel view point based on given images

A Framework of Image Based Rendering Real Scene Sampling System Data Storage System Data representation System Rendering System Synthesized view

The Key Part of IBR The data representation system is the key part of IBR, It determines the other three subsystems. - A taxonomy based on the data representation system

A Taxonomy of IBR The Geometry based data representation The Image based data representation The plenoptic function based data representation

The Geometry Based Data Representation Geometry elements used as data representation in IBR: –polyhedra(Debevec, et. al 1996) –layers (Baker, Szeliski and Anandan 1998) –points(Shade et al. 1998) Similar to traditional Computer Graphics, except the geometry model comes from images

Image Based Data Representation data are treated as a series of images with correspondence relations “ optical flow ” “ morphing map ” forward/ reverse mapping;morphing Examples:. View interpolation (Chen and William,1993).View Morphing(Seitz and Dyer 1998)

Plenoptic Function Based Data Representation Plenoptic function (Adelson and Bergen,1991)

Representative IBR Methods Based on Plenoptic Functions Plenoptic Modeling: 5D Light field/Lumigraph: 4D Concentric Mosaics : 3D Panorama: 2D [L. McMillan 95] [M. Levoy 96, S. J. Gortler 96] [H. Y. Shum 99] [S. E. Chen 95, R. Szeliski 97]

Conclusion The progress of IBR technique is also the progress of new data representation method. We treat an image: –as texture in geometry  texture mapping –as images with correspondence relation  view interpolation /morphing –as light beams  light field –as slit image  concentric mosaics...

Demo of IBR (1) Tour in Dunhuang Art Cave

Demo of IBR (2) Tour in Lingyin Temple in Hangzhou

Image Based Modeling (IBM) PART II

The Common Methods Used for Modeling objects Using Geometry Modeling Software Packages, such as 3D MAX ， Mayer ， SoftImage and AutoCAD etc. to create wireframe models, surface models or volume models. Using 3D Laser Scanners Using Image-based Modeling techniques to create geometry models or appearance models of objects.

A Taxonomy of IBM IBM methods using active cues The active cues are refered to the artificially generated silhouette, which are projected onto the surface of the modeled objects. IBM methods using passive cues The passive cues are refered to the implicit characteristics of the modeled objects, such as geometry features and textures of the objects.

IBM Methods Using Active cues

IBM Methods Using Passive Cues Based on known geometry Based on visual hull Based on light field Based on stereo vision

IBM Methods Using Passive Cues Based on known geometry[Debevec96]

IBM Methods Using Passive Cues Based on visual hull [Wojciech Matusik 2001]

IBM Methods Using Passive Cues Based on Light Field[Marc Pollefeys et. al.]

IBM Methods Using Passive Cues Based on stereo pairs: the goal is to automatically extract a realistic 3D model by freely moving a camera around an object.  Neither the camera motion nor the camera settings have to be known.  The obtained 3D model is a scaled version of the original object.  The surface appearance is obtained from the image sequence as well.

IBM Methods Using Passive Cues How can we get 3D information (depth) from a 2D image? It is impossible to get depth information of the object from a 2D image.

IBM Methods Using Passive Cues It is possible to recover the depth information of the object by using two or more images through triangulation. Reconstruction of 3D point through triangulation

IBM Methods Using Passive Cues But the following information are needed to know for this purpose: Corresponding image points Relative pose of the camera for different views(so called camera extrinsic parameters, i.e. the position and orientation of the camera) Relation between the image points and the corresponding line of sight. It is defined by the camera model, which usually is a pinhole model and cameras intrinsic parameters and extrinsic parameters.

IBM Methods Using Passive Cues Flowchart of the IBM technique based on stereo pairs

Reconstruction of Architectural Models Based on Image Sequence

Camera Calibration Using Corner Structure and Parallel Structure in Real Scene Consisted of at least 4 line segments. 3 of them are perpendicular each other. The forth segment is parallel to one of previous 3 segments. Corner Structure Parallel Structure Consisted of at least 4 line segments. 2 of them are perpendicular each other. Other 2 segments are parallel to previous 2 segments separately.

Calculating focal length based on single image is the directional vector of segment ca is the normal vector of the projective plane is the directional vector of segment AB because We can get (*) This is a quartic equation relative to f, from which we can get one plus real solution of the focal length. Projection of the corner structure Then the directional vectors of OX and OY can be written as an expression relative to the focal length f

Calculating rotation matrix and translation vector based on single image related to SCF From the projection equation of pinhole camera model, we can get: Rotation matrix relative to SCF is: The direction of the translation vector relative to CCF is Projection of the corner structure SCF: structure coordinate frame CCF: camera coordinate frame Translation vector can be deduced:

Optimizing the camera ’ s parameters: f, r and t We choose the distance DIST between the original image point and the reprojected image point as the measurement function for error. Once the initial values of camera parameters were derived, we can optimize these parameters by using other line segments, which are parallel to the segments of the corner structure in two images.

Procedures for optimization  Rotation matrices  Translation vector  Structure condition  Objective function of optimization: To ensure the orthogonality of rotation matrices To ensure the length of the two transla- tion vectors will not be changed in optimization. tlen1 and tlen2 are the modules of the two translation vectors, which can be calculated from the initial solutions. To ensure directions of the segments will not be changed in optimization  Constrained conditions:

Experimental Results Two pictures of kiosk at the campus of Zhejiang University taken by hand-hold camera Camera positions and top-down views of feature points

Error Statistics: Choose the distance between feature points to its epipolar line as the measurement of error. In the ideal situation this distance should be equal to zero. Image Err before optimization (pixels) Err after optimization (pixels) Left Right Metric(Pixel)

Architecture Reconstruction Based on Single Image

Reconstruction of plane Assign base point O and determine its coordinates Assign base plane which contains the base point O as one of its corners. Determine the normal vector of the base plane. Determine unknown neighbor plane of the base plane: choose one point on the common edge of the neighbor planes as the base point of the unknown plane. Determine normal vector of the unknown plane.

Decomposition of sweeping surface into plane patches

Wireframe model

Texture mapping model

Strategy of model merging for multiple scenes Scene structure and 3 camera positions Model merging diagram for multiple scenes

Example of model merging Image sequence Reconstructed model for each image

Main challenges for model merging procedure How to transform all the models in different local coordinates to one world coordinate frame? Model integration: scaling, vertex match? Deletion of overlapped plane?

Reconstructed Tea Box with Texture Mapped

Experiment 1: image sequence

Experiment 1: reconstructed model

Experiment 2: image sequence

Experiment 2: reconstructed model

Thanks a lot