A new approach for modeling and rendering existing architectural scenes from a sparse set of still photographs Combines both geometry-based and image based techniques.

Current geometry-based methods -- a modeling program is used to manually position the elements of the scene. extremely labor-intensive process: surveying the site, locating and digitizing architectural plans, converting existing CAD data; difficult to verify the accuracy of the model; resulting models are noticeably computer-generated.

Image-based system – uses photographs as input and produces photorealistic renderings as output. Some image-based systems rely on the computer vision technique of computational stereopsis to determine the structure of the scene from the multiple photographs available. These systems are only as strong as the underlying stereo algorithms.

Stereo algorithms weaknesses: the photographs need to appear very similar for reliable results to be obtained; use many closely spaced images; in some cases employ significant amounts of user input for each image pair to supervise the stereo algorithm; Systems based on stereo algorithms are nor efficient for creating large-scale, freely navigable virtual environments from photographs.

The new method combines geometry-based and image-based methods and requires only a sparse set of photographs to produce realistic renderings from arbitrary view points. Three new modeling and rendering techniques: photogrammetric modeling; view-dependent texture mapping; model-based stereo.

Photogrammetric Modeling The computer determines the parameters of a hierarchical model of parametric polyhedral primitives to reconstruct the architectural scene. Façade -- interactive modeling program that allows the user to construct a geometric model of a scene from digitized photographs.

Photogrammetric Modeling User: selects a number of photographs; instantiates the components of the model; marks edges in the images; corresponds the edges in the images to the edges in the model. Blocks -- parameterized (length, width and height) geometric primitives such as boxes, prisms, and surfaces of revolution.

Photogrammetric Modeling A photograph of the Campanile, Berkeley’s clock tower.

Photogrammetric Modeling Each coordinate of each vertex of the block is expressed as linear combination of the block’s parameters, relative to an internal coordinate frame. P0=(-width, -height, length) T Each block has an associated bounding box.

Photogrammetric Modeling Blocks are organized in a hierarchical tree structure. Relations -- spatial relationships between blocks. The relation between a block and its parent is represented as a rotation matrix R and a translation vector t.

Photogrammetric Modeling The reconstruction algorithm works by minimizing an objective function that sums the disparity between the projected edges of the model and the edges marked in the images represents the disparity computed for edge feature i.

Photogrammetric Modeling University High School in Urbana, Illinois.

View-Dependent Texture-Mapping Projecting the original photographs from the camera position onto the model. Some parts of the model can shadow others with respect to the camera. Such shadowed regions are determined with an image-space shadow map algorithm. Render the model from a novel point of view -- use multiple images.

View-Dependent Texture-Mapping University High School in Urbana, Illinois.

View-Dependent Texture-Mapping If a pixel is mapped in only one rendering, its value from that rendering is used in the composite. If it is mapped in more than one rendering, the renderer has to decide which image to use. The pixel in the virtual view corresponding to the point on the model is assigned a weighted average of the corresponding pixels in actual views 1 and 2. The weights w1 and w2 are inversely proportional to the magnitude of angles a1 and a2.

View-Dependent Texture-Mapping Unwanted object in the original photograph: mask out the object with a reserved color; set the weights for any pixels corresponding to the masked regions to zero; decrease the weights of the pixels near the boundary as before to minimize seams; any regions in the composite image which are occluded in every projected image are filled in using a hole-filling algorithm.

Model-Based Stereopsis Measures how the actual scene deviates from the approximate model. Places the images into a common frame of reference that makes the stereo correspondence possible even for images taken from relatively far apart. The stereo correspondence information can then be used to render novel views of the scene using image- based rendering techniques. Model-based stereo computes the associated depth map for the key image by determining corresponding points in the key and offset images

Model-Based Stereopsis The warped offset image, produced by projecting the offset image onto the approximate model and viewing it from the position of the key camera. This projection eliminates most of the disparity and foreshortening with respect to the key image, greatly simplifying stereo correspondence.

Model-Based Stereopsis For determining stereo correspondence, a stereo algorithm is used. The warping step makes it dramatically easier to determine the correspondences.

Future Work The use of surfaces of revolution (e.g. domes, columns, and minarets) in the photogrammetric modeling approach. The system should be able to make better use of photographs taken in varying lighting conditions, and it should be able to render images of the scene as it would appear at any time of day, in any weather, and with any configuration of artificial light.

Future Work Selecting which original images to use when rendering a novel view of the scene. This problem is especially difficult when the available images are taken at arbitrary locations. The weighting function still allows seams to appear in renderings and does not consider issues arising from image resampling. Another form of view selection is required to choose which pairs of images should be matched to recover depth in the model-based stereo algorithm.

THANK YOU