Model Refinement from Planar Parallax Anthony DickRoberto Cipolla Department of Engineering University of Cambridge

Model Refinement from Planar Parallax - BMVC'99 2 Model acquisition Recovery of 3D scene structure from one or more images Source: Photobuilder “The devil’s in the detail”

Model Refinement from Planar Parallax - BMVC'99 3 Model refinement We concentrate on approximately planar models of architectural scenes

Model Refinement from Planar Parallax - BMVC'99 4 Planar parallax Two projections of a planar scene are related by a 2D homography H Points not belonging to the plane do not obey this homography –displaced by an additional parallax vector along epipolar line Image 1 Scene Image 2

Model Refinement from Planar Parallax - BMVC'99 5 Model refinement algorithm Initial model Reference Image Homography definition Reconstruction Offset Images Camera Calibration Correspondence PredictionPlane Fitting Refined model OK? YES NO

Model Refinement from Planar Parallax - BMVC'99 6 Homography Definition 4 correspondences required to estimate H (8 DOF) Use linear technique Initial model Reference Image Homography definition Offset Images Camera Calibration

Model Refinement from Planar Parallax - BMVC'99 7 Camera calibration World coordinate system Reference camera coordinate system Ref. image plane Model plane, n = [0 0 1] Optical axis x y z Homography definition Camera calibration Correspondence Perspective camera model: x y z whereis the same for all images Given n offset images, solve for1 + 5n parameters Given homographies8n parameters

Model Refinement from Planar Parallax - BMVC'99 8 Correspondence Warp offset image Apply wavelet transform to each image Perform multi-resolution matching in wavelet domain –sum squared difference of 6 filter outputs –taking into account the epipolar constraint ReconstructionCamera Calibration Correspondence Prediction

Model Refinement from Planar Parallax - BMVC'99 9 Reconstruction Triangulate to obtain dense depth map Robust fusion of several depth maps if necessary –Use median at each point Special case - near depth discontinuity Model plane Choose camera 2 Choose camera 1 Camera 1Camera 2 Reconstruction Correspondence Plane Fitting OK? Y N

Model Refinement from Planar Parallax - BMVC'99 10 Example - Wreath Reference ImageOffset Image 1 Offset Image 2 Image 1 onlyImage 2 onlyFused reconstruction

Model Refinement from Planar Parallax - BMVC'99 11 Example - Wreath

Model Refinement from Planar Parallax - BMVC'99 12 Example - Gateway Image 1 only Images 1 and 2Images 1, 2 and 3Images 1, 2, 3 and 4

Model Refinement from Planar Parallax - BMVC'99 13 Example - Gateway

Model Refinement from Planar Parallax - BMVC'99 14 Grow planar regions RANSAC fit planes at each iteration Intensity edge fitting to refine region boundaries Use planes (refined model) to predict parallax at next offset image –reduces correspondence search window in planar regions More accurate, requires fewer images Plane fitting & prediction ReconstructionCorrespondence PredictionPlane Fitting Refined model OK? Outlier Seed point

Model Refinement from Planar Parallax - BMVC'99 15 Example - Church wall Reference Image Image 1Image 2 Using image 1 Using images 1 and 2

Model Refinement from Planar Parallax - BMVC'99 16 Example - Church wall

Model Refinement from Planar Parallax - BMVC'99 17 Conclusion Model refinement avoids some problems of other methods –little reliance on user –low number of initial parameters Future directions: –simplify correspondence problem by modelling surface reflectance –stitch together multiple façades –model selection