Computer Vision Computer Vision based Hole Filling Chad Hantak COMP December 9, 2003

Computer Vision Background Last Class (Range Data) Brief Recap –DeltaSphere acquires depth samples –Algorithms turn these samples into a polygonal mesh

Computer Vision Holes Again Last Class Scanner can only get information it “sees” –Occlusion –Non-reflective objects

Computer Vision Vision Technique Fill holes in polygonal mesh based on photographs of the scene –Given: Scan of scene (hole-filled mesh) Photographs of room from different views and locations Correlation between points in photographs and points on mesh –Method: Use image segmentation to extract connectivity information from photographs Based on determined connectivity add polygonal information to mesh

Computer Vision Implementation Fill in large planar regions –Create a more complete model –Does not work on a minute scale Three separate utilities –Correspondence Creator Link the photographs to the scan data –MATLAB Analysis Utility Image processing Polygon creation –Display Program Overlay the original scan with new polygonal information

Computer Vision Working Example Demonstrate the pipeline by example Simple scene was setup –One plane partially occluding three other planes

Computer Vision Working Example (cont.) Photographs of occluded planes

Computer Vision Step 1 - Correspondence Mark 6 correspondences Mark component in photograph to insert Mark 3 points on plane in model

Computer Vision Step 2 – MATLAB Region labeling on photograph –Canny Edge Detection –Morphological Operations (Dilation / Erosion) –Region Labeling Boundary of component –Convex Hull Find the camera projection matrix –Direct Linear Transformation (DLT) Create the polygon –Image points -> Mesh points

Computer Vision Step 2 – MATLAB: Region Labeling Canny Edge Detection –Detects strong and weak Edges –Include weak edges only if connected to a strong edge –Canny, J., A Computational Approach to Edge Detection, IEEE PAMI 8(6) , November, 1986.

Computer Vision Step 2 – MATLAB: Region Labeling Need components with closed boundaries for accurate region labeling Morphological Operations –Image processing based on shapes –Many different types Grow lines vertically and horizontally Remove extraneous information without separating objects Remove endpoints of lines without removing small objects

Computer Vision Step 2 – MATLAB: Region Labeling Region labeling marks the different bounded components with unique labels

Computer Vision Step 2 – MATLAB: Boundary Find the boundary of the component to add to mesh Use the component’s specified point as a start, find and walk along region’s border This implementation takes the convex hull of those points

Computer Vision Step 2 – MATLAB: Camera Matrix Image points have to be turned into world points Using 6 correspondences find the camera’s projection matrix Direct Linear Transformation (DLT) Objective Given n≥6 3D to 2D point correspondences {X i ↔x i }, determine the 3D projection matrix P such that x i =PX i Algorithm (i)For each correspondence X i ↔x i compute A i. Usually only two first rows needed. (ii)Assemble n 2x12 matrices A i into a single 2 n x12 matrix A (iii)Obtain SVD of A. Solution for p is last column of V (iv)Determine P from p

Computer Vision Step 2 – MATLAB: Camera Matrix Camera Matrix –Takes world points to image points –Holds the camera’s intrinsic information, orientation, and world position “Inverted” Camera Matrix –Cannot easily go the other way –Best which can be done is turn an image point into a ray in the world originating from the camera’s position –World point for an image point lies along this ray

Computer Vision Step 3 – MATLAB: Create Polygon For each image point on the boundary –Determine the world ray for this point from the camera matrix –User specified a plane in mesh the polygon lies on –Intersect ray with the plane to find the world point Triangulate the world points of the polygon Output PLY file containing the polygon

Computer Vision Results OriginalFilled

Computer Vision Limitations Good Lighting –Hard shadows create false edges –Saturation washes out edges

Computer Vision Limitations Weak borders –Not perfectly straight –Result of edge detection & morphological operations

Computer Vision Future Work Try it on some real data Straighter border edges –Canny Edge Detection combined with multiple line fittings through RANSAC Remove mesh plane specification –No longer require the user to specify plane where the polygon lies in model –Extract depth information from multiple photos and epipolar geometry Object recognition –Template matching –Database of models