3D Single Image Scene Reconstruction For Video Surveillance Systems Patrick Denis, Francisco Estrada and James Elder pdenis@elderlab.yorku.ca Goal Results Compute 3D models of urban scenes from surveillance cameras. Manhattan World Manhattan World Vanishing Points Using homogeneous coordinates [4] for 3D points, we have: Typical man-made urban structures are built as rectangles that lie at right angles to each other. Urban scenes that conform to an orthogonal three-dimensional grid are said to follow a “Manhattan World Assumption” [1]. Manhattan World rectangles must lie in one of the three canonical families of planes defined by the grid. Due to perspective projection, these rectangles appear in the image as general quadrilaterals. A vanishing point is a point in the image where parallel lines intersect. The 3D orientation of these lines is given by the orientation of the line passing through the optical centre and the vanishing point [2]. Calculate the normal to the interpretation plane using the optical centre, and two homogeneous image points of a line segment (u1 & u2) Calculate the plane  using the vanishing point vector , a homogeneous image point u1 and Compute the intersection u3 between  and the line L2 defined by the join of u2 and the optical centre o using Plücker [4] notation Calculate the angles of the triangle defined by the optical centre o, u1 and u3 Find  and , the distances from the optical centre to the 3D endpoints of the line segment. Orientation of lines and Vanishing Points Vanishing point Introduction We suggest that the strong constraints of an urban environment (the “Manhattan World Assumption”) and the internal parameters from a single camera are sufficient to infer enough of the three-dimensional context to produce a 3D rendering useful for visual surveillance. The 3D model is only reconstructed up to a projective scale. The following steps are proposed to render an image of an urban environment as a three-dimensional model: 1) Extract the quadrilaterals from monocular video frames. 2) Infer the 3D orientation and location of the quadrilaterals. 3) Render the scene to the user. At this time, quadrilaterals are identified interactively. We plan to automate this step in the future Optical Centre Health, Nursing & Environmental Studies building @ York University Line through optical centre and vanishing point Parallel Image lines Figure 2. The line through the optical centre and the vanishing point has the same 3D orientation as the parallel lines identified as meeting at the vanishing point. where & 3D Finding world coordinates of an image line A line in the image can be generated by an infinite number of lines in space. The triangular plane defined by two rays that begin at the optical centre of the camera and continue through the two endpoints of the line in the image is known as the interpretation plane [3]. Any line segment in space that extends from one ray to the other in the interpretation plane will project to the same image line. Each of these lines is identified by its orientation and length. Ross building @ York University Geometrical Representation to Find World Coordinates for an Image Line Relationship between Image Line and 3D Lines Conclusion Image plane |u1| 2 u1 |u1| The effectiveness of urban visual surveillance may be enhanced if activities can be rendered within a three-dimensional context. Preliminary results show that semi-automatic inference of 3D models of urban buildings from monocular imagery is feasible, based on the Manhattan World assumption and the laws of perspective projection. Models are inferred up to a global scale factor. 2 g 1 L2 1 L1 o |u2| u3 3 Interpretation plane Image plane |u2| Image Line Figure 4. Necessary steps to find the world coordinates of a line given the orientation and length. Figure 1. User interface created to define the set of quadrilaterals following the Manhattan World Constraint. Figure 1. User interface created to define the set of quadrilaterals following the Manhattan World Constraint. Figure 3. A line in the image may project from any one of many lines in space. Each line in 3D space has a unique orientation  and length L. Creating a Model Creating a Model 3D References Texture Mapping For purposes of texture mapping, we estimate a homography for each quadrilateral in the image that will map pixels from the image to a rectangle for the rendered environment. To make a 3D model from a set of identified quadrilaterals in an image we must: Identify the orientation of the Manhattan World relative to the camera optical centre with the use of Vanishing Points. Find 3D world coordinates of an image line. Find the remaining world coordinates for all defined quadrilaterals. Extract the quadrilaterals in the image suitable for textures rendering. To make a 3D model from a set of identified quadrilaterals in an image we must: Identify the orientation of the Manhattan World relative to the camera optical centre with the use of Vanishing Points. Find 3D world coordinates of an image line. Find the remaining world coordinates for all defined quadrilaterals. Extract the quadrilaterals in the image suitable for textures rendering. [1] Coughlan, J. and Yuille A.L., “Manhattan World: Orientation and Outlier Detection by Bayesian Inference,” in Neural Computation, vol. 15, no. 5, pages 1063-1088, 2003 [2] Collins R.T. and Weiss R.S. “Vanishing Point Calculation as a Statistical Inference on the Unit Sphere,” In Proceedings of the Third International Conference on Computer Vision, pages 400-403, December 1990 [3] Stephen T. Barnard, “Interpreting Perspective Images,” in Artificial Intelligence, vol. 21, pages 435-462, 1982 [4] Hartley R. and Zisserman A., “Multiple View Geometry in Computer Vision,” Cambridge University Press, Second Edition, 2003 The 3D orientation of an image lines is determined using the vanishing point. The length of a line is determined up to a global scale factor. Figure 5. A homography maps an image quadrilateral to a rectangle.