1. 1 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram 3D Scene Reconstruction from Aerial Video Prudhvi Gurram, Eli Saber, Harvey Rhody Chester F. Carlson Center for Imaging Science Rochester Institute of Technology DIRS Group Meeting February 8 th, 2008

2. 2 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Outline Motivation Block Diagram Preprocessing Ray Interpolation Object Modeling Conclusions and Future Work

3. 3 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Outline Motivation Block Diagram Preprocessing Ray Interpolation Object Modeling Conclusions and Future Work

4. 4 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Motivation Objective Extraction of 3D geometry of a scene from multi-modal data sets Possible Approaches Manual Interpretation of Stereo Imagery (Very intensive and time consuming for large areas in the order of days or even months) Automated processing of video frames to build stereo mosaics for the extraction of 3D geometry Combine this with information from LIDAR to improve the accuracy of the 3D Scene. Combine the 3D coordinates with material properties from Hyperspectral imaging to render a 3D Scene which conforms both geometrically and radiometrically to real world

5. 5 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram High-Resolution Video Lidar Data Spectral Imagery Spectrally-Accurate Scene Model Motivation Rapidly construct radiometrically-correct scene models based on multi-sensor data for use in DIRSIG synthetic scene generation

6. 6 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Outline Motivation Block Diagram Preprocessing Ray Interpolation Object Modeling Conclusions and Future Work

7. 7 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Video Frames Pre-processing of the video frames Ray Interpolation Object Extraction and Modeling 3D Structure Exterior Orientation (EO) and Interior Orientation (IO) parameters Orientation-corrected video frames Stereo Mosaics Block Diagram

8. 8 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Outline Motivation Block Diagram Preprocessing Ray Interpolation Object Modeling Conclusions and Future Work

9. 9 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Pre-processing of Video Frames Correct the orientation of the frames so that all the frames have same orientation as the first frame. Observed motion parallax of objects is due to translational motion of camera only. Effects of translational motion of camera in Z direction are lost during digitization process. A world point can be expressed in camera coordinate system with Rotation matrix R and camera center at T as

10. 10 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Pre-processing (Contd…) Considering the first frame as reference frame (rotation matrix is an identity matrix) The image coordinates in any frame i are transformed by matrix A, to observe only translational motion in the sensor The image coordinates of this point are given by (Interior Orientation parameters are embedded in matrix K

11. 11 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Outline Motivation Block Diagram Preprocessing Ray Interpolation Object Modeling Conclusions and Future Work

12. 12 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Introduction Why are we using Parallel Ray Interpolation? To convert the view from perspective view to parallel-perspective view To use motion parallax information (while creating mosaics) To make the stereo mosaics seamless Perspective viewParallel view

13. 13 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Introduction - Animated Perspective ViewParallel ViewParallel View from Perspective View Using Fixed Lines

14. 14 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Ray Interpolation Viewpoint 2 Viewpoint 1 Interpolated Viewpoint Image (Mosaic) Plane Point in the image plane from viewpoint 1 Point in the image plane from viewpoint 2 Point in the image plane from the interpolated viewpoint Acknowledgement: Zhigang Zhu et al., City College of New York, New York City, NY

15. 15 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Fast PRISM Algorithm Block Diagram for building Left or Right Stereo Mosaic Fixed Line Matching Curve Control Points PRISM Matching Curve Control Points Stitching Curve Control Points on Mosaic Fixed Line Control Points Fixed Line Control Points Global Transformation Global Transformation Fixed Line Control Points on Mosaic Fixed Line Control Points On Mosaic Destination Triangles Source Triangles Source Triangles Destination Triangles Affine Transformation Affine Transformation Stitched Slices In Mosaic Frame K Frame K+1 Preparing Input Images for Geometric Transformation Geometric Transformations Transformed Image Data Input Video Frames Color Code

16. 16 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Geometry Frame 1 Frame 2

17. 17 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Geometry Fixed Lines Image Frame

18. 18 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Geometry Frame 1: Frame 2: Fixed Line Overlapped Region

19. 19 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Geometry Frame 1: Frame 2: Source Triangles

20. 20 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Interpolating points along the stitching line Viewpoint 2 Viewpoint 1 Interpolated Viewpoint Image (Mosaic) Plane Point in the image plane from viewpoint 1 Point in the image plane from viewpoint 2 Point in the image plane from the interpolated viewpoint

21. 21 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Geometry Destination Triangles in the Left Mosaic

22. 22 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Motion Parallax Frame 1 Frame 2 Interpolated Frame (before triangulation)

23. 23 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Interpolation Frame 1 Frame 2 Overlay of Frames 1 and 2 Interpolation

24. 24 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Interpolation Frame 1 Frame 2 Interpolation

25. 25 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Triangulation Problem Frame 1 Frame 2 Interpolated Frame (after triangulation)

26. 26 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Triangulation Artifacts Before After

27. 27 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Modified Triangulation Make sure that none of the triangles include regions with different motion parallax Find edges of different regions and align the sides of triangles with the edges But aerial images are too noisy to obtain continuous edges around different objects Use segmentation The inner boundary of each segment forms an edge of a region/object

28. 28 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Overlapped region Frame 1: Frame 2:

29. 29 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Segmented images Segmented Frame 1: Segmented Frame 2:

30. 30 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Frame k Segments in Overlapped Region One of the segments Significant points using Convex Hull around the segment Triangulation

31. 31 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Frame k+1 Matching curve The other part of the segment between matching curve and fixed line Triangulation

32. 32 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Mosaic “Orphan” Pixels Orphan pixels filled Using a constraint inherent in the Parallel-Perspective view Parallel view in dominant motion direction and Perspective view in the other direction Do not consider motion parallax along x direction X Y Direction

33. 33 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Results – Set 1 Motion Parallax Frame 1Frame 2 Fast PRISMModified PRISM

34. 34 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Results – Set 2 Fast PRISM Modified PRISM Visual Artifact Corrected by our method DIRcm X0.1 Y4 Z0 X Y Camera Motion Direction Z

35. 35 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Results – Set 3 Visual artifact Corrected by our method RIT imagery – Collected by WASP Lite at an altitude of 3000ft and Frames captured at 3Hz frequency 1 814 3948 78 Frames

36. 36 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Stereo Mosaic – Fast PRISM

37. 37 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Stereo Mosaic – Modified PRISM Publications: 1.P. Gurram, E. Saber, and H. Rhody, “A Novel Triangulation Method for Building Parallel-Perspective Stereo Mosaics”, Proceedings of Electronic Imaging Symposium, SPIE, San Jose, CA, January 2007. 2.P. Gurram, E. Saber, and H. Rhody, “Segment-based Mesh Design for Building Parallel-Perspective Stereo Mosaics”, To be submitted to IEEE Transactions on Image Processing

38. 38 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Outline Motivation Block Diagram Preprocessing Ray Interpolation Object Modeling Conclusions and Future Work

39. Stereo Geometry 39 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Image Courtesy: Z. Zhu, A. Hanson and E. Riseman, “Generalized Parallel-Perspective Stereo Mosaics from Airborne Video,” IEEE Transactions on Pattern Analysis and Machine Intelligence, Vol. 26, pp. 226 – 237, 2004.

40. Find disparity between the points Disparity Distance from the Sensor to the point H Z Height of the point from ground – the coordinate of any pixel in the right and left mosaics in the dominant motion direction respectively – the baseline of the stereo mosaics

41. 41 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram 3D from Passive Video Left mosaic, right mosaic and Nadir mosaic are built by controlling the angle of the parallel view of the mosaics 1 5 10 15 20 25 30 Image Plane Scene Viewpoints Left mosaicNadir mosaicRight mosaic Sensor motion

42. 42 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Object extraction and modeling Nadir mosaicSegmentation Tree regions Building surfaces extraction using height information Each surface Boundary of each surface Right mosaic Left mosaic Plane fit for each surface using disparity between mosaics Corners through Curvature Scale Space Edges through line fit between corners CAD model of each building DTM Digital Elevation Model (DEM) Reconstructed scene

43. 43 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Results … … Raw Video Frames 1 2 20 29 After Pre- processing … … Video collected over Center for Imaging Science,RIT using WASP LT at an altitude of 1000 ft and with an overlap of about 90% to 98% between frames

44. 44 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Stereo Mosaics Nadir LeftRight Stereo Pair

45. 45 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Object extraction Nadir Mosaic Tree regions Segmented Nadir Mosaic

46. 46 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Object modeling Noise Solar shadow Problems in 3D model of a building due to solar shadow and noise in images Hypotheses of symmetry in the building and flat surfaces on the roof of the building Reconstructed building from different perspectives Publications: 1.P. Gurram, E. Saber, and H. Rhody, “A Novel Triangulation Method for Building Parallel- Perspective Stereo Mosaics”, Proceedings of Electronic Imaging Symposium, SPIE, San Jose, CA, January 2007. 2.P. Gurram, S. Lach, E. Saber, H. Rhody, and J. Kerekes, “3D Scene Reconstruction through a Fusion of Passive Video and Lidar Imagery”, Proceedings of 36th Applied Imagery and Pattern Recognition Workshop (AIPR'06), Washington, D. C., 2006

47. 47 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Situation 1: Edges not Visible in Video Solar shadow Noise There is no information in these cases as one planar surface merges with a neighboring surface at a different height during segmentation Video Lidar Raw Lidar CAD Model from Lidar Good Edges and Planes

48. 48 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Situation 2: Lidar Data is Undersampled The reconstruction begins to break down Edges misrepresented or missed altogether Segmentation Fails Video Good Edges, Corners, and Planes Lidar

49. 49 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Situation 3: Textureless Non-planar Surfaces No variation of disparity on the textureless surface Fit a spherical model to the data using Levenberg- Marquardt algorithm Video Lidar RIT Observatory

50. 50 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Outline Motivation Block Diagram Preprocessing Ray Interpolation Object Modeling Conclusions and Future Work

51. 51 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Conclusions and Future Work Verification process is only as accurate as camera parameters are. Model is being overfitted to the data. Optimize the model according to the uncertainty in data using Bayesian networks. Evidence from other modalities like lidar data can be used to refine the model.

52. 52 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Bayesian or Belief network Image Understanding Algorithms (Visual and Lidar) Control System Belief Network

53. 53 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram An example of belief network Region BuildingsTrees Terrain Feature 1 Feature 2 Feature 3 Feature 4 Feature 5 Feature op. Known Structure Known Data Learning from data

54. 54 DIRS Meeting, February 8 th, 2008 Prudhvi Gurram Thank you!