1. Dana Cobzas-PhD thesis Image-Based Models with Applications in Robot Navigation Dana Cobzas Supervisor: Hong Zhang

2. Dana Cobzas-PhD thesis 3D Modeling in Computer Graphics  Graphics model: 3D detailed geometric model of a scene  Goal: rendering new views New view Real scene Geometric model + texture Rendering Aquisition Range sensors Modelers [Pollefeys & van Gool]

3. Dana Cobzas-PhD thesis Mapping in Mobile Robotics Robot Map sensors Map Building Localization/ Tracking Navigation environment  Navigation map: representation of the navigation space  Goal: tracking/localizing the robot

4. Dana Cobzas-PhD thesis Same objective: How to model existing scenes? Traditional geometry-based approaches: = geometric model + surface model + light model - Modeling complex real scenes is slow - Achieving photorealism is difficult - Rendering cost is related to scene complexity + Easy to combine with traditional graphics Alternative approach: image-based modeling: = non-geometric model from images - Difficult to acquire real scenes - Difficult to integrate with traditional graphics + Achieving photo-realism is easier if starting from real photos + Rendering cost is independent on scene complexity In this work we combine the advantages of both for mobile robotics localization and predictive display

5. Dana Cobzas-PhD thesis This thesis Investigates the applicability of IBMR techniques in mobile robotics. Questions addressed:  Is it possible to use an IBM as navigation map for mobile robotics?  Do they provide desired accuracy for the specific applications – localization and tracking?  What advantages do they offer compared to traditional geometric-based models?

6. Dana Cobzas-PhD thesis Approach Solution:  Reconstructed geometric model combined with image information  2 models Model1: calibrated: panorama with depth Model2: uncalibrated: geometric model with dynamic texture  Applications in localization/tracking and predictive display

7. Dana Cobzas-PhD thesis Model1: Panoramic model

8. Dana Cobzas-PhD thesis Model1: Overview Standard panorama: - no parallax, reprojection from the same viewpoint Solution – adding depth/disparity information: 1.Using two panoramic images for stereo 2.Depth from standard planar image stereo 3.Depth from laser range-finder

9. Dana Cobzas-PhD thesis Depth from stereo Cylindrical image-based panoramic models + depth map  Trinocular Vision System (Point Gray Research)

10. Dana Cobzas-PhD thesis Depth from laser range-finder 180 degrees panoramic mosaic Corresponding range data (spherical representation) Data from different sensors: requires data registration  CCD camera  Laser rangefinder  Pan unit

11. Dana Cobzas-PhD thesis Model 1: Applications Absolute localization:  Input: image+depth  Features: planar patches vertical lines  Input: intensity image  Assumes: approximate pose  Features: vertical lines Incremental localization: Predictive display:

12. Dana Cobzas-PhD thesis Model 2: Geometric model with dynamic texture

13. Dana Cobzas-PhD thesis Model 2: Overview Input images Model Applications Geometric model Dynamic texture Tracking Rendering

14. Dana Cobzas-PhD thesis Tracked features Structure from motion algorithm poses structure Geometric structure

15. Dana Cobzas-PhD thesis Dynamic texture I 1 t I Input Images Re-projected geometry Texture Variability basis

16. Dana Cobzas-PhD thesis 3D SSD Tracking  Goal: determine camera motion (rot+transl) from image differences  Assumes: sparse geometric model of the scene  Features: planar patches past motion current motion past warp current warp differential warp differential motion initial motion 3D Model

17. Dana Cobzas-PhD thesis Tracking example

18. Dana Cobzas-PhD thesis Tracking and predictive display  Goal: track robot 3D pose along a trajectory  Input: geometric model (acquired from images) and initial pose  Features: planar patches

19. Dana Cobzas-PhD thesis Thesis contributions Contrast calibrated and uncalibrated methods for capturing scene geometry and appearance from images: panoramic model with depth data (calibrated) geometric model with dynamic texture (uncalibrated) Demonstrate the use of the models as navigation maps with applications in mobile robotics absolute localization incremental localization model-based tracking predictive display

20. Dana Cobzas-PhD thesis Thesis questions  What advantages do they offer compared to traditional geometric based models?  The image information is used to solve data association problem.  Model renderings are used for predicting robot location for a remote user.  Do they provide desired accuracy for the specific applications – localization, tracking?  The geometric model (reconstructed from images) is used for localization/tracking algorithms. The accuracy of the algorithm depends on the accuracy of the reconstructed model.  The model accuracy can also be improved during navigation as different levels of accuracy are needed depending on the location (large space/narrow space) – future work.  Is it possible to use an image-based model as navigation map for mobile robotics?  A combination of geometric and image-based model can be used as navigation map.

21. Dana Cobzas-PhD thesis Comparison with current approaches Mobile Robotics Map + Image information for data association + Complete model that can be rendered – closer to human perception - Concurrent localization and matching (SLAM-Durrant-Whyte) - Invariant features (light, occlusion) (SIFT-Lowe) - Uncertainty in feature location (localization algorithms) Graphics Model (dynamic texture model-hybrid image+geometric model) + Easy acquisition: non-calibrated camera (raysets, geometric models) + Photorealism (geometric models) + Traditional rendering using the geometric model (raysets) -Automatic feature detection for tracking – larger scenes -Denser geometric model (relief texture) -Light-invariance (geometric models, photogrammetry)

22. Dana Cobzas-PhD thesis Future work Mobile Robotics Map  Improve map during navigation  Different ‘map resolutions’ depending on robot pose  Incorporate uncertainty in robot pose and features  Light, occlusion invariant features  Predictive display: control robot’s motion by ‘pointing’ o ‘dragging’ in image space Graphics Model (dynamic texture)  Automatic feature detection for tracking  Light-invariant model  Compose multiple models into a scene based on intuitive geometric constraints  Detailed geometry (range information from images)