IIIT HYDERABAD Image-based walkthroughs from partial and incremental scene reconstructions Kumar Srijan Syed Ahsan Ishtiaque C. V. Jawahar Center for Visual Information Technology, IIIT-Hyderabad Sudipta N. Sinha Microsoft Research, Redmond

IIIT HYDERABAD Introduction

IIIT HYDERABAD Problem Efficiently organize and browse these huge image collections? Keep Incorporating an incoming stream of images into an existing framework?

IIIT HYDERABAD Related Work World-Wide Media Exchange (WWMX) PhotoCompas Realityflythrough Aspen Movie Map Photowalker Sea of Images Google Streetview Photo Tourism

IIIT HYDERABAD Photo Tourism Computing correspondences Detect Features in each image Match keypoints between each pair of images For each pair, estimate an F-matrix and refine matches Chain pairwise matches into tracks Incremental SfM Select a good initial pair to seed reconstruction Add new images and triangulate new points Bundle adjust Snavely et. al, Photo Tourism: Exploring image collections in 3D Input Images Full Scene Reconstruction

IIIT HYDERABAD Bottlenecks and Issues Global scene reconstruction via incremental structure from motion (Sfm) – Sensitivity to the choice of the initial pair – Cascading of errors – O(N 4 ) in the worst case Snavely et. al, Photo Tourism: Exploring image collections in 3D

IIIT HYDERABAD Bottlenecks and Issues Timing Breakdown Snavely et. al, Photo Tourism: Exploring image collections in 3D Full Scene Reconstruction for Trafalgar Square dataset with 8000 images took > 50 days

IIIT HYDERABAD Our approach “ In a walkthrough, users primarily observe near by overlapping images.” Advantages: – Robustness to errors in incremental SfM module – Worst case linear running time – Scalable – Incremental Independent Partial Scene Reconstructions instead of Global Scene Reconstruction

IIIT HYDERABAD Partial Reconstructions Image Match Compute Matches Refine Matches Compute partial Reconstructions Standard SfM Correct Match Incorrect Match

IIIT HYDERABAD Visualization Interface User interface and navigation Input images Verified neighbors Sample image Partial reconstruction

IIIT HYDERABAD Global vs. Partial Global : Allows transition to any image Partial : Allows transition to a limited number of overlapping images A -> B implies B -> A A B B A

IIIT HYDERABAD Incremental insertion New Image Match Geometric Verification Compute Partial Scene Reconstruction Improve Connectivity

IIIT HYDERABAD Dataset Fort Dataset 5989 images Golconda Fort, Hyderabad

IIIT HYDERABAD Results

IIIT HYDERABAD Results

IIIT HYDERABAD Results Courtyard Dataset with 687 images Initialized with 200 images Added 487 image one by one Largest CC of 674 images.

IIIT HYDERABAD Conclusion Image navigation system based on partial reconstructions can effectively be used to navigate through large collections of images. Robustness to errors Able incorporate more images as they become available.

IIIT HYDERABAD Future Work Complete automation – Download images directly from the internet – Add into the framework

IIIT HYDERABAD Acknowledgements “Photo tourism: Exploring photo collections in 3D“ – Noah Snavely, Cornell University – Steven M. Seitz, University of Washington – Richard Szeliski, Microsoft Research

IIIT HYDERABAD Acknowledgements “Visual Word based Location Recognition in 3D models using Distance Augmented Weighting” – Friedrich Fraundorfer, Marc Pollefeys ETH Zürich – Changchang Wu,Jan- Michael Frahm,Marc Pollefeys - UNC Chapel Hill

IIIT HYDERABAD Thank You Questions