Presentation is loading. Please wait.

Presentation is loading. Please wait.

Estimating Camera Pose from a Single Urban Ground-View Omnidirectional Image and a 2D Building Outline Map Tat-Jen CHAM (with Arridhana Ciptadi, Wei-Chian.

Similar presentations


Presentation on theme: "Estimating Camera Pose from a Single Urban Ground-View Omnidirectional Image and a 2D Building Outline Map Tat-Jen CHAM (with Arridhana Ciptadi, Wei-Chian."— Presentation transcript:

1 Estimating Camera Pose from a Single Urban Ground-View Omnidirectional Image and a 2D Building Outline Map Tat-Jen CHAM (with Arridhana Ciptadi, Wei-Chian Tan, Minh-Tri Pham, Liang-Tien Chia) Center for Multimedia & Network Technology (CeMNet) School of Computer Engineering Nanyang Technological University, Singapore

2 Estimating Camera Pose from a Single Urban Ground-View Omnidirectional Image and a 2D Building Outline Map Tat-Jen CHAM (with Arridhana Ciptadi, Wei-Chian Tan, Minh-Tri Pham, Liang-Tien Chia) Center for Multimedia & Network Technology (CeMNet) School of Computer Engineering Nanyang Technological University, Singapore

3 Urban Landmarks Those easy to recognize Those that aren’t
© Anirudh Koul © kevincole © qureyoon

4 “Back-to-Basics” Map Reading!
An image or images taken from a single location, at probe time A plan-view outline map Won’t consider GPS GPS reception bad in high- rise urban areas GPS can be jammed or spoofed

5 Related Work and Differences
Appearance-based matching in urban areas Robertson & Cipolla BMVC04, Yeh et al. CVPR04, Zhang & Košecká 3DPVT06 General wide-baseline stereo / multi-view (but not targeted for searching through significant-sized datasets) Bay et al. CVPR05, Mičušík et al. CVPR08, Schindler et al. 3DPVT06, Schmid & Zisserman IJCV00, Werner & Zisserman ECCV02 Key differences here: No prior appearance information Only a 2D plan-view geometric map available No stereo / multi-view Images are taken from single location

6 A Geometric Matching Paradigm
Assume buildings are vertical planar extrusions Match building corners in map  vertical corner lines in rectified image Significant building corners Not façade details / painted edges Geometric Signature

7 2D Geometric Image Features
Basic Lines (2D)

8 2½D Geometric Image Features
David Marr’s bottom-up visual perception framework Image Primal Sketch  2½D Sketch  3D model Augmented Lines (2D + adjacent 3D normals)

9 2½D Geometric Image Features
David Marr’s bottom-up visual perception framework Image Primal Sketch  2½D Sketch  3D model Elemental Planes (2D + fixed depth ratios of vertical boundaries)

10 2½D Geometric Image Features
David Marr’s bottom-up visual perception framework Image Primal Sketch  2½D Sketch  3D model Structural Fragments (piecewise 3D structures with unknown scales)

11 Geometric Signatures – Uniqueness Analysis Under Ideal Conditions
Structural Fragments (3D structure with unknown scale) Strong match Elemental Planes (2D + fixed depth ratios) Basic Lines (2D) Augmented Lines (2D + 3D normals) Poor match

12 Overview of Localization Method
BOTTOM-UP Query image Extract vertical corners + normals Recover elemental planes with 3D normals Link into plan-view structural fragments (modulo similarity) Calibration from vanishing points TOP-DOWN 2D map Camera pose Geometric hashing lookup for correspondence candidates Voting-based estimate of optimal camera pose

13 Estimation of Quasi-Manhattan Vanishing Points
Use EM algorithm (Schindler et al. 3DPVT 2006) Details in paper Image rectification  3D verticals become || to image y-axis

14 Vertical Corner Line Hypothesis (VCLH)
Hypotheses for corners of buildings Based on heuristics 3 Categories: Uni-Normal Augmented Line Bi-Normal Augmented Line Basic line

15 Elemental Planes Elemental Plane:
2 VCLHs connected by groups of collinear horizontal edges Same plane normals on linked sides Invariant Depth Ratio:

16 Structural Fragments Structural fragment
Sequence of adjacent elemental planes sharing bi-normal VCLHs Full 3D structure (modulo scale)

17 More Examples Elemental Planes Structural Fragments

18 Matching with Structural Fragments
Exhaustive testing: Correspondence structural fragment of l planes  l linked building edges Best-fit matching with error Consensus support C from other VCLHs Vote in pose-space accumulator array Vote score: Complexity: O(n), n = # of building corners in map 8s per search on Matlab

19 Matching Example with Structural Fragments
Inconsistent matches Consistent matches

20 Experiments – Dataset I
Bronx neighborhood of Woodstock Google Street View images (total 212) 53 unique locations, 4 images per location (shown in quads) Manually created building outline plan view map 111 buildings with 885 corners

21 Experiments – Dataset II
Singapore government housing (HDB) estate Self-collected images (total 120) 30 unique locations, 4 images per location Manually created building outline plan view map 20 mega buildings with 659 corners

22 % of test probes where correct pose is better than this rank
Matching Results Compare probe signature to signatures at 3600 grid locations, and sort matching scores Find rank of ground truth % of test probes where correct pose is better than this rank Selectivity of 0-10% Match ranks

23 Dataset II Example Correct Matches
Example results for matching 3D models are only used for visualizing results

24 Observations This is a start to solving a challenging problem
difficult even for humans Results are mixed: Selectivity is very high 57-70% of correct poses within top-1% selectivity (36 out of 3600) But need to be higher to be end-usable Yet in ideal conditions signatures appear very discriminative Main challenges False VCLH negatives (some) building corners not detected due to poor resolution, etc. False VCLH positives (many) Windows / other façade features often misdetected as corners Architectural designs are seldom perfect extrusions Overhangs, balconies, fire escapes, etc.

25 Concluding Remarks Geometric features can be powerful for discriminating locations Do not always have to rely on prior appearance data Intelligent extension to geometric 2½D features 2D  2D+normals  2D+depth ratios  3D (mod scale) Informal test in ideal conditions show excellent discriminating power Key challenge lies in more robust image analysis Needs robustness to noise and minor deviations from map Future Work Use existing results to bootstrap more advanced (and costly) registration techniques E.g. top-down bundle adjustment working directly on raw image intensities, rather than detected edgels

26 Credits Joint work with Arridhana Ciptadi Wei-Chian Tan Minh-Tri Pham
Clement Liang-Tien Chia Thanks Teck-Khim Ng Zahoor Zafrulla Rudianto Sugiyarto Research Sponsor Project Tacrea Grant Defence Science & Technology Agency (DSTA), Singapore

27 Thank You

28 Scene Assumptions Quasi-Manhattan World
Vertical direction is orthogonal to all horizontal directions Horizontal directions need not be orthogonal to each other Vertical Extrusion Model Each building is a vertical extrusion of a ground-plane cross section Implies buildings have simple vertical planar facades

29 Case 1: Matching Basic Line VCLHs
Possible Procedure: 3 pairwise basic line correspondences between image and map  unique basis for pose RANSAC Randomly select any 3 pairs of line correspondences Verify with other lines Exhaustive search computational cost: n is the number of building corners in a map

30 Case 2: Matching Augmented Line VCLHs
Possible Procedure: 2 pairwise augmented line correspondences for unique pose Pose basis: 2 positions + 1 normal Other excess normals used as quick compatibility test RANSAC Verify with other lines only when compatibility tests passed Exhaustive search computational cost: k<1 is the combined factor for reduced cost of quick compatibility test * unknown % of bases

31 Case 3: Matching with Elemental Planes
Possible Procedure: Exhaustive testing: Correspondence = 1 elemental plane to 1 map building edge Basis for computing pose Voting-based pose estimation Vote in pose-space accumulator array Each vote score from consensus support of other VCLHs Computational cost: m = n is the number of map building edges

32 Geometric Hashing Geometric hashing explored as a rapid shortlisting stage Get correspondence candidates between detected structural fragments and map building contours Offline Preprocessing Phase: For each building contour in map, use each contour edge as basis Other corners of the building are transformed into the canonical frame Each corner logs (building_id, basis) in related hash bin Online Lookup Phase: For each structural fragment in image, select an elemental plane as basis For other VCLHs in the structural fragment, look up hash bins and vote for building and basis. Building + basis with high votes are correspondence candidates

33 Dataset I Results – Geometric Hashing
Use large hash bins to minimize missing correct matches (false negatives) Shortlisting selectivity Speed Without hashing: 8 seconds per image on Matlab With hashing: 4 seconds (actual hashing lookup <1s) # of links per structural fragment 3 4 5 Avg # of correspondence candidates 212.98 91.06 113.33 Selectivity (out of 885) 24.1% 10.3% 12.8%

34 Geometric Signatures – Uniqueness Analysis Under Ideal Conditions
How discriminative are geometric signatures comprising 2½D features? Compute matching score: Signature at an arbitrary spot to other signatures at all locations Under ideal noise-free conditions All building corners are perfectly detectable Generate probe signatures directly from plan-view map Informal analysis Expect uniqueness to be dependent on specific location and design of buildings

35 Geometric Signatures – Uniqueness Analysis Under Ideal Conditions
Structural Fragments (3D structure with unknown scale) Strong match Elemental Planes (2D + fixed depth ratios) Basic Lines (2D) Augmented Lines (2D + 3D normals) Poor match

36 Reduction to 1D Image Matching
Vertical lines parallel to image y-axis Problem reduces to: matching points in a 2D space to a 1D point signature 2D camera extrinsic parameters: 2D position of optical center Angle of optical axis Signatures normalized to unit focal length

37 Camera Calibration from Vanishing Points
Manhattan world 3 v.p.’s  3 orthogonal directions Also computes focal length and principal point Quasi-Manhattan world Ground-vertical v.p. orthogonal to other v.p.’s Assume principal point at image center Obtain 3D v.p. directions w.r.t. camera reference frame In reality, we integrate calibration as part of the iterative v.p. estimation process

38 Potential Future Directions
Exploit localized architectural design “language”? priors to improve geometric feature detection in poor quality images predict occluded parts of higher order geometric features that form the local architectural “vocabulary” Investigate if reasonable to have prior distribution that buildings close by have similar geometric designs


Download ppt "Estimating Camera Pose from a Single Urban Ground-View Omnidirectional Image and a 2D Building Outline Map Tat-Jen CHAM (with Arridhana Ciptadi, Wei-Chian."

Similar presentations


Ads by Google