Computer Vision Lab Seoul National University Keyframe-Based Real-Time Camera Tracking Young Ki BAIK Vision seminar : Mar Computer Vision Lab.

Computer Vision Lab Seoul National University Reference Keyframe-Based Real-Time Camera Tracking - ICCV Zilong Dong, Guofeng Zhang, Jiaya Jia Hujun Bao

Computer Vision Lab Seoul National University Outline Introduction System overview Offline map building Online camera tracking Demonstration Conclusion Opitimal keyframe selection Keyframe recognition using vocabulary tree How to use multiple thread

Computer Vision Lab Seoul National University What is the purpose? Real-time camera tracking Condition Known internal camera parameters The map of environment is given… 3D Map measurement Input frame 6-DOF camera pose estimation!! 3D camera pose

Computer Vision Lab Seoul National University System overview Real-time camera tracking Novel keyframe selection algorithm How to build a map How to estimate the camera pose Offline mapping using the conventional MVG method Online camera tracking Efficient key frame recognition method Parallel computing framework

Computer Vision Lab Seoul National University Offline mapping stage reference images Image Acquisition Video data for the target environment

Computer Vision Lab Seoul National University Offline mapping stage Image Acquisition Feature Extraction and Matching SIFT descriptors

Computer Vision Lab Seoul National University Offline mapping stage Image Acquisition Conventional MVG Feature Extraction and Matching 3D Map Building

Computer Vision Lab Seoul National University Offline mapping stage Keyframe set Image AcquisitionKeyframe selection Feature Extraction and Matching 3D Map Building Optimal Keyframe Selection For efficiency of map management

Computer Vision Lab Seoul National University Offline mapping stage Image Acquisition Vocabulary tree Feature Extraction and Matching 3D Map Building Optimal Keyframe Selection Vocabulary Tree Construction For fast keyframe selection in tracking stage… Contribution!!

Computer Vision Lab Seoul National University Why keyframe? Which one is good for a map? To solve the scalability problem of the map All referencesKeyframe A small number of …

Computer Vision Lab Seoul National University Optimal Keyframe Selection Previous methods select manually… using heuristic method… Proposed method select optimal keyframe Containing many salient features as possible Reducing non-distinctiveness in matching Features should be distributed evenly in keyframes

Computer Vision Lab Seoul National University Optimal Keyframe Selection Cost minimization Problem formulation… to find optimal keyframes Completeness TermRedundancy Term weight Saliency of features Distribution of features Distinctiveness

Computer Vision Lab Seoul National University Optimal Keyframe Selection Completeness Term Definition Reference images Keyframes

Computer Vision Lab Seoul National University Optimal Keyframe Selection Completeness Term Definition Feature clusters Reference image set 2D image position

Computer Vision Lab Seoul National University Optimal Keyframe Selection Completeness Term Definition Set of superior features ~ Long-term surviving (or representative) features in reference images

Computer Vision Lab Seoul National University Optimal Keyframe Selection Completeness Term Saliency ~ The larger s(χ) means the higher saliency! DoG map

Computer Vision Lab Seoul National University Optimal Keyframe Selection Completeness Term Distribution The smaller d (χ) → Features are well distributed !! Feature density 31x31 window

Computer Vision Lab Seoul National University Optimal Keyframe Selection Completeness Term =3 : controls the sensitivity of feature density… : the superior feature set in the keyframe set F

Computer Vision Lab Seoul National University Optimal Keyframe Selection Redundancy Term Normalization The number of keyframes which contain feature χ Small E r (F) indicates low redundancy!

Computer Vision Lab Seoul National University Keyframe selection Algorithm Let = 0 Initialization ~

Computer Vision Lab Seoul National University Keyframe selection Algorithm ~ Add Keyframe

Computer Vision Lab Seoul National University Keyframe selection Algorithm ~ Add Keyframe

Computer Vision Lab Seoul National University Keyframe selection Algorithm ~ Check

Computer Vision Lab Seoul National University Keyframe selection Algorithm ~ Add Keyframe

Computer Vision Lab Seoul National University Keyframe selection Algorithm ~ Add Keyframe

Computer Vision Lab Seoul National University Keyframe selection Algorithm ~ Check

Computer Vision Lab Seoul National University Vocabulary Tree Construction For fast keyframe recognition… Root node List Branch = 8, Tree depth = 5 All F Mean patch All feature descriptor

Computer Vision Lab Seoul National University Vocabulary Tree Construction To build each node… Root node List Clustering by K-means method Branch = 8, Tree depth = 5

Computer Vision Lab Seoul National University Vocabulary Tree Construction To build each node… Root node List Clustering by K-means method List Computing mean patch Building keyframe list Branch = 8, Tree depth = 5

Computer Vision Lab Seoul National University Vocabulary Tree Construction Finally… Root node List Clustering by K-means method List Computing mean patch Building keyframe list List Branch = 8, Tree depth = 5

Computer Vision Lab Seoul National University Online tracking stage SIFT Feature Extraction SIFT feature Input image

Computer Vision Lab Seoul National University Online tracking stage SIFT Feature Extraction Vocabulary tree Keyframe Recognition

Computer Vision Lab Seoul National University Online tracking stage SIFT Feature Extraction Keyframe Recognition Keyframe-based matching Input imageSelected F

Computer Vision Lab Seoul National University Online tracking stage SIFT Feature Extraction Keyframe Recognition Keyframe-based matching Camera Pose Estimation 3-point algorithm 3-D map 2-D point

Computer Vision Lab Seoul National University Keyframe recognition With SIFT patches of input image… Root node List Matching input patch & mean patches Vocabulary tree Key frame voting memory

Computer Vision Lab Seoul National University Keyframe recognition With SIFT patches of input image… Root node List Key frame voting memory Move patch And Update voting memory

Computer Vision Lab Seoul National University Keyframe recognition With SIFT patches of input image… Root node List Key frame voting memory Select keyframe with largest voting value

Computer Vision Lab Seoul National University Processing time Process time per frame with a single thread… Frame rate is around 6fps!!!

Computer Vision Lab Seoul National University Processing time Using multiple-thread with pipeline SIFT Feature Extraction Time axis Select Keyframe Matching SIFT Feature Extraction Pose Estimation Thread#1Thread#2 Thread#3Thread#4 Select Keyframe Matching Pose Estimation 20 fps First frame is time consuming!

Computer Vision Lab Seoul National University Demonstration

Computer Vision Lab Seoul National University Conclusion Real-time camera tracking is presented !! The optimal keyframe selection algorithm Real-time tracking is presented! Minimizing the energy formulation! Fast keyframe recognition using vocabulary tree Multiple thread with pipeline structure Limitation Incremental updating of VT is not considered… Proposed process is only available for known environment.

Computer Vision Lab Seoul National University Q&AQ&A