1. Local Invariant Feature Descriptors
Bin Fan
National Laboratory of Pattern Recognition (NLPR)
Institute of Automation, Chinese Academy of Sciences

4. 局部图像特征描述 —— 应用 Wide-Baseline Image Matching
Structure from Motion, Image-based Localization, Image Stitch
Object/Instance/Scene Recognition
Object Detection
Image Retrieval

5. Structure From Motion

6. Object Recognition
Database …

7. Categories of Descriptors
Design method:
Handcrafted Descriptors
Data-driven Descriptors

8. Developments: Handcrafted Descriptors
1999, SIFT [Citation: 23819]
2003, Shape Context
2006, SURF [Citation: 4093]
2008, SMD, DAISY
2009, OSID, CS-LBP
2010, BRIEF, HRI-CSLTP, BiCE
2011, ORB, BRISK, LIOP, MROGH
2012, FREAK, KAZE, SYM
2013, Line Context

9. Developments: Data-driven Descriptors
2004, PCA-SIFT
2007, LDE, Learning descriptor[Brown et al.]
2009, Best DAISY
2012, D-BRIEF, Learning descriptor by convex optimization[Simonyan et al.], BGM/LBGM, LDAHash
2013, BinBoost, SQ-SIFT/DAISY

10. Categories of Descriptors
Design method:
Handcrafted Descriptors
Data-driven Descriptors
Encode information:
Gradient-based Descriptors
Intensity-based Descriptors
Descriptor-based Descriptors

11. Gradient-Based Intensity-Based Descriptor-Based
SIFT、DAISY、BiCE、MROGH、BGM、LBGM、BinBoost、Learning Descriptor[Brown et al., Simonyan et al.]
Intensity-Based
CS-LBP、OSID、BRIEF、ORB、BRISK、FREAK、LDE、D-BREIF、LIOP
Descriptor-Based
LDAHash, LDP[Cai et al.,PAMI’11]

12. Categories of Descriptors
Design method:
Handcrafted Descriptors
Data-driven Descriptors
Encode information:
Gradient-based Descriptors
Intensity-based Descriptors
Descriptor-based Descriptors
Data type:
Floating-point Descriptors
Binary Descriptors

13. Floating-point Descriptors
SIFT、SURF、DAISY、CS-LBP、OSID、MROGH、LIOP、LBGM、LDE…
Binary Descriptors
BiCE、BRIEF、ORB、FREAK、BRISK、BGM、BinBoost、LDAHash、D-BRIEF…

14. Floating point descriptors
Name
Mem.
Com.
Mat.
Dist.
Rob.
Floating point descriptors
SIFT
●●●
SURF ●○ ●●
●●○
DAISY
●●●●○
●●●●
●●●○
LIOP
MROGH
Binary descriptors
BRIEF ○ ●
ORB
FREAK
D-BRIEF
BinBoost

15. Handcrafted Descriptors - SIFT
SIFT Descriptor [Lowe’99]
Binning of Spatial Coordinates and Gradient Orientations
Soft Assignment of Binning
4x4 spatial grids, 8 gradient orientations, 128 dim SIFT
Normalization

16. Handcrafted Descriptors - DAISY
DAISY Descriptor [Tola et al.’08]
Log-polar grid arrangement
Gaussian pooling of histograms of gradient orientations
Efficient for dense computation, but not for sparse keypoints!

17. Descriptor Learning – Data Driven Methods
Brown et al.’s method [CVPR’07, ICCV’07, PAMI’ 12]
Learning
Normalized Patch
Low-level feature extraction
Smooth
Spatial pooling
Post process
Projection
Descriptor

18. Descriptor Learning – Data Driven Methods
Brown et al.’s method [CVPR’07, ICCV’07, PAMI’ 12]
Pre-defined low level features: gradient-based, filter bank based
Pre-defined spatial poolings: SIFT-like, DAISY-like, GLOH-like
Optimized combination of low level feature + spatial pooling
Projection: PCA, LDE …
1st: DAISY-like spatial pooling + filter bank [high Dim]
2nd: DAISY-like spatial pooling + gradient [moderate Dim]
PCA is better than LDE for projecting descriptor

19. Descriptor Learning – Data Driven Methods
Simonyan et al.’s method [ECCV’12]
Learning
Normalized Patch
Gradient map calculation
Smooth
Spatial pooling
Projection
Descriptor
Spatial pooling is constrained to rings
Using L1 regularization to select pooling rings from a large pool
Max-Margin based objective function [convex]
Best reported results in the Brown et al.’s dataset

20. Handcrafted Binary Descriptors
Pioneering work: LBP

21. Handcrafted Binary Descriptors
BRIEF [ECCV’10, PAMI’12]
Construct descriptor by binary tests:
Binary tests:
Pre-defined positions for binary tests:

22. Handcrafted Binary Descriptors - BRIEF
Low memory, Fast to compute and match
Limited performance

23. Handcrafted Binary Descriptors
FREAK [CVPR’12]
Organizing sampling points analogous to retina structure

24. Learning Binary Descriptors
D-BRIEF [ECCV’12]
Linear representation of projection matrix by Box/Gaussian/Rect filters
Approximate projection by filter responses
Efficient computation of Box/Gaussian/Rect filter responses
Binarization after discriminative projection
Extremely compact [only 32bits = 4 bytes]

25. Learning Binary Descriptors
BGM [NIPS’12]
(P1(1), P2(1),c(1))
(P1(2), P2(2),c(2))
(P1(n), P2(n),c(n)) …
Explore gradient orientation maps as weak learners
Each bit is construct by one weak learner
Select discriminative gradient orientation maps by boosting

26. Learning Binary Descriptors
BinBoost [CVPR’13]
Each bit as a linear combination of many gradient orientation maps
Optimization based on boosting
Very compact [64 bits = 8 bytes]

27. Dataset and Evaluation
Different contexts
Image Matching
Object/Instance Recognition
Image Retrieval

28. Dataset and Evaluation: Matching
Oxford dataset [2D scenes]: popular benchmark
K. Mikolajczyk, C. Schmid,  A performance evaluation of local descriptors. PAMI’05 …

29. Dataset and Evaluation: Matching
Oxford dataset [2D scenes]: popular benchmark
Evaluation protocol: recall vs. 1-precision

30. Dataset and Evaluation: Matching
Brown et al.’s dataset [image patches]: widely used for evaluation of learning based descriptors
M. Brown, G. Hua and S. Winder,  Discriminant Learning of Local Image Descriptors. PAMI’12
Three different subsets, each of which has more than 400k patch pairs
Liberty
Notre Dame
Yosemite

31. Dataset and Evaluation: Matching
Brown et al.’s dataset [image patches]: widely used for evaluation of learning based descriptors
Evaluation protocol: False Positive Rate(FPR) vs. Recall

32. Dataset and Evaluation: Recognition
Dataset: Ukbench, ZuBuD, …
Evaluation Protocol: Recognition rate, recall

33. Dataset and Evaluation: Retrieval
Dataset: Oxford/Paris Building, Holidays
Evaluation Protocol: mAP, Precision vs. Recall
AP(Average Precision): Precision across all recalls
mAP: mean AP of all queries

34. Resources OpenCV: http://opencv.org/ VLFeat: http://www.vlfeat.org/
SIFT, SURF, BRISK, BRIEF, ORB, FREAK
VLFeat:
SIFT, LIOP, Covariant Feature Detectors
Oxford VGG:
Authors’ pages…

35. Published Evaluations: Matching
K. Mikolajczyk and C. Schmid,  A Performance Evaluation of Local Descriptors. PAMI’05
P. Moreels and P. Perona,   Evaluation of Features Detectors and Descriptors based on 3D objects. IJCV’07
Anders Lindbjerg Dahl et al., Finding the Best Feature Detector-Descriptor Combination. 3DIMPVT’11
O.Miksik and K. Mikolajczyk, Evaluation of Local Detectors and Descriptors for Fast Feature Matching, ICPR’12
J. Heinly et al., Comparative Evaluation of Binary Features, ECCV’12

36. Published Evaluations: Classification/Recognition
K. Mikolajczyk et al.,  Local Features for Object Class Recognition. ICCV’05
E. Seemann et al.,   An Evaluation of Local Shape-Based Features for Pedestrian Detection. BMVC’05
M. Stark and B. Schiele, How Good are Local Features for Classes of Geometric Objects. ICCV’07
J. Zhang et al., Local Features and Kernels for Classification of Texture and Object Categories: A Comprehensive Study, IJCV’07
K. E. A. Van de Sande et al., Evaluation of Color Descriptors for Object and Scene Recognition, PAMI’10

37. Joint work with Zhenhua Wang
Our Work
Feature Description by Intensity Order Pooling
Local Intensity Order Pattern
Joint work with Zhenhua Wang

38. Feature Description by Intensity Order Pooling

39. Category of handcrafted descriptors
With a reference orientation:
SIFT, SURF, DAISY, CS-LBP …

40. Category of handcrafted descriptors
With a reference orientation:
SIFT, SURF, DAISY, CS-LBP …
+: encode spatial information, high discriminability
-: sensitive to orientation estimation error

41. Match vs. Orientation error
64%
36%

42. Category of handcrafted descriptors
With a reference orientation:
SIFT, SURF, DAISY, CS-LBP …
+: encode spatial information, high discriminability
-: sensitive to orientation estimation error
Distinctiveness
Robustness

43. Category of handcrafted descriptors
255/2π r
Without a reference orientation:
RIFT, Spin image
+: inherently rotation invariance, robust to orientation estimation error
-: discard some spatial information, limited discriminability

44. Category of handcrafted descriptors
Distinctiveness
Robustness
Without a reference orientation:
RIFT, Spin image
+: inherently rotation invariance, robust to orientation estimation error
-: discard some spatial information, limited discriminablity

45. Category of handcrafted descriptors
With a reference orientation:
SIFT, SURF, DAISY, CS-LBP …
+: encode spatial information, high discriminability
-: sensitive to orientation estimation error
Distinctiveness
Robustness
Without a reference orientation:
RIFT, Spin image
+: inherently rotation invariance, robust to orientation estimation error
-: discard some spatial information, limited discriminablity

46. Category of handcrafted descriptors
With a reference orientation:
SIFT, SURF, DAISY, CS-LBP …
+: encode spatial information, high discriminability
-: sensitive to orientation estimation error
Distinctiveness
Distinctiveness
Robustness
Robustness
Without a reference orientation:
RIFT, Spin image
+: inherently rotation invariance, robust to orientation estimation error
-: discard some spatial information, limited discriminablity

47. Construct a local coordinate for low-level feature computation
Our Solution
Construct a local coordinate for low-level feature computation
Gradient orientation maps [SIFT]
Center-symmetrical binary pattern [CS-LBP]

48. Our Solution
Pool low-level features by intensity orders … … …… …… …

49. Our Solution
Using multiple support regions

50. Using multiple support regions

51. Gradient orientation maps -> MROGH
Center-symmetrical binary pattern -> MRRID
Code:

52. Experiments Multiple Support Regions vs. Single Support Region MROGH
MRRID
SIFT
SR-i: Results of using the i-th support region
MR: Results of using multiple support region
Averaged results over 140 image pairs

53. Hessian-Affine, Viewpoint change
Experiments
Image Matching – Oxford Dataset
Hessian-Affine, Viewpoint change

54. Harris-Affine, Image Blur
Experiments
Image Matching – Oxford Dataset
Harris-Affine, Image Blur

55. Experiments Object Recognition: Datasets: 53 Objects, ZuBuD, Ukbench
265 images of 53 objects
Each object has 5 images of different viewpoints

56. Experiments Object Recognition: Datasets: 53 Objects, ZuBuD, Ukbench
1005 images of 201 buildings in the Zurich city
Each building has 5 images of different viewpoints, across seasons

57. Experiments Object Recognition: Datasets: 53 Objects, ZuBuD, Ukbench
10200 images of 2550 objects [first 4000 images used here]
Each object has 4 images of different viewpoints

58. Experiments 53 Objects Object Recognition: ZuBuD Ukbench RIFT SIFT
DAISY
MROGH
MRRID
37.0%
52.2%
61.2%
72.5%
57.4%
ZuBuD
66.8%
75.5%
83.1%
88.1%
78.6%
Ukbench
34.0%
48.2%
58.3%
74.0%
57.5%

59. Experiments
Recognition examples: 53 Objects
input images

60. Experiments
Recognition examples: ZuBuD
input images

61. Experiments
Recognition examples: Ukbench
input images

62. Local Intensity Order Pattern

63. Local Intensity Order Pattern
Explore the relative intensity relationship among neighboring points
Rotationally invariant computation of neighboring points’ intensities
Intensity order based pooling
Code:

64. Experiments
Image Matching: Oxford dataset

65. Experiments
Image Matching: Oxford dataset

66. Experiments
Image Matching: Complex Brightness Change

67. Questions?
Thank you