6How to handle sets of features? Want to compare, index, cluster, etc. localrepresentations, but:• Each instance is unordered set of vectors• Varying number of vectors per instance
7Comparing sets of local features Previous strategies:Match features individually, vote on small sets to verifyExplicit search for one-to- one correspondencesBag-of-words: Compare frequencies of prototype features
8optimal partial matching Pyramid match kernelOptimal match: O(m3)Pyramid match: O(mL)m = # featuresL = # levels in pyramidoptimal partial matching
9Pyramid match: main idea Feature space partitions serve to “match” the local descriptors within successively wider regions.descriptor space
10Pyramid match: main idea Histogram intersection counts number of possible matches at a given partitioning.
11Image search with matching- sensitive hash functions • Main idea:– Map point sets to a vector space in such a way that a dot product reflects partial match similarity (normalized PMK value).– Exploit random hyperplane properties toconstruct matching-sensitive hash functions.– Perform approximate similarity search onhashed examples.
12Locality Sensitive Hashing (LSH) Guarantee “approximate”-nearest neighbors in sub-linear time, given appropriate hash functions.Randomized LSHfunctionsXiNhr1…rk<< NQ110101hr1…rk110111Q111101
13LSH functions for dot products The probability that a random hyperplane separates two unit vectors depends on the angle between them:Corresponding hash function:A)High dot product:unlikely to splitB)Lower dot product: likely to split
14Metric learningThere are various ways to judge appearance/shape similarity…but often we know more about (some) data than just their appearance.
15Metric learningExploit partially labeled data and/or (dis)similarity constraints to construct more useful distance functionCan dramatically boost performance on clustering, indexing, classification tasks.Various existing techniques
16Fast similarity search for learned metrics • Goal:– Maintain query time guarantees while performing approximate search with a learned metric• Main idea:– Learn Mahalanobis distance parameterization– Use it to affect distribution from which random hash functions are selected• LSH functions that preserve the learned metric• Approximate NN search with existing methods
17Fast Image Search for Learned Metrics Learn a Malhanobis metric for LSHh( ) = h( )h( ) ≠ h( )It should be unlikely that a hash function will split examples like those having similarity constraints……but likely that it splits those having dissimilarity constraints.
18Summary • Local image features useful, important to handle efficiently • Introduced scalable methods to allow fast similarity search methods with– Local feature matching– Learned Mahalanobis metrics• Key idea: design hash functions that encode matching process, or the constraints provided