Scalability of Local Image Descriptors Björn Þór Jónsson Department of Computer Science Reykjavík University Joint work with: Laurent Amsaleg (IRISA-CNRS) Herwig Lejsek Friðrik Heiðar Ásmundsson

Image Search Approaches Text based – Keywords, inverted indices Global descriptors – Color, texture, … Local descriptors

Local Image Descriptors Based on Interest Points – Hundreds of descriptors per image Invariant to many image modifications – Scaling, rotation, … Many Variants – RDTQ (n=24) – SIFT (n=128) – PCA-SIFT (n=36) x1x1 x2x2 x3x3 x4x4 xnxn x1x1

Copyright Protection Stolen & modified images on Web Local descriptors can identify originals 30,000 images  35 million SIFT desc’s Need efficient database support

Database Support Problems – High dimensionality – Many query descriptors State-of-the-art: – L 2 : R-tree, Pyramid tech., VA-file,... – Ranking: Median rank aggregation – Local: Clustering based, LSH, work at INA – Best: A well coded sequential scan 30,000 images  ~10 minutes

Scalability Challenges Efficiency – Response time – Throughput Effectiveness – Detection rate – False positives Descriptor Creation – Number of descriptors – Creation time

Outline Application: Local Descriptors Motivation: Lack of Database Support Key Issues: Efficiency & Effectiveness Effectiveness: The Eff² Descriptors Indexing: PvS-framework and NV-tree Optimization: Stop Rules Scalability: The NV-Network

Eff 2 Descriptors Goals – More descriptors at higher scale – Handle most modifications well Improvement of the SIFT method – “Only” 72 Dimensions – Scale-space norm. (gamma correction) – Improved edge filtering – ~800 most invariant descriptors Eff 2 = Effective x Efficient

Copyright violation workload – 108 original images – 20 Stirmark + 6 “hard” modifications – Almost 3K “pirated” images Most images are OK – 1K images not in the database Two indexed collections – 30K images – 300K images Experimental Setup

Modification Examples Original JPEG 15 MEDIAN 9 ROTSCAL COTR 1 CONV 2 CONV 4 CONV 5

Result Quality: 30K Images SIFT: – ~ 35M descriptors – Matches fewer descriptors – 240 misses – 12 false positives Eff 2 : – ~ 20M descriptors – Matches more descriptors – 32 misses (CONV 5) – 4 false positives

Scalability: 300K Images 30K images: – ~ 20M descriptors – 32 misses – 4 false positives 300K images: – ~ 200+M descriptors – Matches 10% fewer descriptors – 90 misses – 3 false positives

Median Rank Aggregation B + -tree... P1P1 P2P2 PdPd PvS Index Query Descriptor Fagin et al., ACM SIGMOD 2003 Turns d-dim search into many 1-dim searches (random projections) Uses ranking rather than distance Does not work for local descriptors

PvS-Index Creation Descriptor Collection S1S1 S2S2 S3S3 P2P2 P1P1 … P3P3 Projections vs. Segmentations Tree index Leafs are I/O sized (128KB) Goal: One leaf per query descriptor

PvS-Index Search S1S1 S2S2 S3S3 … B + -tree S2S2 P1P1 P2P2 X X P3P3 Query Descriptor

PvS Index Query Descriptor PvS-Framework Summary Repeated projections and segmentations –Partitions collection into leafs of one I/O Uses median rank aggregation High quality results with 3 indices –Three disk I/Os per query descriptor –800 query desc in <40 seconds (1 disk) Query time independent of collection size –Index size is exponential, but disk is cheap

300K Images Revisited 30K images: – ~ 20M descriptors – 3 x 3 GB indices – 3 x 25 mins to index – Typical query: 40 sec 300K images: – ~ 200+M descriptors – 3 x 56 GB indices – 3 x 6 hours to index – Typical query: ~Same

PvS-Framework: Issues Three I/Os per query descriptor – Upper bound & Lower bound – With 2 indices, quality suffers – Segments are not “tight” enough Requires median rank aggregation – Low overhead, but patented by IBM Is using one index possible? – 67% fewer I/Os – Smaller memory requirements

The NV-Tree: Techniques Projections and Segmentations, but... Partition based on data distribution – Variable depth  Improved result quality – Flexible index structure  Updates Improved line selection – Based on variance  Better quality Compact approximate storage – Better quality OR Smaller segments – Smaller index

The NV-Tree: Early Results It works! For local descriptor queries – Individual queries require two indices (intersection) For large collections – Reduced likelihood of random votes For “meaningful” descriptors – Near neighbors in not-so-dense areas

Optimization: Stop Rules Better performance for YES/NO answer Based on probability that a query descriptor gets a “random” vote Many early matches: YES Few matches after a while: NO 80-90% better response time

Scalability: NV-Network Motivation: Apply more disks Benefit: Scalability & Throughput Method: Replicate NV-Trees – A single coordinator assigns workload  Tries to maximize buffer hit ratio – Many workers process local descriptors – The coordinator aggregates results Results: Scales linearly with #workers – 2 indices  Larger memory requirements

Some Results Was: 2+ hrs Now: 2+ sec

Scalability of local image descriptors – Efficiency: NV-Tree, NV-Network, Stop rules – Effectiveness: Eff² descriptors – Index and search 200+ million ld’s Future work – Experiments on NV-tree – Combat index size explosion – Multi-query optimization for throughput – Utilize collective memory of NV-Network – Applications Summary