Scalability of Local Image Descriptors Björn Þór Jónsson Department of Computer Science Reykjavík University Joint work with: Laurent Amsaleg (IRISA-CNRS)

Slides:



Advertisements
Similar presentations
When Is Nearest Neighbors Indexable? Uri Shaft (Oracle Corp.) Raghu Ramakrishnan (UW-Madison)
Advertisements

Object Recognition using Local Descriptors Javier Ruiz-del-Solar, and Patricio Loncomilla Center for Web Research Universidad de Chile.
The A-tree: An Index Structure for High-dimensional Spaces Using Relative Approximation Yasushi Sakurai (NTT Cyber Space Laboratories) Masatoshi Yoshikawa.
Aggregating local image descriptors into compact codes
Presented by Xinyu Chang
School of Computer Science and Engineering Finding Top k Most Influential Spatial Facilities over Uncertain Objects Liming Zhan Ying Zhang Wenjie Zhang.
Searching on Multi-Dimensional Data
Mining Distance-Based Outliers in Near Linear Time with Randomization and a Simple Pruning Rule Stephen D. Bay 1 and Mark Schwabacher 2 1 Institute for.
Similarity Search in High Dimensions via Hashing
A NOVEL LOCAL FEATURE DESCRIPTOR FOR IMAGE MATCHING Heng Yang, Qing Wang ICME 2008.
Low Complexity Keypoint Recognition and Pose Estimation Vincent Lepetit.
Mining Time Series.
A Generic Framework for Handling Uncertain Data with Local Correlations Xiang Lian and Lei Chen Department of Computer Science and Engineering The Hong.
Fast High-Dimensional Feature Matching for Object Recognition David Lowe Computer Science Department University of British Columbia.
WISE: Large Scale Content-Based Web Image Search Michael Isard Joint with: Qifa Ke, Jian Sun, Zhong Wu Microsoft Research Silicon Valley 1.
Modern Information Retrieval
Algorithms for Nearest Neighbor Search Piotr Indyk MIT.
Distributed Search over the Hidden Web Hierarchical Database Sampling and Selection Panagiotis G. Ipeirotis Luis Gravano Computer Science Department Columbia.
Efficient Processing of Top-k Spatial Keyword Queries João B. Rocha-Junior, Orestis Gkorgkas, Simon Jonassen, and Kjetil Nørvåg 1 SSTD 2011.
1 Jun Wang, 2 Sanjiv Kumar, and 1 Shih-Fu Chang 1 Columbia University, New York, USA 2 Google Research, New York, USA Sequential Projection Learning for.
High-Dimensional Similarity Search using Data-Sensitive Space Partitioning ┼ Sachin Kulkarni 1 and Ratko Orlandic 2 1 Illinois Institute of Technology,
Video Google: Text Retrieval Approach to Object Matching in Videos Authors: Josef Sivic and Andrew Zisserman ICCV 2003 Presented by: Indriyati Atmosukarto.
Detecting Image Region Duplication Using SIFT Features March 16, ICASSP 2010 Dallas, TX Xunyu Pan and Siwei Lyu Computer Science Department University.
Distinctive image features from scale-invariant keypoints. David G. Lowe, Int. Journal of Computer Vision, 60, 2 (2004), pp Presented by: Shalomi.
Video Google: Text Retrieval Approach to Object Matching in Videos Authors: Josef Sivic and Andrew Zisserman University of Oxford ICCV 2003.
Scale Invariant Feature Transform (SIFT)
1 An Empirical Study on Large-Scale Content-Based Image Retrieval Group Meeting Presented by Wyman
Multiple Object Class Detection with a Generative Model K. Mikolajczyk, B. Leibe and B. Schiele Carolina Galleguillos.
Improving Min/Max Aggregation over Spatial Objects Donghui Zhang, Vassilis J. Tsotras University of California, Riverside ACM GIS’01.
Internet-scale Imagery for Graphics and Vision James Hays cs195g Computational Photography Brown University, Spring 2010.
School of Information Technology & Electrical Engineering Multiple Feature Hashing for Real-time Large Scale Near-duplicate Video Retrieval Jingkuan Song*,
Performance Tuning on Multicore Systems for Feature Matching within Image Collections Xiaoxin Tang*, Steven Mills, David Eyers, Zhiyi Huang, Kai-Cheung.
Bag of Visual Words for Image Representation & Visual Search Jianping Fan Dept of Computer Science UNC-Charlotte.
Alignment and Matching
1 A Bayesian Method for Guessing the Extreme Values in a Data Set Mingxi Wu, Chris Jermaine University of Florida September 2007.
A Metadata Based Approach For Supporting Subsetting Queries Over Parallel HDF5 Datasets Vignesh Santhanagopalan Graduate Student Department Of CSE.
The X-Tree An Index Structure for High Dimensional Data Stefan Berchtold, Daniel A Keim, Hans Peter Kriegel Institute of Computer Science Munich, Germany.
Copyright Protection of Images Based on Large-Scale Image Recognition Koichi Kise, Satoshi Yokota, Akira Shiozaki Osaka Prefecture University.
A Statistical Approach to Speed Up Ranking/Re-Ranking Hong-Ming Chen Advisor: Professor Shih-Fu Chang.
1 Wenguang WangRichard B. Bunt Department of Computer Science University of Saskatchewan November 14, 2000 Simulating DB2 Buffer Pool Management.
Video Google: A Text Retrieval Approach to Object Matching in Videos Josef Sivic and Andrew Zisserman.
80 million tiny images: a large dataset for non-parametric object and scene recognition CS 4763 Multimedia Systems Spring 2008.
The Simigle Image Search Engine Wei Dong
Efficient EMD-based Similarity Search in Multimedia Databases via Flexible Dimensionality Reduction / 16 I9 CHAIR OF COMPUTER SCIENCE 9 DATA MANAGEMENT.
1 Virtual Machine Memory Access Tracing With Hypervisor Exclusive Cache USENIX ‘07 Pin Lu & Kai Shen Department of Computer Science University of Rochester.
CS 347Notes101 CS 347 Parallel and Distributed Data Processing Distributed Information Retrieval Hector Garcia-Molina Zoltan Gyongyi.
CS848 Similarity Search in Multimedia Databases Dr. Gisli Hjaltason Content-based Retrieval Using Local Descriptors: Problems and Issues from Databases.
1 CSIS 7101: CSIS 7101: Spatial Data (Part 1) The R*-tree : An Efficient and Robust Access Method for Points and Rectangles Rollo Chan Chu Chung Man Mak.
Indexing OLAP Data Sunita Sarawagi Monowar Hossain York University.
Query by Image and Video Content: The QBIC System M. Flickner et al. IEEE Computer Special Issue on Content-Based Retrieval Vol. 28, No. 9, September 1995.
Video Google: Text Retrieval Approach to Object Matching in Videos Authors: Josef Sivic and Andrew Zisserman University of Oxford ICCV 2003.
Rethinking Choices for Multi-dimensional Point Indexing You Jung Kim and Jignesh M. Patel University of Michigan.
Keogh, E. , Chakrabarti, K. , Pazzani, M. & Mehrotra, S. (2001)
Efficient Multi-User Indexing for Secure Keyword Search
CSCI5570 Large Scale Data Processing Systems
Parallel Programming By J. H. Wang May 2, 2017.
Video Google: Text Retrieval Approach to Object Matching in Videos
Database Applications (15-415) DBMS Internals- Part III Lecture 15, March 11, 2018 Mohammad Hammoud.
Evaluation of Relational Operations: Other Operations
Cheng-Ming Huang, Wen-Hung Liao Department of Computer Science
Spatial Online Sampling and Aggregation
Degree-aware Hybrid Graph Traversal on FPGA-HMC Platform
Fast and Exact K-Means Clustering
Panagiotis G. Ipeirotis Luis Gravano
Evaluation of Relational Operations: Other Techniques
Video Google: Text Retrieval Approach to Object Matching in Videos
Minwise Hashing and Efficient Search
Presented by Xu Miao April 20, 2005
Evaluation of Relational Operations: Other Techniques
Efficient Aggregation over Objects with Extent
Presentation transcript:

Scalability of Local Image Descriptors Björn Þór Jónsson Department of Computer Science Reykjavík University Joint work with: Laurent Amsaleg (IRISA-CNRS) Herwig Lejsek Friðrik Heiðar Ásmundsson

Image Search Approaches Text based – Keywords, inverted indices Global descriptors – Color, texture, … Local descriptors

Local Image Descriptors Based on Interest Points – Hundreds of descriptors per image Invariant to many image modifications – Scaling, rotation, … Many Variants – RDTQ (n=24) – SIFT (n=128) – PCA-SIFT (n=36) x1x1 x2x2 x3x3 x4x4 xnxn x1x1

Copyright Protection Stolen & modified images on Web Local descriptors can identify originals 30,000 images  35 million SIFT desc’s Need efficient database support

Database Support Problems – High dimensionality – Many query descriptors State-of-the-art: – L 2 : R-tree, Pyramid tech., VA-file,... – Ranking: Median rank aggregation – Local: Clustering based, LSH, work at INA – Best: A well coded sequential scan 30,000 images  ~10 minutes

Scalability Challenges Efficiency – Response time – Throughput Effectiveness – Detection rate – False positives Descriptor Creation – Number of descriptors – Creation time

Outline Application: Local Descriptors Motivation: Lack of Database Support Key Issues: Efficiency & Effectiveness Effectiveness: The Eff² Descriptors Indexing: PvS-framework and NV-tree Optimization: Stop Rules Scalability: The NV-Network

Eff 2 Descriptors Goals – More descriptors at higher scale – Handle most modifications well Improvement of the SIFT method – “Only” 72 Dimensions – Scale-space norm. (gamma correction) – Improved edge filtering – ~800 most invariant descriptors Eff 2 = Effective x Efficient

Copyright violation workload – 108 original images – 20 Stirmark + 6 “hard” modifications – Almost 3K “pirated” images Most images are OK – 1K images not in the database Two indexed collections – 30K images – 300K images Experimental Setup

Modification Examples Original JPEG 15 MEDIAN 9 ROTSCAL COTR 1 CONV 2 CONV 4 CONV 5

Result Quality: 30K Images SIFT: – ~ 35M descriptors – Matches fewer descriptors – 240 misses – 12 false positives Eff 2 : – ~ 20M descriptors – Matches more descriptors – 32 misses (CONV 5) – 4 false positives

Scalability: 300K Images 30K images: – ~ 20M descriptors – 32 misses – 4 false positives 300K images: – ~ 200+M descriptors – Matches 10% fewer descriptors – 90 misses – 3 false positives

Median Rank Aggregation B + -tree... P1P1 P2P2 PdPd PvS Index Query Descriptor Fagin et al., ACM SIGMOD 2003 Turns d-dim search into many 1-dim searches (random projections) Uses ranking rather than distance Does not work for local descriptors

PvS-Index Creation Descriptor Collection S1S1 S2S2 S3S3 P2P2 P1P1 … P3P3 Projections vs. Segmentations Tree index Leafs are I/O sized (128KB) Goal: One leaf per query descriptor

PvS-Index Search S1S1 S2S2 S3S3 … B + -tree S2S2 P1P1 P2P2 X X P3P3 Query Descriptor

PvS Index Query Descriptor PvS-Framework Summary Repeated projections and segmentations –Partitions collection into leafs of one I/O Uses median rank aggregation High quality results with 3 indices –Three disk I/Os per query descriptor –800 query desc in <40 seconds (1 disk) Query time independent of collection size –Index size is exponential, but disk is cheap

300K Images Revisited 30K images: – ~ 20M descriptors – 3 x 3 GB indices – 3 x 25 mins to index – Typical query: 40 sec 300K images: – ~ 200+M descriptors – 3 x 56 GB indices – 3 x 6 hours to index – Typical query: ~Same

PvS-Framework: Issues Three I/Os per query descriptor – Upper bound & Lower bound – With 2 indices, quality suffers – Segments are not “tight” enough Requires median rank aggregation – Low overhead, but patented by IBM Is using one index possible? – 67% fewer I/Os – Smaller memory requirements

The NV-Tree: Techniques Projections and Segmentations, but... Partition based on data distribution – Variable depth  Improved result quality – Flexible index structure  Updates Improved line selection – Based on variance  Better quality Compact approximate storage – Better quality OR Smaller segments – Smaller index

The NV-Tree: Early Results It works! For local descriptor queries – Individual queries require two indices (intersection) For large collections – Reduced likelihood of random votes For “meaningful” descriptors – Near neighbors in not-so-dense areas

Optimization: Stop Rules Better performance for YES/NO answer Based on probability that a query descriptor gets a “random” vote Many early matches: YES Few matches after a while: NO 80-90% better response time

Scalability: NV-Network Motivation: Apply more disks Benefit: Scalability & Throughput Method: Replicate NV-Trees – A single coordinator assigns workload  Tries to maximize buffer hit ratio – Many workers process local descriptors – The coordinator aggregates results Results: Scales linearly with #workers – 2 indices  Larger memory requirements

Some Results Was: 2+ hrs Now: 2+ sec

Scalability of local image descriptors – Efficiency: NV-Tree, NV-Network, Stop rules – Effectiveness: Eff² descriptors – Index and search 200+ million ld’s Future work – Experiments on NV-tree – Combat index size explosion – Multi-query optimization for throughput – Utilize collective memory of NV-Network – Applications Summary