1. I. Problem  Improve large-scale retrieval / classification accuracy  Incorporate spatial relationship between the features in the image  Oxford 5K Dataset II. Approach  Use a mining algorithm to find Frequent Itemsets (phrases)  Use transactions to encode spatial information among features  Different geometric configurations to capture spatial information Spatial Encodings for Visual Phrases Using Data Mining Techniques Ivette Carreras Haroon Idrees University of Central Florida III. Bag of Visual Words V. Visual Phrases  Multiple words make up a phrase  Different configurations to capture spatial information IV. Data Mining  Algorithms used in large market basket types of data Quadrant 1 Quadrant 4 Quadrant 2 Quadrant 3 Single Circle Transaction Format  Four Quadrants › Prefixes : 1000, 2000, 3000, 4000 › 5000 for the origin  Single Circle › No prefixes  Three Circles › Prefixes : 1000, 2000, 3000 › 4000 for the origin VII. Qualitative Results  Phrases capturing fences/tiles (length = 6)  Phrases capturing window/arches (length = 3 & 4) References: Extract RegionsCompute descriptorsFind clusters and frequencies Compute distance matrix Faces Bikes Wild cats  Ranking (Phrases vs. Bag of Words) (ivette.carreras@knights.ucf.edu)ivette.carreras@knights.ucf.edu (haroon.idrees@knights.ucf.edu)haroon.idrees@knights.ucf.edu Circle 1 Circle 2 Circle 3 Extract features & quantize For each word, find k-NN Encode configurations Mine phrases with support s Sort phrases by length Build Bag of Visual Phrases VIII. Quantitative Results Phrase Percentage BoW (mAP) BoVP_Q (mAP) BoVP_1C (mAP) 10%17.7213.7419.47 25%21.9916.5322.75 40%24.6118.0625.24 50%25.5618.9625.94 65%26.4019.3726.42 80%27.4420.2126.85 85%27.4920.2126.89 90%27.4220.1926.80 95%27.1320.4926.50 97%26.6720.5026.28 98%26.6820.4926.42 ConfigurationsmAP Bag of Words26.95% Bag of Phrases – Quadrants20.68% Bag of Phrases –1 Circle24.77% Bag of Phrases – 3 Circles17.65% BoW & BoP_Q27.09% BoW & BoP_1C27.69% BoW & BoP_3C23.87% BoW & BoP_Q & BoP_3C24.02% BoW & BoP_1C & BoP_3C24.63% BoW & BoP_Q & BoP_1C27.74% 20k40k60k95k Single Circle (mAP)22.55%23.69%24.06%24.77%  Statistics of Phrases PhrasesBoVP single circle Length 298112 Length 32782 Length 494 Length 55 Total100989 Set of items Transactions 1 FIM Frequent Itemsets {Beer, Bread, Jelly, Milk, PeanutButter} TransactionItems t1t1 Bread, Jelly, PeanutButter t2t2 Bread, PeanutButter t3t3 Bread, Milk, PeanutButter, Beer t4t4 Beer, Bread t5t5 Beer, Milk Apriori, Eclat {Bread, PeanutButter} – 3/5 {Beer, Milk} – 2/5  Selection of Phrases - Frequency  Selection of Phrases - Entropy  Results * http://www.di.ens.fr/willow/events/cvml2010/materials/INRIA_summer_school_2010_Cordelia_ bof_classification.pdf 1. Philbin, J., Chum, O., Isard, M., Sivic, J., and Zisserman, A. Object retrieval with large vocabularies and fast spatial matching. In CVPR (2007). 2. T. Quack, V. Ferrari, and L. Van Gool. Video mining with frequent itemset configurations. In CIVR'06, 2006. 3. Zhang, Y., Jia, Z., Chen, T.: Image retrieval with geometry-preserving visual phrases. In: CVPR. (2011)