Recognition using Nearest Neighbor (or kNN) Modified from various sources including http://people.csail.mit.edu/torralba/courses/6.869/6.869.computervision.htm

Recall ---Object Recognition Object detection and recognition is formulated as a classification problem. The image is partitioned into a set of overlapping windows … and a decision is taken at each window about if it contains a target object or not. Decision boundary Computer screen Background In some feature space Where are the screens? Bag of image patches Object detection and recognition is formulated as a classification problem. The image is partitioned into a set of overlapping windows, and a decision is taken at each window about if it contains a target object or not. Each window is represented by extracting a large number of features that encode information such as boundaries, textures, color, spatial structure. The classification function, that maps an image window into a binary decision, is learnt using methods such as SVMs, boosting or neural networks.

Recognition Techniques…a few Nearest neighbor Neural networks 106 examples LeCun, Bottou, Bengio, Haffner 1998 Rowley, Baluja, Kanade 1998 … Shakhnarovich, Viola, Darrell 2003 Berg, Berg, Malik 2005 … Support Vector Machines and Kernels Conditional Random Fields Many more references missing here. These are big fields on its own. Big emphasis on the past on face detection and character recognition. It provided a clear problem with a well defined visual category and many applications. It allowed making progress on efficient techniques. Guyon, Vapnik Heisele, Serre, Poggio, 2001 … McCallum, Freitag, Pereira 2000 Kumar, Hebert 2003 …

Formulation Formulation: binary classification Classification function … x1 x2 x3 … xN xN+1 xN+2 … xN+M Features x = y = -1 +1 -1 -1 Labels ? ? ? Training data: each image patch is labeled as containing the object or background Test data In many cases, F(x) will give a real value and the classification is perform with a Threshold. If F(x)>0 then y =+1 Where belongs to some family of functions Classification function Minimize misclassification error (Not that simple: we need some guarantees that there will be generalization)

Two ways to thing of “modeling an object” in its feature space Parametric model Here we fit one or more parametric equations to the feature space to model our “object” Non-parametric model We do not use an explicit parametric model, more sample oriented or statistical. z

Parametric models Subspace of monkeys Subspace of natural images Nearest Neighbor IS NOT a Parametric model Subspace of monkeys Subspace of natural images Space of all images Parametric model of monkeys

Non-parametric Approach !!! HIGH DIMENSIONAL !!! !!! HIGH DIMENSIONAL !!! Subspace of monkeys Subspace of natural images z Query image Space of all images

Non-parametric Approach !!! HIGH DIMENSIONAL !!! !!! HIGH DIMENSIONAL !!! Subspace of monkeys Subspace of natural images Query image Space of all images

Non-parametric Classifier Nearest-neighbors For each query, obtain sibling set (neighbors) 3 different types of distance metric Hand-designed, use whole image

Metric 1 - Dssd _ 2 Sum of squared differences (SSD) To give invariance to illumination: Each image normalized to be zero mean, unit variance 2 Image 1 Image 2 _

Nearest Neighbors in 80 million images 105 Size of dataset 106 108

Differing density of images

Metric 2 - Dwarp _ 2 SSD but allow small transformations Find min using gradient descent Image 2 Translation: Scalings: Horizontal flip: _ Image 1 SSD Transformations

Metric 3 - Dshift _ 2 As per Warping but also allow sub-window shifts Start with warped version of image 2, as per Dwarp 2 _ Image 1 Image 2 Transformed

Metric 3 - Dshift _ 2 As per Warping but also allow sub-window shifts Start with warped version of image 2, as per Dwarp 2 _ Transformed

Metric 3 - Dshift _ 2 As per Warping but also allow sub-window shifts Start with warped version of image 2, as per Dwarp 2 _

Metric 3 - Dshift _ 2 As per Warping but also allow sub-window shifts Local sub-window

Metric 3 - Dshift _ 2 As per Warping but also allow sub-window shifts Quick since images are so small 2 _ Local sub-window

Metric 3 - Dshift As per Warping but also allow sub-window shifts Tried various sizes of sub-window  1x1 (i.e. single pixel) worked best 2 _ Local sub-window

Comparison of metrics SSD

Neighbors with Different Metrics

Examples Solving vision by brute force Normalized correlation scores

How Many Images Are There? Note: D1=DSSD

How Does Dssd Relate to Semantic Distance? Subject 1 Subject 2 Subject 3

How Does Dssd Relate to Semantic Distance? Subject 1 Subject 2 Subject 3