1. The topic discovery models
Discovering Objects and their Location in Images
Josef Sivic1, Bryan C. Russell2, Alexei A. Efros3, Andrew Zisserman1 and William T. Freeman2
MIT
CMU
1Oxford University MIT 3Carnegie Mellon University
CMU
MIT
Introduction
The topic discovery models
Improving localization using doublets
Goal: Discover visual object categories and their segmentation given a
collection of unlabelled images
Probabilistic Latent Semantic Analysis (pLSA) [Hofmann’99]
Form a new vocabulary from pairs of locally co-occurring regions
w … visual words d … documents (images) z … topics (‘objects’)
Doublet formation
Doublet example I
Doublet examle II
pLSA graphical model
P(z|d) and P(w|z) are multinomial distributions
pLSA Model fitting:
Find topic vectors P(w|z) common to all documents and mixture coefficients P(z|d) specific to each document. Fit model by maximizing likelihood of data using EM.
All detected visual words
Singlet segmentation
Singlet segmentation
Doublet segmentation
Approach: 1) Represent an image as a collection of visual words
2) Apply topic discovery models from statistical text analysis
Latent Dirichlet Allocation (LDA) [Blei et al.’03]
Experiment II: MIT dataset
2873 images, learn 10 topics
Treat multinomial weights over topics as random variables. Fit model using Gibbs sampling [Griffiths and Steyvers’04].
4 of the 10 learned topics shown by the 5 most probable images for each topic
Image representation
Represent an image as a histogram of “visual words”
LDA graphical model
“Buildings”
“Trees / Grass” 2 1
...
Results
Results shown only for pLSA. LDA had very similar performance.
Experiment I: Caltech Dataset
Histogram of visual words
“Computers”
“Bookshelves”
Four object categories: faces, motorbikes, airplanes and cars rear (total of 3,190 images) and 900 background images
Example Images with multiple objects
Detect affine covariant regions
Represent each region by a SIFT descriptor
Build visual vocabulary by k-means clustering (K~1,000)
Assign each region to the nearest cluster centre
Mikolajczyk and Schmid’02, Schaffalitzky and Zisserman’02, Matas et al. ’02, Lowe’99, Sivic and Zisserman’03
Image Classification
Assign each image to a topic with the highest P(z|d)
Learn K = (5,6,7) topics
Background is better modelled by multiple topics
Pre-learning background topics on a separate bg dataset improves results
Performance on novel images is comparable with weakly supervised method of [Fergus et al.’03]
Examples of visual words
Confusion tables (K=5,6,7) learned topics
Experiment III: Application to image retrieval
Learn topic vectors on Caltech database
Represent new query image in terms of learned topic vectors
Segmentation
Five samples from a ‘motorbike’ visual word
Visual Polysemy. Single visual word occurring on different (but locally similar) parts on different object categories.
For a given word wi in document dj examine posterior probability over topics.
pLSA
Retrieve images within Caltech database
Raw word histograms
Visual words colour coded according to the topic with the highest probability
Query image
Five samples from an ‘airplane’ visual word
Visual Synonyms. Two different visual words representing a similar part of an object (wheel of a motorbike).
Faces
Motorbikes
Airplanes
Retrieve images in movie Pretty Woman
Precision – Recall plot
Cars
Overview
Background I
Background II
Retrieved images using pLSA ‘object’ coefficients P(z|d)
Find visual words
Form histograms
Background III
Example face segmentation
Pretty Woman (6,641 keyframes)
Retrieved images using visual word histograms
Represent each keyframe using topic vectors learned on Caltech database
Discover topics
Example motorbike segmentation
Example airplane segmentation