1. Yilin Wang 11/5/2009

2. Background Labeling Problem Labeling: Observed data set (X) Label set (L) Inferring the labels of the data points Most vision problems can be posed as labeling problems Stereo matching Image segmentation Image restoration

3. Examples of Labeling Problem Stereo Matching For a pixel in image 1, where is the corresponding pixel in image 2? Picture source: S. Lazebnik Label set: Differences (disparities) between corresponding pixels

4. Examples of Labeling Problem Image Segmentation To partition an image into multiple disjoint regions. Picture source: http://mmlab.ie.cuhk.edu.hk Label set: Region IDs

5. Examples of Labeling Problem Image Restoration To "compensate for" or "undo" defects which degrade an image. Picture source: http://www.photorestoration.co.nz Label set: Restored Intensities

6. Background Image Labeling Given an image, the system should automatically partition it into semantically meaningful areas each labeled with a specific object class Cow Lawn Plane Sky Building Tree

7. Image Labeling Problem Given : the observed data from an input image, where is the data from site (pixel or block) of the image set S A pre-defined label set Let be the corresponding labels at the image sites, we want to find proper L to maximize the conditional probability :

8. Which kinds of information can be used for labeling? Features from individual sites Intensity, color, texture, … Interactions with neighboring sites Contextual information Vegetation Sky or Building?

9. Two types of interactions Interaction with neighboring labels (Spatial smoothness of labels) neighboring sites tend to have similar labels(except at the discontinuities) Interactions with neighboring observed data Building Sky

10. Information for Image Labeling Let be the label of the site of the image set S, and N i be the neighboring sites of site i Three kinds of information for image labeling Features from local site Interaction with neighboring labels Interaction with neighboring observed data Picture source: S. Xiang site iS-{i}NiNi

11. Markov Random Fields (MRF) Markov Random Fields (MRFs) are the most popular models to incorporate local contextual constraints in labeling problems Let be the label of the site of the image set S, and N i be the neighboring sites of site i The label set L ( ) is said to be a MRF on S w.r.t. a neighborhood N iff the following condition is satisfied: Markov property: Maintain global spatial consistency by only considering relatively local dependencies !

12. Markov-Gibbs Equivalence Let l be a realization of, then P(l) has an explicit formulation (Gibbs distribution): where Energy function Z: a normalizing factor, called the partition function T: a constant Clique C k ={{i,i’,i’’,…}|i,i’,i’’,… are neighbors to one another} Potential functions represent a priori knowledge of interactions between labels of neighboring sites

13. Auto-Model With clique potentials up to two sites, the energy takes the form When and, where G i (·) are arbitrary functions (or constants) and are constants reflecting the pairwise interaction between i and i’, the energy is Such models are called auto-models (Besag 1974)

14. Parameter Estimation Give the functional form of the auto-model How to specify its parameters ?

15. Maximum Likelihood Estimation Given a realization l of a MRF, the maximum likelihood (ML) estimate maximizes the conditional probability P(l | θ) (the likelihood of θ), that is: By Bayesian rules: The prior P(θ) is assumed to be flat when the prior information is totally unavailable. In this case, the MAP estimation reduces to the ML estimation.

16. Maximum Likelihood Estimation The likelihood function is in the Gibbs form where However, the computation of Z(θ) is intractable even for moderately sized problems because there are a combinatorial number of elements in the configuration space L.

17. Maximum Pseudo-Likelihood Assumption: and are independent. Notice that the pseudo-likelihood does not involve the partition function Z. {α, β} can be obtained by solving

18. Inference Recall that in image labeling, we want to find L such that maximizes the posterior, by Bayesian rules: Where prior probability: Let and then: posterior energy likelihood energy prior energy

19. MAP-MRF Labeling Maximizing a posterior probability is equivalent to minimizing the posterior energy: Steps of MAP: Picture source: S. Xiang N C

20. MRF for Image Labeling Difficulties and disadvantages Very strict independence assumptions : The interactions among label are modeled by the priori term (P(L)), and are independent of the observation data, which prohibits one from modeling data- dependent interactions in labels.

21. Conditional Random Fields Let G = (S, E) be a graph, then (X, L) is said to be a Conditional Random Field (CRF) if, when conditioned on X, the random variables obey the Markov property with respect to the graph: where S-{i} is the set of all sites in the graph except the site i, Ni is the set of neighbors of the site i in G. Compare with MRF:

22. CRF According to the Markov-Gibbs equivalence, we have If only up to pairwise clique potentials are nonzero, the posterior probability P(L| X) has the form where −V1 and −V2 are called the association and interaction potentials, respectively, in the CRF literature

23. CRF vs. MRF MRF is a generative model(Two-step) Infer likelihood and prior Then use Bayes theorem to determine posterior CRF is a discriminative model(One-step) Directly infer posterior

24. CRF vs. MRF More differences between the CRF and MRF MRF: CRF: In CRF, both Association and Interaction Potentials are functions of all the observation data as well as that of the labels

25. Discriminative Random Fields The Discriminative Random Field (DRF) is a special type of CRF with two extensions. First, a DRF is defined over 2D lattices (such as the image grid) Second, the unary (association) and pairwise (interaction) potentials therein are designed using local discriminative classifiers Kumar, S. and M. Hebert: `Discriminative Random Fields: A Discriminative Framework for Contextual Interaction in Classification'. ICCV 2003

26. DRF Formulation of DRF where and are called association potential and interaction potential Picture source: S. Xiang

27. Association Potential is modeled using a local discriminative model that outputs the association of the site i with class l i as: where f i (.) is a linear function that maps an patch centered at site i to a feature vector. Picture source: S. Srihari

28. Association Potential For binary classification (l i = -1 or 1), the posterior at site i is modeled using a logistic function: Since l i = -1 or 1, the probability can be compactly expressed as: Finally, the association potential is defined as: Picture source: S. Srihari

29. Interaction Potential The interaction potential can be seen as a measure of how the labels at neighboring sites i and i' should interact given the observed image X. Given the features at two different sites, a pairwise discriminative model is defined as: where is a function that maps an patch centered at site i to a feature vector, is a new feature vector, and v are model parameters is a measure of how likely site i and i’ have the same label given image X

30. Interaction Potential The interaction potential is modeled using data-dependent term along with constant smoothing term The first term is a data-independent smoothing term, similar to the auto-model The second term is a [-1, 1] mapping of the pairwise logistic function, which ensures that both terms have the same range Ideally, the data-dependent term will act as a discontinuity adaptive model that will moderate smoothing when the data from two sites is 'different'.

31. Discussion of I(l i,l i’, X) Suppose,, and Also for simplicity, we assume Then If only considering, will never choose b. The second term is used to compensate the effect of the smoothness assumption. Oversmoothed!

32. Parameter Estimation θ={w,v,β,K} Maximum likelihood estimation In the conventional maximum-likelihood approach, the evaluation of Z is an NP-hard problem. Approximate evaluation of partition function Z by pseudo-likelihood where m indexes over the training images and M is the total number of training images, and Subject to 0≤K≤1

33. Inference Objective function: Iterated Conditional Modes (ICM) algorithm Given an initial label configuration, ICM maximizes the local conditional probabilities iteratively, i.e. ICM yields local maximum of the posterior and has been shown to give reasonably good results

34. Experiment Task: detecting man-made structures in natural scenes Database Corel (training: 108 images, test: 129 images) Each image was divided in non-overlapping 16*16 pixels blocks Compared methods Logistic MRF DRF

35. Experiment Results Detection Rates (DR) and False Positives (FP) Superscript ‘-’ indicates no neighborhood data interaction was used. K = 0 indicates the absence of the data independent term in the interaction potential in DRF. The DRF reduces false positives from the MRF by more than 48%.

36. Experiment Results For similar detection rate, DRF has the lower false positives Detection rate of DRF is higher than that of MRF for similar false positives

37. Conclusion of DRF Pros Provide the benefits of discriminative models Demonstrate good performance Cons Although the model outperforms traditional MRFs, it is not strong enough to capture long range correlations among the labels due to the rigid lattice based structure which allows for only pairwise interactions

38. Problem Local information can be confused when there are large overlaps between different classes Sky or Water ? Solution: utilizing the global contextual information to improve the performance

39. Multiscale Conditional Random Field (mCRF) Considering features in different scales Local Features (site) Regional Label Features (small patch) Global Label Features (big patch or the whole image) The conditional probability P(L|X) is formulated by features in different scales s: where He, X., R. Zemel, and M. Carreira-Perpinan: 2004, `Multiscale conditional random fields for image labelling'. IEEE Int. Conf. CVPR.

40. Local Features The local feature of site i is represented by the outputs of several filters. The aim is to associate the patch with one of a predefined set of labels.

41. Local Classifier Here a multilayer perceptron is used as the local classifier. Independently at each site i, the local classifier produces a conditional distribution over label variable l i given filter outputs x i within an image patch centered on site (pixel) i: where λ are the classifier parameters.

42. Regional Label Features Encoding a particular constraint between the image and the labels within a region of the image Sample pattern: ground pixels (brown) above water pixels (cyan)

43. Global Label Features Operate at a coarser resolution, specifying common value for a patch of sites in the label field. Sample pattern: sky pixels (blue) at the top of the image, hippo pixels (red) in the middle, and water pixels (cyan) near the bottom.

44. Feature Function Global label features are trained by Restricted Boltzmann Machines (RBM) two layers: label sites (L) and features (f) features and labels are fully inter-connected, with no intra-layer connections where w a is the parameter connecting hidden global label feature f a and label sites L The joint distribution of the global label feature model is:

45. Feature Function By marginalizing out the hidden variables (f), the global component of the model is: Similarly, the regional component of the model can be represented as: By multiplicatively combining component conditional distributions:

46. Parameter Estimation and Inference Parameter Estimation The conditional model is trained discriminatively based on the Conditional Maximum Likelihood (CML) criterion, which maximizes the log conditional likelihood: Inference Maximum Posterior Marginals (MPM):

47. Experiment Results Database Corel (100 images with 7 labels) Sowerby (104 images with 8 labels) Compared methods Single classifier (MLP) MRF mCRF

48. Labeling Results

49. Conclusion of mCRF Pros Formulating the image labeling problem into a multiscale CRF model Combining the local and larger scale contextual information in a unique framework Cons Including additional classifiers operating at different scales into the mCRF framework introduces a large number of model parameters The model assumes conditional independence of hidden variables given the label field

50. More CRF models Hierarchical Conditional Random Field (HCRF) –S. Kumar and M. Hebert. A hierarchical field framework for unified context-based classification. 2005 –Jordan Reynolds and Kevin Murphy. Figure-ground segmentation using a hierarchical conditional random field. 2007 Tree Structured Conditional Random Fields (TCRF) –P. Awasthi, A. Gagrani, and B. Ravindran, Image Modeling using Tree Structured Conditional Random Fields. 2007

51. Reference Li, S. Z.: 2009, `Markov Random Field Modeling in Image Analysis’. Springer, 2009 Kumar, S. and M. Hebert: 2003, `Discriminative Random Fields: A Discriminative Framework for Contextual Interaction in Classification'. in proc. IEEE International Conference on Computer Vision (ICCV) He, X., R. Zemel, and M. Carreira-Perpinan: 2004, `Multiscale conditional random fields for image labelling'. IEEE Int. Conf. CVPR.

52. End Thanks!