Presentation is loading. Please wait.

Presentation is loading. Please wait.

HOPS: Efficient Region Labeling using Higher Order Proxy Neighborhoods Albert Y. C. Chen 1, Jason J. Corso 1, and Le Wang 2 1 Dept. of Computer Science.

Similar presentations


Presentation on theme: "HOPS: Efficient Region Labeling using Higher Order Proxy Neighborhoods Albert Y. C. Chen 1, Jason J. Corso 1, and Le Wang 2 1 Dept. of Computer Science."— Presentation transcript:

1 HOPS: Efficient Region Labeling using Higher Order Proxy Neighborhoods Albert Y. C. Chen 1, Jason J. Corso 1, and Le Wang 2 1 Dept. of Computer Science and Engineering 2 Dept. of Geography University at Buffalo, The State University of New York

2 Image Labeling via Energy Minimization Image labeling: assigning an object class (e.g. sky, water, tree, grass) to each pixel. Energy minimization, or equivalently, posterior probability maximization, is the standard approach for solving image labeling problems. Markov Random Fields or Conditional Random Fields, are the typical models used for energy minimization. Vision and Perceptual Machines Lab Albert Y. C. Chen, Jason J. Corso, Le Wang Department of Computer Science and Engineering 1

3 Typical Labeling Results versus the more ideal labeling by HOPS Department of Computer Science and Engineering Vision and Perceptual Machines Lab Albert Y. C. Chen, Jason J. Corso, Le Wang Input Image to be labeledLabels using std. 1 st orderMore Idea labels 2

4 Markov Random Field (MRF) in image labeling Undirected-Graphs with cond. independent nodes. The labels of hidden nodes x i are chosen to minimize the global energy. A hidden nodes x i is dependent only on other hidden nodes within their Markov blanket (in practice, only first-order neighborhood is used). Typical energy used: (bias) (local evidence) (Markov blanket) (E1: unary term) (E2: binary term) Department of Computer Science and Engineering Vision and Perceptual Machines Lab Albert Y. C. Chen, Jason J. Corso, Le Wang 3

5 Image Labeling via Energy Minimization – Previous Approaches Simulated Annealing: Can converge to global optimum in theory, but is extremely slow in practice (several hours to several days). Approximation Algorithms: Belief Propagation: energy reduction done by updating with the messages passed along the nodes. Performance is to some degree un-expectable on loopy graphs. Graph-Cuts: takes large leap in energy space, but is not guaranteed to converge on all energy functions. Graph-Shifts: adaptive hierarchies are used to better represent the underlying data, while shifts are used to efficiently minimize the energy. Department of Computer Science and Engineering Vision and Perceptual Machines Lab Albert Y. C. Chen, Jason J. Corso, Le Wang * For a detailed comparison of the performances, please refer to: Richard Szelisk et al., A Comparative Study of Energy Minimization Methods for Markov Random Fields with Smoothness-Based Priors, PAMI 2008 4

6 Graph-Shifts: Its hierarchy and shifts Department of Computer Science and Engineering Vision and Perceptual Machines Lab Albert Y. C. Chen, Jason J. Corso, Le Wang 1. Build the Adaptive Hierarchy 2.A SHIFT occurs. It can happen at any level of the adaptive hierarchy. 3. Repeat until the overall energy is minimized. 5

7 Graph-Shifts: Its Effectiveness and Efficiency Department of Computer Science and Engineering Vision and Perceptual Machines Lab Albert Y. C. Chen, Jason J. Corso, Le Wang Top: from J. J. Corso et al., IPMI 2007; Bottom: from J. J. Corso et al., CVPR 2008 6

8 Are MRF Blankets using first-order neighborhoods sufficient? Algorithms with first-order neighborhood can be myopic, thus producing unnecessary label changes and even noisy/incorrect labeling results. However, first-order neighborhood have been widely used by previous methods because: Higher-order neighborhood increases the inter- nodal connectivity. Extending the neighborhood to the nth order increases the comp. time by |N 2 |.|N 3 |. … |N i | HOPS, inspired by the Belief Propagation algorithm, approximates the energies of higher- order neighbors using the first order neighborhood. Department of Computer Science and Engineering Vision and Perceptual Machines Lab Albert Y. C. Chen, Jason J. Corso, Le Wang 7

9 HOPS (Higher Order Proxy NeighborhoodS) Instead of calculating the binary energies between {μ, τ i | τ i : μs higher order neighbors} directly: The binary energy between ν i (μs 1 st order neighbors) and their 1 st order neighbors τ j are passed to μ, to approximate the higher order binary energy between {μ, τ i }. Department of Computer Science and Engineering Vision and Perceptual Machines Lab Albert Y. C. Chen, Jason J. Corso, Le Wang 8

10 HOPS (Higher Order Proxy NeighborhoodS) – why it works and how it works When MRF with smoothness based prior is used and higher order neighborhood is requested, it means that: The label of a node μ and its 1 st order neighbors ν are likely to be equivalent. Thus, the approximated higher order binary energy of τ will likely be the same as the direct calculation between {μ, τ}. First order binary energies are cached at each node n. Thus when its neighboring node m is trying to approximate higher order energies, n would work as a proxy and pass its cached energy to m. 9 Department of Computer Science and Engineering Vision and Perceptual Machines Lab Albert Y. C. Chen, Jason J. Corso, Le Wang

11 Natural Image Labeling Results Department of Computer Science and Engineering Vision and Perceptual Machines Lab Albert Y. C. Chen, Jason J. Corso, Le Wang Graph-Shifts using first order neighborhood* Graph-Shifts with Higher Order Proxy NeighborhoodS 10 * J. J. Corso et al., CVPR 2008

12 Discussion 11 Quality/Accuracy of the Labeling The number of small noisy labels is greatly reduced. Object boundaries are better followed in the labeled image The labeled images are much closer to what a human expert would produce. The computation time for one shift have increased only linearly. However, since redundant shifts are effectively avoided: The number of shifts required until convergence decreases by an average of 60%. The overall convergence time is reduced by 30%. Department of Computer Science and Engineering Vision and Perceptual Machines Lab Albert Y. C. Chen, Jason J. Corso, Le Wang

13 What about other types of Energy function and problem sets? Aerial photo and airborne LiDAR (light detection and ranging) - labeling results: HOPS gets approximately the same accuracy rate yet constantly converge 30% faster than standard 1 st order neighbor ones. Other spatial constraints that relies on context of a larger neighborhood can be added (such as shape information) into the model to achieve better results. 12 Department of Computer Science and Engineering Vision and Perceptual Machines Lab Albert Y. C. Chen, Jason J. Corso, Le Wang

14 Conclusion 13 HOPS produces aesthetically and quantitatively better labeling results compared to those using standard first order neighbors. HOPS estimates higher order energies in a recursive and cached manner, which induces little additional computational cost without increasing the node connectivity of the graph. HOPS constantly converges 30% faster while used in the Graph-Shifts algorithm, since more context information is incorporated and redundant / incorrect label changes are more likely to be avoided. Department of Computer Science and Engineering Vision and Perceptual Machines Lab Albert Y. C. Chen, Jason J. Corso, Le Wang


Download ppt "HOPS: Efficient Region Labeling using Higher Order Proxy Neighborhoods Albert Y. C. Chen 1, Jason J. Corso 1, and Le Wang 2 1 Dept. of Computer Science."

Similar presentations


Ads by Google