1. The University of Ontario CS 4487/9687 Algorithms for Image Analysis Multi-Label Image Analysis Problems

2. The University of Ontario CS 4487/9687 Algorithms for Image Analysis Topic overview n Multi-label problems: Stereo, restoration, texture synthesis, multi-object segmentation n Types of pair-wise pixel interactions Convex interactions Discontinuity preserving interactions n Energy minimization algorithms: Ishikawa (convex) a-expansions (robust metric interactions) ICM, simulated annealing, message passing, etc. (general) Extra Reading: Szeliski Ch 3.7

3. The University of Ontario Example of a binary labeling problem Object/background segmentation (topics 4-6) is an example of binary labeling problem feasible labels at any pixel S S S S

4. The University of Ontario Example of a binary labeling problem Object/background segmentation (topics 4-6) is an example of binary labeling problem feasible labels at any pixel labeling of image pixels or, equivalently, S S S S For convenience this topic uses L p (label at p) instead of S p (segment at p)

5. The University of Ontario depth map Stereo (topic 7) is an example of image labeling problem with non-binary labels Example of a multi-label problem feasible disparities at any pixel or, equivalently, labeling of image pixels In topic 7 we used equivalent notation

6. The University of Ontario Remember: stereo with s-t graph cuts [Roy&Cox’98] x y

7. The University of Ontario Remember: stereo with s-t graph cuts [Roy&Cox’98] s t cut L(p) p “cut” x y labels x y Disparity labels

8. The University of Ontario Multi-label energy minimization with s-t graph cuts [Ishikawa 1998, 2003] n Exact optimization for convex pair-wise potentials V(dL) dL=Lp-Lq V(dL) dL=Lp-Lq graph construction for linear interactions extends to “convex” interactions works only for 1D labels

9. The University of Ontario Q: Are convex models good enough for general labeling problems in vision? A: Unfortunately, NO (see the following discussion)

10. The University of Ontario observed noisy image I I p along one scan line in the image Reconstruction in Vision: (a basic example) image labeling L (restored intensities) L = {L 1, L 2,..., L n } I = {I 1, I 2,..., I n } How to compute L from I ? L

11. The University of Ontario observed noisy image I I p along one scan line in the image Reconstruction in Vision: (a basic example) image labeling L (restored intensities) L = {L 1, L 2,..., L n } I = {I 1, I 2,..., I n } How to compute L from I ? L

12. The University of Ontario Energy minimization (discrete approach) n Markov Random Fields (MRF) framework weak membrane model (Geman&Geman’84, Blake&Zisserman’83,87) discontinuity preserving potentials Blake&Zisserman’83,87 spatial regularization data fidelity piece-wise smooth labeling

13. The University of Ontario Basic pairwise potentials V(α,β) : n Convex regularization gradient descent works exact polynomial algorithms (Ishikawa) n TV regularization (extreme case of convex) a bit harder (non-differentiable) global minima algorithms (Ishikawa, Hochbaum, Nikolova et al.) n Robust regularization (“discontinuity-preserving”) bounded potentials (e.g. truncated convex) NP-hard, many local minima good approximations (message passing, a-expansion) total variation

14. The University of Ontario Robust pairwise regularization n Robust regularization (“discontinuity-preserving”) bounded potentials (e.g. truncated convex) NP-hard, many local minima good approximations (message passing, a-expansion) piece-wise smooth labeling

15. The University of Ontario Robust pairwise regularization n Robust regularization (“discontinuity-preserving”) bounded potentials (e.g. Ising or Potts model) NP-hard, many local minima provably good approximations (a-expansion) maxflow/mincut algorithms piece-wise smooth labeling “perceptual grouping” piece-wise constant labeling weak membrane

16. The University of Ontario Potts model (piece-wise constant labeling) n Robust regularization (“discontinuity-preserving”) bounded potentials (e.g. Ising or Potts model) NP-hard, many local minima provably good approximations (a-expansion) maxflow/mincut algorithms

17. The University of Ontario Left eye image Right eye image Potts model (piece-wise constant labeling) depth layers n Robust regularization (“discontinuity-preserving”) bounded potentials (e.g. Ising or Potts model) NP-hard, many local minima provably good approximations (a-expansion) maxflow/mincut algorithms

18. The University of Ontario Potts model (piece-wise constant labeling) C n Robust regularization (“discontinuity-preserving”) bounded potentials (e.g. Ising or Potts model) NP-hard, many local minima provably good approximations (a-expansion) maxflow/mincut algorithms

19. The University of Ontario Pairwise interactions V: “convex” vs. “discontinuity-preserving” V(dL) dL=Lp-Lq Potts model robust “ discontinuity preserving ” interactions V V(dL) dL=Lp-Lq “ convex ” interactions V V(dL) dL=Lp-Lq V(dL) dL=Lp-Lq “linear” model (TV) “quadratic” model bounded models (truncated convex) piecewise constant labeling piecewise smooth labeling smooth labeling smooth labeling (with some discontinuity robustness) “most robust” among convex good for perceptual grouping see comparison in the next slides

20. The University of Ontario Pairwise interactions V: “convex” vs. “discontinuity-preserving” NOTE: optimization of restoration energy with quadratic regularization term relates to noise-reduction via mean-filtering (Topic 3) HINT: consider initial labeling L = I. Treat label L p at one pixel p as the only optimization variable reducing our energy to. The optimal value of L p is weighted mean => local optimization of E(L) corresponds to mean-filtering with kernel I L quadratic

21. The University of Ontario Pairwise interactions V: “convex” vs. “discontinuity-preserving” NOTE: optimization of quadratic regularization model relates to noise-reduction via mean-filtering (Topic 3) HINT: consider initial labeling L = I. Treat label L p at one pixel p as the only optimization variable reducing our energy to. The optimal value of L p is weighted mean => local optimization of E(L) corresponds to mean-filtering with kernel quadratic I L

22. The University of Ontario I p blurred edge along one scan line in the image Pairwise interactions V: “convex” vs. “discontinuity-preserving” quadratic (convex) may over-smooth (similarly to mean filtering) quadratic I L NOTE: minimizing the sum of quadratic differences prefers to split one large jump into many small ones a large jump

23. The University of Ontario I p blurred edge along one scan line in the image Pairwise interactions V: “convex” vs. “discontinuity-preserving” histogram equalization linear (TV) I L quadratic (convex) may over-smooth (similarly to mean filtering) linear (TV) NOTE: minimizing the sum of absolute differences does not care how a large jump is split (the sum does not change) => no over-smoothing may create “stair-case” better robustness (similar to median vs. mean ) a large jump

24. The University of Ontario Pairwise interactions V: “convex” vs. “discontinuity-preserving” Potts model may create “false banding” (next slide) histogram equalization Potts I L I p blurred edge along one scan line in the image quadratic (convex) may over-smooth (similarly to mean filtering) linear (TV) may create “stair-case” bounded (e.g. Potts) NOTE: minimizing the sum of bounded differences prefers one large jump to splitting into smaller ones. => restores sharp boundaries. a large jump

25. The University of Ontario convex models discontinuity-preserving models linear (TV) quadrat ic truncated linear truncat ed quadrat ic Potts restoration with: Pairwise Regularization Models (comparison) smooth piecewis e smooth piecewise constant noisy images stair-casing over-smoothing stair-casing over-smoothing banding common artifacts: over-smoothing, stair-casing, banding

26. The University of Ontario code Optimization for “discontinuity preserving” models n NP-hard problem (3 or more labels) two labels can be solved via s-t cuts n a-expansion approximation algorithm [BVZ 1998, PAMI2001] guaranteed approximation quality (Veksler, 2001) –within a factor of 2 from the global minima (Potts model) n Many other (small or large) move making algorithms - a/b swap, jump moves, range moves, fusion moves, etc. n LP relaxations, message passing, e.g. (LBP, TRWS) n Other MRF techniques (simulated annealing, ICM) n Variational methods (e.g. multi-phase level-sets)

27. The University of Ontario other labels a a-expansion move Basic idea is motivated by methods for multi-way cut problem (similar to Potts model) Break computation into a sequence of binary s-t cuts

28. The University of Ontario a-expansion (binary move) optimizies sumbodular set function expansions correspond to subsets (shaded area) L current labeling =

29. The University of Ontario a-expansion (binary move) optimizies sumbodular set function L current labeling 0 1

30. The University of Ontario a-expansion (binary move) optimizies sumbodular set function L current labeling 0 1 0 1

31. The University of Ontario a-expansion (binary move) optimizies sumbodular set function L current labeling 0 1 0 1 Set function is submodular if

32. The University of Ontario a-expansion (binary move) optimizies sumbodular set function L current labeling 0 1 0 1 Set function is submodular if = 0 triangular inequality for ||a-b||=E(a,b)

33. The University of Ontario a-expansion (binary move) optimizies sumbodular set function L current labeling 0 1 0 1 a-expansion moves are submodular if is a metric on the space of labels [Boykov, Veksler, Zabih, PAMI 2001]

34. The University of Ontario examples of metric pairwise interactions FACT (easy to prove): any truncated metric is also a metric But, this is only a special case of L 2 metric for 1D labels. In general, for this metric is defined as We called this linear or TV potential Just check that Truncated L 2 is also a metric Potts is another important example of a metric Quadratic (squared L 2 ) and truncated quadratic potentials are not metrics. Other very good approximation algorithms are available (e.g. TRW, Kolmogorov&Weinright 2006) Note: unlike Ishikawa, a-expansion and other methods (LBP, TRWS, etc.) apply to labels in

35. The University of Ontario a-expansion algorithm 1.Start with any initial solution 2.For each label “a” in any (e.g. random) order 1.Compute optimal a-expansion move (s-t graph cuts) 2.Decline the move if there is no energy decrease 3.Stop when no expansion move would decrease energy

36. The University of Ontario a-expansion moves initial solution -expansion In each a-expansion a given label “a” grabs space from other labels For each move we choose expansion that gives the largest decrease in the energy: binary optimization problem

37. The University of Ontario Multi-way graph cuts stereo vision original pair of “stereo” images depth map ground truth BVZ 1998 KZ 2002 right image left image

38. The University of Ontario a-expansions vs. ICM basic general alternative Iterated Conditional Modes (ICM) Example : consider pair-wise energy Unlike a-expansion, optimizes only over a single pixel at a time => local (pixel-wise) optimization - Consider current labeling L and any given pixel - Treat label L p as the only optimization variable x keeping other labels for fixed. - This reduces our energy to - Select optimal by enumeration and set new label. - Iterate over all pixels until no pixel operation reduces the energy. ICM algorithm

39. The University of Ontario a-expansions vs. simulated annealing - Consider current labeling L and any given pixel - Treat label L p as the only optimization variable x keeping other labels for fixed. - This reduces our energy to - Select optimal by enumeration and set new label. - Iterate over all pixels until no pixel operation reduces the energy. ICM algorithm - incorporates the following randomization strategy into ICM: at each pixel p label is updated randomly according to probabilities x 0 1 2 … n Pr(x) ~... NOTE 1 - lower energy E p (x) gives x more chances to be selected randomization of ICM simulated annealing (SA) NOTE 2 - higher temperature parameter T means more randomness - lower temperature parameter T reduces to ICM (optimal x is always selected) Unlike a-expansion, optimizes only over a single pixel at a time => local (pixel-wise) optimization Typical SA starts with high T and gradually reduces T to zero. Szeliski: appendix B.5.1 [Geman&Geman 1982]

40. The University of Ontario a-expansions vs. simulated annealing initial labeling one pixel local move (ICM or SA) large move ( a-b swap) large move (a-expansion) a- expansion and ICM/SA are all greedy iterative methods but, they converge to different kinds of local minima -ICM/SA solution can not be improved by changing a label of any one pixel to any given label a -a- expansion solution can not be improved by changing any subset of pixels to any given label a small vs. large moves

41. The University of Ontario normalized correlation, start for annealing, 24.7% err a-expansions (BVZ 89,01) 90 seconds, 5.8% err a-expansions vs. simulated annealing

42. The University of Ontario simulated annealing, 19 hours, 20.3% err a-expansions (BVZ 89,01) 90 seconds, 5.8% err a-expansions vs. simulated annealing NOTE 1: ICM and SA are general methods applicable to arbitrary non-metric and high-order energies NOTE 2: now-days there are other general methods based on graph cuts, message passing, relaxations, etc. [BVZ,2001] a-expansion

43. The University of Ontario Other applications Graph-cut textures (Kwatra, Schodl, Essa, Bobick 2003) similar to “image-quilting” (Efros & Freeman, 2001) A B C D E F G H I J A B G D C F H I J E

44. The University of Ontario Other applications Graph-cut textures (Kwatra, Schodl, Essa, Bobick 2003) Google (maps stitching)

45. The University of Ontario Other applications Multi-object Extraction Obvious generalization of binary object extraction technique (Boykov, Jolly, Funkalea 2004)

46. The University of Ontario Other applications Image compositing (Agarwala et al. 2004, see Szeliski Sec 9.3.2)

47. The University of Ontario Block-coordinate descent alternating a-expansion (for segmentation L ) and fitting colors I i Color model fitting (multi-label version of Chan-Vese) Potts model

48. The University of Ontario Block-coordinate descent alternating a-expansion (for segmentation L ) and fitting colors I i Stereo via piece-wise constant plane fitting [Birchfield &Tomasi 1999] Models T = parameters of affine transformations T(p)=a p + b 2x2 2x1 Potts model

49. The University of Ontario Block-coordinate descent alternating a-expansion (for segmentation L ) and fitting colors I i Piece-wise smooth local plane fitting [Olsson et al. 2013] truncated angle-differences non-metric interactions need other optimization

50. The University of Ontario Block-coordinate descent alternating a-expansion (for segmentation L ) and fitting colors I i Signboard segmentation [Milevsky 2013] Labels = planes in RGBXY space C(p) = a x + b Potts model 3x2 3x1

51. The University of Ontario Signboard segmentation [Milevsky 2013] 3x2 3x1 Goal: detection of characters, then text line fitting and translation

52. The University of Ontario Learning pair-wise potentials Structure detection (Kumar and Hebert 2006, see Szeliski Sec 3.7) standard pair-wise potentialslearned pair-wise potentials

53. The University of Ontario Multi-label optimization n 80% of computer vision and bio-medical image analysis are ill-posed labeling problems requiring optimization of regularization energies n Most problems are NP hard n Optimization algorithms is area of active research Google, Microsoft, GE, Siemens, Adobe, etc. LP relaxations [Schlezinger, Kolmogorov, Komodakis, Savchinsky,…] Message passing, e.g. LBP, TRWS [Kolmogorov] Graph Cuts (a-expanson, a/b-swap, fusion, FTR, etc) Variational methods