1. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Higher-Order Clique Reduction in Binary Graph Cut Hiroshi Ishikawa Nagoya City University Department of Information and Biological Sciences

2. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Contribution of this work Reduce any higher-order binary MRF into first order Adds variables Can also be used for multi-label energy, with the Fusion Move technique

3. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Energy Minimization Close to Y Smooth Given Y Find X Assigns X v (= 0 or 1 ) to each pixel v All pixels Neighboring pixels

4. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Energy Minimization Good (Low Energy)Bad (High Energy) Better (Lower Energy)Worse (Higher Energy) ABCD

5. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Energy Minimization Good (Low Energy)Bad (High Energy) 12 Bad 40 Good

6. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Energy Minimization 10 As 0 Ds Better (Lower Energy)Worse (Higher Energy) 8 Bs 3 Cs 10 As 0 Ds 4 Bs 7 Cs ABCD

7. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Third Order (Clique up to 4 pixels) Higher-Order Energy First Order (Clique up to 2 pixels) General Order C : a set of cliques Clique

8. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. First-Order MRF Minimization Graph cuts Greig et al. 89 Boykov et al. CVPR98, PAMI2001( -exp.) Kolmogorov & Zabih. PAMI2004 Belief propagation Felzenszwalb & Huttenlocher. IJCV2006 Meltzer et al. ICCV2005 Tree-reweighted message passing Kolmogorov. PAMI2006

9. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Higher-Order MRF Minimization Graph cuts Kolmogorov & Zabih. PAMI2004 Freedman & Drineas. CVPR2005 Woodford et al. CVPR2008 Kohli et al. PAMI08, Cremers&Grady ECCV06 Rother et al. CVPR2009 Komodakis & Paragios. CVPR2009 Belief propagation Lan et al. ECCV2006 Potetz. CVPR2008

10. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Higher-Order MRF Minimization Graph cuts Kolmogorov & Zabih. PAMI2004 Freedman & Drineas. CVPR2005 Woodford et al. CVPR2008 Kohli et al. PAMI08, Cremers&Grady ECCV06 Rother et al. CVPR2009 Komodakis & Paragios. CVPR2009 Belief propagation Lan et al. ECCV2006 Potetz. CVPR2008

11. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Functions of Binary Variables Pseudo-Boolean function (PBF) Function of binary ( 0 or 1 ) variables Can always write uniquely as a polynomial One variable x : E 0 (1 x) + E 1 x Two variables x, y : E 00 (1 x) (1 y) + E 01 (1 x) y + E 10 x (1 y) + E 11 x y Three variables x, y, z : E 000 (1 x) (1 y) (1 z) + E 001 (1 x) (1 y) z + … + E 111 x y z n th order binary MRF = ( n +1) th degree PBF

12. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. 2 nd -Order (Cubic) Case Kolmogorov & Zabih. PAMI2004 Freedman & Drineas. CVPR2005 Reduce cubic PBF into quadratic one using 0 0 0 x y z B ={0,1} 0 0 1 0 1 1 1 1 1

13. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. 2 nd -Order (Cubic) Case If a < 0 So, in a minimization problem, we can substitute by Thus

14. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Higher-Order Case if a < 0

15. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Higher-Order Case For a > 0 and d > 3, nothing similar is known our contribution Imagine such a formula:

16. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Higher-Order Case For a > 0 and d > 3, nothing similar is known our contribution Imagine such a formula: Notice LHS is symmetric i.e., if we swap the value of two variables, LHS is unchanged So RHS must be symmetric, too.

17. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Symmetric Polynomial Fact Any symmetric polynomial can be written as a polynomial in terms of elementary symmetric polynomials. If f (x, y, z, t) is quadratic symmetric, it can be written with a polynomial P(u,v) :

18. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Quartic (Degree 4) Case

19. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Quartic (Degree 4) Case

20. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Quartic (Degree 4) Case An exhaustive search for a, b, c, d, e yields

21. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Quintic (Degree 5) Case Similarly, and so on, until one can guess…

22. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. General Case where

23. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. General Case For each monomial, the number of new variable is: For instance, general quintic looks like: So the number is exponential in degree

24. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Multiple labels: Fusion Move Assume labels Labeling Y assigns a label Y v to each v Fusion Move Iteratively update Y : 1. Generate a proposed labeling P Lempitsky et al. ICCV2007

25. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Multiple labels: Fusion Move Assume labels Labeling Y assigns a label Y v to each v Fusion Move Iteratively update Y : 1. Generate a proposed labeling P 2. Merge Y and P The merge defines a binary problem: For each v, change Y v to P v or not Lempitsky et al. ICCV2007

26. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Multiple labels: Fusion Move Fusion Move Iteratively update Y : 1. Generate a proposed labeling P 2. Merge Y and P The merge defines a binary problem: For each v, change Y v to P v or not 1 0 0 1 1 0 0 0 1 0 1 1 0 1 0 0 YPX

27. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Multiple labels: Fusion Move Fusion Move Iteratively update Y : 1. Generate a proposed labeling P 2. Merge Y and P The merge defines a binary problem: For each v, change Y v to P v or not 1 0 0 1 1 0 0 0 1 0 1 1 0 1 0 0 YPX 0 0 0 0 0 0 0 1 1 1 1 1 1 0 1 0

28. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Fusion Move with QPBO QPBO (Roof duality) Minimizes submodular E globally. For non-submodular E, assigns each pixel 0, 1, or unlabeled With fusion move, by not changing unlabeled pixels to P, E doesnt increase Hammer et al. 1984, Boros et al. 1991, 2006 Kolmogorov & Rother PAMI2007, Rother et al. CVPR2007

29. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Experiment: Denoising by FoE FoE (Fields of Experts) Roth & Black CVPR2005 A higher-order prior for natural images C : a set of cliques C :

30. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Experiment: Denoising by FoE OriginalNoise-added1 st order3 rd order

31. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Experiment: Denoising by FoE Lan et al.PotetzThis work = 10 = 20 0 5000 10000 15000 20000 25000 30000 35000 40000 45000 24 25 26 27 28 29 30 31 32 PSNR (larger the better)Energy (smaller the better) Lan et al.PotetzThis work Lan et al. ECCV2006 ~8 hours Potetz. CVPR2008~30 mins This work~10 mins

32. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Experiment: Denoising by FoE Energy & PSNR Two proposal generation strategies 22 23 24 25 26 27 050100150200250 PSNR E(×1000) time (sec.) blur & random expansion blur & random expansion

33. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Summary Reduce any higher-order binary MRF into first order Adds variables Number exponential in order For multi-label, can be used with Fusion Move with QPBO

34. CVPR2009: IEEE Computer Society Conference on Computer Vision and Pattern Recognition Miami Beach, Florida. June 20-25. Thank you! Code available at http://www.nsc.nagoya-cu.ac.jp/~hi/ Acknowledgements Stefan Roth, Brian Potetz, and Vladimir Kolmogorov