1. P 3 & Beyond Solving Energies with Higher Order Cliques Pushmeet Kohli Pawan Kumar Philip H. S. Torr Oxford Brookes University CVPR 2007

2. Energy Functions Observed Variables Hidden Variables MAP Inference

3. Energy Functions MAP Inference Energy Minimization

4. Energy Functions Pairwise Energy Functions UnaryPairwise

5. Energy Functions Pairwise Energy Functions UnaryPairwise

6. Energy Functions Pairwise Energy Functions UnaryPairwise Efficient Algorithms for Minimization Message Passing (BP, TRW) Move Making (Expansion/Swap)

7. Energy Functions Pairwise Energy Functions UnaryPairwise Efficient Algorithms for Minimization Message Passing (BP, TRW) Move Making (Expansion/Swap) Restricted Expressive Power!

8. Energy Functions Higher Order Energy Functions UnaryPairwiseHigher order More expressive than pairwise FOE: Field of Experts (Roth & Black CVPR05)

9. Energy Functions Higher Order Energy Functions UnaryPairwiseHigher order Computationally expensive to minimize! Exponential Complexity O(L N ) L = Number of Labels N = Size of Clique

10. Minimizing Higher Order Energies Efficient BP in Higher Order MRFs (Lan, Roth, Huttenlocher & Black, ECCV 06) 2x2 clique potentials for Image Denoising Searched a restricted state space 16 minutes per iteration Pairwise MRF Higher order MRF Noisy Image

11. Energy Functions Higher Order Energy Functions UnaryPairwiseHigher order Our Method Move making algorithm Can handle cliques of thousand of variables Extremely Efficient ( works in seconds)

12. Talk Outline Move making Algorithms Solvable Higher Order Potentials Moves for the P N Potts Model Application: Texture Segmentation

13. Move Making Algorithms Solution Space Energy

14. Move Making Algorithms Search Neighbourhood Current Solution Optimal Move Solution Space Energy

15. Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move x E(x)E(x) xcxc Transformation function T EmEm Move Energy (t)(t) E m (t) = E(T(x c, t)) T(x c, t) = x n = x c + t

16. Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move E(x)E(x) xcxc Transformation function T EmEm Move Energy (t)(t) x E m (t) = E(T(x c, t)) T(x c, t) = x n = x c + t

17. Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move E(x)E(x) xcxc E m (t) = E(T(x c, t)) Transformation function T EmEm Move Energy T(x c, t) = x n = x c + t minimize t* Optimal Move (t)(t) x

18. Computing the Optimal Move Search Neighbourhood Current Solution Optimal Move E(x)E(x) xcxc Transformation function T EmEm Move Energy (t)(t) x Key Characteristic: Search Neighbourhood Bigger the better!

19. Moves using Graph Cuts Expansion and Swap Move Algorithm [Boykov, Veksler, Zabih] Exponential Move Search Space (Good ) Move encoded by binary vector t Move Energy

20. Moves using Graph Cuts Expansion and Swap Move Algorithm [Boykov, Veksler, Zabih] Exponential Move Search Space (Good ) Move encoded by binary vector t Move Energy Optimal move t* in polynomial time Submodular

21. Expansion Move  Expansion Transformation Variables take label  or retain current label Optimal move can be computed for pairwise potentials which are metric. [Boykov, Veksler, Zabih]

22. Expansion Move Sky House Tree Ground Initialize with Tree Status: Expand GroundExpand HouseExpand Sky [Boykov, Veksler, Zabih]

23. Swap Move Optimal move can be computed for pairwise potentials which are semi-metric.  - Swap Transformation Variables labeled  can swap their labels [Boykov, Veksler, Zabih]

24. Swap Move Sky House Tree Ground Swap Sky, House [Boykov, Veksler, Zabih]

25. Moves for Higher Order Potentials Question you should be asking: Can my higher order potential be solved using α-expansions?

26. Moves for Higher Order Potentials Question you should be asking: Show that move energy is submodular for all x c Can my higher order potential be solved using α-expansions?

27. Moves for Higher Order Potentials Question you should be asking: Show that move energy is submodular for all x c Can my higher order potential be solved using α-expansions? Not an easy thing to do!

28. Form of the Higher Order Potentials Moves for Higher Order Potentials Clique Inconsistency function: Pairwise potential: xixi xjxj xkxk xmxm xlxl c Sum Form Max Form

29. Theoretical Results: Swap Move energy is always submodular if non-decreasing concave. See paper for proofs

30. Theoretical Results: Expansion Move energy is always submodular if increasing linear See paper for proofs

31. P N Potts Model

32. c

33. c Cost :  red

34. P N Potts Model c Cost :  max

35. Optimal moves for P N Potts Computing the optimal swap move c Label 3 Label 4 Label 1  Label 2 

36. Optimal moves for P N Potts Computing the optimal swap move c Label 3 Label 4 Case 1 Not all variables assigned label 1 or 2 Label 1  Label 2 

37. Optimal moves for P N Potts Computing the optimal swap move c Label 3 Label 4 Case 1 Not all variables assigned label 1 or 2 Label 1  Label 2 

38. Optimal moves for P N Potts Computing the optimal swap move c Label 3 Label 4 Case 1 Not all variables assigned label 1 or 2 Label 1  Label 2 

39. Optimal moves for P N Potts Computing the optimal swap move c Label 3 Label 4 Case 1 Not all variables assigned label 1 or 2 Move Energy is independent of t c and can be ignored. Label 1  Label 2 

40. Optimal moves for P N Potts Computing the optimal swap move c Label 1  Label 2  Label 3 Label 4 Case 2 All variables assigned label 1 or 2

41. Optimal moves for P N Potts Computing the optimal swap move c Label 3 Label 4 Case 2 All variables assigned label 1 or 2 Can be minimized by solving a st-mincut problem Label 1  Label 2 

42. Solving the Move Energy Add a constant This transformation does not effect the solution

43. Solving the Move Energy Computing the optimal swap move Source Sink v1v1 v2v2 vnvn MsMs MtMt t i = 0 v i  Source Set t j = 1 v j  Sink Set

44. Solving the Move Energy Computing the optimal swap move Case 1: all x i =  (v i  Source ) Cost: Source Sink v1v1 v2v2 vnvn MsMs MtMt

45. Solving the Move Energy Computing the optimal swap move v1v1 v2v2 vnvn MsMs MtMt Cost: Source Sink Case 2: all x i =  (v i  Sink )

46. Solving the Move Energy Computing the optimal swap move Cost: v1v1 v2v2 vnvn MsMs MtMt Source Sink Case 3: all x i =  (v i  Source, Sink )

47. Optimal moves for P N Potts The expansion move energy Similar graph construction. See paper for details

48. Experimental Results Unary (Colour) Pairwise (Smoothness) Higher Order (Texture) Original Image Texture Segmentation

49. Experimental Results Texture Segmentation Unary (Colour) Pairwise (Smoothness) Higher Order (Texture) Colour Histogram Unary Cost: Tree

50. Experimental Results Texture Segmentation Unary (Colour) Pairwise (Smoothness) Higher Order (Texture) Edge Sensitive Smoothness Cost

51. Experimental Results Texture Segmentation Unary (Colour) Pairwise (Smoothness) Higher Order (Texture) Expansion Solution

52. Higher Order Texture Potentials Patch Dictionary (Tree) 5x5 patches

53. Higher Order Texture Potentials Patch Dictionary (Tree) G (c,p s ): L 1 distance between patch p s and pixel set

54. Experimental Results Texture Segmentation Unary (Colour) Pairwise (Smoothness) Higher Order (Texture) Original PairwiseHigher order

55. Experimental Results OriginalSwap (3.2 sec) Expansion (2.5 sec) PairwiseHigher Order Swap (4.2 sec) Expansion (3.0 sec)

56. Experimental Results Original PairwiseHigher Order Swap (4.7 sec) Expansion (3.7sec) Swap (5.0 sec) Expansion (4.4 sec)

57. Conclusions & Future Work Efficient minimization of certain higher order energies Can handle very large cliques Allows more expressive functions Explore other interesting family of potential functions

58. Thanks Questions?