1. GraphCut-based Optimisation for Computer Vision Ľubor Ladický

2. Overview Motivation Min-Cut / Max-Flow (Graph Cut) Algorithm Markov and Conditional Random Fields Random Field Optimisation using Graph Cuts Submodular vs. Non-Submodular Problems Pairwise vs. Higher Order Problems 2-Label vs. Multi-Label Problems Recent Advances in Random Field Optimisation Conclusions

3. Overview Motivation Min-Cut / Max-Flow (Graph Cut) Algorithm Markov and Conditional Random Fields Random Field Optimisation using Graph Cuts Submodular vs. Non-Submodular Problems Pairwise vs. Higher Order Problems 2-Label vs. Multi-Label Problems Recent Advances in Random Field Optimisation Conclusions

4. Image Labelling Problems Image DenoisingGeometry EstimationObject Segmentation Assign a label to each image pixel Building Sky Tree Grass Depth Estimation

5. Image Labelling Problems Labellings highly structured Possible labelling Unprobable labellingImpossible labelling

6. Image Labelling Problems Labellings highly structured Labels highly correlated with very complex dependencies Neighbouring pixels tend to take the same label Low number of connected components Classes present may be seen in one image Geometric / Location consistency Planarity in depth estimation … many others (task dependent)

7. Image Labelling Problems Labelling highly structured Labels highly correlated with very complex dependencies Independent label estimation too hard

8. Image Labelling Problems Labelling highly structured Labels highly correlated with very complex dependencies Independent label estimation too hard Whole labelling should be formulated as one optimisation problem

9. Image Labelling Problems Labelling highly structured Labels highly correlated with very complex dependencies Independent label estimation too hard Whole labelling should be formulated as one optimisation problem Number of pixels up to millions Hard to train complex dependencies Optimisation problem is hard to infer

10. Image Labelling Problems Labelling highly structured Labels highly correlated with very complex dependencies Independent label estimation too hard Whole labelling should be formulated as one optimisation problem Number of pixels up to millions Hard to train complex dependencies Optimisation problem is hard to infer Vision is hard !

11. Image Labelling Problems You can either Change the subject from the Computer Vision to the History of Renaissance Art Vision is hard !

12. Image Labelling Problems You can either Change the subject from the Computer Vision to the History of Renaissance Art or Learn everything about Random Fields and Graph Cuts Vision is hard !

13. Foreground / Background Estimation Rother et al. SIGGRAPH04

14. Foreground / Background Estimation Data termSmoothness term Data term Estimated using FG / BG colour models Smoothness term where Intensity dependent smoothness

15. Foreground / Background Estimation Data termSmoothness term

16. Foreground / Background Estimation Data termSmoothness term How to solve this optimisation problem?

17. Foreground / Background Estimation Data termSmoothness term How to solve this optimisation problem? Transform into min-cut / max-flow problem Solve it using min-cut / max-flow algorithm

18. Overview Motivation Min-Cut / Max-Flow (Graph Cut) Algorithm Markov and Conditional Random Fields Random Field Optimisation using Graph Cuts Submodular vs. Non-Submodular Problems Pairwise vs. Higher Order Problems 2-Label vs. Multi-Label Problems Applications Conclusions

19. Max-Flow Problem source sink 9 5 6 8 4 2 2 2 5 3 5 5 3 1 1 6 2 3 Task : Maximize the flow from the sink to the source such that 1) The flow it conserved for each node 2) The flow for each pipe does not exceed the capacity

20. Max-Flow Problem source sink 9 5 6 8 4 2 2 2 5 3 5 5 3 1 1 6 2 3

21. Max-Flow Problem source sink 9 5 6 8 4 2 2 2 5 3 5 5 3 1 1 6 2 3 flow from the source capacity flow from node i to node j conservation of flow set of edges set of nodes

22. Max-Flow Problem source sink 9 5 6 8 4 2 2 2 5 3 5 5 3 1 1 6 2 3 Ford & Fulkerson algorithm (1956) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph (“subtract” the flow) Find the path in the residual graph End

23. Max-Flow Problem source sink 9 5 6 8 4 2 2 2 5 3 5 5 3 1 1 6 2 3 Ford & Fulkerson algorithm (1956) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph (“subtract” the flow) Find the path in the residual graph End

24. Max-Flow Problem source sink 9 5 6 8 4 2 2 2 5 3 5 5 3 1 1 6 2 3 Ford & Fulkerson algorithm (1956) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph (“subtract” the flow) Find the path in the residual graph End flow = 3

25. Max-Flow Problem source sink 9 5 6-3 8-3 4 2+3 2 2 5-3 3-3 5 5 3 1 1 6 2 3 Ford & Fulkerson algorithm (1956) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph (“subtract” the flow) Find the path in the residual graph End flow = 3 +3

26. Max-Flow Problem source sink 9 5 3 5 4 5 2 2 2 5 5 3 1 1 6 2 3 Ford & Fulkerson algorithm (1956) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph (“subtract” the flow) Find the path in the residual graph End flow = 3 3

27. Max-Flow Problem source sink 9 5 3 5 4 5 2 2 2 5 5 3 1 1 6 2 3 Ford & Fulkerson algorithm (1956) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph (“subtract” the flow) Find the path in the residual graph End flow = 3 3

28. Max-Flow Problem source sink 9 5 3 5 4 5 2 2 2 5 5 3 1 1 6 2 3 Ford & Fulkerson algorithm (1956) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph (“subtract” the flow) Find the path in the residual graph End flow = 6 3

29. Max-Flow Problem source sink 9-3 5 3 5-3 4 5 2 2 2 5 5 3-3 1 1 6 2 Ford & Fulkerson algorithm (1956) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph (“subtract” the flow) Find the path in the residual graph End flow = 6 3 +3

30. Max-Flow Problem source sink 6 5 3 2 4 5 2 2 2 5 5 1 1 6 2 Ford & Fulkerson algorithm (1956) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph (“subtract” the flow) Find the path in the residual graph End flow = 6 3 3 3

31. Max-Flow Problem source sink 6 5 3 2 4 5 2 2 2 5 5 1 1 6 2 Ford & Fulkerson algorithm (1956) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph (“subtract” the flow) Find the path in the residual graph End flow = 6 3 3 3

32. Max-Flow Problem source sink 6 5 3 2 4 5 2 2 2 5 5 1 1 6 2 Ford & Fulkerson algorithm (1956) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph (“subtract” the flow) Find the path in the residual graph End flow = 11 3 3 3

33. Max-Flow Problem source sink 6-5 5-5 3 2 4 5 2 2 2 1+5 1 6 2+5 Ford & Fulkerson algorithm (1956) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph (“subtract” the flow) Find the path in the residual graph End flow = 11 3 3 3

34. Max-Flow Problem source sink 1 3 2 4 5 2 2 2 6 1 6 7 Ford & Fulkerson algorithm (1956) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph (“subtract” the flow) Find the path in the residual graph End flow = 11 3 3 3

35. Max-Flow Problem source sink 1 3 2 4 5 2 2 2 6 1 6 7 Ford & Fulkerson algorithm (1956) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph (“subtract” the flow) Find the path in the residual graph End flow = 11 3 3 3

36. Max-Flow Problem source sink 1 3 2 4 5 2 2 2 6 1 6 7 Ford & Fulkerson algorithm (1956) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph (“subtract” the flow) Find the path in the residual graph End flow = 13 3 3 3

37. Max-Flow Problem source sink 1 3 2-2 4 5 6 1 6 7 Ford & Fulkerson algorithm (1956) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph (“subtract” the flow) Find the path in the residual graph End flow = 13 3 3 3 +2

38. Max-Flow Problem source sink 1 3 4 5 6 1 6 7 Ford & Fulkerson algorithm (1956) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph (“subtract” the flow) Find the path in the residual graph End flow = 13 3 3 3 2 2

39. Max-Flow Problem source sink 1 3 4 5 6 1 6 7 Ford & Fulkerson algorithm (1956) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph (“subtract” the flow) Find the path in the residual graph End flow = 13 3 3 3 2 2

40. Max-Flow Problem source sink 1 3 4 5 6 1 6 7 Ford & Fulkerson algorithm (1956) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph (“subtract” the flow) Find the path in the residual graph End flow = 15 3 3 3 2 2

41. Max-Flow Problem source sink 1 3-2 4-2 5 6 1 6-2 7 Ford & Fulkerson algorithm (1956) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph (“subtract” the flow) Find the path in the residual graph End flow = 15 3 3 3+2 2 2-2 +2

42. Max-Flow Problem source sink 1 1 5 6 1 4 7 Ford & Fulkerson algorithm (1956) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph (“subtract” the flow) Find the path in the residual graph End flow = 15 3 3 5 2 2 2

43. Max-Flow Problem source sink 1 1 5 6 1 4 7 Ford & Fulkerson algorithm (1956) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph (“subtract” the flow) Find the path in the residual graph End flow = 15 3 3 5 2 2 2

44. Max-Flow Problem source sink 1 1 5 6 1 4 7 Ford & Fulkerson algorithm (1956) Find the path from source to sink While (path exists) flow += maximum capacity in the path Build the residual graph (“subtract” the flow) Find the path in the residual graph End flow = 15 3 3 5 2 2 2

45. Max-Flow Problem source sink 1 1 5 6 1 4 7 Ford & Fulkerson algorithm (1956) Why is the solution globally optimal ? flow = 15 3 3 5 2 2 2

46. Max-Flow Problem source sink 1 1 5 6 1 4 7 Ford & Fulkerson algorithm (1956) Why is the solution globally optimal ? 1. Let S be the set of reachable nodes in the residual graph flow = 15 3 3 5 2 2 2 S

47. Max-Flow Problem Ford & Fulkerson algorithm (1956) Why is the solution globally optimal ? 1. Let S be the set of reachable nodes in the residual graph 2. The flow from S to V - S equals to the sum of capacities from S to V – S source sink 9 5 6 8 4 2 2 2 5 3 5 5 3 1 1 6 2 3 S

48. Max-Flow Problem Ford & Fulkerson algorithm (1956) Why is the solution globally optimal ? 1. Let S be the set of reachable nodes in the residual graph 2. The flow from S to V - S equals to the sum of capacities from S to V – S 3. The flow from any A to V - A is upper bounded by the sum of capacities from A to V – A source sink 9 5 6 8 4 2 2 2 5 3 5 5 3 1 1 6 2 3 A

49. Max-Flow Problem flow = 15 Ford & Fulkerson algorithm (1956) Why is the solution globally optimal ? 1. Let S be the set of reachable nodes in the residual graph 2. The flow from S to V - S equals to the sum of capacities from S to V – S 3. The flow from any A to V - A is upper bounded by the sum of capacities from A to V – A 4. The solution is globally optimal source sink 8/9 5/5 5/6 8/8 2/4 0/2 2/2 0/2 5/5 3/3 5/5 3/3 0/1 2/6 0/2 3/3 Individual flows obtained by summing up all paths

50. Max-Flow Problem flow = 15 Ford & Fulkerson algorithm (1956) Why is the solution globally optimal ? 1. Let S be the set of reachable nodes in the residual graph 2. The flow from S to V - S equals to the sum of capacities from S to V – S 3. The flow from any A to V - A is upper bounded by the sum of capacities from A to V – A 4. The solution is globally optimal source sink 8/9 5/5 5/6 8/8 2/4 0/2 2/2 0/2 5/5 3/3 5/5 3/3 0/1 2/6 0/2 3/3

51. Max-Flow Problem source sink 1000 Ford & Fulkerson algorithm (1956) Order does matter 1000 1

52. Max-Flow Problem source sink 1000 Ford & Fulkerson algorithm (1956) Order does matter 1000 1

53. Max-Flow Problem source sink 999 Ford & Fulkerson algorithm (1956) Order does matter 1000 1

54. Max-Flow Problem source sink 999 Ford & Fulkerson algorithm (1956) Order does matter 1000 1

55. Max-Flow Problem source sink 999 Ford & Fulkerson algorithm (1956) Order does matter 999 1

56. Max-Flow Problem source sink 999 Ford & Fulkerson algorithm (1956) Order does matter Standard algorithm not polynomial Breath first leads to O(VE 2 ) Path found in O(E) At least one edge gets saturated The saturated edge distance to the source has to increase and is at most V leading to O(VE) (Edmonds & Karp, Dinic) 999 1

57. Max-Flow Problem source sink 999 Ford & Fulkerson algorithm (1956) Order does matter Standard algorithm not polynomial Breath first leads to O(VE 2 ) (Edmonds & Karp) Various methods use different algorithm to find the path and vary in complexity 999 1

58. Min-Cut Problem source sink 9 5 6 8 4 2 2 2 5 3 5 5 3 1 1 6 2 3 Task : Minimize the cost of the cut 1) Each node is either assigned to the source S or sink T 2) The cost of the edge (i, j) is taken if (i  S) and (j  T)

59. Min-Cut Problem source sink 9 5 6 8 4 2 2 2 5 3 5 5 3 1 1 6 2 3

60. Min-Cut Problem source sink 9 5 6 8 4 2 2 2 5 3 5 5 3 1 1 6 2 3 source setsink set edge costs

61. Min-Cut Problem source sink 9 5 6 8 4 2 2 2 5 3 5 5 3 1 1 6 2 3 source setsink set edge costs S T cost = 18

62. Min-Cut Problem source sink 9 5 6 8 4 2 2 2 5 3 5 5 3 1 1 6 2 3 source setsink set edge costs S T cost = 25

63. Min-Cut Problem source sink 9 5 6 8 4 2 2 2 5 3 5 5 3 1 1 6 2 3 source setsink set edge costs S T cost = 23

64. Min-Cut Problem source sink 9 5 6 8 4 2 2 2 5 3 5 5 3 1 1 6 2 3 x1x1 x2x2 x3x3 x4x4 x5x5 x6x6

65. Min-Cut Problem source sink 9 5 6 8 4 2 2 2 5 3 5 5 3 1 1 6 2 3 x1x1 x2x2 x3x3 x4x4 x5x5 x6x6 transformable into Linear program (LP)

66. Min-Cut Problem source sink 9 5 6 8 4 2 2 2 5 3 5 5 3 1 1 6 2 3 x1x1 x2x2 x3x3 x4x4 x5x5 x6x6 Dual to max-flow problem transformable into Linear program (LP)

67. Min-Cut Problem source sink 9 5 6 8 4 2 2 2 5 3 5 5 3 1 1 6 2 3 x1x1 x2x2 x3x3 x4x4 x5x5 x6x6 After the substitution of the constraints :

68. Min-Cut Problem source sink 9 5 6 8 4 2 2 2 5 3 5 5 3 1 1 6 2 3 x i = 0  x i  S x i = 1  x i  T x1x1 x2x2 x3x3 x4x4 x5x5 x6x6 source edgessink edges Edges between variables

69. Min-Cut Problem source sink 9 5 6 8 4 2 2 2 5 3 5 5 3 1 1 6 2 3 C(x) = 5x 1 + 9x 2 + 4x 3 + 3x 3 (1-x 1 ) + 2x 1 (1-x 3 ) + 3x 3 (1-x 1 ) + 2x 2 (1-x 3 ) + 5x 3 (1-x 2 ) + 2x 4 (1-x 1 ) + 1x 5 (1-x 1 ) + 6x 5 (1-x 3 ) + 5x 6 (1-x 3 ) + 1x 3 (1-x 6 ) + 3x 6 (1-x 2 ) + 2x 4 (1-x 5 ) + 3x 6 (1-x 5 ) + 6(1-x 4 ) + 8(1-x 5 ) + 5(1-x 6 ) x1x1 x2x2 x3x3 x4x4 x5x5 x6x6

70. Min-Cut Problem C(x) = 5x 1 + 9x 2 + 4x 3 + 3x 3 (1-x 1 ) + 2x 1 (1-x 3 ) + 3x 3 (1-x 1 ) + 2x 2 (1-x 3 ) + 5x 3 (1-x 2 ) + 2x 4 (1-x 1 ) + 1x 5 (1-x 1 ) + 6x 5 (1-x 3 ) + 5x 6 (1-x 3 ) + 1x 3 (1-x 6 ) + 3x 6 (1-x 2 ) + 2x 4 (1-x 5 ) + 3x 6 (1-x 5 ) + 6(1-x 4 ) + 8(1-x 5 ) + 5(1-x 6 ) source sink 9 5 6 8 4 2 2 2 5 3 5 5 3 1 1 6 2 3 x1x1 x2x2 x3x3 x4x4 x5x5 x6x6

71. Min-Cut Problem C(x) = 2x 1 + 9x 2 + 4x 3 + 2x 1 (1-x 3 ) + 3x 3 (1-x 1 ) + 2x 2 (1-x 3 ) + 5x 3 (1-x 2 ) + 2x 4 (1-x 1 ) + 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 5x 6 (1-x 3 ) + 1x 3 (1-x 6 ) + 3x 6 (1-x 2 ) + 2x 4 (1-x 5 ) + 3x 6 (1-x 5 ) + 6(1-x 4 ) + 5(1-x 5 ) + 5(1-x 6 ) + 3x 1 + 3x 3 (1-x 1 ) + 3x 5 (1-x 3 ) + 3(1-x 5 ) source sink 9 5 6 8 4 2 2 2 5 3 5 5 3 1 1 6 2 3 x1x1 x2x2 x3x3 x4x4 x5x5 x6x6

72. Min-Cut Problem C(x) = 2x 1 + 9x 2 + 4x 3 + 2x 1 (1-x 3 ) + 3x 3 (1-x 1 ) + 2x 2 (1-x 3 ) + 5x 3 (1-x 2 ) + 2x 4 (1-x 1 ) + 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 5x 6 (1-x 3 ) + 1x 3 (1-x 6 ) + 3x 6 (1-x 2 ) + 2x 4 (1-x 5 ) + 3x 6 (1-x 5 ) + 6(1-x 4 ) + 5(1-x 5 ) + 5(1-x 6 ) + 3x 1 + 3x 3 (1-x 1 ) + 3x 5 (1-x 3 ) + 3(1-x 5 ) 3x 1 + 3x 3 (1-x 1 ) + 3x 5 (1-x 3 ) + 3(1-x 5 ) = 3 + 3x 1 (1-x 3 ) + 3x 3 (1-x 5 ) source sink 9 5 6 8 4 2 2 2 5 3 5 5 3 1 1 6 2 3 x1x1 x2x2 x3x3 x4x4 x5x5 x6x6

73. Min-Cut Problem C(x) = 3 + 2x 1 + 9x 2 + 4x 3 + 5x 1 (1-x 3 ) + 3x 3 (1-x 1 ) + 2x 2 (1-x 3 ) + 5x 3 (1-x 2 ) + 2x 4 (1-x 1 ) + 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 5x 6 (1-x 3 ) + 1x 3 (1-x 6 ) + 3x 6 (1-x 2 ) + 2x 4 (1-x 5 ) + 3x 6 (1-x 5 ) + 6(1-x 4 ) + 5(1-x 5 ) + 5(1-x 6 ) + 3x 3 (1-x 5 ) source sink 9 5 3 5 4 5 2 2 2 5 5 3 1 1 6 2 3 3 x1x1 x2x2 x3x3 x4x4 x5x5 x6x6

74. Min-Cut Problem C(x) = 3 + 2x 1 + 9x 2 + 4x 3 + 5x 1 (1-x 3 ) + 3x 3 (1-x 1 ) + 2x 2 (1-x 3 ) + 5x 3 (1-x 2 ) + 2x 4 (1-x 1 ) + 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 5x 6 (1-x 3 ) + 1x 3 (1-x 6 ) + 3x 6 (1-x 2 ) + 2x 4 (1-x 5 ) + 3x 6 (1-x 5 ) + 6(1-x 4 ) + 5(1-x 5 ) + 5(1-x 6 ) + 3x 3 (1-x 5 ) source sink 9 5 3 5 4 5 2 2 2 5 5 3 1 1 6 2 3 3 x1x1 x2x2 x3x3 x4x4 x5x5 x6x6

75. Min-Cut Problem C(x) = 3 + 2x 1 + 6x 2 + 4x 3 + 5x 1 (1-x 3 ) + 3x 3 (1-x 1 ) + 2x 2 (1-x 3 ) + 5x 3 (1-x 2 ) + 2x 4 (1-x 1 ) + 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 5x 6 (1-x 3 ) + 1x 3 (1-x 6 ) + 2x 5 (1-x 4 ) + 6(1-x 4 ) + 2(1-x 5 ) + 5(1-x 6 ) + 3x 3 (1-x 5 ) + 3x 2 + 3x 6 (1-x 2 ) + 3x 5 (1-x 6 ) + 3(1-x 5 ) 3x 2 + 3x 6 (1-x 2 ) + 3x 5 (1-x 6 ) + 3(1-x 5 ) = 3 + 3x 2 (1-x 6 ) + 3x 6 (1-x 5 ) source sink 9 5 3 5 4 5 2 2 2 5 5 3 1 1 6 2 3 3 x1x1 x2x2 x3x3 x4x4 x5x5 x6x6

76. Min-Cut Problem C(x) = 6 + 2x 1 + 6x 2 + 4x 3 + 5x 1 (1-x 3 ) + 3x 3 (1-x 1 ) + 2x 2 (1-x 3 ) + 5x 3 (1-x 2 ) + 2x 4 (1-x 1 ) + 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 5x 6 (1-x 3 ) + 1x 3 (1-x 6 ) + 2x 5 (1-x 4 ) + 6(1-x 4 ) + 3x 2 (1-x 6 ) + 3x 6 (1-x 5 ) + 2(1-x 5 ) + 5(1-x 6 ) + 3x 3 (1-x 5 ) source sink 6 5 3 2 4 5 2 2 2 5 5 1 1 6 2 3 3 3 x1x1 x2x2 x3x3 x4x4 x5x5 x6x6

77. Min-Cut Problem C(x) = 6 + 2x 1 + 6x 2 + 4x 3 + 5x 1 (1-x 3 ) + 3x 3 (1-x 1 ) + 2x 2 (1-x 3 ) + 5x 3 (1-x 2 ) + 2x 4 (1-x 1 ) + 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 5x 6 (1-x 3 ) + 1x 3 (1-x 6 ) + 2x 5 (1-x 4 ) + 6(1-x 4 ) + 3x 2 (1-x 6 ) + 3x 6 (1-x 5 ) + 2(1-x 5 ) + 5(1-x 6 ) + 3x 3 (1-x 5 ) source sink 6 5 3 2 4 5 2 2 2 5 5 1 1 6 2 3 3 3 x1x1 x2x2 x3x3 x4x4 x5x5 x6x6

78. Min-Cut Problem C(x) = 11 + 2x 1 + 1x 2 + 4x 3 + 5x 1 (1-x 3 ) + 3x 3 (1-x 1 ) + 7x 2 (1-x 3 ) + 2x 4 (1-x 1 ) + 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 6x 3 (1-x 6 ) + 2x 5 (1-x 4 ) + 6(1-x 4 ) + 3x 2 (1-x 6 ) + 3x 6 (1-x 5 ) + 2(1-x 5 ) + 3x 3 (1-x 5 ) source sink 1 3 2 4 5 2 2 2 6 1 6 7 3 3 3 x1x1 x2x2 x3x3 x4x4 x5x5 x6x6

79. Min-Cut Problem C(x) = 11 + 2x 1 + 1x 2 + 4x 3 + 5x 1 (1-x 3 ) + 3x 3 (1-x 1 ) + 7x 2 (1-x 3 ) + 2x 4 (1-x 1 ) + 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 6x 3 (1-x 6 ) + 2x 5 (1-x 4 ) + 6(1-x 4 ) + 3x 2 (1-x 6 ) + 3x 6 (1-x 5 ) + 2(1-x 5 ) + 3x 3 (1-x 5 ) source sink 1 3 2 4 5 2 2 2 6 1 6 7 3 3 3 x1x1 x2x2 x3x3 x4x4 x5x5 x6x6

80. Min-Cut Problem C(x) = 13 + 1x 2 + 4x 3 + 5x 1 (1-x 3 ) + 3x 3 (1-x 1 ) + 7x 2 (1-x 3 ) + 2x 1 (1-x 4 ) + 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 6x 3 (1-x 6 ) + 2x 4 (1-x 5 ) + 6(1-x 4 ) + 3x 2 (1-x 6 ) + 3x 6 (1-x 5 ) + 3x 3 (1-x 5 ) source sink 1 3 4 5 6 1 6 7 3 3 3 2 2 x1x1 x2x2 x3x3 x4x4 x5x5 x6x6

81. Min-Cut Problem C(x) = 13 + 1x 2 + 2x 3 + 5x 1 (1-x 3 ) + 3x 3 (1-x 1 ) + 7x 2 (1-x 3 ) + 2x 1 (1-x 4 ) + 1x 5 (1-x 1 ) + 3x 5 (1-x 3 ) + 6x 3 (1-x 6 ) + 2x 4 (1-x 5 ) + 6(1-x 4 ) + 3x 2 (1-x 6 ) + 3x 6 (1-x 5 ) + 3x 3 (1-x 5 ) source sink 1 3 4 5 6 1 6 7 3 3 3 2 2 x1x1 x2x2 x3x3 x4x4 x5x5 x6x6

82. Min-Cut Problem C(x) = 15 + 1x 2 + 4x 3 + 5x 1 (1-x 3 ) + 3x 3 (1-x 1 ) + 7x 2 (1-x 3 ) + 2x 1 (1-x 4 ) + 1x 5 (1-x 1 ) + 6x 3 (1-x 6 ) + 6x 3 (1-x 5 ) + 2x 5 (1-x 4 ) + 4(1-x 4 ) + 3x 2 (1-x 6 ) + 3x 6 (1-x 5 ) source sink 1 1 5 6 1 4 7 3 3 5 2 2 2 x1x1 x2x2 x3x3 x4x4 x5x5 x6x6

83. Min-Cut Problem C(x) = 15 + 1x 2 + 4x 3 + 5x 1 (1-x 3 ) + 3x 3 (1-x 1 ) + 7x 2 (1-x 3 ) + 2x 1 (1-x 4 ) + 1x 5 (1-x 1 ) + 6x 3 (1-x 6 ) + 6x 3 (1-x 5 ) + 2x 5 (1-x 4 ) + 4(1-x 4 ) + 3x 2 (1-x 6 ) + 3x 6 (1-x 5 ) source sink 1 1 5 6 1 4 7 3 3 5 2 2 2 x1x1 x2x2 x3x3 x4x4 x5x5 x6x6

84. Min-Cut Problem C(x) = 15 + 1x 2 + 4x 3 + 5x 1 (1-x 3 ) + 3x 3 (1-x 1 ) + 7x 2 (1-x 3 ) + 2x 1 (1-x 4 ) + 1x 5 (1-x 1 ) + 6x 3 (1-x 6 ) + 6x 3 (1-x 5 ) + 2x 5 (1-x 4 ) + 4(1-x 4 ) + 3x 2 (1-x 6 ) + 3x 6 (1-x 5 ) source sink 1 1 5 6 1 4 7 3 3 5 2 2 2 x1x1 x2x2 x3x3 x4x4 x5x5 x6x6 cost = 0 min cut S T cost = 0 All coefficients positive Must be global minimum S – set of reachable nodes from s

85. Min-Cut Problem C(x) = 15 + 1x 2 + 4x 3 + 5x 1 (1-x 3 ) + 3x 3 (1-x 1 ) + 7x 2 (1-x 3 ) + 2x 1 (1-x 4 ) + 1x 5 (1-x 1 ) + 6x 3 (1-x 6 ) + 6x 3 (1-x 5 ) + 2x 5 (1-x 4 ) + 4(1-x 4 ) + 3x 2 (1-x 6 ) + 3x 6 (1-x 5 ) source sink 1 1 5 6 1 4 7 3 3 5 2 2 2 x1x1 x2x2 x3x3 x4x4 x5x5 x6x6 cost = 0 min cut S T All coefficients positive Must be global minimum T – set of nodes that can reach t (not necesarly the same)

86. Overview Motivation Min-Cut / Max-Flow (Graph Cut) Algorithm Markov and Conditional Random Fields Random Field Optimisation using Graph Cuts Submodular vs. Non-Submodular Problems Pairwise vs. Higher Order Problems 2-Label vs. Multi-Label Problems Recent Advances in Random Field Optimisation Conclusions

87. Markov and Conditional RF Markov / Conditional Random fields model probabilistic dependencies of the set of random variables

88. Markov and Conditional RF Markov / Conditional Random fields model conditional dependencies between random variables Each variable is conditionally independent of all other variables given its neighbours

89. Markov and Conditional RF Markov / Conditional Random fields model conditional dependencies between random variables Each variable is conditionally independent of all other variables given its neighbours Posterior probability of the labelling x given data D is : partition functionpotential functionscliques

90. Markov and Conditional RF Markov / Conditional Random fields model conditional dependencies between random variables Each variable is conditionally independent of all other variables given its neighbours Posterior probability of the labelling x given data D is : Energy of the labelling is defined as : partition functionpotential functionscliques

91. Markov and Conditional RF The most probable (Max a Posteriori (MAP)) labelling is defined as:

92. Markov and Conditional RF The most probable (Max a Posteriori (MAP)) labelling is defined as: The only distinction (MRF vs. CRF) is that in the CRF the conditional dependencies between variables depend also on the data

93. Markov and Conditional RF The most probable (Max a Posteriori (MAP)) labelling is defined as: The only distinction (MRF vs. CRF) is that in the CRF the conditional dependencies between variables depend also on the data Typically we define an energy first and then pretend there is an underlying probabilistic distribution there, but there isn’t really (Pssssst, don’t tell anyone)

94. Overview Motivation Min-Cut / Max-Flow (Graph Cut) Algorithm Markov and Conditional Random Fields Random Field Optimisation using Graph Cuts Submodular vs. Non-Submodular Problems Pairwise vs. Higher Order Problems 2-Label vs. Multi-Label Problems Recent Advances in Random Field Optimisation Conclusions

95. Standard CRF Energy Pairwise CRF models Data termSmoothness term

96. Graph Cut based Inference The energy / potential is called submodular (only) if for every pair of variables :

97. Graph Cut based Inference For 2-label problems x  {0, 1} :

98. Graph Cut based Inference For 2-label problems x  {0, 1} : Pairwise potential can be transformed into : where

99. Graph Cut based Inference For 2-label problems x  {0, 1} : Pairwise potential can be transformed into : where

100. Graph Cut based Inference For 2-label problems x  {0, 1} : Pairwise potential can be transformed into : where

101. Graph Cut based Inference After summing up :

102. Graph Cut based Inference After summing up : Let :

103. Graph Cut based Inference After summing up : Let :Then :

104. Graph Cut based Inference After summing up : Let :Then : Equivalent st-mincut problem is :

105. Foreground / Background Estimation Data termSmoothness term Data term Estimated using FG / BG colour models Smoothness term where Intensity dependent smoothness

106. Foreground / Background Estimation source sink Rother et al. SIGGRAPH04

107. Extendable to (some) Multi-label CRFs each state of multi-label variable encoded using multiple binary variables the cost of every possible cut must be the same as the associated energy the solution obtained by inverting the encoding Graph Cut based Inference Obtain solution Graph Cut Build Graph Encoding multi-label energy solution

108. Dense Stereo Estimation Left Camera ImageRight Camera ImageDense Stereo Result For each pixel assigns a disparity label : z Disparities from the discrete set {0, 1,.. D}

109. Dense Stereo Estimation Data term Left image feature Shifted right image feature Left Camera ImageRight Camera Image

110. Dense Stereo Estimation Data term Left image feature Shifted right image feature Smoothness term

111. Dense Stereo Estimation Ishikawa PAMI03 Encoding sour ce xi0xi0 xiDxiD xk0xk0 xkDxkD... x k 1 x k 2 source sink.

112. Dense Stereo Estimation Ishikawa PAMI03 Unary Potential sour ce xi0xi0 xiDxiD xk0xk0 xkDxkD... x k 1 x k 2 ∞ ∞ ∞ ∞ ∞ ∞ ∞ ∞ source sink. cost = + The cost of every cut should be equal to the corresponding energy under the encoding

113. Dense Stereo Estimation Ishikawa PAMI03 Unary Potential sour ce xi0xi0 xiDxiD xk0xk0 xkDxkD... x k 1 x k 2 ∞ ∞ ∞ ∞ ∞ ∞ ∞ ∞ source sink. cost = + The cost of every cut should be equal to the corresponding energy under the encoding

114. Dense Stereo Estimation Ishikawa PAMI03 Unary Potential sour ce xi0xi0 xiDxiD xk0xk0 xkDxkD... x k 1 x k 2 ∞ ∞ ∞ ∞ ∞ ∞ ∞ ∞ source sink. cost = ∞ The cost of every cut should be equal to the corresponding energy under the encoding

115. Dense Stereo Estimation Ishikawa PAMI03 Pairwise Potential sour ce xi0xi0 xiDxiD xk0xk0 xkDxkD... x k 1 x k 2 ∞ ∞ ∞ ∞ ∞ ∞ ∞ ∞ K K K K source sink. cost = + + K The cost of every cut should be equal to the corresponding energy under the encoding

116. Dense Stereo Estimation Ishikawa PAMI03 sour ce xi0xi0 xiDxiD xk0xk0 xkDxkD... x k 1 x k 2 ∞ ∞ ∞ ∞ ∞ ∞ ∞ ∞ K K K K source sink. cost = + + D K Pairwise Potential The cost of every cut should be equal to the corresponding energy under the encoding

117. Dense Stereo Estimation See [Ishikawa PAMI03] for more details sour ce xi0xi0 xiDxiD xk0xk0 xkDxkD... x k 1 x k 2 ∞ ∞ ∞ ∞ ∞ ∞ ∞ ∞ source sink. Extendable to any convex cost Convex function

118. Graph Cut based Inference Move making algorithms Original problem decomposed into a series of subproblems solvable with graph cut In each subproblem we find the optimal move from the current solution in a restricted search space Update solution Graph Cut Build Graph Encoding Propose move Initial solution solution

119. Graph Cut based Inference Example : [Boykov et al. 01] α-expansion – each variable either keeps its old label or changes to α – all α-moves are iteratively performed till convergence Transformation function : α  -swap – each variable taking label α or  can change its label to α or  – all α  -moves are iteratively performed till convergence

120. Graph Cut based Inference Sufficient conditions for move making algorithms : (all possible moves are submodular) α  -swap :semi-metricity Proof:

121. Graph Cut based Inference Sufficient conditions for move making algorithms : (all possible moves are submodular) α-expansion :metricity Proof:

122. Object-class Segmentation Data termSmoothness term Data term Smoothness term where Discriminatively trained classifier

123. Object-class Segmentation Original ImageInitial solution grass

124. Object-class Segmentation Original ImageInitial solutionBuilding expansion grass building

125. Object-class Segmentation Original ImageInitial solutionBuilding expansion Sky expansion grass building sky

126. Object-class Segmentation Original ImageInitial solutionBuilding expansion Sky expansion Tree expansion grass building sky tree

127. Object-class Segmentation Original ImageInitial solutionBuilding expansion Sky expansion Tree expansionFinal Solution grass building sky tree aeroplane

128. Graph Cut based Inference Range-expansion [Kumar, Veksler & Torr 1] – Expansion version of range swap moves – Each varibale can change its label to any label in a convex range or keep its old label Range-(swap) moves [Veksler 07] – Each variable in the convex range can change its label to any other label in the convex range – all range-moves are iteratively performed till convergence

129. Dense Stereo Reconstruction Data term Smoothness term Same as before Left Camera ImageRight Camera ImageDense Stereo Result

130. Dense Stereo Reconstruction Data term Smoothness term Same as before Left Camera ImageRight Camera ImageDense Stereo Result Convex range Truncation

131. Graph Cut based Inference Original ImageInitial Solution

132. Graph Cut based Inference Original ImageInitial SolutionAfter 1 st expansion

133. Graph Cut based Inference Original ImageInitial Solution After 2 nd expansion After 1 st expansion

134. Graph Cut based Inference Original ImageInitial Solution After 2 nd expansion After 1 st expansion After 3 rd expansion

135. Graph Cut based Inference Original ImageInitial Solution After 2 nd expansion After 1 st expansion After 3 rd expansionFinal solution

136. Graph Cut based Inference Extendible to certain classes of binary higher order energies Higher order terms have to be transformable to the pairwise submodular energy with binary auxiliary variables z C Higher order term Pairwise term Update solution Graph Cut Transf. to pairwise subm. Encoding Propose move Initial solution solution

137. Graph Cut based Inference Extendible to certain classes of binary higher order energies Higher order terms have to be transformable to the pairwise submodular energy with binary auxiliary variables z C Higher order term Pairwise term Example : Kolmogorov ECCV06, Ramalingam et al. DAM12

138. Graph Cut based Inference Enforces label consistency of the clique as a weak constraint Input image Pairwise CRFHigher order CRF Kohli et al. CVPR07

139. Graph Cut based Inference Update solution Graph Cut Transf. to pairwise subm. Encoding Propose move Initial solution solution If the energy is not submodular / graph-representable – Overestimation by a submodular energy – Quadratic Pseudo-Boolean Optimisation (QPBO)

140. Graph Cut based Inference Update solution Graph Cut Overestim. by pairwise subm. Encoding Propose move Initial solution solution Overestimation by a submodular energy (convex / concave) – We find an energy E’(t) s.t. – it is tight in the current solutionE’(t 0 ) = E(t 0 ) – Overestimates E(t) for all tE’(t) ≥ E(t) – We replace E(t) by E’(t) – The moves are not optimal, but quaranteed to converge – The tighter over-estimation, the better Example :

141. Graph Cut based Inference Quadratic Pseudo-Boolean Optimisation – Each binary variable is encoded using two binary variables x i and x i s.t. x i = 1 - x i _ _

142. Graph Cut based Inference Quadratic Pseudo-Boolean Optimisation source sink xixi xjxj xixi xjxj _ _ – Energy submodular in x i and x i – Solved by dropping the constraint x i = 1 - x i – If x i = 1 - x i the solution for x i is optimal _ _ _

143. Overview Motivation Min-Cut / Max-Flow (Graph Cut) Algorithm Markov and Conditional Random Fields Random Field Optimisation using Graph Cuts Submodular vs. Non-Submodular Problems Pairwise vs. Higher Order Problems 2-Label vs. Multi-Label Problems Recent Advances in Random Field Optimisation Conclusions

144. Motivation To propose formulations enforcing certain structure properties of the output useful in computer vision – Label Consistency in segments as a weak constraint – Label agreement over different scales – Label-set consistency – Multiple domains consistency To propose feasible Graph-Cut based inference method

145. Structures in CRF Taskar et al. 02 – associative potentials Woodford et al. 08 – planarity constraint Vicente et al. 08 – connectivity constraint Nowozin & Lampert 09 – connectivity constraint Roth & Black 09 – field of experts Woodford et al. 09 – marginal probability Delong et al. 10 – label occurrence costs

146. Object-class Segmentation Problem Aims to assign a class label for each pixel of an image Classifier trained on the training set Evaluated on never seen test images

147. Pairwise CRF models Standard CRF Energy for Object Segmentation Cannot encode global consistency of labels!! Local context Ladický, Russell, Kohli, Torr ECCV10

148. Thing – Thing Stuff - Stuff Stuff - Thing [ Images from Rabinovich et al. 07 ] Encoding Co-occurrence Co-occurrence is a powerful cue [Heitz et al. '08] [Rabinovich et al. ‘07] Ladický, Russell, Kohli, Torr ECCV10

149. Thing – Thing Stuff - Stuff Stuff - Thing Encoding Co-occurrence Co-occurrence is a powerful cue [Heitz et al. '08] [Rabinovich et al. ‘07] Proposed solutions : 1. Csurka et al. 08 - Hard decision for label estimation 2. Torralba et al. 03 - GIST based unary potential 3. Rabinovich et al. 07 - Full-connected CRF [ Images from Rabinovich et al. 07 ] Ladický, Russell, Kohli, Torr ECCV10

150. So... What properties should these global co-occurence potentials have ? Ladický, Russell, Kohli, Torr ECCV10

151. Desired properties 1. No hard decisions Ladický, Russell, Kohli, Torr ECCV10

152. Desired properties 1. No hard decisions Incorporation in probabilistic framework Unlikely possibilities are not completely ruled out Ladický, Russell, Kohli, Torr ECCV10

153. Desired properties 1. No hard decisions 2. Invariance to region size Ladický, Russell, Kohli, Torr ECCV10

154. Desired properties 1. No hard decisions 2. Invariance to region size Cost for occurrence of {people, house, road etc.. } invariant to image area Ladický, Russell, Kohli, Torr ECCV10

155. Desired properties 1. No hard decisions 2. Invariance to region size The only possible solution : Local context Global context Cost defined over the assigned labels L(x) L(x)={,, } Ladický, Russell, Kohli, Torr ECCV10

156. Desired properties 1. No hard decisions 2. Invariance to region size 3. Parsimony – simple solutions preferred L(x)={ building, tree, grass, sky } L(x)={ aeroplane, tree, flower, building, boat, grass, sky }  Ladický, Russell, Kohli, Torr ECCV10

157. Desired properties 1. No hard decisions 2. Invariance to region size 3. Parsimony – simple solutions preferred 4. Efficiency Ladický, Russell, Kohli, Torr ECCV10

158. Desired properties 1. No hard decisions 2. Invariance to region size 3. Parsimony – simple solutions preferred 4. Efficiency a) Memory requirements as O(n) with the image size and number or labels b) Inference tractable Ladický, Russell, Kohli, Torr ECCV10

159. Torralba et al.(2003) – Gist-based unary potentials Rabinovich et al.(2007) - complete pairwise graphs Csurka et al.(2008) - hard estimation of labels present Previous work Ladický, Russell, Kohli, Torr ECCV10

160. Zhu & Yuille 1996 – MDL prior Bleyer et al. 2010 – Surface Stereo MDL prior Hoiem et al. 2007 – 3D Layout CRF MDL Prior Delong et al. 2010 – label occurence cost Related work C(x) = K |L(x)| C(x) = Σ L K L δ L (x) Ladický, Russell, Kohli, Torr ECCV10

161. Zhu & Yuille 1996 – MDL prior Bleyer et al. 2010 – Surface Stereo MDL prior Hoiem et al. 2007 – 3D Layout CRF MDL Prior Delong et al. 2010 – label occurence cost Related work C(x) = K |L(x)| C(x) = Σ L K L δ L (x) All special cases of our model Ladický, Russell, Kohli, Torr ECCV10

162. Inference for Co-occurence Label indicator functions Co-occurence representation Ladický, Russell, Kohli, Torr ECCV10

163. Inference for Co-occurence Move Energy Cost of current label set Ladický, Russell, Kohli, Torr ECCV10

164. Inference for Co-occurence Move Energy Decomposition to α-dependent and α-independent part α-independentα-dependent Ladický, Russell, Kohli, Torr ECCV10

165. Inference for Co-occurence Move Energy Decomposition to α-dependent and α-independent part Either α or all labels in the image after the move Ladický, Russell, Kohli, Torr ECCV10

166. Inference for Co-occurence Move Energy submodularnon-submodular Ladický, Russell, Kohli, Torr ECCV10

167. Inference for Co-occurence Move Energy non-submodular Non-submodular energy overestimated by E'(t) E'(t) = E(t) for current solution E'(t)  E(t) for any other labelling Ladický, Russell, Kohli, Torr ECCV10

168. Inference for Co-occurence Move Energy non-submodular Non-submodular energy overestimated by E'(t) E'(t) = E(t) for current solution E'(t)  E(t) for any other labelling Occurrence - tight Ladický, Russell, Kohli, Torr ECCV10

169. Inference for Co-occurence Move Energy non-submodular Non-submodular energy overestimated by E'(t) E'(t) = E(t) for current solution E'(t)  E(t) for any other labelling Co-occurrence overestimation Ladický, Russell, Kohli, Torr ECCV10

170. Inference for Co-occurence Move Energy non-submodular Non-submodular energy overestimated by E'(t) E'(t) = E(t) for current solution E'(t)  E(t) for any other labelling General case [See the paper] Ladický, Russell, Kohli, Torr ECCV10

171. Inference for Co-occurence Move Energy non-submodular Non-submodular energy overestimated by E'(t) E'(t) = E(t) for current solution E'(t)  E(t) for any other labelling Quadratic representation Ladický, Russell, Kohli, Torr ECCV10

172. Potential Training for Co-occurence Ladický, Russell, Kohli, Torr ICCV09 Generatively trained Co-occurence potential Approximated by 2 nd order representation

173. Results on MSRC data set Input Image Comparisons of results with and without co-occurence Input Image Without Co-occurence With Co-occurence Without Co-occurence With Co-occurence Ladický, Russell, Kohli, Torr ICCV09

174. Pixel CRF Energy for Object-class Segmentation Pairwise CRF models Data termSmoothness term Input ImagePairwise CRF Result Shotton et al. ECCV06

175. Pixel CRF Energy for Object-class Segmentation Pairwise CRF models Data termSmoothness term Lacks long range interactions Results oversmoothed Data term information may be insufficient

176. Segment CRF Energy for Object-class Segmentation Pairwise CRF models Data termSmoothness term Input ImageUnsupervised Segmentation Shi, Malik PAMI2000, Comaniciu, Meer PAMI2002, Felzenschwalb, Huttenlocher, IJCV2004,

177. Segment CRF Energy for Object-class Segmentation Pairwise CRF models Data termSmoothness term Input ImagePairwise CRFResult Batra et al. CVPR08, Yang et al. CVPR07, Zitnick et al. CVPR08, Rabinovich et al. ICCV07, Boix et al. CVPR10

178. Segment CRF Energy for Object-class Segmentation Pairwise CRF models Data termSmoothness term Allows long range interactions Does not distinguish between instances of objects Cannot recover from incorrect segmentation

179. Segment CRF Energy for Object-class Segmentation Pairwise CRF models Data termSmoothness term Allows long range interactions Does not distinguish between instances of objects Cannot recover from incorrect segmentation

180. Segment CRF Energy for Object-class Segmentation Robust P N approach Pairwise CRFSegment consistency term Input Image Kohli, Ladický, Torr CVPR08 Higher order CRF

181. Segment CRF Energy for Object-class Segmentation Robust P N approach Pairwise CRFSegment consistency term Kohli, Ladický, Torr CVPR08

182. Segment CRF Energy for Object-class Segmentation Robust P N approach Pairwise CRFSegment consistency term Kohli, Ladický, Torr CVPR08 No dominant label

183. Segment CRF Energy for Object-class Segmentation Robust P N approach Pairwise CRFSegment consistency term Kohli, Ladický, Torr CVPR08 Dominant label l

184. Segment CRF Energy for Object-class Segmentation Robust P N approach Pairwise CRFSegment consistency term Kohli, Ladický, Torr CVPR08 Enforces consistency as a weak constraint Can recover from incorrect segmentation Can combine multiple segmentations Does not use features at segment level

185. Segment CRF Energy for Object-class Segmentation Robust P N approach Pairwise CRFSegment consistency term Kohli, Ladický, Torr CVPR08 Transformable to pairwise graph with auxiliary variable

186. Segment CRF Energy for Object-class Segmentation Robust P N approach Pairwise CRFSegment consistency term Ladický, Russell, Kohli, Torr ICCV09 where

187. Segment CRF Energy for Object-class Segmentation Robust P N approach Pairwise CRFSegment consistency term Input Image Ladický, Russell, Kohli, Torr ICCV09 Higher order CRF

188. Associative Hierarchical CRFs Ladický, Russell, Kohli, Torr ICCV09 Can be generalized to Hierarchical model Allows unary potentials for region variables Allows pairwise potentials for region variables Allows multiple layers and multiple hierarchies

189. Associative Hierarchical CRFs Ladický, Russell, Kohli, Torr ICCV09 AH CRF Energy for Object-class Segmentation Pairwise CRFHigher order term

190. Associative Hierarchical CRFs Ladický, Russell, Kohli, Torr ICCV09 AH CRF Energy for Object-class Segmentation Pairwise CRFHigher order term Higher order term recursively defined as Segment unary term Segment pairwise term Segment higher order term

191. Associative Hierarchical CRFs Ladický, Russell, Kohli, Torr ICCV09 AH CRF Energy for Object-class Segmentation Pairwise CRFHigher order term Higher order term recursively defined as Segment unary term Segment pairwise term Segment higher order term Why is this generalisation useful?

192. Associative Hierarchical CRFs Ladický, Russell, Kohli, Torr ICCV09 Let us analyze the case with one segmentation potentials only over segment level

193. Associative Hierarchical CRFs Ladický, Russell, Kohli, Torr ICCV09 Let us analyze the case with one segmentation potentials only over segment level 1) Minimum is segment-consistent

194. Associative Hierarchical CRFs Ladický, Russell, Kohli, Torr ICCV09 Let us analyze the case with one segmentation potentials only over segment level 1)Minimum is segment-consistent 2)Cost of every segment-consistent CRF equal to the cost of pairwise CRF over segments

195. Associative Hierarchical CRFs Ladický, Russell, Kohli, Torr ICCV09 Let us analyze the case with one segmentation potentials only over segment level Equivalent to pairwise CRF over segments!

196. Associative Hierarchical CRFs Ladický, Russell, Kohli, Torr ICCV09 Merges information over multiple scales Easy to train potentials and learn parameters Allows multiple segmentations and hierarchies Allows long range interactions Limited (?) to associative interlayer connections

197. Inference for AH-CRF Graph Cut based move making algorithms [Boykov et al. 01] α-expansion transformation function for base layer Ladický (thesis) 2011

198. Inference for AH-CRF Graph Cut based move making algorithms [Boykov et al. 01] α-expansion transformation function for base layer Ladický (thesis) 2011 α-expansion transformation function for auxiliary layer (2 binary variables per node)

199. Inference for AH-CRF Ladický (thesis) 2011 Interlayer connection between the base layer and the first auxiliary layer:

200. Inference for AH-CRF Ladický (thesis) 2011 Interlayer connection between the base layer and the first auxiliary layer: The move energy is: where

201. Inference for AH-CRF Ladický (thesis) 2011 The move energy for interlayer connection between the base and auxiliary layer is: Can be transformed to binary submodular pairwise function:

202. Inference for AH-CRF Ladický (thesis) 2011 The move energy for interlayer connection between auxiliary layers is: Can be transformed to binary submodular pairwise function:

203. Inference for AH-CRF Ladický (thesis) 2011 Graph constructions

204. Inference for AH-CRF Ladický (thesis) 2011 Pairwise potential between auxiliary variables :

205. Inference for AH-CRF Ladický (thesis) 2011 Move energy if x = x : cd Can be transformed to binary submodular pairwise function:

206. Inference for AH-CRF Ladický (thesis) 2011 Move energy if x ≠ x : cd Can be transformed to binary submodular pairwise function:

207. Inference for AH-CRF Ladický (thesis) 2011 Graph constructions

208. Potential training for Object class Segmentation Pixel unary potential Unary likelihoods based on spatial configuration (Shotton et al. ECCV06) Classifier trained using boosting Ladický, Russell, Kohli, Torr ICCV09

209. Potential training for Object class Segmentation Segment unary potential Ladický, Russell, Kohli, Torr ICCV09 Classifier trained using boosting (resp. Kernel SVMs)

210. Results on MSRC dataset

211. Results on Corel dataset

212. Results on VOC 2010 dataset Input ImageResultInput ImageResult

213. VOC 2010-Qualitative Input ImageResultInput ImageResult

214. CamVid-Qualitative Our Result Input Image Ground Truth Brostow et al.

215. Quantitative comparison MSRC dataset Corel dataset VOC2009 dataset

216. Quantitative comparison Ladický, Russell, Kohli, Torr ICCV09 MSRC dataset VOC2009 dataset

217. Dense Stereo Reconstruction Unary Potential Unary Cost dependent on the similarity of patches, e.g.cross correlation Disparity = 0

218. Dense Stereo Reconstruction Unary Cost dependent on the similarity of patches, e.g.cross correlation Unary Potential Disparity = 5

219. Dense Stereo Reconstruction Unary Cost dependent on the similarity of patches, e.g.cross correlation Unary Potential Disparity = 10

220. Dense Stereo Reconstruction Encourages label consistency in adjacent pixels Cost based on the distance of labels Linear TruncatedQuadratic Truncated Pairwise Potential

221. Dense Stereo Reconstruction Does not work for Road Scenes ! Original Image Dense Stereo Reconstruction

222. Does not work for Road Scenes ! Patches can be matched to any other patch for flat surfices Different brightness in cameras

223. Dense Stereo Reconstruction Does not work for Road Scenes ! Patches can be matched to any other patch for flat surfices Different brightness in cameras Could object recognition for road scenes help? Recognition of road scenes is relatively easy

224. Joint Dense Stereo Reconstruction and Object class Segmentation sky Object class and 3D location are mutually informative Sky always in infinity (disparity = 0) 

225. Joint Dense Stereo Reconstruction and Object class Segmentation buildingcarsky Object class and 3D location are mutually informative Sky always in infinity (disparity = 0) Cars, buses & pedestrians have their typical height

226. Joint Dense Stereo Reconstruction and Object class Segmentation buildingcarsky Object class and 3D location are mutually informative Sky always in infinity (disparity = 0) Cars, buses & pedestrians have their typical height Road and pavement on the ground plane road

227. Joint Dense Stereo Reconstruction and Object class Segmentation buildingcarsky Object class and 3D location are mutually informative Sky always in infinity (disparity = 0) Cars, buses & pedestrians have their typical height Road and pavement on the ground plane Buildings and pavement on the sides road

228. Joint Dense Stereo Reconstruction and Object class Segmentation buildingcarsky Object class and 3D location are mutually informative Sky always in infinity (disparity = 0) Cars, buses & pedestrians have their typical height Road and pavement on the ground plane Buildings and pavement on the sides Both problems formulated as CRF Joint approach possible? road

229. Joint Formulation Each pixels takes label z i = [ x i y i ]  L 1  L 2 Dependency of x i and y i encoded as a unary and pairwise potential, e.g. strong correlation between x = road, y = near ground plane strong correlation between x = sky, y = 0 Correlation of edge in object class and disparity domain

230. Joint Formulation Weighted sum of object class, depth and joint potential Joint unary potential based on histograms of height Object layer Disparity layer Joint unary links Unary Potential

231. Joint Formulation Pairwise Potential Object class and depth edges correlated Transitions in depth occur often at the object boundaries Object layer Disparity layer Joint pairwise links

232. Joint Formulation

233. Inference Standard α-expansion Each node in each expansion move keeps its old label or takes a new label [x L1, y L2 ], Possible in case of metric pairwise potentials

234. Inference Standard α-expansion Each node in each expansion move keeps its old label or takes a new label [x L1, y L2 ], Possible in case of metric pairwise potentials Too many moves! ( |L1| |L2| ) Impractical !

235. Inference Projected move for product label space One / Some of the label components remain(s) constant after the move Set of projected moves α-expansion in the object class projection Range-expansion in the depth projection

236. Dataset Leuven Road Scene dataset Contained 3 sequences 643 pairs of images We labelled 50 training + 20 test images Object class (7 labels) Disparity (100 labels) Publicly available http://cms.brookes.ac.uk/research/visiongroup/files/Leuven.zip Left cameraRight cameraObject GTDisparity GT

237. Qualitative Results Large improvement for dense stereo estimation Minor improvement in object class segmentation

238. Quantitative Results Dependency of the ratio of correctly labelled pixels within the maximum allowed error delta

239. We proposed : New models enforcing higher order structure Label-consistency in segments Consistency between scale Label-set consistency Consistency between different domains Graph-Cut based inference methods Source code http://cms.brookes.ac.uk/staff/PhilipTorr/ale.htm There is more to come ! Summary

240. Questions? Thank you