1. 2008 International Conference on Multimedia & Expo GRAPH CUTS BY USING LOCAL TEXTURE FEATURES OF WAVELET COEFFICIENT FOR IMAGE SEGMENTATION Keita Fukuda, Tetsuya Takiguchi, Yasuo Ariki Graduate School of Engineering, Kobe University, Japan

2. 2008 International Conference on Multimedia & Expo2 Table of contents Introduction conventional method, problem with graph cuts Proposed method iterated graph cuts using texture and smoothing and detail of each method Experiments Conclusion and Future Work

3. 2008 International Conference on Multimedia & Expo3 Introduction Extracting the foreground objects in static images is one of the most fundamental tasks (Image Segmentation Problem). “bkg” “obj”

4. 2008 International Conference on Multimedia & Expo4 Background Recently, the image segmentation problem is formalized as an optimal solution problem. e.g. Snakes, Level Set Method, Graph Cuts Y.Boykov, M.P.Jolly,“Interactive graph cuts for optimal boundary & region segmentation of object in N-D images”, IEEE International Conference on Computer Vision and Pattern Recognition possible to compute global optimal solution The cost function is general enough to include both region and boundary properties of the segments Advantage of Graph Cuts S T O B

5. 2008 International Conference on Multimedia & Expo5 Graph Cuts S T O B min cut O B Assigned the corresponding label Searching the minimum cost cut t-link n-link Background likelihood S:”obj” T:”bkg” O B Neighbor similarity A 3-by-4 image Segmentation result Object likelihood Construct a graph

6. 2008 International Conference on Multimedia & Expo6 Problem (1) with GC It has been difficult to segment images that include complex noisy edges.  To solve this problem, using iterated graph cuts based on smoothing. T. Nagahashi, H.Fujiyoshi, and T.Kanade ， “ Image Segmentation Using Iterated Graph Cuts Based on Multi-scale Smoothing ， ”ACCV2007 Noise exists in strong edge

7. 2008 International Conference on Multimedia & Expo7 Problem (2) with GC It is difficult to segment images with an object whose color is similar to the background.  To solve this problem, employing the texture likelihood as well as color likelihood. The color of object is similar to that of background Iterated Graph Cuts based on local texture features as well as low frequency features of wavelet coefficient. Proposed method

8. 2008 International Conference on Multimedia & Expo8 To solve problem (2) with GC, high pass subbands (LHk,HLk,HHk) are used for t-link. Local texture features are defined from them. The likelihood are derived from local texture features as well as color. And the prior probabilities are defined using the previous segmentation result. After initializing the level k, the input image is decomposed into subbands using multiresolution wavelet analysis. To solve problem (1) with GC, low pass subband (LLk) is used for n-link. The obtained neighbor similarity is set to n-link as edge cost. Proposed Method LL,LH,HL,HH are obtained Graph Cuts Segmentation Prior probability GMM(color+texture)update k ← k-1 LL: Smoothing LH,HL,HH: Local texture features Multiresolution analysis (level k) seed Input Output t-link n-link Graph Cuts segmentation is carried out, and these processes are repeated until k = 0.

9. 2008 International Conference on Multimedia & Expo9 Multiresolution analysis downsampled HH1LH1 HL1 LH2HH2 HL2 LL2 Level 2 LH1 HL1 HH1 LL1 Level 1 LL: low-pass information LH: vertical HL: horizontal HH: diagonal orientation Subbands information for n-link for t-link Input imageLevel 1Level 2 Input image downsampled

10. 2008 International Conference on Multimedia & Expo10 n-link pq B {p,q} is large when p and q are similar, it is close to 0 when p and q are very different. Global to local segmentation can be performed by iterated Graph Cuts with multiresolution analysis using from coarse to fine level for n-link. finecoarse Level 3Level 2Level 1

11. 2008 International Conference on Multimedia & Expo11 Local texture features LH (level 1)HL (level 1) Local texture features are defined by averaging the absolute wavelet coefficient in the window (3×3) surrounding pixel p HH (level 1) Local texture features d: wavelet coefficient k: level Local texture features T p are larger in complex region, and smaller in flat region.

12. 2008 International Conference on Multimedia & Expo12 t-link Distance trans. Pr(O) Distance trans. Pr(B) S:”obj” T:”bkg” p － ln Pr(B|Y p ) － ln Pr(O|Y p ) Color (RGB) :C p Local texture:T p Pr(Y p |O) Color (RGB) :C p Local texture:T p Pr(Y p |B) Background Object GMM Object likelihood Background likelihood

13. 2008 International Conference on Multimedia & Expo13 Graph Cuts Segmentation Edge cost of the graph The boundary between the object and the background is found by searching the minimum cost cut on the graph S T O B mincut Yuri Boykov and Vladimir Kolmogorov, “An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy Minimization in Computer Vision”

14. 2008 International Conference on Multimedia & Expo14 Summarize proposed method S:”obj ” T:”bkg” O B t-link ( texture with color ) n-link Segmentation result S:”obj ” T:”bkg” O B k Multiresolution at level k k-1 Multiresolution at level k-1 Multiresolution level k-1 is set to k-1 The prior probability

15. 2008 International Conference on Multimedia & Expo15 Experimental Condition  We compared using the same labels: method (1) Interactive Graph Cuts [Boykov 04] method (2) Iterated Graph Cuts using smoothing [Nagahashi 07] ProposedIterated Graph Cuts using smoothing and texture  Berkley Image database (50 images)  The segmentation error rate is defined as follow: Mask image Output imageError (in red and blue ) =

16. 2008 International Conference on Multimedia & Expo16 Experimental Result k Level kmethod (1)method (2)Proposed 14.623.783.17 24.622.952.62 34.622.951.98  The proposed method improved the error rate compared to two conventional methods.  In this database, two conventional methods could achieve the low error rate. ⇒ The examples with high error rate are shown from next page. Err (%) in segmentation result at multiresolution level k

17. 2008 International Conference on Multimedia & Expo17 Experimental result (1) Effect of smoothing 6.27 %1.16 %0.85 % 0.79 %3.69 %0.83 % The method (2) and proposed method can achieve the better image segmentation for the images with complex edges than the method (1). Method (1)Method (2)Proposed

18. 2008 International Conference on Multimedia & Expo18 Experimental result (2) 10.44 %8.28 %2.86 % 11.38 %16.64 %2.15 % Effect of local texture features Method (1)Method (2)Proposed The proposed method can achieve the better image segmentation for the images with object colors similar to background than method (1) and(2)

19. 2008 International Conference on Multimedia & Expo19 Discussion  To solve problem (1) : method (2), proposed  Prior probability The brief shape information can be given from previous segmentation result  Smoothing removing local strong edges.  To solve problem (2) ： proposed  Local texture features for the image with object colors similar to background.

20. 2008 International Conference on Multimedia & Expo20 Summary Graph Cuts by using local texture features of wavelet coefficient for image segmentation. New concept: Graph Cuts segmentation based on local texture features as well as smoothing process. In future work:  The weight to texture and color for segmentation  Others texture features.

21. 2008 International Conference on Multimedia & Expo21 Thank you for your attention !

22. 2008 International Conference on Multimedia & Expo22 The cost function f E(f) The soft constraints that we impose of label f are described by the cost function E(f) as follow: The regional term assumes that the individual penalties for assigning pixel p to labels. The boundary term assumes that a penalty for a discontinuity between p and q

23. 2008 International Conference on Multimedia & Expo23 Minimum cost cut Ｓ Ｔ 10 2 2 2 6 6 5 3 9 7 Ｓ Ｔ 2 2 2 6 6 5 3 9 7 3/34/10 5/5 0/6 5/6 8/9 2/7 2/2 0/2 1. Graph Construct Edge label = capacity Edge = pipe  Goal : max flow from S to T Ｓ Ｔ 2 2 2 6 6 5 3 9 7 2 8 Example of a graphResidual GraphCurrent flow 3. Current flow from residual graph e.g. 8/9 = (flow) / (capacity) 2. Residual graph Find augmenting paths until finished.

24. 2008 International Conference on Multimedia & Expo24 Minimum cost cut Ｓ Ｔ 10 2 2 2 6 6 5 3 9 7 1. Graph Construct Edge label = capacity Edge = pipe  Goal : max flow from S to T 3. Current flow from residual graph e.g. 8/9 = (flow) / (capacity) 4. Minimum cost cut Saturated edges are found from 2. Each node is assigned for the corresponding labels. Ｓ Ｔ 10 2 2 2 6 6 5 3 9 7 3/34/10 5/5 0/6 5/6 8/9 2/7 2/2 0/2 mincut STT Yuri Boykov and Vladimir Kolmogorov, “An Experimental Comparison of Min-Cut/Max-Flow Algorithms for Energy Minimization in Computer Vision” 2. Residual graph Find augmenting paths until finished.

25. 2008 International Conference on Multimedia & Expo25 Experimental result (3) for the unclear boundary images. The proposed method is effective for images which have the unclear boundary between object and background. Method (1)Method (2)Proposed

26. 2008 International Conference on Multimedia & Expo26 Time cost The size of image ： 650×400 method(1)method(2)Proposed 10.8 sec56.8 sec82.5 sec The proposed method carried out 3 times graph cuts segmentation. And Gaussian Mixture Model is derived from 6 dimensional features. It is because the proposed method is slow. Time cost

27. 2008 International Conference on Multimedia & Expo27 Discussion (2)  Remained problem The proposed method is ineffective for a flat image such as artificial images due to a few edges. Future work include optimization of the weight to texture and color for segmentation.

28. 2008 International Conference on Multimedia & Expo28 Noisy edge problem If the strong edges exist, n-link edge cost is small. (easy to pass the n-link edge) pq S T pq S T n-link edge cost is large. (hardly pass the n-link edge) Else,

29. 2008 International Conference on Multimedia & Expo29 Influence of seeds position Graph cuts technique is quite stable and normally produces the same results regardless of particular seed positioning within the same image.

30. 2008 International Conference on Multimedia & Expo30 Prior probability dDistance d from the border between the object and background is normalized from 0.5 to 1.0. The prior probability is defined by using d ojb and d bkg. Prior probability

31. 2008 International Conference on Multimedia & Expo31 First time segmentation S:”obj” T:”bkg” p － ln Pr(Y p |B) － ln Pr(Y p |O) Pr(Y p |O) Pr(Y p |B) Color (RGB) :C p Local texture:T p GMM Color (RGB) :C p Local texture:T p GMM “obj” seeds “bkg” seeds

32. 2008 International Conference on Multimedia & Expo32 Multiresolution level At level 3 The low pass subband LL3 is 1 / 64 * the original image. The low pass subband LL4 is 1 / 256 * the original image. At level 2

33. 2008 International Conference on Multimedia & Expo33 Parameter Graph Cuts parameter : λ = 0.05 σ = 12.0 K The constant K is larger than sum of all n-link costs not to label the opposite label. Gaussian Mixture Model: 5 components

34. 2008 International Conference on Multimedia & Expo34 Local Texture features

35. 2008 International Conference on Multimedia & Expo35 Results of proposed method

36. 2008 International Conference on Multimedia & Expo36 Result of proposed method

37. 2008 International Conference on Multimedia & Expo37 Question Please repeat the question. Thank you for your question. Are you asking about … ? What you see here is … I believe that … To be honest, we never looked into possibility. The question was … There is not the particular deep meaning. Not yet Thank you for your advice.

38. 2008 International Conference on Multimedia & Expo38 追加スライド n-linkの問題について ユーザの引き方に問題は？ 数パーセントでも違いがはっきりしている パラメータの説明 Level4がない理由 問題点 一回目のセグメンテーション

39. 2008 International Conference on Multimedia & Expo39 Minimum cost cut Ｓ Ｔ 10 2 2 2 6 6 5 3 9 7 10+5+6+9 = 30 2+2+5+3 = 12 (mincut) 10+2+5+6+9 = 32

40. 2008 International Conference on Multimedia & Expo40 Ｓ Ｔ 2 2 2 6 6 5 3 9 7 2 8 inputlabeledmaskoutputerror