Download presentation

Presentation is loading. Please wait.

Published byLeanna Champlin Modified about 1 year ago

1
Segmentation from Examples By: A’laa Kryeem Lecturer: Hagit Hel-Or

2
What is Segmentation from Examples ? Segment an image based on one (or more) correctly segmented image(s) assumed to be from the same domain Effective when making a semantic segmentation

3
Why to use Examples The example defines the granularity of the desired output Give us the ability to characterize meaningful parts in the image Using example allow us to use non-parametric model

4
The example defines the granularity of the desired output: Training image Test Image Desired Segmentation Induced Segmentation

5
Why to use Examples The example defines the granularity of the desired output Give us the ability to characterize meaningful parts in the image

6
Give us the ability to characterize meaningful parts in the image

7
Semantic Segmentation from an example We want to segment an image into semantically meaningful parts Required in various applications

8
Semantic Segmentation from an example We want to segment an image into semantically meaningful parts Required in various applications Problems: Meaningful parts are often too complex Semantic interpretation is highly subjective, depending on both the application, and the user

9
Meaningful parts are often too complex

10
Semantic Segmentation from an example We want to segment an image into semantically meaningful parts Required in various applications Problems: Meaningful parts are often too complex Semantic interpretation is highly subjective, depending on both the application, and the user

11
Example of image different segmentation

12
Semantic Segmentation from an example So, How to achieve semantic segmentation Getting segmented training image(s) as input

13
Training set

14
Semantic Segmentation from an example So, How to achieve semantic segmentation Getting segmented training image(s) as input Using non-parametric representation

15
non-parametric model Each semantic part is represented by a set of square patches

16
Semantic Segmentation from an example So, How to achieve semantic segmentation Getting segmented training image(s) as input Using non-parametric representation Over-segmenting the Test image into small fragments

17
Over segmented image

18
Semantic Segmentation from an example So, How to achieve semantic segmentation Getting segmented training image(s) as input Using non-parametric representation Over-segmenting the Test image into small fragments Compute costs for fragment-label pairs

19
(fragment,label) cost example ?

20
Semantic Segmentation from an example So, How to achieve semantic segmentation Getting segmented training image(s) as input Using non-parametric representation Over-segmenting the Test image into small fragments Compute costs for fragment-label pairs Graph-cuts multi-label optimization

21
Why do we need graph-cuts Graph-cuts optimization is used to label each fragment in a globally optimal manner

22
Training set Test image Fragmentation Fragments Patch sets Classification Classification scores Graph-Cuts optimization Result

23
Over segmenting Fragment: small arbitrarily-shaped and simply-connected pixel clusters We assume that small homogeneous regions always belong to the same semantic part

24
Over segmenting Fragment: small arbitrarily-shaped and simply connected pixel clusters We assume that small homogeneous regions always belong to the same semantic part Advantages: Enforces a locally coherent labeling Reduces the computational complexity

25
Graph-cuts multi-label optimization For each fragment we have k cost values, we need to determine which one is the optimal Using expanded version of the graph-cuts we saw at Jad’s lecture, where we may have more than two labels (background, object)

26
Algorithm for semantic segmentation 1.Pixel labeling costs 2.Fragmentation 3.Fragment labeling costs 4.Graph-cuts optimization

27
Algorithm for semantic segmentation 1.Pixel labeling costs 2.Fragmentation 3.Fragment labeling costs 4.Graph-cuts optimization

28
Pixel labeling costs Given I train and L train Representing each label(segment) in L train by a set of square patches We get k sets {S l } l=1,…,k one for each label

29
Pixel labeling costs (cont.) Next, we define φ (p, l) for each (pixel,label) pair The cost of assigning label l to pixel p I test P’S l p:pixel at I test l : label in L train ssd(P,P’) is the sum of squared distances between P,P’ M:mxmx3 P:mXm neighborhood centered at p P’:mxm patch

30

31
Algorithm for semantic segmentation 1.Pixel labeling costs 2.Fragmentation 3.Fragment labeling costs 4.Graph-cuts optimization

32
Fragmentation We partition I test into small,color- homogeneous regions using mean shift segmentation

33
Fragmentation We partition I test into small,color- homogeneous regions using mean shift segmentation Fragment size is adjusted according to I test.(fragments are smaller in more detailed areas of I test, and larger in more homogeneous regions)

34
Fragmentation We partition I test into small,color- homogeneous regions using mean shift segmentation Fragment size is adjusted according to I test.(fragments are smaller in more detailed areas of I test, and larger in more homogeneous regions) Fragment boundaries align with edges in the image

35
Fragmentation (cont.) Random colorizationDetailed close-up

36
Algorithm for semantic segmentation 1.Pixel labeling costs 2.Fragmentation 3.Fragment labeling costs 4.Graph-cuts optimization

37
Fragment labeling costs Voting scheme in order to compute labeling costs of each fragment For each fragment f I test we pick a few representative pixels: Rep(f)={p i f } i=I,…,R f R f is proportional to |f|

38
Fragment labeling costs (cont.)

39
Fragment labeling vs. pixel labeling

40
Enforces a locally coherent labeling Training image Training seg. Input image Fragment labeling Pixel labeling

41
Algorithm for semantic segmentation 1.Pixel labeling costs 2.Fragmentation 3.Fragment labeling costs 4.Graph-cuts optimization

42
Graph-cuts optimization After calculating labeling cost for all image fragments we get k images. Image i describes the cost assigning each pixel at the test image to label i fragment labeling costs. Costs range in the interval [0,1]

43
Graph-cuts optimization Now for each pixel p I test we have a labeling cost We need to find L test the globally optimal labeling Requirements: Minimizes the total labeling cost Consistent with presence (or absence) of edges

44
Graph-cuts optimization (cont.)

45

46
Intuition:

47

48
Graph-cuts optimization (cont.) Finally L test is determined by solving L test =min L E(L) Fragment labeling Labeling after Graph- Cuts Optimization

49
Multi-label graph-cut Colored nodes:labels Squares : fragments For each (fragment,label) pair we have an edge. Edge weigh according to φ. Edges between two squares weighed according to Ψ.

50
Multi-label graph-cut Induced graph Each fragment connected to a single label.

51
Multi-label graph-cut is NP-complete problem!

52
Training image Training segmentation Input image

53
Algorithm for semantic segmentation 1.Pixel labeling costs 2.Fragmentation 3.Fragment labeling costs 4.Graph-cuts optimization

54
Algorithm results Training set a b c

55
Bear results invariant to the number of instances of each semantic part within the image, and insensitive to the shape of each part. We can’t separate multiple objects belonging to the same label (c).

56
Algorithm results Training set a b c d

57
Summary

58
Thank You For Listening

59
References Inducing Semantic Segmentation from an Example, Yaar Schnitman, Yaron Caspi, Daniel Cohen-Or, and Dani Lischinski. "Segmentation by Example“, Sameer Agarwal and Serge Belongie. Christoudias, C.M., Georgescu, B.: Edge detection and image segmentation (edison) system. Boykov, Y., Veksler, O., Zabih, R.: Fast approximate energy minimization via graph cuts. IEEE Trans. Pattern Anal. Mach. Intell. 23 (2001) 1222–1239

Similar presentations

© 2017 SlidePlayer.com Inc.

All rights reserved.

Ads by Google